Background worker

The sweeps that run off the request path: payouts, subscriptions, fee realization, checkpoints, the outbox/inbox relay, and the lifecycle jobs — accrual drain, re-proof, orphans, archive, and retention.

Source src/worker/index.tsSWEEP_NAMESsrc/worker/payouts.ts#L50advanceDuePayoutssrc/worker/relay.ts#L53relayOutbox

The synchronous core posts to the ledger and returns an Outcome on the request path. But plenty of work outlives a single request.

A payout waits on an external rail. A subscription bills over time. A promo grant expires. A platform fee only becomes the platform’s once its refund window closes. The outbox holds events that still need to drain to the dispatcher.

The worker advances each of these one safe step at a time, off the request path. It owns no new behavior: it pushes recorded work toward its next state and reports what moved.

The sweeps

Each worker cycle runs a fixed set of jobs, named in SWEEP_NAMES. The array order is both the run order and the order results come back in.

Here is what each sweep does:

Two pairs are deliberately adjacent. feeSweep runs right after treasury: treasury only measures the surplus, then feeSweep moves it. And checkpointVerify runs before checkpoint, so the old snapshot is checked against the ledger before a fresh one overwrites it.

How a sweep is shaped

Every sweep claims a bounded batch of due rows, advances each row one step, and returns a summary that buckets the rows by outcome. The cap per pass is limit; the current time is now.

The payout sweep is the clearest example. It claims the due payout sagas and pushes each one forward:

const summary = await advanceDuePayouts(store, ctx, { now, limit });
// → { submitted: [...], deadLettered: [...], retrying: [...] }

A RESERVED payout is submitted to the provider and moves to SUBMITTED. Settlement does not happen here; it arrives through the provider’s settlement webhook. A SUBMITTED payout is watched: when the processor adapter offers the optional payoutStatus probe, the sweep acts on its answer first — a provider-reported FAILED or RETURNED releases the reserve promptly, a reported SETTLED blocks the force-fail so a lost settlement webhook cannot end in a double-pay, and a reported PENDING defers the timeout while the rail is still working. Otherwise the sweep only forces a SUBMITTED payout to fail when it has waited past maxPayoutAgeMs for a webhook that never came.

When a payout is dead-lettered, the worker posts the exact reverse of the request-time reservation in the same transaction, so the seller’s reserved credits are returned rather than stranded.

Each row is isolated

A sweep handles its rows one at a time, and each row runs inside its own error boundary. One broken row can’t stop the others in the batch.

A failure that looks temporary (a flaky network or database) bumps an attempt counter and retries next run. A permanent failure, or one that has retried too many times, is set aside (dead-lettered) so the batch never wedges on a row that can’t progress.

The recorded reason names the failing subsystem. A raw throw from an injected port — the outbox dispatcher, the inbox applier, a float or reconcile feed — is stamped PROVIDER.FAILURE; STORE.FAILURE stays reserved for the storage layer. The dead-letter reason and the worker.*.failed logs are the operator’s paging signal, so they point at the right owner.

The whole sweep is isolated too. runSweeps wraps each job, so a job that throws is recorded as a failed result against just that job, and the other sweeps still run. The run never throws.

Six sweeps can be skipped

Most sweeps always run. Six carry an optional capability or opt-in and short-circuit when it’s absent, each reporting the fixed idle summary IDLE_SUMMARIES declares for it — the same shape a real run with nothing to do produces, so a dashboard never special-cases the gap.

floatCoverage needs a float feed (the payout rail’s wallet balance) — the economy-edge bridge supplies one when enabled (the packages). With none configured, it returns an empty successful summary and the internal treasury backing check stands alone. The check is alert-level either way: a shortfall is logged and counted, never posted against.

relay needs a dispatcher to deliver through. With none configured, it returns an empty successful summary and pending rows stay in the outbox for a later run.

const summary = await relayOutbox(store, ctx, { dispatcher, limit });
// → { relayed: [...], failed: [...], deadLettered: [...] }

drainInbox needs an economy handle to submit through. With none, it likewise short-circuits and leaves pending inbox rows in place.

The three lifecycle jobs are host opt-ins, configured through the worker’s defaults (below): orphans runs with an orphans options object (and only settles with settleOlderThanMs set — without it the sweep reports and touches nothing), archive runs with an archive sink and checkpointOlderThanMs, and retention runs with at least one horizon set. Archival owns what each option means and how to choose the bounds.

The relay and the inbox

The relay drains the outbox — that page owns the pattern: why every event commits with its posting, and why delivery is at-least-once. The sweep’s own contract is the batch: it claims up to limit pending events, sends each through the dispatcher, and marks the delivered ones done. The receiver must drop duplicates by event id; the bundled @pwngh/taskq bridge does exactly that, enqueuing each delivery keyed by the event id (the packages).

A delivery that keeps failing bumps the row’s attempt count and retries, up to config.maxOutboxAttempts (default 10). At the cap the event is dead-lettered, so one poison event can’t block the events behind it.

The inbox is the inbound mirror. Where the relay delivers committed money moves outward, drainInbox applies received events inward, submitting each stored Operation through the same economy a direct caller hits.

A re-applied inbox row is deduped by the stored idempotencyKey (the provider event id), so a second apply resolves to a duplicate Outcome rather than a second posting. A rejected Outcome (a terminal business “no” like INSUFFICIENT_FUNDS) is dead-lettered, since retrying the same doomed apply every sweep would never succeed.

Driving the worker

A host builds the worker with createWorker(ports, economy) and drives it with sweep, which runs every sweep once over a shared request and returns the per-job batch plus the txn ids the run committed. Every field of the request is optional — now defaults to the clock, limit to the worker’s default:

const worker = createWorker(ports, economy);
let { batch, postings } = await worker.sweep();

The optional third argument, WorkerDefaults, binds steady-state arguments once — the dispatcher, float and reconcile feeds, the batch limit, and the lifecycle opt-ins (orphans, archive, retention); any SweepRequest field overrides them per sweep:

const worker = createWorker(ports, economy, {
  orphans: { settleOlderThanMs: 180_000 },
  retention: { idempotencyOlderThanMs: 90 * 24 * 60 * 60 * 1000 },
});

start(everyMs, request) runs the jobs on a timer and returns a stop function:

const stop = worker.start(60_000);

start is always present: it drives the ticks through ports.scheduler when one is set and a real timer otherwise, so start and stop stay on the same code path either way.

Treasury and checkpoints

The treasury sweep is measure-only: it sums the custodial credits, converts to required USD at par, and compares against TRUST_CASH. A shortfall is logged and counted, never posted against.

See solvency for what backing means and how the same check runs inside the prover.

feeSweep is the write the treasury sweep doesn’t do. On each run it realizes the full amount the platform is allowed to take (the smaller of its cash surplus and its matured revenue) and skips cleanly when that amount is zero. A draw that would dip into users’ money throws COMMINGLING and posts nothing.

The checkpoint pair maintains the ledger’s tamper-evidence. checkpoint seals a fresh signed snapshot of the per-account hash chains; checkpointVerify re-checks the previous one against the live ledger. A mismatch is a tamper signal, recorded on the summary and logged, not thrown. See ledger integrity for the chain and checkpoint mechanics.

Troubleshooting

My payout never settles

The payouts sweep only submits: it converts the reserve and calls the rail, advancing the saga to SUBMITTED. Settlement is a separate event that arrives as the provider’s verified webhook, maps to a settlePayout operation, and applies through the inbox drain — so a saga parked in SUBMITTED is waiting on the provider, not on you.

await worker.sweep({ limit: 50 }); // the payouts sweep submits due RESERVED sagas
// SUBMITTED → SETTLED happens when the provider's webhook arrives, not here

If the webhook never comes, the saga doesn’t hang forever: past maxPayoutAgeMs with no provider evidence to the contrary, the sweep force-fails it and the reserve returns to the seller. The payout saga owns the state machine.

A sweep keeps retrying the same row

Each claimed row retries up to its attempt cap, then dead-letters with the failing subsystem named in the reason — a PROVIDER.FAILURE dead-letter is the rail misbehaving, a STORE.FAILURE is your database. Read the reason before restarting anything; the row tells you whose outage it was.