Runbooks

One page per SignatureName, in the same shape: symptoms, detection (exact metrics and thresholds), what the supervisor already did, the manual steps, and when to escalate.

• stuck-saga

  A payout saga stopped advancing — the one classic automatic remediation: a guarded worker sweep, verified by re-reading the saga.

• outbox-backlog

  Pending events aging past threshold — the relay is down or poisoned; the supervisor re-drives the relay alone and verifies against the next gauge.

• inbox-dead-letter

  Inbound provider events dead-lettered before applying — the supervisor revives the oldest dead rows and lets the normal drain re-apply them idempotently.

• integrity-mismatch

  The checkpoint reverify failed: rows changed behind the hash chain. The supervisor proves once, escalates with the evidence, pauses the worker, and latches containment — never fixes.

• treasury-breach

  The backing invariant is breached: custodial CREDIT exceeds what the trust or float covers. Escalate-only — a solvency signal, never auto-fixed.

• engine-stall

  Connection acquires started and none completed: the pool is starved or the database is unresponsive. Submits queue rather than fail, so nothing errors on its own.

• signal-silence

  A signal the host declared as periodic has gone quiet — the process that should emit it is down, paused, or wedged, and nothing downstream will say so on its own.

• retry-exhaustion

  Retry budgets are running out: submits failed to their callers after every retry. Unlike a deadlock storm, users are seeing errors, not just latency.

• deadlock-storm

  A burst of lock conflicts absorbed inside the retry budget — invisible to callers, visible as retry pressure. Advisory only.

• velocity-anomaly

  Risk-gate rejections spiking — a fraud signal first: card testing or credit farming trips the per-user velocity gates long before a legitimate surge does.

• webhook-replay-storm

  The same provider eventIds keep arriving: a provider misconfiguration or a replay attack. The gates held — the counter is what makes the storm visible.

• checkpoint-seal-slow

  Merkle sealing is trending slower — a table-growth problem brewing, caught while it is a trend and not an outage. Advisory only.