Skip to main content

Incident response

One page per incident is too many pages. This is the set, in severity order, each with what to do first.

Before anything else: an offline Edge is not an outage. Deposits are still detected, tracked and recorded; webhooks arrive late rather than being lost. Check Edge liveness before escalating.

Identity rotation detected​

Severity: critical. Do not resolve by approving.

A different mnemonic is mounted against your account. Address generation is blocked on the affected chains.

First action: determine whether anyone intended to change the mnemonic. If not, reject the rotation — rejecting is reversible and changes nothing.

Full runbook: Identity rotation.

Webhooks quiet while balances move​

Severity: critical. Ziklag-side, and we page on it — but you may see it first.

The symptom on your side is deposits appearing in GET /v1/transactions and on the console while no webhook arrives for them. Inside Cloud that is one metric, chainos_event_publication_incomplete, and it is the highest-severity signal in the system: the ledger is advancing while notifications stall.

Your first action: confirm it is not your endpoint. Check the dead-letter queue and the delivery log on the console — a handler returning 5xx produces the same silence and is far more common.

curl -s -H "X-API-Key: $CHAINOS_EDGE_KEY" \
  http://localhost:8787/v1/webhooks/deliveries?status=failed | jq '.data.items[0]'

If deliveries are not being attempted at all, escalate with the detail below. Do not wait it out: Event retention behind replay is 7 days, and an event that ages out has to be reconciled against the chain instead (Reconciliation).

SELECT listener_id, count(*), min(publication_date)
  FROM event_publication
 WHERE completion_date IS NULL
 GROUP BY listener_id;

The listener_id names the consumer that is not completing.

Recovery is a restart of processor with republish enabled, which re-delivers everything incomplete. Then reconcile — Reconciliation — because you cannot assume nothing was missed.

A confirmed deposit reversed​

Severity: critical, and genuinely rare.

A reversal below threshold is normal reorg behaviour and produces no deposit_confirmed. A reversal after deposit_confirmed means a reorg deeper than 12 Ethereum blocks, 128 Polygon blocks or 3 Bitcoin blocks — which is a consensus-level event on the chain, not a ChainOS fault.

First action: confirm it on a public explorer before touching your ledger. The explorerUrl on the transaction record links directly.

Then:

  1. Freeze outbound activity for the affected customer, and for the chain if more than one customer is affected.
  2. Check chain news. A reorg that deep is public and will be discussed.
  3. Do not auto-reverse the credit. If the transaction is re-included — which it usually is — you will have reversed a credit that stands. Hold the position and wait.
  4. Reconcile the affected addresses against the chain once the chain has settled.
  5. Mail Ziklag with the txid and the block numbers. This is worth investigating on our side even when the cause is external.

Split-brain: two Edge leaders​

Severity: high.

Two sessions claiming leader, or a follower reporting a lease epoch above the leader's.

First action: confirm it, because every per-session health indicator will be green.

curl -s $CLOUD/v1/edge/sessions -H "Authorization: Bearer $TOKEN" \
  | jq '.data[] | { podHint, role, leaseEpoch, liveness }'

Cloud fences this — a frame with a stale epoch is refused with 409 LEASE_EPOCH_STALE and the stream closed — so the stale leader cannot deliver. The incident is that your network partitioned, which is worth knowing about for its own sake.

  1. Find the partition. Usually a network policy change, a node-level issue, or an overlay problem.
  2. Do not restart both replicas at once. Restart the one with the lower epoch; it will re-acquire as a follower.
  3. Check for duplicate deliveries in the window. Your handler should be idempotent on event.id — Idempotency — and this is the event that tests it.

Outbox poisoned​

Severity: high.

{ "outboxDepth": 4193, "outboxPoisoned": true }

One write keeps failing and is blocking every write behind it, in order.

First action: look at what your endpoint returns for the head of the queue. The delivery history for the webhook carries the status code.

  • A 4xx from your endpoint. Your handler is rejecting a payload it should accept. A signature verification bug is the usual cause — see Webhooks.
  • A TLS failure. An expired certificate on your endpoint, or a CA the Edge does not trust.
  • A payload your handler cannot parse. Check for an event type you did not expect; identity_rotation_detected is delivered regardless of subscription and a handler that switches exhaustively over a smaller set will throw on it.

Fix the endpoint. The queue drains in order once deliveries succeed. It can take a while at depth, and that is correct rather than a hang.

Dead-letter queue growing​

Severity: high if non-empty. The 7-day clock matters.

Events that exhausted the retry ladder — delivery, 2m, 5m, 30m.

First action: fix the endpoint, then replay. An automatic replay against a still-broken endpoint just re-runs the ladder.

curl -s $CLOUD/v1/webhooks/deliveries/dead-letters -H "Authorization: Bearer $TOKEN"
curl -X POST $CLOUD/v1/webhooks/deliveries/{id}/replay -H "Authorization: Bearer $TOKEN"

Event retention behind replay is 7 days. Anything older cannot be replayed and has to be reconciled against the chain instead. A dead-letter queue that has been accumulating for a week needs action, not a backlog ticket.

A chain is unavailable​

Severity: medium, and often expected.

503 CHAIN_UNAVAILABLE means ChainOS cannot currently reach that chain. Note that any chain not enabled for your account reports this permanently by design — that is not a fault to investigate, and enabling a chain is a support request rather than an incident.

What keeps working: derivation, and therefore address issuance — it needs no RPC. What does not: balances, fee estimates and broadcast.

A zero balance is never reported in this case. A confident zero is indistinguishable from an emptied wallet, so the API refuses to give you one.

First action: confirm which chains are configured, in the console, before assuming a fault.

Suspected compromise of the Edge or its cluster​

Severity: critical. Treat it as possible key exposure until shown otherwise.

What contains it depends on the keystore mode, and it is worth knowing in advance:

  • Sealed mode (awskms, azurekv): revoke the Edge role's kms:Decrypt (or the unwrap role assignment), or disable the key. Running replicas keep signing — they do not call the KMS after boot — so this does not take withdrawals down. What it does is make every restart fail, including a restart an attacker triggers on a pod they control to obtain the seed. Then find out who could act as the Edge's identity and who had pods/exec or node access.
  • mnemonic mode: there is nothing to revoke. Whoever read the mounted secret has the phrase. Assume it is exposed.
  • Either mode, if exposure is likely: pause autonomous spending (unset the EDGE_SPEND_CAP_* variables and restart), revoke the zkl_* keys your applications use, and plan a move of funds to a new mnemonic — a rotation, with the human approval it requires (Identity rotation).

The alarms that would have caught it are in Monitoring.

The Edge will not start​

Severity: medium. Withdrawals are blocked; deposits are not.

Go straight to Troubleshooting, which is ordered by frequency. The most common are a mnemonic file the container cannot read and a data directory on ephemeral storage. In a sealed mode, a KMS refusal after someone revoked the role — perhaps deliberately, as above — looks exactly like this; check the audit log before restoring access.

Subscription lapsed​

Severity: medium. New addresses are refused; nothing else changes.

A console banner and escalating warnings first, then 403 NO_SUBSCRIPTION on new address generation.

Existing addresses stay monitored and webhooks keep being delivered, indefinitely. A billing state must never cause funds to become unmonitored, and it does not.

Full detail: Bands and tiers.

Escalating to Ziklag​

Include:

  • The meta.requestId from a failing response — it is what lets us find the request.
  • The txid and chain, for anything transaction-related.
  • The rotation id, for an identity rotation.
  • Edge version, and whether the request was live or sandbox.
  • For the Edge: /v1/edge/health, /v1/edge/lease, /v1/edge/outbox and the last 500 log lines.

Never send a mnemonic, a private key or an unmasked API key. Support does not need them and cannot use them. If a diagnostic dump contains a mnemonic, you are using PASS_PHRASE rather than PASS_PHRASE_FILE, which is its own finding — see Configuration.

Security issues go to security@ziklagconsulting.com, not to support.