Skip to main content

Reconciliation

Webhooks are how you react. Reconciliation is how you know.

At-least-once delivery with a replayable dead-letter queue is a strong guarantee, and it is still not a substitute for periodically checking that what your ledger says matches what the chain says. Every fintech that has been running a while has found at least one gap this way, and the ones that found it in a nightly job rather than in a customer complaint are the ones with a nightly job.

Three records — your ledger, ChainOS and the chain — with the three ways a gap between them shows up and what each one means.

What can actually diverge​

Ranked by how often it happens:

CauseSymptom
Your handler acked an event and threw before committingChainOS shows a confirmed deposit, your ledger does not
An event dead-lettered and was never replayedSame
Your handler double-credited a duplicate deliveryYour ledger shows more than the chain
An address was archived while a deposit was in flightThe chain has funds ChainOS is not monitoring
A deposit went to an address you never issued through ChainOSThe chain has funds nothing is monitoring
A reorg reversed a confirmed transactionGenuinely rare at these thresholds. A real incident

Note what is not on that list: ChainOS missing a deposit to a monitored address. Chain event streams are backed by a reconciliation sweep on the Cloud side that re-scans monitored addresses, so a dropped stream event is picked up rather than lost. If you find one that was missed, that is a defect worth a ticket with the txid.

The daily routine​

Three comparisons, cheapest first.

1. Transaction count and sum, per chain, per day​

curl -s "$EDGE/transactions?chain=eth&direction=inbound&status=confirmed&from=2026-08-18&to=2026-08-19&size=500" \
  -H "X-API-Key: $CHAINOS_EDGE_KEY" \
  | jq '{ n: (.data.content | length),
          total: ([.data.content[].amount | tonumber] | add) }'

Compare against the same window in your ledger. A count that matches and a sum that does not means an amount was mis-parsed somewhere — usually a Number where a BigInt belonged.

tonumber in that snippet is for eyeballing, not for accounting

jq uses IEEE 754 doubles. It will silently mangle a wei amount. For the real comparison, sum in your own code with an arbitrary-precision integer type. Nothing in ChainOS parses a monetary value into a float and neither should your reconciliation.

2. Balance per address​

curl -s -X POST $EDGE/balances/batch \
  -H "X-API-Key: $CHAINOS_EDGE_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "chain": "eth", "addresses": ["0x3fC9…7FAD", "0x8a1c…"] }'

Batch it — five hundred separate calls is five hundred rate-limited requests, and one batch is one.

{ "confirmed": "1000000", "available": "1000000", "reserved": "0", "decimals": 6 }

Reconcile against confirmed, not available. available has the chain's locked reserve deducted, which is real money that exists and is simply not spendable — on XRP the base reserve, on Solana the rent-exempt minimum, on Stellar the account's own minimum balance. A reconciliation against available will report every XRP address as 1 XRP short and every Stellar address at least 1 XLM short, forever.

A chain whose RPC is unreachable returns 503 CHAIN_UNAVAILABLE rather than a zero. Treat that as "unknown" and retry, never as "empty" — a confident zero is indistinguishable from an emptied wallet, which is why the API refuses to give you one.

3. Address inventory​

curl -s "$EDGE/addresses?size=1000" -H "X-API-Key: $CHAINOS_EDGE_KEY" \
  | jq -r '.data.content[] | [.chain, .address, .derivationPath, .userRef, .status] | @csv'

Every address ChainOS holds for you, with its absolute derivation path. Two things to check:

  • Every active address maps to exactly one live customer in your systems. Two customers on one address means an address was issued twice and one mapping was lost.
  • Every customer who should have an address has one, and only one per chain. A duplicate here is the signature of a retried POST /v1/addresses without an Idempotency-Key.

Keep this export. It is the mapping from address to customer, and ChainOS is currently the only place it lives — the mnemonic recovers funds without ChainOS, but not who they belong to. See Mnemonic management.

When you find a gap​

Missing in your ledger, present in ChainOS. The likely cause is an event you acked and did not apply.

  1. Find it: GET /v1/transactions/{id} for the full record.
  2. Check the delivery history for the corresponding webhook. If it shows a 2xx, your handler received it and lost it.
  3. Check the dead-letter queue. If it is there, fix the endpoint and replay.
  4. If the event is older than the 7-day event retention it cannot be replayed. Apply the correction from the transaction record directly, and record that you did so — a manual ledger entry with no audit trail is its own problem later.

Present in your ledger, missing in ChainOS. Almost always a double-credit on your side. Search your dedup table for two rows with the same (txid, outputIndex); if there are none, check whether your credit is keyed on event.id rather than on the deposit — Bitcoin fires deposit_confirmed three times, once per threshold, and a ledger that credits per event credits three times. See Idempotency.

On the chain, unknown to ChainOS. A deposit to an address that was archived, or to an address not issued through ChainOS at all. Neither is monitored, so no webhook exists and none will. The funds are still yours and still recoverable from your mnemonic plus the derivation path. Do not un-archive and expect the historic deposit to appear — monitoring resumes forward, it does not backfill.

Automating it​

  • Run it daily, on yesterday's closed window rather than on today's open one. Reconciling a window that is still receiving deposits produces discrepancies that resolve themselves and train everyone to ignore the report.
  • Alert on a non-empty result, not on a threshold. "Three discrepancies is normal" is how a real one gets missed.
  • Use the console's CSV export for a manual check — every list screen exports the same data the API returns, which is useful when someone needs to look without a token.
  • Reconcile sandbox too, in CI. A reconciliation job with a bug is worse than none, and sandbox is where you find out.

What Ziklag watches on your behalf​

One metric predicts a reconciliation gap before it becomes one, and it is on our side of the line:

chainos_event_publication_incomplete

A non-zero value that does not fall back to zero within seconds means the ledger is advancing while notifications stall — customers' balances are changing and nobody is being told. It pages us. You will see it as webhooks going quiet while balances still move, which is why the reconciliation job above is worth running even though we are watching: it is the check that does not depend on us. See Monitoring.