Monitoring

Ziklag operates Cloud. You monitor your Edge and your own webhook endpoint. The division is worth stating precisely, because the gap between the two is where an incident hides — so this page covers both halves, including what we watch on your behalf.

The one metric

chainos_event_publication_incomplete

Alert on this first, and treat it as the highest-severity signal in the system.

ChainOS's transactional event-publication registry guarantees that an event committed alongside a ledger write is eventually delivered. This gauge counts publications that have been recorded and not yet completed.

A brief non-zero value is normal — it is the window between commit and delivery. A value that does not fall back to zero within seconds means the ledger is advancing while notifications stall: customers' balances are changing on-chain, being written to the database, and nobody is being told.

No other metric shows this. Delivery success rates look fine, because nothing is being attempted. Consumer lag looks fine, because nothing was produced. The API is healthy. Only this gauge is wrong.

- alert: ChainOsEventPublicationStalled
  expr: chainos_event_publication_incomplete > 0
  for: 2m
  labels: { severity: critical }
  annotations:
    summary: Ledger advancing while notifications stall
    runbook: /docs/operations/incident-response#event-publication-stalled

Two minutes, not thirty seconds — the normal commit-to-delivery window is well under a second and a tighter window produces noise on a busy instance.

Edge-side, whatever your deployment

These are yours even on SaaS.

Signal	Source	Alert when
Edge liveness	GET /v1/edge/readiness, or the edge_offline webhook	Down beyond your tolerance
Outbox depth	GET /v1/edge/outbox	Growing monotonically for more than a few minutes
Outbox poisoned	Same endpoint, outboxPoisoned	Immediately — one write is blocking the queue
Leader present	GET /v1/edge/lease	No leader for more than the lease TTL
Lease epoch agreement	GET /v1/edge/sessions	Two sessions claim leader, or epochs disagree
Chain readiness	GET /v1/edge/readiness	Any chain amber (quarantined) — always. Red — if you use it
Container health	GET /v1/edge/health	Standard liveness probe

Amber readiness is never routine. quarantined means an identity rotation was detected — a different mnemonic is mounted against your account — and address generation is blocked until an owner approves it. Page someone. See Identity rotation.

A growing outbox almost always means your own webhook endpoint is refusing deliveries. Check that before you look at the Edge.

Your webhook endpoint

The part nobody instruments and everybody should.

Signal	Alert when
Delivery success rate	Below your normal. A drop means retries, and retries mean latency
Handler p99 latency	Rising. A slow handler becomes a delivery timeout, which becomes a duplicate
Duplicate rate	Spiking. Normal is a trickle; a spike means a rebalance or a replay
Dead-letter queue depth	Non-zero. Not "above a threshold"
Dedup-table insert conflicts	Useful as a health signal, not an error — it is idempotency working

Delivery history per webhook, with status codes and durations for every attempt, is in the console and at GET /v1/webhooks/{id}/deliveries.

Alert on a non-empty dead-letter queue, not on a depth threshold

"Three dead letters is normal" is how a real one gets missed. The event retention behind replay is 7 days; an event that ages out cannot be replayed and has to be reconciled against the chain instead. See Reconciliation.

What Ziklag watches, by role

Not yours to configure, and listed because an operations review will ask. These are the Micrometer series we alert on inside Cloud; each row is a signal with a standing alert behind it.

Ingest

Metric	Watch for
chainos_stream_events_received_total	Flat when it should not be — the provider stopped pushing
chainos_stream_signature_failures_total	Any. A signature failure is either a misconfigured secret or something worse
chainos_reconciliation_lag_blocks	Growing. The sweep that backstops the stream is falling behind

Processor

Metric	Watch for
Consumer lag on the event bus	Sustained growth on chain.events.raw or chain.events.matched
chainos_confirmations_tracked	Flat while deposits arrive
chainos_ledger_write_errors_total	Any

Dispatcher

Metric	Watch for
chainos_webhook_delivery_total{outcome}	The failure ratio
Lag on webhook.retry.2m / 5m / 30m	Growth means the ladder is backing up
webhook.dlq depth	Non-zero

Gateway

Metric	Watch for
chainos_edge_sessions_active	Zero when it should not be
chainos_edge_lease_epoch	Advancing rapidly — the lease is flapping between replicas
chainos_edge_signature_failures_total	Any

Infrastructure

Database connection saturation and replication lag, event-bus replication health, and disk headroom on the log directories.

What not to page anyone about

Signal discipline matters more than coverage. These are noisy and mean nothing on their own:

• 503 CHAIN_UNAVAILABLE on a single chain. Chain infrastructure has blips. Alert on it lasting, not on it happening. Note that a chain not enabled for your account reports this permanently by design, which is not a blip.
• A single failed webhook delivery. That is what the retry ladder is for.
• A single INSUFFICIENT_GAS. It is the normal state of an ERC-20 deposit address, not a fault.
• An Edge restart. Enrolment resumes, the outbox drains, and webhooks arrive late rather than being lost.
• A brief non-zero event-publication count. As above — it is the commit-to-delivery window.
• ALREADY_ENROLLED or RESYNC_REQUIRED once. Both are automatic recovery paths working.

A minimal alert set

If you are only going to configure five:

1. chainos_event_publication_incomplete > 0 for 2m — critical.
2. Any chain readiness quarantined — critical.
3. Dead-letter queue non-empty — high.
4. No Edge leader for longer than the lease TTL — high.
5. Outbox depth growing monotonically for 5m — high.

Everything else can be a dashboard.

Logging

Structured JSON, one line per event, with the requestId that also appears in meta.requestId on every API response. Correlate on it.

What is never logged: mnemonics, private keys, extended public keys, API keys beyond a masked prefix, webhook secrets, or TOTP secrets. If you find one of those in a log, that is a security finding — report it to security@ziklagconsulting.com.