Webhooks

Nine events, HMAC-signed, at-least-once, delivered Cloud → your Edge → your endpoint.

Because the last hop is from your own Edge inside your own network, your endpoint needs no public ingress. That is usually the single biggest difference between integrating ChainOS and integrating a hosted provider.

Register an endpoint

curl -X POST $EDGE/webhooks \
  -H "X-API-Key: $CHAINOS_EDGE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://core.internal.bank.example/webhooks/chainos",
    "events": ["deposit_detected", "deposit_confirmed",
               "withdrawal_confirmed", "withdrawal_failed"]
  }'

{
  "success": true,
  "data": {
    "id": "wh_01J8XKG",
    "url": "https://core.internal.bank.example/webhooks/chainos",
    "events": ["deposit_detected", "deposit_confirmed", "withdrawal_confirmed", "withdrawal_failed"],
    "secret": "whsec_4f2a…",
    "active": true
  }
}

The secret is shown once. It is the HMAC key for every delivery to that endpoint. Store it with your other secrets now; there is no way to read it back, only to rotate it.

The nine events

Event	Fired when
deposit_detected	Inbound transaction first seen, 1 confirmation
deposit_confirmed	Inbound reaches a configured threshold. Fires once per threshold
withdrawal_broadcast	Outbound accepted by the network
withdrawal_confirmed	Outbound reaches its threshold
withdrawal_failed	Outbound reverted or rejected
address_activated	XRP address funded past the reserve; Stellar account created by its first payment; first use on other chains
edge_offline	Your Edge is down beyond the threshold. Account-level, not a chain event
edge_online	Your Edge recovered
identity_rotation_detected	Security event. Always delivered, regardless of your subscription filter

That last row is not a subscription you can opt out of. A different mnemonic is mounted against your account; you are being told whether you asked to be or not.

Verifying the signature

Every delivery carries X-Webhook-Signature: the hex-encoded HMAC-SHA256 of the raw request body under your endpoint's secret.

Verify against the bytes you received

Not against a re-serialised object. Not against JSON.stringify(req.body). Not against a Jackson or Gson round-trip.

Re-serialising produces the same bytes most of the time — which is what makes this bug so unpleasant. It fails only when key ordering, whitespace or number formatting happens to differ, so it passes every test you write and then rejects a fraction of production deliveries with no pattern anyone can see.

Capture the raw body before any parser touches it.

Node — Express

import crypto from 'node:crypto';
import express from 'express';

const app = express();

app.post('/webhooks/chainos',
  // express.raw, not express.json — the Buffer is the point.
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const expected = crypto
      .createHmac('sha256', process.env.CHAINOS_WEBHOOK_SECRET)
      .update(req.body)                     // Buffer, not parsed JSON
      .digest('hex');

    const given = req.header('x-webhook-signature') ?? '';

    // Length check first: timingSafeEqual throws on a length mismatch.
    if (given.length !== expected.length ||
        !crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(given))) {
      return res.status(401).send('bad signature');
    }

    const evt = JSON.parse(req.body.toString('utf8'));

    // Ack fast, work later. See "Respond quickly" below.
    enqueue(evt);
    res.sendStatus(200);
  });

Java — Spring

@PostMapping(path = "/webhooks/chainos", consumes = MediaType.APPLICATION_JSON_VALUE)
ResponseEntity<Void> receive(@RequestBody byte[] rawBody,
                            @RequestHeader("X-Webhook-Signature") String signature)
        throws Exception {

    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(secret.getBytes(UTF_8), "HmacSHA256"));
    String expected = HexFormat.of().formatHex(mac.doFinal(rawBody));

    // Constant-time comparison. String.equals leaks timing information.
    if (!MessageDigest.isEqual(expected.getBytes(UTF_8), signature.getBytes(UTF_8))) {
        return ResponseEntity.status(401).build();
    }

    events.enqueue(mapper.readValue(rawBody, WebhookEvent.class));
    return ResponseEntity.ok().build();
}

byte[] as the parameter type, not a DTO. Binding to a DTO gives Spring the parsed object and throws the bytes away.

Python — Flask

import hashlib, hmac, os
from flask import Flask, request, abort

app = Flask(__name__)
SECRET = os.environ["CHAINOS_WEBHOOK_SECRET"].encode()

@app.post("/webhooks/chainos")
def receive():
    raw = request.get_data()                       # bytes, before any json parsing
    expected = hmac.new(SECRET, raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, request.headers.get("X-Webhook-Signature", "")):
        abort(401)
    enqueue(request.get_json())
    return "", 200

Go

func receive(w http.ResponseWriter, r *http.Request) {
    raw, err := io.ReadAll(r.Body)
    if err != nil { http.Error(w, "", http.StatusBadRequest); return }

    m := hmac.New(sha256.New, []byte(os.Getenv("CHAINOS_WEBHOOK_SECRET")))
    m.Write(raw)
    expected := hex.EncodeToString(m.Sum(nil))

    if !hmac.Equal([]byte(expected), []byte(r.Header.Get("X-Webhook-Signature"))) {
        http.Error(w, "bad signature", http.StatusUnauthorized)
        return
    }

    var evt Event
    if err := json.Unmarshal(raw, &evt); err != nil { http.Error(w, "", 400); return }
    enqueue(evt)
    w.WriteHeader(http.StatusOK)
}

Respond quickly, then work

Acknowledge with a 2xx as soon as the signature verifies and the event is durably queued on your side. Do the ledger write, the customer notification and the downstream calls afterwards.

An endpoint that does its full processing inline before responding turns every slow dependency into a delivery timeout, and a timeout puts you on the retry ladder — where the same event arrives again while the first one is still being processed. That is the shape of most double-credit bugs.

Delivery guarantees

At-least-once. Duplicates are possible after a broker rebalance or an Edge reconnect, and your consumers must be idempotent on event.id. This is the documented contract, not an edge case — see Idempotency.

Ordered per address. The event topic is partitioned by chain:address and consumed by a single partition owner, so events for one address arrive in order. Ordering across addresses is not guaranteed and must not be relied on.

The retry ladder, using separate queue topics rather than in-process timers so a dispatcher restart does not lose scheduled retries:

A flapping endpoint can therefore be up to about 37 minutes behind before an event dead-letters.

The dead-letter queue

Visible in the console with a replay action, and via the API:

curl -s $CLOUD/v1/webhooks/deliveries/dead-letters -H "Authorization: Bearer $TOKEN"
curl -X POST $CLOUD/v1/webhooks/deliveries/{id}/replay -H "Authorization: Bearer $TOKEN"

Replay is a deliberate operator action, not automatic. Fix the endpoint first, then replay — an automatic replay against a still-broken endpoint just re-runs the ladder.

The event retention behind this is 7 days. Events older than that cannot be replayed, so a dead-letter queue that has been accumulating for a week needs attention rather than a note in a backlog. Reconcile against the chain for anything beyond the window; see Reconciliation.

Inspecting deliveries

curl -s "$EDGE/webhooks/wh_01J8XKG/deliveries?size=50" -H "X-API-Key: $CHAINOS_EDGE_KEY"

Every attempt, with its HTTP status code, duration and attempt number. The console shows the same per webhook. This is the first place to look when your handler is behaving and events still are not arriving — the status code is usually the whole answer.

If nothing arrives

In order:

1. Is your Edge the leader? Only the leader relays. GET /v1/edge/lease.
2. Is the outbox draining? GET /v1/edge/outbox. A depth that only grows means your endpoint is refusing. outboxPoisoned means one write keeps failing and blocking the queue behind it.
3. Is your endpoint returning 2xx? Check the delivery history above.
4. Is the event in your subscription? A webhook subscribed to deposit_confirmed only will never see deposit_detected.
5. Is it dead-lettered?

Troubleshooting has the full sequence.

Rotating a secret

curl -X PUT $EDGE/webhooks/wh_01J8XKG \
  -H "X-API-Key: $CHAINOS_EDGE_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "rotateSecret": true }'

The new secret is returned once and takes effect immediately, so deliveries in flight at that moment are signed with the old one. Accept both secrets for a few minutes during a rotation, then drop the old one.