High availability

An Edge scales horizontally. Every replica serves the full local API, and exactly one may deliver webhooks — two would double-deliver every event.

Why replicas need no coordination

The Edge identity key is derived from the mnemonic, at m/83696968'/0'/0', rather than enrolled. Ten containers started from the same mnemonic all compute the same identity, so they present to Cloud as one Edge with no registration handshake and no shared state between them.

That is the whole reason horizontal scaling is easy here. Nothing has to be distributed because nothing is negotiated.

In a sealed keystore mode, every replica unseals independently at boot. Each one asks the KMS to unwrap the envelope with its own workload identity, once, and never again until it restarts. The KMS is therefore a start-up dependency, not a signing dependency: running replicas keep signing through a KMS outage or a revoked role, and a replica that restarts during one does not come back until the KMS answers. Scale up before a planned KMS change, not during it. Expect one Decrypt (or UnwrapKey) per replica start in the audit log, and alert on more.

The leader lease

Cloud grants the lease, not the Edge fleet. A partial unique index is the election:

CREATE UNIQUE INDEX idx_one_leader ON edge_sessions(account_id, environment)
    WHERE is_leader;

The database decides. No consensus protocol, no external coordinator, no fourth container.

Leaders and followers are identical from your application's point of view. Both serve the full API on 8787 and both sign. The difference is only which one holds the stream to Cloud and therefore relays webhooks and answers pool requests for Solana and Stellar.

Epoch fencing

Each grant carries a monotonically increasing epoch, and Cloud rejects any frame presenting a stale one with 409 LEASE_EPOCH_STALE, closing the stream.

This is the part that matters under a network partition. A replica cut off from Cloud may still believe it holds the lease — it has no way to know otherwise — but it cannot act, because Cloud has already issued a higher epoch to whoever took over. The stale leader's frames are refused before they are read.

Without the epoch, a partition that healed would produce two replicas both convinced they were the leader, both delivering, and every webhook arriving twice.

Failover

Graceful shutdown releases the lease, so a planned restart fails over in under a second rather than waiting for the TTL. An unplanned loss — a killed pod, a partition — waits for EDGE_LEASE_TTL_SECONDS (45 by default, which is one missed heartbeat of slack) before another replica can acquire.

How many replicas

Replicas	What you get
1	Correct. Webhooks queue in the outbox during a restart and drain after
2	Continuous delivery through a rolling restart
3+	Headroom, and safe with EDGE_OUTBOX_MODE=disabled

Two is the usual answer. Go to three if you have disabled the outbox, because then a rolling restart with two replicas can briefly leave zero leaders and there is nowhere for the undelivered writes to go.

EDGE_LEADER_ELIGIBLE=false pins a replica as follower-only — useful for a replica in a different region that you want serving reads without becoming the delivery path.

Configuration for a fleet

spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: edge
          env:
            - name: PASS_PHRASE_FILE
              value: /run/secrets/chainos/mnemonic     # the same mnemonic on every replica
            - name: EDGE_POD_HINT
              valueFrom:
                fieldRef: { fieldPath: metadata.name } # display only, never identity
            - name: EDGE_LEASE_TTL_SECONDS
              value: "45"
      terminationGracePeriodSeconds: 20                # room to release the lease

The same mnemonic on every replica

Every replica must mount the same mnemonic. A replica with a different one derives a different identity, and Cloud treats that as an identity rotation: it quarantines the keys and pages your account owner. That is the correct behaviour and it will happen the moment a misconfigured replica starts.

Each replica still needs its own persistent data directory — a volumeClaimTemplate on a StatefulSet, not a shared volume. Two processes writing one bbolt file corrupt it.

Watching for split-brain

Settings → Edge sessions in the console lists every active session with its role, version, liveness, lease epoch, outbox mode, last heartbeat and pod hint. Ended sessions are reaped after an hour, so the list reflects the running fleet rather than an accumulating history.

The lease epoch is the column to look at. Every per-session health indicator will be green during a split-brain — each replica genuinely is healthy — and the epoch is the only thing that shows two replicas disagreeing about who leads. It is surfaced for exactly this reason.

curl -s $CLOUD/v1/edge/sessions -H "Authorization: Bearer $TOKEN" \
  | jq '.data[] | { podHint, role, leaseEpoch, liveness, outboxDepth }'

Expect one leader and the rest follower, all reporting the same leaseEpoch. Two sessions claiming leader, or a follower reporting an epoch above the leader's, is a genuine incident — see Incident response.

What a fleet does not fix

• Signing still needs at least one live replica. Zero live replicas means 423 EDGE_OFFLINE on anything requiring a signature.
• Pool replenishment is leader-only. Followers do not derive pool addresses, so a fleet with no leader and an exhausted pool cannot issue Solana or Stellar addresses.
• Deposits are unaffected either way. Monitoring, confirmation tracking and webhook generation are entirely Cloud-side. A completely offline Edge means webhooks arrive late, never that they are lost.
• A shared mnemonic is a shared blast radius. Replicas do not compartmentalise key material; they multiply the number of processes holding it. That is the trade for the no-coordination property, and it is why the hardening in Installation matters more as you add replicas.
• Mixed keystore modes are compatible and should be brief. A rollout from mnemonic to a sealed mode runs both for a while; they present the same identity and derive the same addresses. The security posture is the weakest replica's, so finish the rollout and remove the plaintext secret. The portal shows each session's keystore.