Skip to main content

Edge troubleshooting

Ordered by how often each one occurs, not by severity.

Start at the Edge, not at the console​

Before the first successful sync the console can tell you nothing useful: Cloud has no extended public keys for you, so every chain is red with "No Edge has connected yet."

The Edge's own status page carries the actual reason and is served before any sync:

http://localhost:8787/          # status page, in a browser
GET  /v1/edge/health            # machine-readable
GET  /v1/edge/lease             # lease state and epoch
GET  /v1/edge/outbox            # depth, and whether it is poisoned

Then the log:

docker logs --tail 200 chainos-edge

The container will not start​

mnemonic file not readable​

The container's uid cannot read the mounted file.

docker exec chainos-edge ls -l /run/secrets/chainos_mnemonic   # if the image had a shell
stat -c '%a %U %G' ./secrets/mnemonic.txt                      # on the host

The image is scratch and has no shell, so check on the host: mode 0400 and owned by the uid the container runs as (10001 by default). A file owned by root at 0400 is unreadable to a non-root container, which is the most common form of this.

invalid BIP-39 checksum​

The phrase is not a valid mnemonic. In order of likelihood: a trailing newline included in the phrase, a typo in one word, or two words transposed.

# A trailing newline is the usual culprit — `printf`, not `echo`.
xxd ./secrets/mnemonic.txt | tail -1

The Edge validates the checksum at boot and refuses rather than continuing, which is deliberate: a mnemonic with a typo derives perfectly valid addresses that you do not hold the keys to.

EDGE_DATA_DIR is on ephemeral storage​

The data directory is a tmpfs or an emptyDir. The Edge refuses, because an outbox that loses its contents on restart looks durable and is not.

Use a named Docker volume or a StatefulSet with a volumeClaimTemplate. If you genuinely accept the loss, set EDGE_OUTBOX_MODE=disabled explicitly and read what else that costs.

mlock failed​

The Edge locks the memory region holding the seed so it cannot be paged to swap, and fails rather than continuing with a swappable seed.

  • Docker: add --cap-add IPC_LOCK, or raise --ulimit memlock.
  • Kubernetes: securityContext.capabilities.add: ["IPC_LOCK"].
  • Host: check ulimit -l; a low memlock limit on the host caps what the container can lock regardless of capabilities.

API_KEY has an unrecognised prefix​

The key must begin flk_live_ or flk_test_. A zkl_* key is one the Edge issues to your application — it is not a Ziklag credential and cannot be used here. See API conventions.

It starts but never syncs​

503 UPSTREAM_UNAVAILABLE in the log​

The Edge cannot reach Cloud. Every connection is outbound from the Edge, so this is egress:

# From the Edge's network namespace or an equivalent pod
curl -sv https://api.chainos.cloud/actuator/health

Check the egress proxy, the allowlist, and TLS interception — a corporate MITM proxy breaks the mTLS channel specifically, and it fails at handshake rather than with a clear message. If CHAINOS_CLOUD_URL is set at all, check it: it exists for staging and a stale value left in a production manifest points the Edge at nothing.

409 ALREADY_ENROLLED​

Cloud already has a completed enrolment for this identity and a full sync would register nothing new. The Edge handles this automatically by switching to POST /v1/edge/resume, so a single occurrence in the log during recovery is normal.

Repeated occurrences mean the Edge lost its local enrolment cache and is re-attempting a full sync each boot. Usually that means the data directory is not actually persisting — check the volume — or EDGE_OUTBOX_MODE=disabled, which has nowhere to cache it.

409 RESYNC_REQUIRED​

Cloud has no enrolment for this Edge — restored from a backup, or a sync that died partway. The Edge drops its stale local copy and enrols again automatically. No action needed.

401 EDGE_SIGNATURE_SKEW​

The Edge's clock is more than 300 seconds from Cloud's. Fix NTP on the host. This is not a ChainOS setting to relax; the window exists to bound replay.

401 EDGE_SIGNATURE_INVALID or 409 IDENTITY_ROTATION_PENDING​

These are security events, not transient failures. A different mnemonic is mounted than the one Cloud pinned. Do not retry, do not restart in a loop, and do not approve the rotation without knowing why the identity changed.

Go to Identity rotation.

409 CROSS_ENVIRONMENT_IDENTITY​

The same mnemonic is in use in both live and sandbox. Generate a separate sandbox phrase — see Environments.

It syncs but a chain is not ready​

Check readiness rather than guessing:

curl -s $CLOUD/v1/edge/readiness -H "Authorization: Bearer $TOKEN" | jq
StateMeaningAddress generation
not_ready (red)No extended public key registered, or for Solana and Stellar no poolBlocked
ready (green)Registered and healthyAllowed
quarantined (amber)Rotation detected, awaiting owner approvalBlocked
pool_low (orange)Pooled chains only — Solana or Stellar below 20%Allowed, replenish requested

A chain that is ready but whose balances return 503 CHAIN_UNAVAILABLE is a Cloud-side problem: the chain RPC endpoint is down or not configured. Derivation needs no RPC, which is why address issuance keeps working while balances do not. Note that a zero is never reported in this case — a confident zero is indistinguishable from an emptied wallet.

Webhooks are not arriving​

In this order:

  1. Is the Edge the leader? GET /v1/edge/lease. Only the leader relays. A fleet with no leader delivers nothing — see High availability.
  2. Is the outbox draining? GET /v1/edge/outbox. A depth that only grows means your endpoint is refusing deliveries. outboxPoisoned means a single write keeps failing and is blocking the queue behind it.
  3. Is your endpoint returning 2xx? Delivery history is in the console per webhook, with the status code and duration of every attempt.
  4. Are you on the retry ladder? Delivery, then 2m, 5m, 30m, then the dead-letter queue. A flapping endpoint can be up to 37 minutes behind before it dead-letters.
  5. Is it in the dead-letter queue? The console lists it with a replay action.

507 EDGE_OUTBOX_FULL means the local queue reached capacity. Fix the endpoint; the queue drains in order once deliveries succeed.

Signing fails​

ErrorCause
423 EDGE_OFFLINENo live Edge replica. Also raised for Solana and Stellar address allocation when the pool is exhausted
423 EDGE_NO_LEADERReplicas are live but none holds the lease. Check for a partition
INSUFFICIENT_FUNDSNot enough of the asset being sent
INSUFFICIENT_GASEnough of the asset, not enough native currency for the fee. The common one
INSUFFICIENT_RESERVENominally enough, but the send would breach a locked reserve (XRP, Solana, Stellar)
DESTINATION_OPT_IN_REQUIREDStellar only. The destination holds no trustline for the asset, and only it can establish one
422 CHAIN_REJECTEDThe node refused the transaction. The node's reason is in error.details

The three insufficient-funds codes are separate because they have three different remedies. See Withdrawals.

Diagnostics to collect before opening a ticket​

docker logs --tail 500 chainos-edge > edge.log
curl -s http://localhost:8787/v1/edge/health  > health.json
curl -s http://localhost:8787/v1/edge/lease   > lease.json
curl -s http://localhost:8787/v1/edge/outbox  > outbox.json
docker inspect chainos-edge \
  | jq '.[0].Config.Env | map(select(startswith("PASS_PHRASE") == false))' > env.json

The jq filter drops PASS_PHRASE*. Check the output before you send it anyway — and if you find your mnemonic in there, you are using PASS_PHRASE instead of PASS_PHRASE_FILE, which is its own finding.

Include the meta.requestId from any failing API response. It is what lets support find the request on our side.