Edge troubleshooting

Ordered by how often each one occurs, not by severity.

Start at the Edge, not at the console

Before the first successful sync the console can tell you nothing useful: Cloud has no extended public keys for you, so every chain is red with "No Edge has connected yet."

The Edge's own status page carries the actual reason and is served before any sync:

http://localhost:8787/          # status page, in a browser
GET  /v1/edge/health            # machine-readable
GET  /v1/edge/lease             # lease state and epoch
GET  /v1/edge/outbox            # depth, and whether it is poisoned

Then the log:

docker logs --tail 200 chainos-edge

The container will not start

mnemonic file not readable

The container's uid cannot read the mounted file.

docker exec chainos-edge ls -l /run/secrets/chainos_mnemonic   # if the image had a shell
stat -c '%a %U %G' ./secrets/mnemonic.txt                      # on the host

The image is scratch and has no shell, so check on the host: mode 0400 and owned by the uid the container runs as (10001 by default). A file owned by root at 0400 is unreadable to a non-root container, which is the most common form of this.

invalid BIP-39 checksum

The phrase is not a valid mnemonic. In order of likelihood: a trailing newline included in the phrase, a typo in one word, or two words transposed.

# A trailing newline is the usual culprit — `printf`, not `echo`.
xxd ./secrets/mnemonic.txt | tail -1

The Edge validates the checksum at boot and refuses rather than continuing, which is deliberate: a mnemonic with a typo derives perfectly valid addresses that you do not hold the keys to.

EDGE_DATA_DIR is on ephemeral storage

The data directory is a tmpfs or an emptyDir. The Edge refuses, because an outbox that loses its contents on restart looks durable and is not.

Use a named Docker volume or a StatefulSet with a volumeClaimTemplate. If you genuinely accept the loss, set EDGE_OUTBOX_MODE=disabled explicitly and read what else that costs.

cannot lock key material into memory; raise RLIMIT_MEMLOCK

On Linux the Edge keeps the seed in a small buffer of its own that the kernel is told never to page to swap and never to include in a core dump. Locking that buffer needs a few kilobytes of RLIMIT_MEMLOCK and no capability — the default limit (64 KiB on older kernels, 8 MiB from 5.16) covers it. When even that is refused, the Edge refuses to start: a seed that can be paged to a swap device is not one it will hold.

• Docker: something has set --ulimit memlock to (nearly) zero. Remove it, or set --ulimit memlock=65536:65536.
• Kubernetes: the node's container runtime sets the limit. Check the runtime's LimitMEMLOCK (containerd's systemd unit) or the kubelet's configuration.
• Host: ulimit -l shows the limit the runtime inherits.

Do not add the IPC_LOCK capability. Earlier versions asked for it because they locked the whole process; this one does not need it, and Pod Security restricted forbids it.

cannot mark the process non-dumpable / cannot disable core dumps

At startup the Edge makes itself non-dumpable — so another process in the same pod, or a kubectl debug container, cannot read /proc/<pid>/environ or /proc/<pid>/mem — and forbids core files. Neither needs privilege, so this failing means the runtime is blocking prctl or setrlimit through a custom seccomp profile. Use RuntimeDefault.

this AWS identity may not call kms:Decrypt with …

The Edge reached KMS and was refused. Check, in order: the Edge's pod is using the ServiceAccount bound to the role (EKS Pod Identity association or IRSA annotation); the key policy names that role; and the policy's kms:EncryptionContext:environment condition matches the envelope's environment. CloudTrail records the refused call with the principal that made it. If the role was revoked deliberately — as an incident containment step — this is the expected result: a running Edge keeps signing, a restarted one does not start.

enclave: kms:Decrypt for this enclave was refused

KMS refused to release the data key to this enclave. In order of likelihood:

• The measurements do not match the key policy. Compare the PCR0 in the Edge's log — or nitro-cli describe-enclaves on the instance — with the policy's kms:RecipientAttestation:* conditions. A new release needs its new PCR0 added first; the guided installer's upgrade does that.
• The enclave is in debug mode. Its PCRs are all zeros and a correct policy refuses it. Remove --debug-mode (ENCLAVE_DEBUG).
• The task role lacks the grant, or the encryption-context environment condition does not match the envelope.
• Decrypt was revoked on purpose, as incident containment. Running Edges keep signing; restarts stop here. Check CloudTrail before restoring it.

EDGE_KEYSTORE=nitro takes its autonomous-spend policy from the sealed envelope

One of EDGE_SPEND_CAP_*, EDGE_SPEND_WINDOW, EDGE_SWEEP_TREASURY_ALLOWLIST or CHAINOS_FORWARDER_CONTRACTS is set. In nitro mode the enclave enforces the policy sealed with the seed and would ignore these, so the Edge refuses rather than appear to apply them. Remove them; to change the policy, reseal with seal-mnemonic --provider=nitro --policy=<file>.

capped spending is frozen on N chain(s) until …

The enclave restarted without a signed ledger snapshot from the last two minutes — the first start, a restart after a longer outage, a snapshot that failed verification, or an EDGE_DATA_DIR that did not survive the restart. It cannot tell which spends it may have missed, so gas top-ups and forwards pause on every capped chain for one spend window. Withdrawals, sweeps and consolidations are unaffected. Nothing needs doing: it lifts at the time in the message. To avoid it on routine restarts, keep EDGE_DATA_DIR on a durable volume and restart promptly.

cannot reach the enclave

The Edge could not connect to the enclave. Check nitro-cli describe-enclaves shows it RUNNING with the CID in EDGE_NITRO_CID; in a container, that EDGE_NITRO_SOCKET names the socket the launcher's bridge created on the shared volume. A container that dials vsock directly fails with operation not permitted — Docker's seccomp profile refuses it — which is what the socket bridge is for.

KMS returned plaintext to a recipient request; refusing it

KMS should answer an attested request only with CiphertextForRecipient. The enclave refuses anything else, because a plaintext key in the response would have been readable by the parent that relayed it. Report it.

this Azure identity may not perform keys/unwrap on …

The Edge reached Key Vault and was refused. Check that the pod uses the ServiceAccount federated with the managed identity (Workload Identity), and that the identity holds the unwrap-only role on the key — on Managed HSM, in its local RBAC rather than Azure RBAC.

… is not an RSA-HSM key / … is a 2048-bit key

seal-mnemonic refuses a software-protected or short Key Vault key. Create an RSA-HSM key of 3072 bits or more, with only the wrapKey and unwrapKey operations, in a Premium vault or a Managed HSM.

EDGE_AZUREKV_KEY_ID must include the key version

Use the full identifier, ending in the version. An unversioned identifier would follow key rotation and stop opening the envelope.

AZURE_CLIENT_SECRET is set

azurekv takes credentials only from Workload Identity or a managed identity. Remove the client secret or certificate path from the environment.

the envelope was not sealed under … and this encryption context

KMS could not decrypt the wrapped key with this key and context: the envelope was sealed under a different key, or its context was edited. Re-seal with seal-mnemonic.

the key … is disabled or pending deletion

Re-enable the key, or cancel the deletion, in KMS. If the key is gone, re-seal under a new key from the offline backup of the phrase.

the envelope was sealed for a different environment

The envelope's environment is live and the API key is flk_test_…, or the reverse. The Edge refuses before calling KMS, so this never shows up as a KMS error. Seal a separate envelope per environment.

the envelope was sealed under …, and this Edge is configured for …

EDGE_AWSKMS_KEY_ID and the envelope's keyRef differ. One of them is wrong; the Edge will not decrypt with whichever key an envelope happens to name.

the sealed mnemonic does not derive the identity recorded in its envelope

The envelope opened, but the phrase inside it is not the one the custodian sealed for this Edge — the wrong file is mounted, or envelopes for two wallets were mixed up. Nothing reached Cloud. Mount the right envelope.

AWS_ACCESS_KEY_ID is set

A sealed mode takes AWS credentials only from the workload identity. Remove the static keys from the environment and bind the pod's ServiceAccount to the Edge role (EKS Pod Identity or IRSA).

… takes the mnemonic from PASS_PHRASE_SEALED_FILE, but PASS_PHRASE is also set

Remove PASS_PHRASE and PASS_PHRASE_FILE — and the plaintext secret they came from. A plaintext copy beside the sealed one is exactly what sealing removes.

API_KEY has an unrecognised prefix

The key must begin flk_live_ or flk_test_. A zkl_* key is one the Edge issues to your application — it is not a Ziklag credential and cannot be used here. See API conventions.

It starts but never syncs

503 UPSTREAM_UNAVAILABLE in the log

The Edge cannot reach Cloud. Every connection is outbound from the Edge, so this is egress:

# From the Edge's network namespace or an equivalent pod
curl -sv https://api.chainos.cloud/actuator/health

Check the egress proxy, the allowlist, and TLS interception — a corporate MITM proxy breaks the mTLS channel specifically, and it fails at handshake rather than with a clear message. If CHAINOS_CLOUD_URL is set at all, check it: it exists for staging and a stale value left in a production manifest points the Edge at nothing.

409 ALREADY_ENROLLED

Cloud already has a completed enrolment for this identity and a full sync would register nothing new. The Edge handles this automatically by switching to POST /v1/edge/resume, so a single occurrence in the log during recovery is normal.

Repeated occurrences mean the Edge lost its local enrolment cache and is re-attempting a full sync each boot. Usually that means the data directory is not actually persisting — check the volume — or EDGE_OUTBOX_MODE=disabled, which has nowhere to cache it.

409 RESYNC_REQUIRED

Cloud has no enrolment for this Edge — restored from a backup, or a sync that died partway. The Edge drops its stale local copy and enrols again automatically. No action needed.

401 EDGE_SIGNATURE_SKEW

The Edge's clock is more than 300 seconds from Cloud's. Fix NTP on the host. This is not a ChainOS setting to relax; the window exists to bound replay.

401 EDGE_SIGNATURE_INVALID or 409 IDENTITY_ROTATION_PENDING

These are security events, not transient failures. A different mnemonic is mounted than the one Cloud pinned. Do not retry, do not restart in a loop, and do not approve the rotation without knowing why the identity changed.

Go to Identity rotation.

409 CROSS_ENVIRONMENT_IDENTITY

The same mnemonic is in use in both live and sandbox. Generate a separate sandbox phrase — see Environments.

It syncs but a chain is not ready

Check readiness rather than guessing:

curl -s $CLOUD/v1/edge/readiness -H "Authorization: Bearer $TOKEN" | jq

State	Meaning	Address generation
not_ready (red)	No extended public key registered, or for Solana and Stellar no pool	Blocked
ready (green)	Registered and healthy	Allowed
quarantined (amber)	Rotation detected, awaiting owner approval	Blocked
pool_low (orange)	Pooled chains only — Solana or Stellar below 20%	Allowed, replenish requested

A chain that is ready but whose balances return 503 CHAIN_UNAVAILABLE is a Cloud-side problem: the chain RPC endpoint is down or not configured. Derivation needs no RPC, which is why address issuance keeps working while balances do not. Note that a zero is never reported in this case — a confident zero is indistinguishable from an emptied wallet.

Webhooks are not arriving

In this order:

1. Is the Edge the leader? GET /v1/edge/lease. Only the leader relays. A fleet with no leader delivers nothing — see High availability.
2. Is the outbox draining? GET /v1/edge/outbox. A depth that only grows means your endpoint is refusing deliveries. outboxPoisoned means a single write keeps failing and is blocking the queue behind it.
3. Is your endpoint returning 2xx? Delivery history is in the console per webhook, with the status code and duration of every attempt.
4. Are you on the retry ladder? Delivery, then 2m, 5m, 30m, then the dead-letter queue. A flapping endpoint can be up to 37 minutes behind before it dead-letters.
5. Is it in the dead-letter queue? The console lists it with a replay action.

507 EDGE_OUTBOX_FULL means the local queue reached capacity. Fix the endpoint; the queue drains in order once deliveries succeed.

Signing fails

Error	Cause
423 EDGE_OFFLINE	No live Edge replica. Also raised for Solana and Stellar address allocation when the pool is exhausted
423 EDGE_NO_LEADER	Replicas are live but none holds the lease. Check for a partition
INSUFFICIENT_FUNDS	Not enough of the asset being sent
INSUFFICIENT_GAS	Enough of the asset, not enough native currency for the fee. The common one
INSUFFICIENT_RESERVE	Nominally enough, but the send would breach a locked reserve (XRP, Solana, Stellar)
DESTINATION_OPT_IN_REQUIRED	Stellar only. The destination holds no trustline for the asset, and only it can establish one
422 CHAIN_REJECTED	The node refused the transaction. The node's reason is in error.details

The three insufficient-funds codes are separate because they have three different remedies. See Withdrawals.

Diagnostics to collect before opening a ticket

docker logs --tail 500 chainos-edge > edge.log
curl -s http://localhost:8787/v1/edge/health  > health.json
curl -s http://localhost:8787/v1/edge/lease   > lease.json
curl -s http://localhost:8787/v1/edge/outbox  > outbox.json
docker inspect chainos-edge \
  | jq '.[0].Config.Env | map(select(startswith("PASS_PHRASE") == false))' > env.json

The jq filter drops PASS_PHRASE*. Check the output before you send it anyway — and if you find your mnemonic in there, you are using PASS_PHRASE instead of PASS_PHRASE_FILE, which is its own finding.

Include the meta.requestId from any failing API response. It is what lets support find the request on our side.