Files
proxmox/docs/runbooks/MULTI_CHAIN_EXECUTION_INCIDENT_RESPONSE.md
defiQUG fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
docs: Ledger Live integration, contract deploy learnings, NEXT_STEPS updates
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands
- CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround
- CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check
- NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere
- MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates
- LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-12 15:46:57 -08:00

1.1 KiB

Multi-Chain Execution — Incident Response

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation


Scope: Intent API, EO, TRPE, Nonce Service, Mirroring Service, Chain Adapters (138, 651940, public).

Severity

  • P1: Execution API down or circuit breaker stuck open.
  • P2: Mirroring or proof API degraded.
  • P3: Single-chain adapter (e.g. 138 or 651940 RPC) failing.

Immediate actions

  1. Circuit breaker open

    • Check /v1/health and /v1/metrics.
    • Fix upstream (RPC, DB); circuit resets after 60s with no new errors.
    • To force-close: restart service.
  2. RPC failures

    • Verify RPC URLs (CHAIN_138_RPC_URL, CHAIN_651940_RPC_URL).
    • Use chain adapter healthCheck(); switch to fallback RPC if configured.
  3. Reorg detected

    • Follow runbook MULTI_CHAIN_EXECUTION_REORG_HANDLING.md.
    • Roll back SAL to last common ancestor; re-execute steps within TTL if applicable.
  4. Key compromise

    • Follow runbook MULTI_CHAIN_EXECUTION_KEY_ROTATION.md.

Escalation

  • P1: Page on-call; consider kill-switch if data integrity at risk.
  • P2/P3: Ticket and fix within SLA.