Files
proxmox/docs/runbooks/RECOVERY_PROCEDURES.md
defiQUG cb47cce074 Complete markdown files cleanup and organization
- Organized 252 files across project
- Root directory: 187 → 2 files (98.9% reduction)
- Moved configuration guides to docs/04-configuration/
- Moved troubleshooting guides to docs/09-troubleshooting/
- Moved quick start guides to docs/01-getting-started/
- Moved reports to reports/ directory
- Archived temporary files
- Generated comprehensive reports and documentation
- Created maintenance scripts and guides

All files organized according to established standards.
2026-01-06 01:46:25 -08:00

1.9 KiB

Recovery Procedures

Purpose: Step-by-step recovery procedures for various failure scenarios


🔄 Transaction Recovery

Stuck Transaction

Scenario: Transaction pending for extended period

Recovery:

  1. Check transaction status

    cast tx <tx_hash> --rpc-url $RPC_URL
    
  2. Check nonce

    cast nonce <address> --rpc-url $RPC_URL
    
  3. Retry with higher gas

    bash scripts/retry-failed-transactions.sh
    

💰 Balance Recovery

Insufficient Balance

Scenario: Not enough ETH or LINK for operations

Recovery:

  1. Check current balances

    bash scripts/check-bridge-status.sh
    
  2. Transfer funds

    • Transfer ETH to deployer address
    • Transfer LINK tokens if needed
  3. Verify balances

    cast balance <address> --rpc-url $RPC_URL
    

🔗 Chain Configuration Recovery

Missing Destination Chain

Scenario: Destination chain not configured

Recovery:

  1. Verify chain selector

  2. Add destination

    bash scripts/configure-bridge-destinations.sh
    
  3. Verify configuration

    bash scripts/verify-bridge-configuration.sh
    

🌐 Network Recovery

RPC Outage

Scenario: RPC endpoint unavailable

Recovery:

  1. Check RPC status

    cast block-number --rpc-url $RPC_URL
    
  2. Switch to backup RPC

    • Update RPC_URL in .env
    • Restart services
  3. Verify connectivity

    bash scripts/health-check.sh
    

🔒 Security Recovery

Unauthorized Access

Scenario: Suspicious activity detected

Recovery:

  1. Pause bridge immediately

    cast send <BRIDGE_ADDRESS> "pause()" --rpc-url $RPC_URL --private-key $PRIVATE_KEY
    
  2. Review logs

    tail -1000 logs/alerts-*.log
    
  3. Rotate keys if needed

  4. Resume after security review


Last Updated: $(date)