# CCIP Recovery Procedures

## Overview

This document provides recovery procedures for CCIP failures and outages.

## Recovery Scenarios

### Scenario 1: Router Failure

**Symptoms**: Router service is down or unresponsive

**Recovery Steps**:

1. **Check Router Status**
   ```bash
   kubectl get pods -n besu-network -l app=ccip-router
   kubectl describe pod <router-pod> -n besu-network
   ```

2. **Restart Router**
   ```bash
   kubectl delete pod <router-pod> -n besu-network
   # Wait for new pod to start
   kubectl get pods -n besu-network -l app=ccip-router
   ```

3. **Verify Recovery**
   ```bash
   # Test router connectivity
   cast call $CCIP_ROUTER "getSupportedTokens(uint64)" $CHAIN_SELECTOR --rpc-url $RPC_URL
   ```

4. **Resume Operations**
   - Monitor message sending
   - Verify message delivery
   - Check for backlog

### Scenario 2: Contract Failure

**Symptoms**: Sender or receiver contract is not functioning

**Recovery Steps**:

1. **Identify Issue**
   ```bash
   # Check contract state
   cast call $SENDER "paused()" --rpc-url $RPC_URL
   cast call $RECEIVER "processedMessages(bytes32)" $MESSAGE_ID --rpc-url $RPC_URL
   ```

2. **Pause Operations** (if needed)
   ```solidity
   // Call pause function if available
   sender.pause();
   ```

3. **Fix Contract**
   - Update configuration
   - Fix bugs if any
   - Deploy new version if needed

4. **Resume Operations**
   ```solidity
   // Unpause if paused
   sender.unpause();
   ```

5. **Verify Recovery**
   - Test message sending
   - Verify message receiving
   - Monitor for issues

### Scenario 3: Message Backlog

**Symptoms**: Messages queued but not being processed

**Recovery Steps**:

1. **Assess Backlog**
   ```bash
   # Check pending messages
   cast logs --from-block latest-1000 --address $SENDER --rpc-url $RPC_URL | grep MessageSent
   ```

2. **Identify Cause**
   - Check router status
   - Verify LINK balance
   - Check target chain status

3. **Clear Backlog**
   - Fix underlying issue
   - Process messages in order
   - Monitor processing

4. **Prevent Future Backlog**
   - Increase processing capacity
   - Improve error handling
   - Add monitoring

### Scenario 4: Data Corruption

**Symptoms**: Invalid messages or corrupted data

**Recovery Steps**:

1. **Identify Corrupted Messages**
   - Review message logs
   - Check for invalid formats
   - Identify affected messages

2. **Isolate Issue**
   - Pause message processing
   - Prevent further corruption
   - Assess impact

3. **Recover Data**
   - Resend valid messages
   - Skip corrupted messages
   - Update data if possible

4. **Prevent Recurrence**
   - Fix encoding/decoding
   - Add validation
   - Improve error handling

### Scenario 5: Network Partition

**Symptoms**: Cannot communicate with target chain

**Recovery Steps**:

1. **Verify Connectivity**
   ```bash
   # Test target chain connectivity
   curl -X POST $TARGET_CHAIN_RPC_URL -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
   ```

2. **Wait for Recovery**
   - Monitor network status
   - Check chain status pages
   - Wait for partition to resolve

3. **Resume Operations**
   - Verify connectivity restored
   - Process queued messages
   - Resume normal operations

## Recovery Verification

### Post-Recovery Checks

1. **Functionality**
   - Test message sending
   - Verify message receiving
   - Check fee calculation

2. **Performance**
   - Check message latency
   - Verify throughput
   - Monitor error rates

3. **Data Integrity**
   - Verify message order
   - Check for duplicates
   - Validate message content

## Prevention

### Best Practices

1. **Monitoring**
   - Set up comprehensive monitoring
   - Configure alerts
   - Regular health checks

2. **Redundancy**
   - Deploy multiple router instances
   - Use backup contracts
   - Maintain backup configurations

3. **Testing**
   - Regular disaster recovery drills
   - Test recovery procedures
   - Update procedures based on tests

4. **Documentation**
   - Keep runbooks updated
   - Document lessons learned
   - Share knowledge

## References

- [CCIP Operations Runbook](ccip-operations.md)
- [CCIP Incident Response](ccip-incident-response.md)
- [CCIP Troubleshooting](../docs/CCIP_TROUBLESHOOTING.md)

