# Oracle Recovery Procedures

## Recovery Scenarios

### Stale Oracle Recovery

**Scenario:** Oracle has not updated in > 5 minutes

**Steps:**
1. Identify stale oracle
2. Check transmitter status
3. Verify data sources
4. Force update if needed
5. Monitor recovery

**Commands:**
```bash
# Check oracle state
cast call $AGGREGATOR_ADDRESS "latestRoundData()" --rpc-url $RPC_URL

# Force update
cast send $AGGREGATOR_ADDRESS "updateAnswer(uint256)" $PRICE \
  --rpc-url $RPC_URL \
  --private-key $PRIVATE_KEY
```

### Failed Transmitter Recovery

**Scenario:** Transmitter is unavailable

**Steps:**
1. Identify failed transmitter
2. Remove failed transmitter
3. Add backup transmitter
4. Verify updates resume
5. Monitor recovery

**Commands:**
```bash
# Remove failed transmitter
cast send $AGGREGATOR_ADDRESS "removeTransmitter(address)" $FAILED_TRANSMITTER \
  --rpc-url $RPC_URL \
  --private-key $ADMIN_KEY

# Add backup transmitter
cast send $AGGREGATOR_ADDRESS "addTransmitter(address)" $BACKUP_TRANSMITTER \
  --rpc-url $RPC_URL \
  --private-key $ADMIN_KEY
```

### Data Source Failure Recovery

**Scenario:** Data source is unavailable

**Steps:**
1. Identify failed data source
2. Switch to backup data source
3. Verify updates resume
4. Monitor recovery
5. Document incident

### CCIP Failure Recovery

**Scenario:** CCIP messages failing

**Steps:**
1. Check CCIP router status
2. Verify LINK balance
3. Check message queue
4. Retry failed messages
5. Monitor recovery

## Recovery Verification

### Verify Oracle Updates

```bash
# Check latest round
cast call $AGGREGATOR_ADDRESS "latestRoundData()" --rpc-url $RPC_URL

# Monitor updates
watch -n 5 "cast call $AGGREGATOR_ADDRESS 'latestRoundData()' --rpc-url $RPC_URL"
```

### Verify Transmitter Status

```bash
# Check transmitter list
cast call $AGGREGATOR_ADDRESS "getTransmitters()" --rpc-url $RPC_URL

# Check transmitter status
cast call $AGGREGATOR_ADDRESS "isTransmitter(address)" $TRANSMITTER --rpc-url $RPC_URL
```

## Post-Recovery

1. **Document Incident**: Document what happened
2. **Root Cause Analysis**: Identify root cause
3. **Preventive Measures**: Implement preventive measures
4. **Update Runbooks**: Update runbooks if needed
5. **Monitor**: Continue monitoring

