Complete markdown files cleanup and organization
- Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
This commit is contained in:
128
docs/runbooks/BRIDGE_OPERATIONS_RUNBOOK.md
Normal file
128
docs/runbooks/BRIDGE_OPERATIONS_RUNBOOK.md
Normal file
@@ -0,0 +1,128 @@
|
||||
# Bridge Operations Runbook
|
||||
|
||||
**Purpose**: Step-by-step procedures for common bridge operations
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Bridge Transfer Procedure
|
||||
|
||||
### Standard Transfer
|
||||
|
||||
1. **Check System Status**
|
||||
```bash
|
||||
bash scripts/health-check.sh
|
||||
```
|
||||
|
||||
2. **Verify Balances**
|
||||
```bash
|
||||
bash scripts/check-bridge-status.sh
|
||||
```
|
||||
|
||||
3. **Estimate Fees**
|
||||
```bash
|
||||
bash scripts/fee-management.sh estimate
|
||||
```
|
||||
|
||||
4. **Execute Transfer**
|
||||
```bash
|
||||
bash scripts/bridge-with-dynamic-gas.sh weth9 1.0 bsc
|
||||
```
|
||||
|
||||
5. **Monitor Transfer**
|
||||
```bash
|
||||
bash scripts/monitor-bridge-transfers.sh <transaction_hash>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚨 Emergency Procedures
|
||||
|
||||
### Pause Bridge Operations
|
||||
|
||||
1. **Access Bridge Contract**
|
||||
```bash
|
||||
cast send <BRIDGE_ADDRESS> "pause()" --rpc-url $RPC_URL --private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
2. **Verify Pause**
|
||||
```bash
|
||||
cast call <BRIDGE_ADDRESS> "paused()" --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
3. **Resume Operations**
|
||||
```bash
|
||||
cast send <BRIDGE_ADDRESS> "unpause()" --rpc-url $RPC_URL --private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Transaction Stuck
|
||||
|
||||
1. **Check Transaction Status**
|
||||
```bash
|
||||
cast tx <transaction_hash> --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
2. **Retry with Higher Gas**
|
||||
```bash
|
||||
bash scripts/retry-failed-transactions.sh
|
||||
```
|
||||
|
||||
3. **Check Nonce**
|
||||
```bash
|
||||
cast nonce <address> --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
### Low Balance
|
||||
|
||||
1. **Check Balances**
|
||||
```bash
|
||||
bash scripts/check-bridge-status.sh
|
||||
```
|
||||
|
||||
2. **Add Funds**
|
||||
- Transfer ETH to deployer address
|
||||
- Transfer LINK tokens if needed
|
||||
|
||||
### RPC Issues
|
||||
|
||||
1. **Test RPC Connectivity**
|
||||
```bash
|
||||
cast block-number --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
2. **Check RPC Health**
|
||||
```bash
|
||||
bash scripts/health-check.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### Daily Checks
|
||||
|
||||
```bash
|
||||
# Run automated monitoring
|
||||
bash scripts/automated-monitoring.sh
|
||||
|
||||
# Generate daily report
|
||||
bash scripts/generate-bridge-report.sh daily
|
||||
```
|
||||
|
||||
### Weekly Review
|
||||
|
||||
```bash
|
||||
# Generate weekly report
|
||||
bash scripts/generate-bridge-report.sh weekly
|
||||
|
||||
# Run test suite
|
||||
bash scripts/test-suite.sh all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: $(date)
|
||||
|
||||
129
docs/runbooks/INCIDENT_RESPONSE_RUNBOOK.md
Normal file
129
docs/runbooks/INCIDENT_RESPONSE_RUNBOOK.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Incident Response Runbook
|
||||
|
||||
**Purpose**: Procedures for responding to bridge system incidents
|
||||
|
||||
---
|
||||
|
||||
## 🚨 Incident Classification
|
||||
|
||||
### Critical (P0)
|
||||
- Bridge contract not accessible
|
||||
- RPC endpoint completely down
|
||||
- All destination chains unavailable
|
||||
- Security breach detected
|
||||
|
||||
### High (P1)
|
||||
- Single destination chain unavailable
|
||||
- High transaction failure rate
|
||||
- Balance issues preventing transfers
|
||||
|
||||
### Medium (P2)
|
||||
- Performance degradation
|
||||
- Monitoring system down
|
||||
- Documentation issues
|
||||
|
||||
### Low (P3)
|
||||
- Minor configuration issues
|
||||
- Documentation updates needed
|
||||
|
||||
---
|
||||
|
||||
## 📋 Incident Response Procedure
|
||||
|
||||
### 1. Detection
|
||||
|
||||
**Automated Monitoring**:
|
||||
```bash
|
||||
bash scripts/automated-monitoring.sh
|
||||
```
|
||||
|
||||
**Manual Check**:
|
||||
```bash
|
||||
bash scripts/health-check.sh
|
||||
```
|
||||
|
||||
### 2. Assessment
|
||||
|
||||
**Gather Information**:
|
||||
```bash
|
||||
# System status
|
||||
bash scripts/health-check.sh
|
||||
|
||||
# Recent transactions
|
||||
bash scripts/monitor-bridge-transfers.sh
|
||||
|
||||
# Error logs
|
||||
tail -100 logs/alerts-$(date +%Y%m%d).log
|
||||
```
|
||||
|
||||
### 3. Containment
|
||||
|
||||
**Pause Operations if Needed**:
|
||||
```bash
|
||||
# Pause bridge
|
||||
cast send <BRIDGE_ADDRESS> "pause()" --rpc-url $RPC_URL --private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
### 4. Resolution
|
||||
|
||||
**Follow Specific Procedures**:
|
||||
- See troubleshooting section in Bridge Operations Runbook
|
||||
- Check logs for error patterns
|
||||
- Verify configuration
|
||||
|
||||
### 5. Recovery
|
||||
|
||||
**Resume Operations**:
|
||||
```bash
|
||||
# Unpause bridge
|
||||
cast send <BRIDGE_ADDRESS> "unpause()" --rpc-url $RPC_URL --private-key $PRIVATE_KEY
|
||||
|
||||
# Verify system
|
||||
bash scripts/test-suite.sh all
|
||||
```
|
||||
|
||||
### 6. Post-Incident
|
||||
|
||||
**Documentation**:
|
||||
- Document incident details
|
||||
- Update runbooks if needed
|
||||
- Review monitoring alerts
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Common Incidents
|
||||
|
||||
### RPC Outage
|
||||
|
||||
**Symptoms**: Cannot connect to RPC endpoint
|
||||
|
||||
**Response**:
|
||||
1. Check RPC endpoint status
|
||||
2. Verify network connectivity
|
||||
3. Switch to backup RPC if available
|
||||
4. Contact infrastructure team
|
||||
|
||||
### Bridge Contract Issue
|
||||
|
||||
**Symptoms**: Bridge contract calls failing
|
||||
|
||||
**Response**:
|
||||
1. Verify contract address
|
||||
2. Check contract code
|
||||
3. Verify network status
|
||||
4. Check for contract upgrades
|
||||
|
||||
### High Failure Rate
|
||||
|
||||
**Symptoms**: Many transactions failing
|
||||
|
||||
**Response**:
|
||||
1. Check gas prices
|
||||
2. Verify balances
|
||||
3. Check destination chain status
|
||||
4. Review recent changes
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: $(date)
|
||||
|
||||
120
docs/runbooks/RECOVERY_PROCEDURES.md
Normal file
120
docs/runbooks/RECOVERY_PROCEDURES.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Recovery Procedures
|
||||
|
||||
**Purpose**: Step-by-step recovery procedures for various failure scenarios
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Transaction Recovery
|
||||
|
||||
### Stuck Transaction
|
||||
|
||||
**Scenario**: Transaction pending for extended period
|
||||
|
||||
**Recovery**:
|
||||
1. Check transaction status
|
||||
```bash
|
||||
cast tx <tx_hash> --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
2. Check nonce
|
||||
```bash
|
||||
cast nonce <address> --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
3. Retry with higher gas
|
||||
```bash
|
||||
bash scripts/retry-failed-transactions.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💰 Balance Recovery
|
||||
|
||||
### Insufficient Balance
|
||||
|
||||
**Scenario**: Not enough ETH or LINK for operations
|
||||
|
||||
**Recovery**:
|
||||
1. Check current balances
|
||||
```bash
|
||||
bash scripts/check-bridge-status.sh
|
||||
```
|
||||
|
||||
2. Transfer funds
|
||||
- Transfer ETH to deployer address
|
||||
- Transfer LINK tokens if needed
|
||||
|
||||
3. Verify balances
|
||||
```bash
|
||||
cast balance <address> --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Chain Configuration Recovery
|
||||
|
||||
### Missing Destination Chain
|
||||
|
||||
**Scenario**: Destination chain not configured
|
||||
|
||||
**Recovery**:
|
||||
1. Verify chain selector
|
||||
2. Add destination
|
||||
```bash
|
||||
bash scripts/configure-bridge-destinations.sh
|
||||
```
|
||||
|
||||
3. Verify configuration
|
||||
```bash
|
||||
bash scripts/verify-bridge-configuration.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🌐 Network Recovery
|
||||
|
||||
### RPC Outage
|
||||
|
||||
**Scenario**: RPC endpoint unavailable
|
||||
|
||||
**Recovery**:
|
||||
1. Check RPC status
|
||||
```bash
|
||||
cast block-number --rpc-url $RPC_URL
|
||||
```
|
||||
|
||||
2. Switch to backup RPC
|
||||
- Update RPC_URL in .env
|
||||
- Restart services
|
||||
|
||||
3. Verify connectivity
|
||||
```bash
|
||||
bash scripts/health-check.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔒 Security Recovery
|
||||
|
||||
### Unauthorized Access
|
||||
|
||||
**Scenario**: Suspicious activity detected
|
||||
|
||||
**Recovery**:
|
||||
1. Pause bridge immediately
|
||||
```bash
|
||||
cast send <BRIDGE_ADDRESS> "pause()" --rpc-url $RPC_URL --private-key $PRIVATE_KEY
|
||||
```
|
||||
|
||||
2. Review logs
|
||||
```bash
|
||||
tail -1000 logs/alerts-*.log
|
||||
```
|
||||
|
||||
3. Rotate keys if needed
|
||||
4. Resume after security review
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: $(date)
|
||||
|
||||
Reference in New Issue
Block a user