proxmox/docs/04-configuration/DEPLOYMENT_CHECKLIST.md

# Besu Configuration Deployment Checklist

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Date**: 2026-01-17  
**Purpose**: Step-by-step checklist for deploying cleaned Besu configurations to running nodes

---

## Pre-Deployment Verification

### ✅ Step 1: Verify Configuration Readiness

```bash
# Run automated readiness check
./scripts/verify-deployment-readiness.sh
```

**Expected**: All checks pass ✅

**Manual verification**:
- [ ] All 16 configuration files exist
- [ ] All configs validated (0 errors)
- [ ] No deprecated options detected
- [ ] Deployment scripts executable

---

### ✅ Step 2: Validate Configurations

```bash
# Validate all configs
./scripts/validate-besu-config.sh

# Expected output:
# Total files validated: 16
# Passed: 16
# Failed: 0
```

**Verify**:
- [ ] All configs pass validation
- [ ] No syntax errors
- [ ] No deprecated options
- [ ] All required options present

---

### ✅ Step 3: Audit Configurations

```bash
# Compare configs to templates
./scripts/audit-besu-configs.sh

# Expected output:
# Total configs audited: 11
# Matching templates: 10+
# Drift detected: 0
```

**Verify**:
- [ ] All configs match templates (or expected differences documented)
- [ ] No unexpected drift detected

---

## Deployment Process

### 📋 Step 4: Review Deployment Plan

**Deployment Order** (rolling deployment):
1. **Validators** (1000-1004) - First
2. **Sentries** (1500-1503) - Second  
3. **RPC Nodes** (2500+) - Last

**Process per Node**:
1. Backup existing config
2. Copy cleaned config to node
3. Validate config on node
4. Restart service gracefully
5. Verify service started successfully
6. Wait 30 seconds before next node

---

### 🧪 Step 5: Dry-Run Deployment

```bash
# Preview deployment (no actual changes)
./scripts/deploy-besu-configs.sh --dry-run
```

**Review**:
- [ ] All nodes identified correctly
- [ ] Config mappings correct (VMID → config file)
- [ ] Service names correct
- [ ] Deployment order correct (validators → sentries → RPC)

**Fix any issues** before proceeding.

---

### 🚀 Step 6: Deploy to Validators

```bash
# Deploy to validators (1000-1004)
./scripts/deploy-besu-configs.sh
```

**Monitor during deployment**:
- [ ] Each validator config backed up
- [ ] Config copied successfully
- [ ] Service restarts successfully
- [ ] No errors in deployment output

**Wait** 30 seconds between validators.

**After validators complete**:
- [ ] Check validator services: `pct exec 1000 -- systemctl status besu-validator.service`
- [ ] Verify logs: `pct exec 1000 -- journalctl -u besu-validator.service -n 20`
- [ ] Check logging level (should be WARN, minimal logs)
- [ ] No configuration errors in logs

---

### 🚀 Step 7: Deploy to Sentries

**After validators stable** (wait 5-10 minutes):

```bash
# Deploy to sentries (1500-1503)
./scripts/deploy-besu-configs.sh
```

**Monitor during deployment**:
- [ ] Each sentry config backed up
- [ ] Config copied successfully
- [ ] Service restarts successfully
- [ ] No errors in deployment output

**Wait** 30 seconds between sentries.

**After sentries complete**:
- [ ] Check sentry services: `pct exec 1500 -- systemctl status besu-sentry.service`
- [ ] Verify logs: `pct exec 1500 -- journalctl -u besu-sentry.service -n 20`
- [ ] Check logging level (should be INFO, detailed logs)
- [ ] No configuration errors in logs

---

### 🚀 Step 8: Deploy to RPC Nodes

**After sentries stable** (wait 5-10 minutes):

```bash
# Deploy to RPC nodes (2500+)
./scripts/deploy-besu-configs.sh
```

**Monitor during deployment**:
- [ ] Each RPC config backed up
- [ ] Config copied successfully
- [ ] Service restarts successfully
- [ ] No errors in deployment output

**Wait** 30 seconds between RPC nodes.

**After RPC nodes complete**:
- [ ] Check RPC services: `pct exec 2500 -- systemctl status besu-rpc.service`
- [ ] Verify logs: `pct exec 2500 -- journalctl -u besu-rpc.service -n 20`
- [ ] Check logging level (should be WARN, minimal logs)
- [ ] No configuration errors in logs

---

## Post-Deployment Verification

### ✅ Step 9: Immediate Verification (0-1 hour)

**Service Status**:
```bash
# Check all validator services
for vmid in 1000 1001 1002 1003 1004; do
    echo "VMID $vmid:"
    pct exec $vmid -- systemctl is-active besu-validator.service
done

# Check all sentry services
for vmid in 1500 1501 1502 1503; do
    echo "VMID $vmid:"
    pct exec $vmid -- systemctl is-active besu-sentry.service
done

# Check all RPC services
for vmid in 2500 2501 2502 2503 2505 2506 2507 2508; do
    echo "VMID $vmid:"
    pct exec $vmid -- systemctl is-active besu-rpc.service
done
```

**Expected**: All services `active` ✅

**Configuration Verification**:
```bash
# Verify logging levels on sample nodes
# Validator (should be WARN)
pct exec 1000 -- grep "^logging" /etc/besu/config-validator.toml

# Sentry (should be INFO)
pct exec 1500 -- grep "^logging" /etc/besu/config-sentry.toml

# RPC (should be WARN)
pct exec 2500 -- grep "^logging" /etc/besu/config-rpc-core.toml
```

**Expected**:
- Validators: `logging="WARN"`
- Sentries: `logging="INFO"`
- RPC: `logging="WARN"`

**Error Checking**:
```bash
# Check for configuration errors
for vmid in 1000 1500 2500; do
    echo "VMID $vmid:"
    pct exec $vmid -- journalctl -u besu-*.service --since "10 minutes ago" | grep -i "error\|unknown option" | head -5
done
```

**Expected**: No configuration errors ✅

---

### ✅ Step 10: Functional Verification (1-6 hours)

**Validator Verification**:
- [ ] Validators participating in consensus
- [ ] No validator errors in logs
- [ ] Validators synced with network

**Sentry Verification** (Archive Nodes):
- [ ] Sentries serving archive queries
- [ ] Test historical query: `curl -X POST http://192.168.11.150:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x0000000000000000000000000000000000000000","0x100"],"id":1}'`
- [ ] Sentries synced with network

**RPC Verification**:
- [ ] RPC endpoints responding
- [ ] Test RPC: `curl -X POST http://192.168.11.250:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'`
- [ ] CORS working (if applicable)

**Network Connectivity**:
- [ ] Validators connected to sentries
- [ ] Sentries connected to validators and external peers
- [ ] RPC nodes connected to internal peers

---

### ✅ Step 11: Performance Monitoring (6-48 hours)

**Resource Usage**:
- [ ] Memory usage within expected ranges
- [ ] CPU usage normal
- [ ] Disk I/O acceptable

**Service Stability**:
- [ ] No restart loops
- [ ] Services stable for 24+ hours
- [ ] No configuration drift detected

**Performance Metrics**:
- [ ] Block propagation normal
- [ ] RPC response times acceptable
- [ ] Archive queries performing well

---

## Troubleshooting

### Issue: Service Fails to Start

**Check**:
```bash
pct exec <VMID> -- systemctl status besu-*.service
pct exec <VMID> -- journalctl -u besu-*.service -n 50
```

**Common Causes**:
1. Configuration syntax error
2. Deprecated options still present
3. Invalid option values

**Fix**:
1. Restore from backup: `/etc/besu/config-*.toml.backup.*`
2. Validate config: `./scripts/validate-besu-config.sh`
3. Review error logs for specific issues

---

### Issue: Logging Level Not Applied

**Check**:
```bash
# Verify config file
pct exec <VMID> -- grep "^logging" /etc/besu/config-*.toml

# Check actual logs
pct exec <VMID> -- journalctl -u besu-*.service -n 20
```

**Fix**:
1. Verify config file was updated
2. Ensure service was restarted
3. Check file permissions

---

### Issue: Configuration Drift

**Check**:
```bash
./scripts/audit-besu-configs.sh
```

**Fix**:
1. Re-deploy from templates
2. Verify no manual changes were made
3. Document any intentional differences

---

## Rollback Procedure

If deployment fails or issues occur:

### Immediate Rollback

```bash
# For each node, restore from backup
pct exec <VMID> -- cp /etc/besu/config-*.toml.backup.<timestamp> /etc/besu/config-*.toml
pct exec <VMID> -- systemctl restart besu-*.service
```

### Full Rollback

1. Stop deployment immediately
2. Restore all nodes from backups (created by deploy script)
3. Restart all services
4. Verify services running correctly
5. Investigate issues before retrying

---

## Deployment Sign-Off

### Pre-Deployment Sign-Off

- [ ] All configurations validated
- [ ] Dry-run successful
- [ ] Deployment plan reviewed
- [ ] Rollback procedure understood
- [ ] Monitoring plan ready

**Sign-off**: _________________ **Date**: ___________

---

### Post-Deployment Sign-Off

**After 48 hours of stable operation**:

- [ ] All services running correctly
- [ ] No configuration errors
- [ ] Logging levels correct
- [ ] Network connectivity normal
- [ ] Performance metrics acceptable
- [ ] No configuration drift detected

**Sign-off**: _________________ **Date**: ___________

---

## Related Documentation

- `BESU_DEPLOYMENT_MONITORING.md` - Detailed monitoring guide
- `BESU_CONFIGURATION_GUIDE.md` - Configuration reference
- `BESU_IMPLEMENTATION_COMPLETE.md` - Implementation summary

---

**Last Updated**: 2026-01-17  
**Status**: Ready for Deployment