proxmox/docs/archive/completion/RPC_TROUBLESHOOTING_COMPLETE.md

# RPC Troubleshooting - Complete Summary

**Date**: $(date)  
**Issue**: RPC-01 (VMID 2500) troubleshooting and resolution

---

## 🔍 Issue Identified

RPC-01 (VMID 2500) was experiencing multiple issues preventing proper operation:

1. **Missing Configuration File**: Service expected `/etc/besu/config-rpc.toml` but only `config-rpc-public.toml` existed
2. **Service File Mismatch**: Service file referenced wrong config file name
3. **Database Corruption**: Corrupted `DATABASE_METADATA.json` file preventing startup
4. **Missing Required Files**: Genesis, static-nodes, and permissions files in wrong locations
5. **Database Directory Missing**: `/data/besu/database/` directory did not exist

---

## ✅ Resolution Steps Taken

### 1. Configuration File Fix

**Problem**: Service expected `config-rpc.toml` but only `config-rpc-public.toml` existed

**Solution**:
```bash
pct exec 2500 -- cp /etc/besu/config-rpc-public.toml /etc/besu/config-rpc.toml
pct exec 2500 -- chown besu:besu /etc/besu/config-rpc.toml
```

### 2. Service File Update

**Problem**: Service file referenced `config-rpc-public.toml` instead of `config-rpc.toml`

**Solution**:
```bash
pct exec 2500 -- sed -i 's|config-rpc-public.toml|config-rpc.toml|g' /etc/systemd/system/besu-rpc.service
pct exec 2500 -- systemctl daemon-reload
```

### 3. Database Corruption Fix

**Problem**: Corrupted `DATABASE_METADATA.json` causing startup failures

**Solution**:
```bash
pct exec 2500 -- systemctl stop besu-rpc.service
pct exec 2500 -- rm -f /data/besu/DATABASE_METADATA.json
pct exec 2500 -- rm -rf /data/besu/database/*
```

### 4. Required Files Setup

**Problem**: Genesis, static-nodes, and permissions files in `/etc/besu/` but config expects `/genesis/` and `/permissions/`

**Solution**:
```bash
# Create directories
pct exec 2500 -- mkdir -p /genesis /permissions

# Copy files
pct exec 2500 -- cp /etc/besu/genesis.json /genesis/
pct exec 2500 -- cp /etc/besu/static-nodes.json /genesis/
pct exec 2500 -- cp /etc/besu/permissions-nodes.toml /permissions/

# Set ownership
pct exec 2500 -- chown -R besu:besu /genesis /permissions
```

### 5. Database Directory Creation

**Problem**: Database directory missing

**Solution**:
```bash
pct exec 2500 -- mkdir -p /data/besu/database
pct exec 2500 -- chown -R besu:besu /data/besu
```

### 6. Service Restart

**Solution**:
```bash
pct exec 2500 -- systemctl start besu-rpc.service
```

---

## ✅ Verification Results

### Service Status

- **Status**: ✅ Active (running)
- **Process**: ✅ Besu process running (PID 327821)
- **Uptime**: Stable since restart

### Network Ports

- **8545 (HTTP RPC)**: ✅ Listening
- **8546 (WebSocket RPC)**: ✅ Listening
- **30303 (P2P)**: ✅ Listening
- **9545 (Metrics)**: ✅ Listening

### Network Connectivity

- **RPC Endpoint**: ✅ Responding
- **Chain ID**: ✅ 138 (correct)
- **Block Production**: ✅ Active (syncing blocks)
- **Peers**: ✅ Connected to 5 peers

### Block Sync Status

- **Current Block**: > 11,200 (at time of fix)
- **Sync Status**: ✅ Actively syncing
- **Import Rate**: Processing blocks successfully

---

## 🛠️ Tools Created

### 1. Troubleshooting Script

**File**: `scripts/troubleshoot-rpc-2500.sh`

**Features**:
- Container status check
- Network configuration verification
- Service status check
- Configuration file validation
- Required files check
- Port listening check
- RPC endpoint test
- Process verification
- Error log analysis

### 2. Fix Script

**File**: `scripts/fix-rpc-2500.sh`

**Features**:
- Automated configuration file creation
- Deprecated option removal
- Service file update
- Required files setup
- Service restart
- Verification

### 3. Documentation

**Files Created**:
- `docs/09-troubleshooting/RPC_2500_TROUBLESHOOTING.md` - Complete guide
- `docs/09-troubleshooting/RPC_2500_QUICK_FIX.md` - Quick reference
- `docs/09-troubleshooting/RPC_2500_TROUBLESHOOTING_SUMMARY.md` - Summary

---

## 📋 Configuration Fixes Applied

### Template Updates

**File**: `smom-dbis-138-proxmox/templates/besu-configs/config-rpc.toml`

**Changes**:
- ✅ Removed `log-destination` (deprecated)
- ✅ Removed `max-remote-initiated-connections` (deprecated)
- ✅ Removed `trie-logs-enabled` (deprecated)
- ✅ Removed `accounts-enabled` (deprecated)
- ✅ Removed `database-path` (deprecated)
- ✅ Removed `rpc-http-host-allowlist` (deprecated)

### Installation Script Updates

**File**: `smom-dbis-138-proxmox/install/besu-rpc-install.sh`

**Changes**:
- ✅ Changed service to use `config-rpc.toml` (not `config-rpc-public.toml`)
- ✅ Updated template file name
- ✅ Removed deprecated options from template
- ✅ Fixed file paths (`/genesis/` instead of `/etc/besu/`)

---

## ✅ Current Status

**RPC-01 (VMID 2500)**: ✅ **FULLY OPERATIONAL**

- Service: Active and stable
- Network: Connected and syncing
- RPC: Accessible and responding
- All ports: Listening correctly

---

## 🔄 Next Steps

### Immediate

1. ✅ RPC-01 fixed and operational
2. ⏳ Verify RPC-02 (VMID 2501) status
3. ⏳ Verify RPC-03 (VMID 2502) status

### Short-term

1. Apply same fixes to RPC-02 and RPC-03 if needed
2. Verify all RPC nodes are in sync
3. Test load balancing across RPC nodes

---

## 📚 Related Documentation

- [RPC 2500 Troubleshooting Guide](/docs/09-troubleshooting/RPC_2500_TROUBLESHOOTING.md)
- [RPC 2500 Quick Fix](/docs/09-troubleshooting/RPC_2500_QUICK_FIX.md)
- [Deployment Readiness](../../03-deployment/DEPLOYMENT_READINESS.md)

---

**Resolution Date**: $(date)  
**Status**: ✅ **RESOLVED**