proxmox/docs/09-troubleshooting/RPC_2500_TROUBLESHOOTING.md

# RPC-01 (VMID 2500) Troubleshooting Guide

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Container**: besu-rpc-1  
**VMID**: 2500  
**IP Address**: 192.168.11.250  
**Expected Ports**: 8545 (HTTP), 8546 (WS), 30303 (P2P), 9545 (Metrics)

---

## 🔍 Quick Diagnostic

Run the automated troubleshooting script:

```bash
cd /home/intlc/projects/proxmox
./scripts/troubleshoot-rpc-2500.sh
```

---

## 📋 Common Issues & Solutions

### Issue 1: Container Not Running

**Symptoms**:
- `pct status 2500` shows "stopped"
- Cannot connect to container

**Solution**:
```bash
# Start container
pct start 2500

# Check why it stopped
pct config 2500
pct logs 2500
```

---

### Issue 2: Service Not Active

**Symptoms**:
- Container running but service inactive
- `systemctl status besu-rpc.service` shows failed/stopped

**Diagnosis**:
```bash
# Check service status
pct exec 2500 -- systemctl status besu-rpc.service

# Check recent logs
pct exec 2500 -- journalctl -u besu-rpc.service -n 50 --no-pager
```

**Common Causes**:

#### A. Configuration File Missing
**Error**: `Unable to read TOML configuration, file not found`

**Solution**:
```bash
# Check if config exists
pct exec 2500 -- ls -la /etc/besu/config-rpc.toml

# If missing, copy from template
pct push 2500 /path/to/config-rpc.toml /etc/besu/config-rpc.toml
```

#### B. Deprecated Configuration Options
**Error**: `Unknown options in TOML configuration file`

**Solution**:
Remove deprecated options from config:
- `log-destination`
- `max-remote-initiated-connections`
- `trie-logs-enabled`
- `accounts-enabled`
- `database-path`
- `rpc-http-host-allowlist`

**Fix**:
```bash
# Edit config file
pct exec 2500 -- nano /etc/besu/config-rpc.toml

# Or use sed to remove deprecated options
pct exec 2500 -- sed -i '/^log-destination/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^max-remote-initiated-connections/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^trie-logs-enabled/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^accounts-enabled/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^database-path/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^rpc-http-host-allowlist/d' /etc/besu/config-rpc.toml

# Restart service
pct exec 2500 -- systemctl restart besu-rpc.service
```

#### C. RPC Not Enabled
**Error**: Service starts but RPC endpoint not accessible

**Solution**:
```bash
# Check if RPC is enabled
pct exec 2500 -- grep "rpc-http-enabled" /etc/besu/config-rpc.toml

# Enable if disabled
pct exec 2500 -- sed -i 's/rpc-http-enabled=false/rpc-http-enabled=true/' /etc/besu/config-rpc.toml

# Restart service
pct exec 2500 -- systemctl restart besu-rpc.service
```

---

### Issue 3: RPC Endpoint Not Responding

**Symptoms**:
- Service is active
- Ports not listening
- Cannot connect to RPC

**Diagnosis**:
```bash
# Check if ports are listening
pct exec 2500 -- ss -tlnp | grep -E "8545|8546"

# Test RPC endpoint
pct exec 2500 -- curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```

**Solutions**:

#### A. Check RPC Configuration
```bash
# Verify RPC is enabled and configured correctly
pct exec 2500 -- grep -E "rpc-http|rpc-ws" /etc/besu/config-rpc.toml
```

Expected:
```toml
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
```

#### B. Check Firewall
```bash
# Check if firewall is blocking
pct exec 2500 -- iptables -L -n | grep -E "8545|8546"

# If needed, allow ports
pct exec 2500 -- iptables -A INPUT -p tcp --dport 8545 -j ACCEPT
pct exec 2500 -- iptables -A INPUT -p tcp --dport 8546 -j ACCEPT
```

#### C. Check Host Allowlist
```bash
# Check allowlist configuration
pct exec 2500 -- grep "rpc-http-host-allowlist" /etc/besu/config-rpc.toml

# If too restrictive, update to allow all or specific IPs
pct exec 2500 -- sed -i 's/rpc-http-host-allowlist=.*/rpc-http-host-allowlist=["*"]/' /etc/besu/config-rpc.toml
```

---

### Issue 4: Network Configuration

**Symptoms**:
- Wrong IP address
- Cannot reach container from network

**Diagnosis**:
```bash
# Check IP address
pct exec 2500 -- ip addr show eth0

# Check Proxmox config
pct config 2500 | grep net0
```

**Solution**:
```bash
# Update IP in Proxmox config (if needed)
pct set 2500 -net0 name=eth0,bridge=vmbr0,ip=192.168.11.250/24,gw=192.168.11.1

# Restart container
pct stop 2500
pct start 2500
```

---

### Issue 5: Missing Required Files

**Symptoms**:
- Service fails to start
- Errors about missing genesis or static nodes

**Diagnosis**:
```bash
# Check required files
pct exec 2500 -- ls -la /genesis/genesis.json
pct exec 2500 -- ls -la /genesis/static-nodes.json
pct exec 2500 -- ls -la /permissions/permissions-nodes.toml
```

**Solution**:
```bash
# Copy files from source project
# (Adjust paths as needed)
pct push 2500 /path/to/genesis.json /genesis/genesis.json
pct push 2500 /path/to/static-nodes.json /genesis/static-nodes.json
pct push 2500 /path/to/permissions-nodes.toml /permissions/permissions-nodes.toml

# Set correct ownership
pct exec 2500 -- chown -R besu:besu /genesis /permissions

# Restart service
pct exec 2500 -- systemctl restart besu-rpc.service
```

---

### Issue 6: Database/Storage Issues

**Symptoms**:
- Service starts but crashes
- Errors about database corruption
- Disk space issues

**Diagnosis**:
```bash
# Check disk space
pct exec 2500 -- df -h

# Check database directory
pct exec 2500 -- ls -la /data/besu/database/

# Check for corruption errors in logs
pct exec 2500 -- journalctl -u besu-rpc.service | grep -i "database\|corrupt"
```

**Solution**:
```bash
# If database is corrupted, may need to resync
# (WARNING: This will delete local blockchain data)
pct exec 2500 -- systemctl stop besu-rpc.service
pct exec 2500 -- rm -rf /data/besu/database/*
pct exec 2500 -- systemctl start besu-rpc.service
```

---

## 🔧 Manual Diagnostic Commands

### Check Service Status
```bash
pct exec 2500 -- systemctl status besu-rpc.service
```

### View Service Logs
```bash
# Real-time logs
pct exec 2500 -- journalctl -u besu-rpc.service -f

# Last 100 lines
pct exec 2500 -- journalctl -u besu-rpc.service -n 100 --no-pager

# Errors only
pct exec 2500 -- journalctl -u besu-rpc.service | grep -iE "error|fail|exception"
```

### Check Configuration
```bash
# View config file
pct exec 2500 -- cat /etc/besu/config-rpc.toml

# Validate config syntax
pct exec 2500 -- besu --config-file=/etc/besu/config-rpc.toml --help 2>&1 | head -20
```

### Test RPC Endpoint
```bash
# From container
pct exec 2500 -- curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# From host (if accessible)
curl -X POST http://192.168.11.250:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```

### Check Process
```bash
# Check if Besu process is running
pct exec 2500 -- ps aux | grep besu

# Check process details
pct exec 2500 -- ps aux | grep besu | head -1
```

### Check Network Connectivity
```bash
# Check IP
pct exec 2500 -- ip addr show

# Test connectivity to other nodes
pct exec 2500 -- ping -c 3 192.168.11.100  # Validator
pct exec 2500 -- ping -c 3 192.168.11.150  # Sentry
```

---

## 🔄 Restart Procedures

### Soft Restart (Service Only)
```bash
pct exec 2500 -- systemctl restart besu-rpc.service
```

### Hard Restart (Container)
```bash
pct stop 2500
sleep 5
pct start 2500
```

### Full Restart (With Config Reload)
```bash
# Stop service
pct exec 2500 -- systemctl stop besu-rpc.service

# Verify config
pct exec 2500 -- cat /etc/besu/config-rpc.toml

# Start service
pct exec 2500 -- systemctl start besu-rpc.service

# Check status
pct exec 2500 -- systemctl status besu-rpc.service
```

---

## 📊 Expected Configuration

### Configuration File Location
- **Path**: `/etc/besu/config-rpc.toml`
- **Type**: Core RPC node configuration

### Key Settings
```toml
# Network
network-id=138
p2p-host="0.0.0.0"
p2p-port=30303

# RPC HTTP
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","WEB3"]
rpc-http-cors-origins=["*"]

# RPC WebSocket
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
rpc-ws-api=["ETH","NET","WEB3"]
rpc-ws-origins=["*"]

# Metrics
metrics-enabled=true
metrics-port=9545
metrics-host="0.0.0.0"

# Data
data-path="/data/besu"
genesis-file="/genesis/genesis.json"
static-nodes-file="/genesis/static-nodes.json"
permissions-nodes-config-file="/permissions/permissions-nodes.toml"
```

---

## ✅ Verification Checklist

After troubleshooting, verify:

- [ ] Container is running
- [ ] Service is active
- [ ] IP address is 192.168.11.250
- [ ] Port 8545 is listening
- [ ] Port 8546 is listening
- [ ] Port 30303 is listening (P2P)
- [ ] Port 9545 is listening (Metrics)
- [ ] RPC endpoint responds to `eth_blockNumber`
- [ ] No errors in recent logs
- [ ] Configuration file is valid
- [ ] All required files exist

---

## 📚 Related Documentation

- [Besu Configuration Guide](/docs/01-getting-started/README.md)
- [RPC Node Types Architecture](../05-network/RPC_NODE_TYPES_ARCHITECTURE.md)
- [Network Troubleshooting](TROUBLESHOOTING_FAQ.md)
- [BESU_CONFIGURATION_GUIDE.md](../04-configuration/BESU_CONFIGURATION_GUIDE.md) · [CHAIN138_BESU_CONFIGURATION.md](../06-besu/CHAIN138_BESU_CONFIGURATION.md)

---

**Last Updated**: $(date)