proxmox/docs/09-troubleshooting/RPC_2500_TROUBLESHOOTING.md

RPC-01 (VMID 2500) Troubleshooting Guide

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Container: besu-rpc-1
VMID: 2500
IP Address: 192.168.11.250
Expected Ports: 8545 (HTTP), 8546 (WS), 30303 (P2P), 9545 (Metrics)

---

🔍 Quick Diagnostic

Run the automated troubleshooting script:

cd /home/intlc/projects/proxmox
./scripts/troubleshoot-rpc-2500.sh

---

📋 Common Issues & Solutions

Issue 1: Container Not Running

Symptoms:

• pct status 2500 shows "stopped"
• Cannot connect to container

Solution:

# Start container
pct start 2500

# Check why it stopped
pct config 2500
pct logs 2500

---

Issue 2: Service Not Active

Symptoms:

• Container running but service inactive
• systemctl status besu-rpc.service shows failed/stopped

Diagnosis:

# Check service status
pct exec 2500 -- systemctl status besu-rpc.service

# Check recent logs
pct exec 2500 -- journalctl -u besu-rpc.service -n 50 --no-pager

Common Causes:

A. Configuration File Missing

Error: Unable to read TOML configuration, file not found

Solution:

# Check if config exists
pct exec 2500 -- ls -la /etc/besu/config-rpc.toml

# If missing, copy from template
pct push 2500 /path/to/config-rpc.toml /etc/besu/config-rpc.toml

B. Deprecated Configuration Options

Error: Unknown options in TOML configuration file

Solution: Remove deprecated options from config:

• log-destination
• max-remote-initiated-connections
• trie-logs-enabled
• accounts-enabled
• database-path
• rpc-http-host-allowlist

Fix:

# Edit config file
pct exec 2500 -- nano /etc/besu/config-rpc.toml

# Or use sed to remove deprecated options
pct exec 2500 -- sed -i '/^log-destination/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^max-remote-initiated-connections/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^trie-logs-enabled/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^accounts-enabled/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^database-path/d' /etc/besu/config-rpc.toml
pct exec 2500 -- sed -i '/^rpc-http-host-allowlist/d' /etc/besu/config-rpc.toml

# Restart service
pct exec 2500 -- systemctl restart besu-rpc.service

C. RPC Not Enabled

Error: Service starts but RPC endpoint not accessible

Solution:

# Check if RPC is enabled
pct exec 2500 -- grep "rpc-http-enabled" /etc/besu/config-rpc.toml

# Enable if disabled
pct exec 2500 -- sed -i 's/rpc-http-enabled=false/rpc-http-enabled=true/' /etc/besu/config-rpc.toml

# Restart service
pct exec 2500 -- systemctl restart besu-rpc.service

---

Issue 3: RPC Endpoint Not Responding

Symptoms:

• Service is active
• Ports not listening
• Cannot connect to RPC

Diagnosis:

# Check if ports are listening
pct exec 2500 -- ss -tlnp | grep -E "8545|8546"

# Test RPC endpoint
pct exec 2500 -- curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Solutions:

A. Check RPC Configuration

# Verify RPC is enabled and configured correctly
pct exec 2500 -- grep -E "rpc-http|rpc-ws" /etc/besu/config-rpc.toml

Expected:

rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546

B. Check Firewall

# Check if firewall is blocking
pct exec 2500 -- iptables -L -n | grep -E "8545|8546"

# If needed, allow ports
pct exec 2500 -- iptables -A INPUT -p tcp --dport 8545 -j ACCEPT
pct exec 2500 -- iptables -A INPUT -p tcp --dport 8546 -j ACCEPT

C. Check Host Allowlist

# Check allowlist configuration
pct exec 2500 -- grep "rpc-http-host-allowlist" /etc/besu/config-rpc.toml

# If too restrictive, update to allow all or specific IPs
pct exec 2500 -- sed -i 's/rpc-http-host-allowlist=.*/rpc-http-host-allowlist=["*"]/' /etc/besu/config-rpc.toml

---

Issue 4: Network Configuration

Symptoms:

• Wrong IP address
• Cannot reach container from network

Diagnosis:

# Check IP address
pct exec 2500 -- ip addr show eth0

# Check Proxmox config
pct config 2500 | grep net0

Solution:

# Update IP in Proxmox config (if needed)
pct set 2500 -net0 name=eth0,bridge=vmbr0,ip=192.168.11.250/24,gw=192.168.11.1

# Restart container
pct stop 2500
pct start 2500

---

Issue 5: Missing Required Files

Symptoms:

• Service fails to start
• Errors about missing genesis or static nodes

Diagnosis:

# Check required files
pct exec 2500 -- ls -la /genesis/genesis.json
pct exec 2500 -- ls -la /genesis/static-nodes.json
pct exec 2500 -- ls -la /permissions/permissions-nodes.toml

Solution:

# Copy files from source project
# (Adjust paths as needed)
pct push 2500 /path/to/genesis.json /genesis/genesis.json
pct push 2500 /path/to/static-nodes.json /genesis/static-nodes.json
pct push 2500 /path/to/permissions-nodes.toml /permissions/permissions-nodes.toml

# Set correct ownership
pct exec 2500 -- chown -R besu:besu /genesis /permissions

# Restart service
pct exec 2500 -- systemctl restart besu-rpc.service

---

Issue 6: Database/Storage Issues

Symptoms:

• Service starts but crashes
• Errors about database corruption
• Disk space issues

Diagnosis:

# Check disk space
pct exec 2500 -- df -h

# Check database directory
pct exec 2500 -- ls -la /data/besu/database/

# Check for corruption errors in logs
pct exec 2500 -- journalctl -u besu-rpc.service | grep -i "database\|corrupt"

Solution:

# If database is corrupted, may need to resync
# (WARNING: This will delete local blockchain data)
pct exec 2500 -- systemctl stop besu-rpc.service
pct exec 2500 -- rm -rf /data/besu/database/*
pct exec 2500 -- systemctl start besu-rpc.service

---

🔧 Manual Diagnostic Commands

Check Service Status

pct exec 2500 -- systemctl status besu-rpc.service

View Service Logs

# Real-time logs
pct exec 2500 -- journalctl -u besu-rpc.service -f

# Last 100 lines
pct exec 2500 -- journalctl -u besu-rpc.service -n 100 --no-pager

# Errors only
pct exec 2500 -- journalctl -u besu-rpc.service | grep -iE "error|fail|exception"

Check Configuration

# View config file
pct exec 2500 -- cat /etc/besu/config-rpc.toml

# Validate config syntax
pct exec 2500 -- besu --config-file=/etc/besu/config-rpc.toml --help 2>&1 | head -20

Test RPC Endpoint

# From container
pct exec 2500 -- curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# From host (if accessible)
curl -X POST http://192.168.11.250:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Check Process

# Check if Besu process is running
pct exec 2500 -- ps aux | grep besu

# Check process details
pct exec 2500 -- ps aux | grep besu | head -1

Check Network Connectivity

# Check IP
pct exec 2500 -- ip addr show

# Test connectivity to other nodes
pct exec 2500 -- ping -c 3 192.168.11.100  # Validator
pct exec 2500 -- ping -c 3 192.168.11.150  # Sentry

---

🔄 Restart Procedures

Soft Restart (Service Only)

pct exec 2500 -- systemctl restart besu-rpc.service

Hard Restart (Container)

pct stop 2500
sleep 5
pct start 2500

Full Restart (With Config Reload)

# Stop service
pct exec 2500 -- systemctl stop besu-rpc.service

# Verify config
pct exec 2500 -- cat /etc/besu/config-rpc.toml

# Start service
pct exec 2500 -- systemctl start besu-rpc.service

# Check status
pct exec 2500 -- systemctl status besu-rpc.service

---

📊 Expected Configuration

Configuration File Location

• Path: /etc/besu/config-rpc.toml
• Type: Core RPC node configuration

Key Settings

# Network
network-id=138
p2p-host="0.0.0.0"
p2p-port=30303

# RPC HTTP
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","WEB3"]
rpc-http-cors-origins=["*"]

# RPC WebSocket
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
rpc-ws-api=["ETH","NET","WEB3"]
rpc-ws-origins=["*"]

# Metrics
metrics-enabled=true
metrics-port=9545
metrics-host="0.0.0.0"

# Data
data-path="/data/besu"
genesis-file="/genesis/genesis.json"
static-nodes-file="/genesis/static-nodes.json"
permissions-nodes-config-file="/permissions/permissions-nodes.toml"

---

✅ Verification Checklist

After troubleshooting, verify:

• Container is running
• Service is active
• IP address is 192.168.11.250
• Port 8545 is listening
• Port 8546 is listening
• Port 30303 is listening (P2P)
• Port 9545 is listening (Metrics)
• RPC endpoint responds to eth_blockNumber
• No errors in recent logs
• Configuration file is valid
• All required files exist

---

📚 Related Documentation

• Besu Configuration Guide
• RPC Node Types Architecture
• Network Troubleshooting
• Besu Configuration Issues

---

Last Updated: $(date)