# RPC Node Testing - Final Report **Last Updated:** 2026-01-31 **Document Version:** 1.0 **Status:** Active Documentation --- **Date**: 2026-01-17 **Test Type**: Comprehensive Full Testing --- ## Executive Summary ### Current Status - **Containers Running**: 11/12 (2301 has startup error) - **RPC Responding**: 2/12 nodes (2101, 2201) - **Block Synchronization**: ✅ Perfect (0 block difference between responding nodes) - **Peer Connections**: 2 nodes with peers - **Chain ID**: ✅ All nodes on Chain 138 (0x8a) ### Issues Identified 1. **Most nodes not responding to RPC** - Services may still be starting or have configuration issues 2. **Container 2301 startup failure** - Requires investigation 3. **Services restarting repeatedly** - Some nodes showing high restart counts (49, 58, 63, 66785!) 4. **Permissioning errors** - VMID 2305 showed "not in nodes-allowlist" error (fixed by redeployment) --- ## Detailed Test Results ### Test 1: Container Status | VMID | Status | Name | |------|--------|------| | 2101 | ✅ Running | besu-rpc-core-1 | | 2201 | ✅ Running | besu-rpc-public-1 | | 2301 | ❌ Stopped | besu-rpc-private-1 (startup error) | | 2303 | ✅ Running | besu-rpc-ali-0x8a | | 2304 | ✅ Running | besu-rpc-ali-0x1 | | 2305 | ✅ Running | besu-rpc-luis-0x8a | | 2306 | ✅ Running | besu-rpc-luis-0x1 | | 2307 | ✅ Running | besu-rpc-putu-0x8a | | 2308 | ✅ Running | besu-rpc-putu-0x1 | | 2401 | ✅ Running | besu-rpc-thirdweb-0x8a-1 | | 2402 | ✅ Running | besu-rpc-thirdweb-0x8a-2 | | 2403 | ✅ Running | besu-rpc-thirdweb-0x8a-3 | **Result**: 11/12 containers running --- ### Test 2: RPC Connectivity #### Responding Nodes (2/12) - ✅ **2101 (192.168.11.211)**: Block 1135876 - ✅ **2201 (192.168.11.221)**: Block 1135876 #### Non-Responding Nodes (10/12) - ❌ 2301 (192.168.11.232) - Container stopped - ❌ 2303 (192.168.11.233) - Service starting - ❌ 2304 (192.168.11.234) - Service starting - ❌ 2305 (192.168.11.235) - Service restarting (permissioning error) - ❌ 2306 (192.168.11.236) - Service starting - ❌ 2307 (192.168.11.237) - Service restarting - ❌ 2308 (192.168.11.238) - Service restarting - ❌ 2401 (192.168.11.241) - Service restarting (66785 restarts!) - ❌ 2402 (192.168.11.242) - Service starting - ❌ 2403 (192.168.11.243) - Service starting **Result**: 2/12 nodes responding --- ### Test 3: Block Synchronization **Responding Nodes Block Numbers**: - 2101: Block 1135876 - 2201: Block 1135876 **Synchronization Status**: ✅ **Perfect** - Block difference: 0 - Both nodes are at the same block height **Note**: Once other nodes come online, they will need to sync to the current block height. --- ### Test 4: Peer Connections - **2101**: 1 peer connected ✅ - **2201**: 1 peer connected ✅ - **Other nodes**: 0 peers (not responding to RPC) **Result**: 2 nodes with peer connections --- ### Test 5: Sync Status - **2101**: ✅ Synced - **2201**: ✅ Synced - **Other nodes**: Unknown (not responding) **Result**: 2/12 nodes synced --- ### Test 6: Chain ID Verification - **2101**: Chain ID 0x8a (138) ✅ - **2201**: Chain ID 0x8a (138) ✅ - **Other nodes**: Could not verify (not responding) **Result**: All responding nodes on correct chain --- ## Critical Issues Found ### 1. Container 2301 Startup Failure **Error**: `lxc_init: 845 Failed to run lxc.hook.pre-start` **Impact**: Container cannot start **Action Required**: - Investigate pre-start hook - Check container configuration - May need to fix or recreate container --- ### 2. Services Restarting Repeatedly **Affected Nodes**: 2304, 2305, 2307, 2308, 2401, 2402, 2403 **Restart Counts Observed**: - 2304: 49 restarts - 2305: Permissioning error (fixed) - 2307: 63 restarts - 2308: 58 restarts - 2401: 66,785 restarts! ⚠️ **CRITICAL** **Possible Causes**: - Configuration errors - Permissioning issues (partially fixed) - Missing files - Port conflicts **Action Required**: - Investigate VMID 2401 (extremely high restart count) - Check Besu logs for all restarting services - Verify all configuration files are correct --- ### 3. RPC Services Not Responding **Status**: 10/12 nodes not responding to RPC calls **Possible Causes**: - Services still starting (Besu can take 5-10 minutes) - Services crashing and restarting - Configuration issues preventing startup - Port binding issues **Action Required**: - Wait longer for services to fully start - Check logs for startup errors - Verify network configuration --- ## Configuration Files Status ### static-nodes.json - ✅ **Status**: Valid with 15 unique enodes - ✅ **Deployed**: To all 11 running nodes - ✅ **Uniqueness**: All enode IDs verified unique ### permissions-nodes.toml - ✅ **Status**: Valid with 15 enodes (matches static-nodes.json) - ✅ **Deployed**: To all 11 running nodes - ✅ **Match**: All enodes from static-nodes.json are in permissions-nodes.toml --- ## Working Nodes ### Fully Operational (2/12) 1. **2101 (192.168.11.211)** - besu-rpc-core-1 - ✅ Container running - ✅ RPC responding - ✅ Block: 1135876 - ✅ 1 peer connected - ✅ Synced - ✅ Chain ID: 0x8a 2. **2201 (192.168.11.221)** - besu-rpc-public-1 - ✅ Container running - ✅ RPC responding - ✅ Block: 1135876 - ✅ 1 peer connected - ✅ Synced - ✅ Chain ID: 0x8a --- ## Recommendations ### Immediate Actions 1. **Wait for Services to Start** - Besu services can take 5-10 minutes to fully start - Monitor services and retest after 10-15 minutes 2. **Investigate VMID 2401** - Extremely high restart count (66,785) indicates critical issue - Check logs for root cause - May need to stop service and investigate configuration 3. **Fix Container 2301** - Investigate pre-start hook error - Check container configuration - May need to recreate container 4. **Monitor Service Logs** - Check logs for all restarting services - Look for configuration errors - Verify all required files exist ### Follow-up Testing 1. **Wait 10-15 minutes** for all services to fully start 2. **Re-run comprehensive test**: `bash scripts/test-rpc-nodes-complete.sh 192.168.11.10` 3. **Verify block synchronization** across all responding nodes 4. **Check peer connections** improve as nodes come online --- ## Test Scripts - **Comprehensive Test**: `/home/intlc/projects/proxmox/scripts/test-rpc-nodes-complete.sh` - **Enode Verification**: `/home/intlc/projects/proxmox/scripts/verify-unique-enodes.sh` --- ## Conclusion **Current State**: 2/12 nodes fully operational and synchronized **Next Steps**: 1. Wait for services to fully start (10-15 minutes) 2. Investigate and fix restarting services 3. Fix container 2301 startup issue 4. Retest after fixes **Expected Outcome**: Once all services are running, all 12 nodes should be responding and synchronized. --- **Last Updated**: 2026-01-17