# RPC Node Testing Report

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Date**: 2026-01-17  
**Test Type**: Comprehensive RPC Node Testing

---

## Test Results Summary

### Container Status
- **Running**: 11/12 containers
- **Stopped**: 1/12 (2301 - has startup error)

### RPC Connectivity
- **Responding**: 2/12 nodes (2101, 2201)
- **Not Responding**: 10/12 nodes
- **Note**: Many nodes have Besu processes running but RPC not accessible yet (may be starting up)

### Block Synchronization
- **2101 (192.168.11.211)**: Block 1135876
- **2201 (192.168.11.221)**: Block 1135876
- **Status**: ✅ Both responding nodes are perfectly synchronized (0 block difference)

### Peer Connections
- **2101**: 1 peer
- **2201**: 1 peer
- **Other nodes**: 0 peers (not responding to RPC)

### Chain ID
- **All responding nodes**: Chain ID 0x8a (138) ✅

---

## Issues Identified

### 1. Container 2301 Startup Failure
- **Error**: `lxc_init: 845 Failed to run lxc.hook.pre-start`
- **Status**: Container cannot start
- **Action Required**: Investigate and fix startup hook issue

### 2. RPC Services Not Responding
- **Nodes Affected**: 2303, 2304, 2305, 2306, 2307, 2308, 2401, 2402, 2403
- **Status**: Besu processes are running on most nodes, but RPC endpoints not accessible
- **Possible Causes**:
  - Services still starting up (Besu can take several minutes to fully start)
  - Configuration issues
  - Port binding issues
  - Network connectivity issues

### 3. Peer Connection Issues
- **Status**: Only 2 nodes have peer connections
- **Cause**: Most nodes not responding, so cannot form peer connections
- **Note**: Once all nodes are responding, peer connections should improve

---

## Working Nodes

### Fully Operational (2/12)
1. **2101 (192.168.11.211)** - besu-rpc-core-1
   - ✅ RPC responding
   - ✅ Block: 1135876
   - ✅ 1 peer connected
   - ✅ Synced

2. **2201 (192.168.11.221)** - besu-rpc-public-1
   - ✅ RPC responding
   - ✅ Block: 1135876
   - ✅ 1 peer connected
   - ✅ Synced

---

## Next Steps

1. **Wait for Services to Fully Start**
   - Besu services can take 5-10 minutes to fully start and become responsive
   - Monitor services and retest after waiting period

2. **Fix Container 2301**
   - Investigate startup hook error
   - Check container configuration
   - May need to recreate container or fix hook script

3. **Verify Configuration Files**
   - Ensure static-nodes.json and permissions-nodes.toml are correct
   - Verify all enodes are unique
   - Check that files are deployed to all nodes

4. **Monitor Block Sync**
   - Once all nodes are responding, verify they sync to the same block
   - Check peer connections improve as nodes come online

5. **Retest After Full Startup**
   - Wait 10-15 minutes for all services to fully start
   - Run comprehensive test again
   - Verify all 12 nodes are responding and synchronized

---

## Test Scripts Created

- `/home/intlc/projects/proxmox/scripts/test-rpc-nodes-complete.sh` - Comprehensive testing script
- Can be run anytime: `bash scripts/test-rpc-nodes-complete.sh 192.168.11.10`

---

**Last Updated**: 2026-01-17