fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
259 lines
6.8 KiB
Markdown
259 lines
6.8 KiB
Markdown
# RPC Node Testing - Final Report
|
|
|
|
**Last Updated:** 2026-01-31
|
|
**Document Version:** 1.0
|
|
**Status:** Active Documentation
|
|
|
|
---
|
|
|
|
**Date**: 2026-01-17
|
|
**Test Type**: Comprehensive Full Testing
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
### Current Status
|
|
- **Containers Running**: 11/12 (2301 has startup error)
|
|
- **RPC Responding**: 2/12 nodes (2101, 2201)
|
|
- **Block Synchronization**: ✅ Perfect (0 block difference between responding nodes)
|
|
- **Peer Connections**: 2 nodes with peers
|
|
- **Chain ID**: ✅ All nodes on Chain 138 (0x8a)
|
|
|
|
### Issues Identified
|
|
1. **Most nodes not responding to RPC** - Services may still be starting or have configuration issues
|
|
2. **Container 2301 startup failure** - Requires investigation
|
|
3. **Services restarting repeatedly** - Some nodes showing high restart counts (49, 58, 63, 66785!)
|
|
4. **Permissioning errors** - VMID 2305 showed "not in nodes-allowlist" error (fixed by redeployment)
|
|
|
|
---
|
|
|
|
## Detailed Test Results
|
|
|
|
### Test 1: Container Status
|
|
| VMID | Status | Name |
|
|
|------|--------|------|
|
|
| 2101 | ✅ Running | besu-rpc-core-1 |
|
|
| 2201 | ✅ Running | besu-rpc-public-1 |
|
|
| 2301 | ❌ Stopped | besu-rpc-private-1 (startup error) |
|
|
| 2303 | ✅ Running | besu-rpc-ali-0x8a |
|
|
| 2304 | ✅ Running | besu-rpc-ali-0x1 |
|
|
| 2305 | ✅ Running | besu-rpc-luis-0x8a |
|
|
| 2306 | ✅ Running | besu-rpc-luis-0x1 |
|
|
| 2307 | ✅ Running | besu-rpc-putu-0x8a |
|
|
| 2308 | ✅ Running | besu-rpc-putu-0x1 |
|
|
| 2401 | ✅ Running | besu-rpc-thirdweb-0x8a-1 |
|
|
| 2402 | ✅ Running | besu-rpc-thirdweb-0x8a-2 |
|
|
| 2403 | ✅ Running | besu-rpc-thirdweb-0x8a-3 |
|
|
|
|
**Result**: 11/12 containers running
|
|
|
|
---
|
|
|
|
### Test 2: RPC Connectivity
|
|
|
|
#### Responding Nodes (2/12)
|
|
- ✅ **2101 (192.168.11.211)**: Block 1135876
|
|
- ✅ **2201 (192.168.11.221)**: Block 1135876
|
|
|
|
#### Non-Responding Nodes (10/12)
|
|
- ❌ 2301 (192.168.11.232) - Container stopped
|
|
- ❌ 2303 (192.168.11.233) - Service starting
|
|
- ❌ 2304 (192.168.11.234) - Service starting
|
|
- ❌ 2305 (192.168.11.235) - Service restarting (permissioning error)
|
|
- ❌ 2306 (192.168.11.236) - Service starting
|
|
- ❌ 2307 (192.168.11.237) - Service restarting
|
|
- ❌ 2308 (192.168.11.238) - Service restarting
|
|
- ❌ 2401 (192.168.11.241) - Service restarting (66785 restarts!)
|
|
- ❌ 2402 (192.168.11.242) - Service starting
|
|
- ❌ 2403 (192.168.11.243) - Service starting
|
|
|
|
**Result**: 2/12 nodes responding
|
|
|
|
---
|
|
|
|
### Test 3: Block Synchronization
|
|
|
|
**Responding Nodes Block Numbers**:
|
|
- 2101: Block 1135876
|
|
- 2201: Block 1135876
|
|
|
|
**Synchronization Status**: ✅ **Perfect**
|
|
- Block difference: 0
|
|
- Both nodes are at the same block height
|
|
|
|
**Note**: Once other nodes come online, they will need to sync to the current block height.
|
|
|
|
---
|
|
|
|
### Test 4: Peer Connections
|
|
|
|
- **2101**: 1 peer connected ✅
|
|
- **2201**: 1 peer connected ✅
|
|
- **Other nodes**: 0 peers (not responding to RPC)
|
|
|
|
**Result**: 2 nodes with peer connections
|
|
|
|
---
|
|
|
|
### Test 5: Sync Status
|
|
|
|
- **2101**: ✅ Synced
|
|
- **2201**: ✅ Synced
|
|
- **Other nodes**: Unknown (not responding)
|
|
|
|
**Result**: 2/12 nodes synced
|
|
|
|
---
|
|
|
|
### Test 6: Chain ID Verification
|
|
|
|
- **2101**: Chain ID 0x8a (138) ✅
|
|
- **2201**: Chain ID 0x8a (138) ✅
|
|
- **Other nodes**: Could not verify (not responding)
|
|
|
|
**Result**: All responding nodes on correct chain
|
|
|
|
---
|
|
|
|
## Critical Issues Found
|
|
|
|
### 1. Container 2301 Startup Failure
|
|
**Error**: `lxc_init: 845 Failed to run lxc.hook.pre-start`
|
|
|
|
**Impact**: Container cannot start
|
|
|
|
**Action Required**:
|
|
- Investigate pre-start hook
|
|
- Check container configuration
|
|
- May need to fix or recreate container
|
|
|
|
---
|
|
|
|
### 2. Services Restarting Repeatedly
|
|
**Affected Nodes**: 2304, 2305, 2307, 2308, 2401, 2402, 2403
|
|
|
|
**Restart Counts Observed**:
|
|
- 2304: 49 restarts
|
|
- 2305: Permissioning error (fixed)
|
|
- 2307: 63 restarts
|
|
- 2308: 58 restarts
|
|
- 2401: 66,785 restarts! ⚠️ **CRITICAL**
|
|
|
|
**Possible Causes**:
|
|
- Configuration errors
|
|
- Permissioning issues (partially fixed)
|
|
- Missing files
|
|
- Port conflicts
|
|
|
|
**Action Required**:
|
|
- Investigate VMID 2401 (extremely high restart count)
|
|
- Check Besu logs for all restarting services
|
|
- Verify all configuration files are correct
|
|
|
|
---
|
|
|
|
### 3. RPC Services Not Responding
|
|
**Status**: 10/12 nodes not responding to RPC calls
|
|
|
|
**Possible Causes**:
|
|
- Services still starting (Besu can take 5-10 minutes)
|
|
- Services crashing and restarting
|
|
- Configuration issues preventing startup
|
|
- Port binding issues
|
|
|
|
**Action Required**:
|
|
- Wait longer for services to fully start
|
|
- Check logs for startup errors
|
|
- Verify network configuration
|
|
|
|
---
|
|
|
|
## Configuration Files Status
|
|
|
|
### static-nodes.json
|
|
- ✅ **Status**: Valid with 15 unique enodes
|
|
- ✅ **Deployed**: To all 11 running nodes
|
|
- ✅ **Uniqueness**: All enode IDs verified unique
|
|
|
|
### permissions-nodes.toml
|
|
- ✅ **Status**: Valid with 15 enodes (matches static-nodes.json)
|
|
- ✅ **Deployed**: To all 11 running nodes
|
|
- ✅ **Match**: All enodes from static-nodes.json are in permissions-nodes.toml
|
|
|
|
---
|
|
|
|
## Working Nodes
|
|
|
|
### Fully Operational (2/12)
|
|
1. **2101 (192.168.11.211)** - besu-rpc-core-1
|
|
- ✅ Container running
|
|
- ✅ RPC responding
|
|
- ✅ Block: 1135876
|
|
- ✅ 1 peer connected
|
|
- ✅ Synced
|
|
- ✅ Chain ID: 0x8a
|
|
|
|
2. **2201 (192.168.11.221)** - besu-rpc-public-1
|
|
- ✅ Container running
|
|
- ✅ RPC responding
|
|
- ✅ Block: 1135876
|
|
- ✅ 1 peer connected
|
|
- ✅ Synced
|
|
- ✅ Chain ID: 0x8a
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
### Immediate Actions
|
|
1. **Wait for Services to Start**
|
|
- Besu services can take 5-10 minutes to fully start
|
|
- Monitor services and retest after 10-15 minutes
|
|
|
|
2. **Investigate VMID 2401**
|
|
- Extremely high restart count (66,785) indicates critical issue
|
|
- Check logs for root cause
|
|
- May need to stop service and investigate configuration
|
|
|
|
3. **Fix Container 2301**
|
|
- Investigate pre-start hook error
|
|
- Check container configuration
|
|
- May need to recreate container
|
|
|
|
4. **Monitor Service Logs**
|
|
- Check logs for all restarting services
|
|
- Look for configuration errors
|
|
- Verify all required files exist
|
|
|
|
### Follow-up Testing
|
|
1. **Wait 10-15 minutes** for all services to fully start
|
|
2. **Re-run comprehensive test**: `bash scripts/test-rpc-nodes-complete.sh 192.168.11.10`
|
|
3. **Verify block synchronization** across all responding nodes
|
|
4. **Check peer connections** improve as nodes come online
|
|
|
|
---
|
|
|
|
## Test Scripts
|
|
|
|
- **Comprehensive Test**: `/home/intlc/projects/proxmox/scripts/test-rpc-nodes-complete.sh`
|
|
- **Enode Verification**: `/home/intlc/projects/proxmox/scripts/verify-unique-enodes.sh`
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
**Current State**: 2/12 nodes fully operational and synchronized
|
|
|
|
**Next Steps**:
|
|
1. Wait for services to fully start (10-15 minutes)
|
|
2. Investigate and fix restarting services
|
|
3. Fix container 2301 startup issue
|
|
4. Retest after fixes
|
|
|
|
**Expected Outcome**: Once all services are running, all 12 nodes should be responding and synchronized.
|
|
|
|
---
|
|
|
|
**Last Updated**: 2026-01-17
|