proxmox/docs/06-besu/RPC_NODE_PEER_COUNT_ANALYSIS.md

RPC Node Peer Count Analysis

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-18
Status: 📊 ANALYSIS COMPLETE

---

Current Peer Count Status

VMID	IP Address	Block Height	Peers	Status	Analysis
2101	192.168.11.211	1,145,367	7	✅ Good	Well connected
2201	192.168.11.221	1,145,367	7	✅ Good	Well connected
2303	192.168.11.233	1,145,367	7	✅ Good	Well connected
2304	192.168.11.234	1,145,367	7	✅ Good	Well connected
2305	192.168.11.235	1,145,367	7	✅ Good	Well connected
2306	192.168.11.236	1,145,367	7	✅ Good	Well connected
2307	192.168.11.237	1,145,367	7	✅ Good	Well connected
2308	192.168.11.238	1,145,367	7	✅ Good	Well connected
2400	192.168.11.240	1,149,992	2	⚠️ Warning	Under-connected, ahead
2401	192.168.11.241	1,149,992	2	⚠️ Warning	Under-connected, ahead
2402	192.168.11.242	1,149,992	2	⚠️ Warning	Under-connected, ahead
2403	192.168.11.243	600,172	0	⏳ Syncing	Still syncing (expected)

---

Expected Peer Counts

Network Size Context

Current Network: ~19-20 active nodes

• 5 Validators (VMIDs 1000-1004)
• 4 Sentries (VMIDs 1500-1503)
• 12 RPC Nodes (VMIDs 2101, 2201, 2303-2308, 2400-2403)

Recommended Peer Counts

Based on BESU_PERFORMANCE_TUNING.md:

Network Size	RPC (Standard)	RPC (High Traffic)
10-20 nodes	5-7 peers	20-25 peers
20-50 nodes	20-25 peers	30-40 peers
50-100 nodes	25-30 peers	40-50 peers

For Current Network (10-20 nodes):

• Minimum healthy: 2-3 peers
• Recommended: 5-7 peers ✅
• Maximum: 20-25 peers (max-peers setting)

---

Analysis by Node Group

✅ Group 1: Standard RPC Nodes (7 peers) - HEALTHY

Nodes: 2101, 2201, 2303-2308
Peer Count: 7 peers each
Block Height: 1,145,367 (all synchronized)

Status: ✅ EXCELLENT

• All nodes have 7 peers (optimal for network size)
• All nodes are at the same block height (synchronized)
• Well connected to the network
• No issues detected

Connected to:

• Likely connected to: Validators, Sentries, and other RPC nodes
• Network topology is healthy

---

⚠️ Group 2: ThirdWeb RPC Nodes (2 peers) - UNDER-CONNECTED

Nodes: 2400, 2401, 2402
Peer Count: 2 peers each
Block Height: 1,149,992 (ahead by ~4,625 blocks!)

Status: ⚠️ ISSUE DETECTED

Issues:

1. Under-connected: Only 2 peers each (should have 5-7)
2. Block height mismatch: Ahead by ~4,625 blocks
3. Network isolation: Likely only connected to each other

Analysis:

• 2 peers each suggests they're only connected to each other (3 nodes = 2 peers each)
• Ahead block height suggests:
  • They might be on a different fork
  • They might not be receiving blocks from the main network
  • They're syncing from a different source

Possible Causes:

1. Missing from static-nodes.json: Not all main network nodes listed
2. Discovery disabled: Cannot discover other nodes
3. Network partition: Isolated from main network
4. Different bootnodes: Connecting to different network

Impact:

• ⚠️ Reduced redundancy (only 3 nodes connected)
• ⚠️ Potential fork if block height diverges further
• ⚠️ Limited access to network state

---

⏳ Group 3: Syncing Node (0 peers) - EXPECTED

Node: 2403
Peer Count: 0 peers
Block Height: 600,172 (syncing, ~545,000 blocks behind)

Status: ⏳ SYNCING (EXPECTED)

Analysis:

• 0 peers is normal during initial sync
• Node is actively syncing (block 600,172 vs current ~1,145,367)
• Once sync completes, peer count should increase

Expected Behavior:

• Once synced, should have 5-7 peers
• May take time depending on sync speed

---

Root Cause Analysis

Why ThirdWeb Nodes Have Only 2 Peers

Most Likely Cause: Network Isolation

1. static-nodes.json mismatch:

   • ThirdWeb nodes might not have all main network nodes in their static-nodes.json
   • Main network nodes might not have ThirdWeb nodes in their static-nodes.json

2. Discovery configuration:

   • ThirdWeb nodes might have discovery-enabled=false
   • Without discovery, they can only connect to nodes in static-nodes.json

3. Network topology:

   • ThirdWeb nodes might be in a different network segment
   • Firewall rules might be blocking P2P connections (port 30303)

4. Bootnode configuration:

   • Different bootnodes configured
   • Connecting to different network

---

Recommendations

Immediate Actions

1. Fix ThirdWeb Nodes (2400, 2401, 2402)

Priority: 🟠 HIGH - Network isolation and fork risk

Actions:

1. Verify static-nodes.json:

   pct exec 2400 -- cat /var/lib/besu/static-nodes.json | jq .
   pct exec 2401 -- cat /var/lib/besu/static-nodes.json | jq .
   pct exec 2402 -- cat /var/lib/besu/static-nodes.json | jq .
   

   • Should contain all 15 nodes (validators, sentries, all RPC nodes)

2. Verify permissions-nodes.toml:

   pct exec 2400 -- cat /etc/besu/permissions-nodes.toml
   

   • Should contain all nodes

3. Check discovery setting:

   pct exec 2400 -- grep discovery-enabled /etc/besu/*.toml
   

   • Should be discovery-enabled=true for RPC nodes

4. Restart services:

   for vmid in 2400 2401 2402; do
       pct exec $vmid -- systemctl restart besu-rpc.service
   done
   

5. Verify network connectivity:

   # From ThirdWeb nodes, test connection to main network
   pct exec 2400 -- ping -c 1 192.168.11.211
   pct exec 2400 -- telnet 192.168.11.211 30303
   

2. Monitor Syncing Node (2403)

Priority: 🟢 LOW - Expected behavior

Actions:

• Monitor sync progress
• Once synced, verify peer count increases to 5-7
• No immediate action needed

---

Expected Peer Count Summary

For Network Size (10-20 nodes):

Status	Peer Count	Description
✅ Optimal	5-7 peers	Ideal for network size
✅ Good	3-4 peers	Acceptable, but below optimal
⚠️ Warning	2 peers	Under-connected, investigate
❌ Critical	0-1 peers	Isolated or syncing

Current Status:

• ✅ 8 nodes (2101, 2201, 2303-2308): 7 peers - Optimal
• ⚠️ 3 nodes (2400, 2401, 2402): 2 peers - Need investigation
• ⏳ 1 node (2403): 0 peers - Syncing (expected)

---

Verification Commands

Check Peer Count for All Nodes

for ip in 192.168.11.211 192.168.11.221 192.168.11.233 192.168.11.234 \
         192.168.11.235 192.168.11.236 192.168.11.237 192.168.11.238 \
         192.168.11.240 192.168.11.241 192.168.11.242 192.168.11.243; do
  echo -n "$ip: "
  curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
    http://$ip:8545 | jq -r '.result' | xargs printf "%d\n"
done

Check Block Height

for ip in 192.168.11.211 192.168.11.221 192.168.11.233 192.168.11.234 \
         192.168.11.235 192.168.11.236 192.168.11.237 192.168.11.238 \
         192.168.11.240 192.168.11.241 192.168.11.242 192.168.11.243; do
  echo -n "$ip: "
  curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    http://$ip:8545 | jq -r '.result' | xargs printf "%d\n"
done

Check Peer Details (if ADMIN API enabled)

curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}' \
  http://192.168.11.240:8545 | jq -r '.result[].enode'

---

Fix Scripts

Run this to check and fix peer connectivity:

cd /home/intlc/projects/proxmox
./scripts/fix-explorer-and-check-peers.sh

---

Summary

✅ Good Status (8 nodes):

• All have 7 peers (optimal for network size)
• All synchronized at block 1,145,367
• No action needed

⚠️ Needs Attention (3 nodes):

• ThirdWeb nodes (2400, 2401, 2402) have only 2 peers
• They're ahead by ~4,625 blocks (potential fork)
• Need to verify static-nodes.json and discovery configuration

⏳ Expected (1 node):

• Node 2403 has 0 peers (still syncing)
• No action needed until sync completes

---

Next Steps: Verify and fix static-nodes.json on ThirdWeb nodes to increase peer count to 5-7.