proxmox/docs/archive/deployment-reports/COMPLETE_FIX_AND_ANALYSIS.md

Complete Fix & Analysis Summary

Date: 2026-01-18
Status: ✅ All Scripts Created, Ready to Execute

---

Part 1: Explorer 502 Error - Fix

Root Cause

The 502 Bad Gateway error occurs because:

1. NPMplus routes to explorer.d-bis.org → 192.168.11.140:80 (nginx on VMID 5000)
2. Nginx on VMID 5000 tries to proxy to 127.0.0.1:4000 (Blockscout)
3. Blockscout is either:
   • Not running
   • Not listening on port 4000
   • Not accessible from nginx

Immediate Fix

Run from Proxmox host:

cd /home/intlc/projects/proxmox
./scripts/fix-blockscout-nginx-complete.sh

This will:

1. Start Blockscout service
2. Wait for Blockscout to be ready
3. Fix nginx configuration
4. Restart nginx
5. Verify connectivity

Alternative: Direct Route

If Blockscout can be made network-accessible on port 4000:

1. Configure Blockscout to listen on 0.0.0.0:4000 (not just 127.0.0.1:4000)
2. Update NPMplus to route directly to 192.168.11.140:4000 (bypass nginx)

---

Part 2: RPC Node Peer Count Analysis

Expected Peer Counts

For Network Size (10-20 nodes):

• Minimum healthy: 2-3 peers
• Recommended: 5-7 peers ✅
• Maximum: 20-25 peers (based on max-peers=25 setting)

Current Status Analysis

Group	Nodes	Peers	Block Height	Status	Expected
Standard RPC	2101, 2201, 2303-2308	7	1,145,367	✅ Optimal	5-7 ✅
ThirdWeb RPC	2400, 2401, 2402	2	1,149,992	⚠️ Low	5-7 ⚠️
Syncing RPC	2403	0	600,172	⏳ Syncing	5-7 (after sync)

Why Different Peer Counts?

✅ 7 Peers (8 Nodes) - OPTIMAL

Nodes: 2101, 2201, 2303-2308
Why 7 peers?

• Network has ~19-20 active nodes total
• 7 peers provides optimal connectivity and redundancy
• All nodes synchronized at same block height
• Well distributed across validators, sentries, and RPC nodes

Connected to:

• Validators (VMIDs 1000-1004)
• Sentries (VMIDs 1500-1503)
• Other RPC nodes
• Total: 7 peer connections

Status: ✅ PERFECT - No action needed

---

⚠️ 2 Peers (3 Nodes) - UNDER-CONNECTED

Nodes: 2400, 2401, 2402 (ThirdWeb RPC nodes)
Why only 2 peers?

• Network isolation: Only connected to each other (3 nodes = 2 peers each)
• Missing from static-nodes.json: Main network nodes not in their static-nodes.json
• Discovery disabled: Cannot discover other nodes automatically
• Different network segment: Firewall or network configuration issue

Why ahead by ~4,625 blocks?

• Isolated network may be producing its own blocks
• Or syncing from different source
• Potential fork risk if not fixed

Should have: 5-7 peers (connected to main network)

Fix:

# Verify static-nodes.json has all 15 nodes
./scripts/fix-thirdweb-peer-connectivity.sh

# Deploy correct node lists
./scripts/deploy-node-lists-to-all-nodes.sh

# Restart ThirdWeb nodes
for vmid in 2400 2401 2402; do
    pct exec $vmid -- systemctl restart besu-rpc.service
done

---

⏳ 0 Peers (1 Node) - EXPECTED

Node: 2403
Why 0 peers?

• Still syncing (block 600,172 vs current ~1,145,367)
• ~545,000 blocks behind
• 0 peers is normal during initial sync
• Node will connect to peers once sync completes

Should have: 5-7 peers (after sync completes)

Status: ⏳ EXPECTED - No action needed (just wait for sync)

---

Expected Peer Count Summary

For Network Size (10-20 nodes):

Status	Peer Count	Description
✅ Optimal	5-7 peers	Ideal for network size (current: 8 nodes)
✅ Good	3-4 peers	Acceptable, but below optimal
⚠️ Warning	2 peers	Under-connected (current: 3 nodes)
❌ Critical	0-1 peers	Isolated or syncing (current: 1 node)

Current Status:

• ✅ 8 nodes (2101, 2201, 2303-2308): 7 peers - OPTIMAL ✅
• ⚠️ 3 nodes (2400, 2401, 2402): 2 peers - NEEDS FIX ⚠️
• ⏳ 1 node (2403): 0 peers - SYNCING (expected)

---

Complete Fix Script

Run this comprehensive fix script from Proxmox host:

cd /home/intlc/projects/proxmox
./scripts/fix-all-issues-complete.sh

This will:

1. Fix Blockscout and nginx (explorer 502 errors)
2. Analyze all RPC node peer counts
3. Fix ThirdWeb peer connectivity
4. Provide verification and next steps

---

Quick Fix Commands

Fix Explorer (502 Errors)

# From Proxmox host
./scripts/fix-blockscout-nginx-complete.sh

Or manually:

# Start Blockscout
pct exec 5000 -- systemctl start blockscout.service

# Check Blockscout
pct exec 5000 -- curl -I http://127.0.0.1:4000/api/v2/stats

# Restart nginx
pct exec 5000 -- systemctl restart nginx

Fix ThirdWeb Peers (2 → 5-7 peers)

# From Proxmox host
./scripts/fix-thirdweb-peer-connectivity.sh

# Deploy correct node lists
./scripts/deploy-node-lists-to-all-nodes.sh

# Restart services
for vmid in 2400 2401 2402; do
    pct exec $vmid -- systemctl restart besu-rpc.service
done

---

Verification

Test Explorer API

curl -I https://explorer.d-bis.org/api/v2/stats

Should return HTTP 200 (not 502).

Check Peer Counts

for ip in 192.168.11.211 192.168.11.221 192.168.11.233 192.168.11.234 \
         192.168.11.235 192.168.11.236 192.168.11.237 192.168.11.238 \
         192.168.11.240 192.168.11.241 192.168.11.242 192.168.11.243; do
  echo -n "$ip: "
  curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
    http://$ip:8545 | jq -r '.result' | xargs printf "%d\n"
done

Expected:

• 8 nodes: 7 peers each ✅
• 3 nodes: 5-7 peers each (after fix) ⚠️
• 1 node: 5-7 peers (after sync completes) ⏳

---

Summary

✅ Completed

• ✅ All fix scripts created
• ✅ Peer count analysis complete
• ✅ Documentation created

⏳ Action Required

1. Fix Explorer:

   • Run: ./scripts/fix-blockscout-nginx-complete.sh
   • Or start Blockscout and restart nginx manually

2. Fix ThirdWeb Peers:

   • Verify static-nodes.json has all 15 nodes
   • Deploy correct node lists
   • Restart ThirdWeb nodes

3. Monitor Sync:

   • Wait for VMID 2403 to complete sync
   • Verify peer count increases to 5-7 after sync

---

Next Step: Run the fix scripts from Proxmox host to apply all changes.