proxmox/docs/archive/deployment-reports/FIX_ALL_ISSUES_COMPLETE.md

Fix All Issues - Complete Summary

Date: 2026-01-18
Status: ✅ All Scripts Created, Action Required

---

Summary

Issue 1: Explorer 502 Errors ✅ Scripts Created

Problem: Blockscout API returning 502 Bad Gateway
Solution: Direct route from NPMplus → Blockscout:4000 (bypass nginx)
Status: ⏳ Configuration files updated, NPMplus update pending

Issue 2: RPC Node Peer Counts ✅ Analysis Complete

Problem: Different peer counts (0, 2, 7 peers)
Analysis: Documented expected vs actual peer counts
Status: ✅ Analysis complete, fix scripts created

---

Part 1: Explorer Fixes

Created Scripts

1. ✅ scripts/fix-all-explorer-issues.sh - Main fix script
2. ✅ scripts/fix-explorer-and-check-peers.sh - Combined fix + peer check
3. ✅ scripts/fix-blockscout-network-access.sh - Network accessibility fix
4. ✅ scripts/verify-blockscout-port-4000.sh - Port verification
5. ✅ scripts/diagnose-explorer-502-error.sh - Comprehensive diagnostics

Configuration Files Updated

✅ All NPMplus configuration scripts updated to use port 4000
✅ All documentation updated

Action Required

Update NPMplus configuration:

1. Log into NPMplus: https://192.168.0.166:81
2. Find explorer.d-bis.org proxy host
3. Update Forward Port: 80 → 4000
4. Save changes

Or run automated script (from machine with NPMplus access):

cd /home/intlc/projects/proxmox
./scripts/apply-direct-blockscout-route.sh

---

Part 2: RPC Node Peer Count Analysis

Current Status

Group	Nodes	Peers	Block Height	Status	Expected
Standard RPC	2101, 2201, 2303-2308	7	1,145,367	✅ Optimal	5-7 ✅
ThirdWeb RPC	2400, 2401, 2402	2	1,149,992	⚠️ Low	5-7 ⚠️
Syncing RPC	2403	0	600,172	⏳ Syncing	5-7 (after sync)

Expected Peer Counts

For Network Size (10-20 nodes):

• Minimum healthy: 2-3 peers
• Recommended: 5-7 peers ✅
• Maximum: 20-25 peers (based on max-peers=25 setting)

Current Assessment:

• ✅ 8 nodes have 7 peers - OPTIMAL (2101, 2201, 2303-2308)
• ⚠️ 3 nodes have 2 peers - UNDER-CONNECTED (2400, 2401, 2402)
• ⏳ 1 node has 0 peers - SYNCING (2403 - expected)

---

Part 3: Why Different Peer Counts?

✅ 7 Peers (Optimal) - 8 Nodes

Nodes: 2101, 2201, 2303-2308
Status: ✅ Excellent

Analysis:

• All connected to validators, sentries, and other RPC nodes
• Well distributed across the network
• Optimal for network size (10-20 nodes)
• All synchronized at same block height

Why 7 peers?:

• Network has ~19-20 active nodes total
• 7 peers provides good redundancy and connectivity
• Standard RPC nodes configured with max-peers=25
• Discovery enabled allows finding all peers

⚠️ 2 Peers (Under-connected) - 3 Nodes

Nodes: 2400, 2401, 2402 (ThirdWeb RPC nodes)
Status: ⚠️ Warning - Network Isolation

Analysis:

• Only connected to each other (3 nodes = 2 peers each)
• Ahead by ~4,625 blocks (potential fork!)
• Not connected to main network (2101, 2201, 2303-2308, validators, sentries)

Why only 2 peers?:

1. Missing from static-nodes.json: ThirdWeb nodes may not have all main network nodes
2. Discovery disabled: Cannot discover other nodes automatically
3. Network isolation: Firewall or network segmentation
4. Different bootnodes: Connecting to different network

Why ahead in blocks?:

• Isolated network may be producing its own blocks
• Or syncing from different source
• Potential fork risk if not fixed

Should have: 5-7 peers (connected to main network)

⏳ 0 Peers (Syncing) - 1 Node

Node: 2403
Status: ⏳ Expected During Sync

Analysis:

• Still syncing (block 600,172 vs current ~1,145,367)
• 0 peers is normal during initial sync
• Will connect to peers once sync completes

Should have: 5-7 peers (after sync completes)

---

Fix ThirdWeb Nodes (2400, 2401, 2402)

Root Cause

Most likely: static-nodes.json incomplete or discovery disabled

Fix Steps

1. Verify static-nodes.json

# Check each ThirdWeb node
for vmid in 2400 2401 2402; do
    echo "=== VMID $vmid ==="
    pct exec $vmid -- cat /var/lib/besu/static-nodes.json | jq '. | length'
done

Should have: 15 nodes total

• 5 Validators (1000-1004)
• 4 Sentries (1500-1503)
• 6 Standard RPC (2101, 2201, 2303-2308)
• 3 ThirdWeb RPC (2400, 2401, 2402)

2. Verify Discovery Setting

# Check discovery setting
for vmid in 2400 2401 2402; do
    echo "=== VMID $vmid ==="
    pct exec $vmid -- grep discovery-enabled /etc/besu/*.toml
done

Should be: discovery-enabled=true for RPC nodes

3. Update Configuration Files

If static-nodes.json is incomplete:

# Deploy correct static-nodes.json to all nodes
cd /home/intlc/projects/proxmox
./scripts/deploy-node-lists-to-all-nodes.sh

This will:

• Sync static-nodes.json to all nodes
• Sync permissions-nodes.toml to all nodes
• Ensure all nodes have all 15 enodes

4. Enable Discovery (if disabled)

# Enable discovery on ThirdWeb nodes
for vmid in 2400 2401 2402; do
    pct exec $vmid -- sed -i 's/^discovery-enabled=.*/discovery-enabled=true/' /etc/besu/*.toml
    pct exec $vmid -- systemctl restart besu-rpc.service
done

5. Restart Services

# Restart ThirdWeb nodes
for vmid in 2400 2401 2402; do
    pct exec $vmid -- systemctl restart besu-rpc.service
done

6. Verify Fix

# Check peer count after fix
for ip in 192.168.11.240 192.168.11.241 192.168.11.242; do
    echo -n "$ip: "
    curl -s -X POST -H "Content-Type: application/json" \
        --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
        http://$ip:8545 | jq -r '.result' | xargs printf "%d\n"
done

Expected: 5-7 peers each

---

Quick Fix Script

Run this comprehensive script:

cd /home/intlc/projects/proxmox

# Fix explorer and check peers
./scripts/fix-explorer-and-check-peers.sh

# Or analyze peers only
./scripts/analyze-all-rpc-peers.sh

# Or fix ThirdWeb connectivity
./scripts/fix-thirdweb-peer-connectivity.sh

---

Expected Results After Fixes

Explorer

• ✅ No more 502 errors
• ✅ Direct route: NPMplus → Blockscout:4000
• ✅ Faster API response times

RPC Nodes

• ✅ All nodes: 5-7 peers
• ✅ ThirdWeb nodes connected to main network
• ✅ All nodes synchronized at same block height
• ✅ No forks or block height mismatches

---

Summary

✅ Completed

• ✅ Explorer configuration files updated
• ✅ Peer count analysis complete
• ✅ Fix scripts created
• ✅ Documentation created

⏳ Action Required

1. Update NPMplus:

   • Change explorer.d-bis.org port: 80 → 4000
   • Or run: ./scripts/apply-direct-blockscout-route.sh

2. Fix ThirdWeb Nodes:

   • Verify static-nodes.json has all 15 nodes
   • Enable discovery if disabled
   • Restart services
   • Or run: ./scripts/deploy-node-lists-to-all-nodes.sh

3. Monitor:

   • Verify explorer 502 errors resolved
   • Verify ThirdWeb nodes reach 5-7 peers
   • Verify all nodes synchronized

---

Documentation

• Explorer Fixes: docs/04-configuration/FIX_502_ERROR_GUIDE.md
• Direct Route: docs/04-configuration/DIRECT_BLOCKSCOUT_ROUTE_UPDATE.md
• Peer Analysis: docs/06-besu/RPC_NODE_PEER_COUNT_ANALYSIS.md
• Performance Tuning: docs/04-configuration/BESU_PERFORMANCE_TUNING.md

---

Status: ✅ All scripts and documentation ready
Next: Run the fix scripts to apply changes