# Complete Fix & Analysis Summary

**Date**: 2026-01-18  
**Status**: ✅ **All Scripts Created, Ready to Execute**

---

## Part 1: Explorer 502 Error - Fix

### Root Cause

The 502 Bad Gateway error occurs because:
1. **NPMplus** routes to `explorer.d-bis.org` → `192.168.11.140:80` (nginx on VMID 5000)
2. **Nginx on VMID 5000** tries to proxy to `127.0.0.1:4000` (Blockscout)
3. **Blockscout** is either:
   - Not running
   - Not listening on port 4000
   - Not accessible from nginx

### Immediate Fix

**Run from Proxmox host**:

```bash
cd /home/intlc/projects/proxmox
./scripts/fix-blockscout-nginx-complete.sh
```

This will:
1. Start Blockscout service
2. Wait for Blockscout to be ready
3. Fix nginx configuration
4. Restart nginx
5. Verify connectivity

### Alternative: Direct Route

If Blockscout can be made network-accessible on port 4000:

1. **Configure Blockscout** to listen on `0.0.0.0:4000` (not just `127.0.0.1:4000`)
2. **Update NPMplus** to route directly to `192.168.11.140:4000` (bypass nginx)

---

## Part 2: RPC Node Peer Count Analysis

### Expected Peer Counts

**For Network Size (10-20 nodes)**:
- **Minimum healthy**: 2-3 peers
- **Recommended**: **5-7 peers** ✅
- **Maximum**: 20-25 peers (based on max-peers=25 setting)

### Current Status Analysis

| Group | Nodes | Peers | Block Height | Status | Expected |
|-------|-------|-------|--------------|--------|----------|
| **Standard RPC** | 2101, 2201, 2303-2308 | **7** | 1,145,367 | ✅ Optimal | 5-7 ✅ |
| **ThirdWeb RPC** | 2400, 2401, 2402 | **2** | 1,149,992 | ⚠️ Low | 5-7 ⚠️ |
| **Syncing RPC** | 2403 | **0** | 600,172 | ⏳ Syncing | 5-7 (after sync) |

### Why Different Peer Counts?

#### ✅ 7 Peers (8 Nodes) - OPTIMAL

**Nodes**: 2101, 2201, 2303-2308  
**Why 7 peers?**
- Network has ~19-20 active nodes total
- 7 peers provides optimal connectivity and redundancy
- All nodes synchronized at same block height
- Well distributed across validators, sentries, and RPC nodes

**Connected to**:
- Validators (VMIDs 1000-1004)
- Sentries (VMIDs 1500-1503)
- Other RPC nodes
- **Total**: 7 peer connections

**Status**: ✅ **PERFECT** - No action needed

---

#### ⚠️ 2 Peers (3 Nodes) - UNDER-CONNECTED

**Nodes**: 2400, 2401, 2402 (ThirdWeb RPC nodes)  
**Why only 2 peers?**
- **Network isolation**: Only connected to each other (3 nodes = 2 peers each)
- **Missing from static-nodes.json**: Main network nodes not in their static-nodes.json
- **Discovery disabled**: Cannot discover other nodes automatically
- **Different network segment**: Firewall or network configuration issue

**Why ahead by ~4,625 blocks?**
- Isolated network may be producing its own blocks
- Or syncing from different source
- **Potential fork risk** if not fixed

**Should have**: 5-7 peers (connected to main network)

**Fix**:
```bash
# Verify static-nodes.json has all 15 nodes
./scripts/fix-thirdweb-peer-connectivity.sh

# Deploy correct node lists
./scripts/deploy-node-lists-to-all-nodes.sh

# Restart ThirdWeb nodes
for vmid in 2400 2401 2402; do
    pct exec $vmid -- systemctl restart besu-rpc.service
done
```

---

#### ⏳ 0 Peers (1 Node) - EXPECTED

**Node**: 2403  
**Why 0 peers?**
- **Still syncing** (block 600,172 vs current ~1,145,367)
- **~545,000 blocks behind**
- 0 peers is **normal during initial sync**
- Node will connect to peers once sync completes

**Should have**: 5-7 peers (after sync completes)

**Status**: ⏳ **EXPECTED** - No action needed (just wait for sync)

---

## Expected Peer Count Summary

### For Network Size (10-20 nodes):

| Status | Peer Count | Description |
|--------|------------|-------------|
| ✅ **Optimal** | **5-7 peers** | Ideal for network size (current: 8 nodes) |
| ✅ **Good** | **3-4 peers** | Acceptable, but below optimal |
| ⚠️ **Warning** | **2 peers** | Under-connected (current: 3 nodes) |
| ❌ **Critical** | **0-1 peers** | Isolated or syncing (current: 1 node) |

### Current Status:

- ✅ **8 nodes** (2101, 2201, 2303-2308): **7 peers** - **OPTIMAL** ✅
- ⚠️ **3 nodes** (2400, 2401, 2402): **2 peers** - **NEEDS FIX** ⚠️
- ⏳ **1 node** (2403): **0 peers** - **SYNCING** (expected)

---

## Complete Fix Script

Run this comprehensive fix script from Proxmox host:

```bash
cd /home/intlc/projects/proxmox
./scripts/fix-all-issues-complete.sh
```

This will:
1. Fix Blockscout and nginx (explorer 502 errors)
2. Analyze all RPC node peer counts
3. Fix ThirdWeb peer connectivity
4. Provide verification and next steps

---

## Quick Fix Commands

### Fix Explorer (502 Errors)

```bash
# From Proxmox host
./scripts/fix-blockscout-nginx-complete.sh
```

Or manually:
```bash
# Start Blockscout
pct exec 5000 -- systemctl start blockscout.service

# Check Blockscout
pct exec 5000 -- curl -I http://127.0.0.1:4000/api/v2/stats

# Restart nginx
pct exec 5000 -- systemctl restart nginx
```

### Fix ThirdWeb Peers (2 → 5-7 peers)

```bash
# From Proxmox host
./scripts/fix-thirdweb-peer-connectivity.sh

# Deploy correct node lists
./scripts/deploy-node-lists-to-all-nodes.sh

# Restart services
for vmid in 2400 2401 2402; do
    pct exec $vmid -- systemctl restart besu-rpc.service
done
```

---

## Verification

### Test Explorer API

```bash
curl -I https://explorer.d-bis.org/api/v2/stats
```

Should return HTTP 200 (not 502).

### Check Peer Counts

```bash
for ip in 192.168.11.211 192.168.11.221 192.168.11.233 192.168.11.234 \
         192.168.11.235 192.168.11.236 192.168.11.237 192.168.11.238 \
         192.168.11.240 192.168.11.241 192.168.11.242 192.168.11.243; do
  echo -n "$ip: "
  curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
    http://$ip:8545 | jq -r '.result' | xargs printf "%d\n"
done
```

**Expected**:
- 8 nodes: 7 peers each ✅
- 3 nodes: 5-7 peers each (after fix) ⚠️
- 1 node: 5-7 peers (after sync completes) ⏳

---

## Summary

### ✅ Completed
- ✅ All fix scripts created
- ✅ Peer count analysis complete
- ✅ Documentation created

### ⏳ Action Required

1. **Fix Explorer**:
   - Run: `./scripts/fix-blockscout-nginx-complete.sh`
   - Or start Blockscout and restart nginx manually

2. **Fix ThirdWeb Peers**:
   - Verify static-nodes.json has all 15 nodes
   - Deploy correct node lists
   - Restart ThirdWeb nodes

3. **Monitor Sync**:
   - Wait for VMID 2403 to complete sync
   - Verify peer count increases to 5-7 after sync

---

**Next Step**: Run the fix scripts from Proxmox host to apply all changes.