proxmox/docs/04-configuration/NPMPLUS_BACKEND_SERVICES_RESOLUTION.md

# NPMplus Backend Services Resolution

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

## Status: Network Routing Issue Identified

### Completed Tasks ✅

1. **Backend Service Diagnosis** ✅
   - Created `scripts/diagnose-npmplus-backend-services.sh`
   - Verified all 7 backend services are running and responding
   - Identified network routing issue between NPMplus and backend services

2. **Backend Service Fix Script** ✅
   - Created `scripts/fix-npmplus-backend-services.sh`
   - Attempted to start stopped containers (3 containers have config issues)
   - All backend services are actually running and accessible from their hosts

3. **HTTPS Domain Verification** ✅
   - Created `scripts/verify-npmplus-domains-https.sh`
   - Tested all 19 production domains
   - All domains returning 502 errors due to network routing issue

### Root Cause Analysis

**Problem:** NPMplus container (10233) cannot reach backend services on 192.168.11.0/24 network.

**Current Network Configuration:**
- **NPMplus Container:** 
  - Currently on VLAN 11 (tag=11) but not receiving IP address
  - Previously on 192.168.0.0/24 (default network)
  - Needs access to 192.168.11.0/24 for backend services

- **Backend Services:**
  - All on 192.168.11.0/24 (VLAN 11 - MGMT-LAN)
  - Services are running and responding:
    - VMID 5000 (blockscout-1): 192.168.11.140:80 ✅
    - VMID 10130 (dbis-frontend): 192.168.11.130:80 ✅
    - VMID 10150 (dbis-api-primary): 192.168.11.155:3000 ✅
    - VMID 10151 (dbis-api-secondary): 192.168.11.156:3000 ✅
    - VMID 7811 (mim-api-1): 192.168.11.36:80 ✅
    - **Current Active VMIDs:**
      - VMID 2501 (besu-rpc-2): 192.168.11.251:443 ✅ (Currently active)
      - VMID 2502 (besu-rpc-3): 192.168.11.252:443 ✅ (Currently active)
    - **Planned New VMIDs (Scripts Updated, VMIDs Not Yet Created):**
      - VMID 2101 (besu-rpc-core-1): 192.168.11.211:443 ⚠️ (Planned - not yet created)
      - VMID 2201 (besu-rpc-public-1): 192.168.11.221:443 ⚠️ (Planned - not yet created)
      - VMID 2301 (besu-rpc-private-1): 192.168.11.231:443 ⚠️ (Planned - not yet created)
      - VMID 2302 (besu-rpc-private-2): 192.168.11.232:443 ⚠️ (Planned - not yet created)

### Required Fixes

#### Option 1: Configure NPMplus with Static IP on VLAN 11 (Recommended)

1. **Assign static IP to NPMplus container:**
   ```bash
   ssh root@192.168.11.11
   pct set 10233 -net0 name=eth0,bridge=vmbr0,tag=11,ip=192.168.11.166/24,gw=192.168.11.1
   pct stop 10233
   pct start 10233
   ```

2. **Verify connectivity:**
   ```bash
   pct exec 10233 -- ping -c 2 192.168.11.140
   pct exec 10233 -- curl -I http://192.168.11.140:80
   ```

3. **Update NPMplus port forwarding:**
   - Update UDM Pro port forwarding rules to point to new IP (192.168.11.166)
   - Or keep existing IP (192.168.0.166) if dual-homed

#### Option 2: Configure UDM Pro Inter-VLAN Routing

1. **Enable routing between 192.168.0.0/24 and 192.168.11.0/24:**
   - Access UDM Pro web UI: https://192.168.11.1
   - Navigate to: Settings → Networks → Routing
   - Add static route:
     - Destination: 192.168.11.0/24
     - Gateway: 192.168.11.1
     - Interface: VLAN 11

2. **Configure firewall rules:**
   - Allow traffic from 192.168.0.0/24 to 192.168.11.0/24
   - Allow return traffic

3. **Revert NPMplus VLAN assignment:**
   ```bash
   ssh root@192.168.11.11
   pct set 10233 -net0 name=eth0,bridge=vmbr0,hwaddr=BC:24:11:97:68:6C,ip=dhcp,type=veth
   pct stop 10233
   pct start 10233
   ```

#### Option 3: Dual-Home NPMplus (Both Networks)

1. **Add second network interface:**
   ```bash
   ssh root@192.168.11.11
   pct set 10233 -net1 name=eth1,bridge=vmbr0,tag=11,ip=192.168.11.166/24
   pct stop 10233
   pct start 10233
   ```

2. **Configure routing inside container:**
   ```bash
   pct exec 10233 -- ip route add 192.168.11.0/24 dev eth1
   ```

### Verification Steps

After implementing one of the above options:

1. **Test backend connectivity from NPMplus:**
   ```bash
   ssh root@192.168.11.11
   pct exec 10233 -- curl -I http://192.168.11.140:80
   pct exec 10233 -- curl -I http://192.168.11.130:80
   pct exec 10233 -- curl -I http://192.168.11.155:3000
   ```

2. **Test HTTPS domains:**
   ```bash
   bash scripts/verify-npmplus-domains-https.sh
   ```

3. **Check NPMplus logs:**
   ```bash
   ssh root@192.168.11.11
   pct exec 10233 -- docker logs npmplus --tail 50
   ```

### Remaining Tasks

1. **Fix Network Routing** (Choose one option above)
2. **Fix Quirks Mode** (Backend HTML templates need `<!DOCTYPE html>`)
   - This requires updating backend application code
   - Affected services: All frontend applications
   - See: `docs/04-configuration/NPMPLUS_CSP_QUIRKS_MODE_FIX.md`

### Scripts Created

- `scripts/diagnose-npmplus-backend-services.sh` - Diagnose backend service status
- `scripts/fix-npmplus-backend-services.sh` - Start stopped services
- `scripts/verify-npmplus-domains-https.sh` - Verify HTTPS domain accessibility

### Next Steps

1. **Immediate:** Choose and implement one of the network routing options above
2. **After routing fixed:** Re-run `scripts/verify-npmplus-domains-https.sh` to verify 502 errors resolved
3. **Backend fixes:** Update backend HTML templates to include `<!DOCTYPE html>` for Quirks Mode fix

---

**Last Updated:** 2025-01-20  
**Status:** 
- ✅ Scripts updated with new VMID mappings (2101, 2201, 2301, 2302)
- ⚠️ New VMIDs not yet created on system (still using 2501, 2502)
- ⚠️ Network routing configuration required