proxmox/docs/04-configuration/FIXES_PREPARED.md

# Fixes Prepared — Required and Optional

> Historical note: This checklist captured a specific remediation window and intentionally preserves then-current references to older Alltra/HYBX and `250x` RPC layouts. Read it as execution history. For the live fleet and current remediation targets, use `docs/04-configuration/ALL_VMIDS_ENDPOINTS.md`, `docs/04-configuration/RPC_ENDPOINTS_MASTER.md`, and the current verification scripts.

**Last Updated:** 2026-02-07  
**Purpose:** Single checklist of all fixes (required and optional) with copy-paste commands.  
**References:** [CHECKS_AND_FIXES_20260206.md](verification-evidence/CHECKS_AND_FIXES_20260206.md), [NEXT_STEPS_OPERATOR.md](../00-meta/NEXT_STEPS_OPERATOR.md), [UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md](UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md).  
**Consolidated (validators, block/tx, Sentries, RPCs + this):** [FULL_FIXES_PREPARED.md](FULL_FIXES_PREPARED.md).

---

## Summary

| Category | Item | Action | Where |
|----------|------|--------|--------|
| **Required** | UDM Pro port forward (Alltra/HYBX) | Manual | [§ UDM Pro](#1-udm-pro-port-forward-alltrahybx-required) |
| **Required** | Alltra/HYBX 502 (RPC + Cacti) | Verify backends → fix NPMplus or deploy | [§ Alltra/HYBX 502](#2-alltrahybx-502-failures-required) |
| **Optional** | NPMplus certs (remaining Alltra/HYBX hosts) | Script or UI | [§ NPMplus certs](#3-npmplus-certificates-remaining-alltrahybx-optional) |
| **Optional** | Explorer SSL | Manual NPMplus UI | [§ Explorer SSL](#4-explorer-ssl-optional) |
| **Optional** | NPMplus cert 134 (cross-all.defi-oracle.io) | Manual NPMplus UI | [§ Cert 134](#5-npmplus-cert-134-optional) |
| **Optional** | Shellcheck | Install + run | [§ Shellcheck](#6-shellcheck-optional) |
| **Optional** | Env permissions | Re-run if new .env added | [§ Env permissions](#7-env-permissions-optional) |
| **Optional** | Full verification re-run | Script | [§ Re-run verification](#8-re-run-full-verification-optional) |

---

## Required fixes

### 1. UDM Pro port forward (Alltra/HYBX)

**Why:** Alltra/HYBX direct/management access uses 76.53.10.38 → NPMplus at 192.168.11.169. Tunnel traffic goes to primary NPMplus (192.168.11.167); this forward is for direct access to the Alltra/HYBX NPMplus instance.

**Steps:** Add in **UniFi Network** → **Settings** → **Firewall & Security** (or **Networks** → **Port Forwarding**):

| Rule Name | Destination IP | Dest Port | Forward to IP | Forward to Port | Protocol |
|-----------|----------------|-----------|---------------|-----------------|----------|
| NPMplus Alltra/HYBX HTTP | 76.53.10.38 | 80 | 192.168.11.169 | 80 | TCP |
| NPMplus Alltra/HYBX HTTPS | 76.53.10.38 | 443 | 192.168.11.169 | 443 | TCP |
| NPMplus Alltra/HYBX Admin | 76.53.10.38 | 81 | 192.168.11.169 | 81 | TCP |

**Note:** 76.53.10.38 must be assigned on the UDM Pro.

**Verify (from LAN):**
```bash
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.169:80/
curl -s -o /dev/null -w "%{http_code}" -k https://192.168.11.169:81/
```
After port forward (from internet): `curl -s -o /dev/null -w "%{http_code}" http://76.53.10.38:80/`

**Doc:** [UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md](UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md)

---

### 2. Alltra/HYBX 502 failures (required)

**Observed (E2E 2026-02-07):** RPC and HTTPS return 502 for:

- `rpc-alltra.d-bis.org`, `rpc-alltra-2.d-bis.org`, `rpc-alltra-3.d-bis.org`
- `rpc-hybx.d-bis.org`, `rpc-hybx-2.d-bis.org`, `rpc-hybx-3.d-bis.org`
- `cacti-alltra.d-bis.org`, `cacti-hybx.d-bis.org`

**Traffic path:** Cloudflare DNS (CNAME to tunnel) → Cloudflare Tunnel → **primary NPMplus 192.168.11.167:443** → proxy hosts → backends.

**Root cause (choose one or both):**

1. **Backends not running** — Alltra/HYBX RPC (2500–2502, 2503–2505) and Cacti (5201, 5202) containers not deployed or stopped.
2. **NPMplus proxy target wrong** — Proxy hosts on 192.168.11.167 point to wrong IP/port (see [NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md) for correct backends).

**Expected backends (from master plan):**

| Domain type | Backend IP(s) | Port |
|-------------|---------------|------|
| rpc-alltra* | 192.168.11.172, .173, .174 (VMID 2500–2502) | 8545 |
| rpc-hybx*   | 192.168.11.246, .247, .248 (VMID 2503–2505) | 8545 |
| cacti-alltra | 192.168.11.177 (VMID 5201) | 80 |
| cacti-hybx   | 192.168.11.251 (VMID 5202) | 80 |

**Fix steps:**

1. **Verify backends from LAN (Proxmox or jump host):**
   ```bash
   # Alltra RPC
   curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.172:8545
   # HYBX RPC
   curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.246:8545
   # Cacti
   curl -s -o /dev/null -w "%{http_code}" http://192.168.11.177:80/
   curl -s -o /dev/null -w "%{http_code}" http://192.168.11.251:80/
   ```

2. **If backends respond:** In NPMplus (https://192.168.11.167:81) check Proxy Hosts for each Alltra/HYBX hostname: Forward hostname = backend IP, port = 8545 or 80 as above. Save and test.

3. **If backends do not respond:** Deploy or start the Alltra/HYBX containers (2500–2502, 2503–2505, 5201, 5202) per [NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md) and [MISSING_CONTAINERS_LIST.md](../03-deployment/MISSING_CONTAINERS_LIST.md). Then re-check NPMplus proxy targets.

---

## Optional fixes

### 3. NPMplus certificates (remaining Alltra/HYBX) (optional)

Request Let's Encrypt for any Alltra/HYBX proxy host that does not yet have a cert.

**From project root (LAN required; NPMplus API reachable):**
```bash
cd /path/to/proxmox
# First host only (verify before bulk)
FIRST_ONLY=1 NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh
# Then all remaining (no FIRST_ONLY)
NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh
```

**Via SSH to r630-01:**
```bash
bash scripts/run-via-proxmox-ssh.sh request-cert --host 192.168.11.11
```

**Reference:** CHECKS_AND_FIXES: *"For remaining hosts, run: NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh"*

---

### 4. Explorer SSL (optional)

If **https://explorer.d-bis.org** shows "Your connection isn't private":

1. Open NPMplus: **https://192.168.11.167:81** (use `.167` if `.166` refuses; credentials: `NPM_EMAIL`, `NPM_PASSWORD` from `.env`).
2. **SSL Certificates** → Add Let's Encrypt for `explorer.d-bis.org` (DNS Challenge + Cloudflare credential if needed).
3. **Proxy Hosts** → explorer.d-bis.org → **SSL** tab → assign cert, Force SSL, Save.

**Doc:** [EXPLORER_TROUBLESHOOTING.md](EXPLORER_TROUBLESHOOTING.md), [NEXT_STEPS_OPERATOR.md](../00-meta/NEXT_STEPS_OPERATOR.md) § Explorer SSL.

---

### 5. NPMplus cert 134 (optional)

If verification reports **"cert files missing"** for cert ID 134 (cross-all.defi-oracle.io):

1. Open NPMplus: **https://192.168.11.167:81** → **SSL Certificates**.
2. Find **cross-all.defi-oracle.io** → re-save or **Request** Let's Encrypt again to restore cert files on disk.

No automated script; UI only.

---

### 6. Shellcheck (optional)

Install and run optional shellcheck (no failure if not installed):

```bash
# Install (one of)
sudo apt install shellcheck    # Debian/Ubuntu
brew install shellcheck       # macOS

# Run (from project root)
cd /path/to/proxmox
bash scripts/verify/run-shellcheck.sh --optional
# Or without --optional to fail on issues:
bash scripts/verify/run-shellcheck.sh
```

---

### 7. Env permissions (optional)

Re-run if you added new `.env` files and want consistent permissions:

```bash
cd /path/to/proxmox
bash scripts/security/secure-env-permissions.sh
```

Applies `chmod 600` to `.env`, `unifi-api/.env`, `smom-dbis-138/.env`, `dbis_core/.env` where present.

---

### 8. Re-run full verification (optional)

Re-run the full 6-step verification and regenerate source-of-truth:

```bash
cd /path/to/proxmox
bash scripts/verify/run-full-verification.sh
```

Outputs under `docs/04-configuration/verification-evidence/` and updates `docs/04-configuration/INGRESS_SOURCE_OF_TRUTH.json`.

---

## Quick command index

| Goal | Command |
|------|---------|
| UDM Pro Alltra/HYBX | Manual: [UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md](UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md) |
| Request NPMplus certs (first only) | `FIRST_ONLY=1 NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh` |
| Request NPMplus certs (all remaining) | `NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh` |
| Explorer SSL | NPMplus UI → SSL Certificates → explorer.d-bis.org; Proxy Hosts → SSL tab |
| Cert 134 fix | NPMplus UI → SSL Certificates → cross-all.defi-oracle.io → re-save / re-request |
| Shellcheck | `bash scripts/verify/run-shellcheck.sh --optional` |
| Env permissions | `bash scripts/security/secure-env-permissions.sh` |
| Full verification | `bash scripts/verify/run-full-verification.sh` |
| Backup NPMplus | `bash scripts/verify/backup-npmplus.sh` |

---

## Execution order suggestion

1. **Required:** UDM Pro port forward (if you use direct 76.53.10.38 access).
2. **Required:** Diagnose Alltra/HYBX 502 (verify backends, then fix NPMplus or deploy containers).
3. **Optional:** NPMplus certs for remaining Alltra/HYBX hosts.
4. **Optional:** Explorer SSL, cert 134, shellcheck, env permissions, full verification re-run as needed.

Evidence and prior checks: [verification-evidence/CHECKS_AND_FIXES_20260206.md](verification-evidence/CHECKS_AND_FIXES_20260206.md).