Files
proxmox/docs/04-configuration/FIXES_PREPARED.md

210 lines
9.1 KiB
Markdown
Raw Normal View History

# Fixes Prepared — Required and Optional
**Last Updated:** 2026-02-07
**Purpose:** Single checklist of all fixes (required and optional) with copy-paste commands.
**References:** [CHECKS_AND_FIXES_20260206.md](verification-evidence/CHECKS_AND_FIXES_20260206.md), [NEXT_STEPS_OPERATOR.md](../00-meta/NEXT_STEPS_OPERATOR.md), [UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md](UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md).
**Consolidated (validators, block/tx, Sentries, RPCs + this):** [FULL_FIXES_PREPARED.md](FULL_FIXES_PREPARED.md).
---
## Summary
| Category | Item | Action | Where |
|----------|------|--------|--------|
| **Required** | UDM Pro port forward (Alltra/HYBX) | Manual | [§ UDM Pro](#1-udm-pro-port-forward-alltrahybx-required) |
| **Required** | Alltra/HYBX 502 (RPC + Cacti) | Verify backends → fix NPMplus or deploy | [§ Alltra/HYBX 502](#2-alltrahybx-502-failures-required) |
| **Optional** | NPMplus certs (remaining Alltra/HYBX hosts) | Script or UI | [§ NPMplus certs](#3-npmplus-certificates-remaining-alltrahybx-optional) |
| **Optional** | Explorer SSL | Manual NPMplus UI | [§ Explorer SSL](#4-explorer-ssl-optional) |
| **Optional** | NPMplus cert 134 (cross-all.defi-oracle.io) | Manual NPMplus UI | [§ Cert 134](#5-npmplus-cert-134-optional) |
| **Optional** | Shellcheck | Install + run | [§ Shellcheck](#6-shellcheck-optional) |
| **Optional** | Env permissions | Re-run if new .env added | [§ Env permissions](#7-env-permissions-optional) |
| **Optional** | Full verification re-run | Script | [§ Re-run verification](#8-re-run-full-verification-optional) |
---
## Required fixes
### 1. UDM Pro port forward (Alltra/HYBX)
**Why:** Alltra/HYBX direct/management access uses 76.53.10.38 → NPMplus at 192.168.11.169. Tunnel traffic goes to primary NPMplus (192.168.11.167); this forward is for direct access to the Alltra/HYBX NPMplus instance.
**Steps:** Add in **UniFi Network****Settings****Firewall & Security** (or **Networks****Port Forwarding**):
| Rule Name | Destination IP | Dest Port | Forward to IP | Forward to Port | Protocol |
|-----------|----------------|-----------|---------------|-----------------|----------|
| NPMplus Alltra/HYBX HTTP | 76.53.10.38 | 80 | 192.168.11.169 | 80 | TCP |
| NPMplus Alltra/HYBX HTTPS | 76.53.10.38 | 443 | 192.168.11.169 | 443 | TCP |
| NPMplus Alltra/HYBX Admin | 76.53.10.38 | 81 | 192.168.11.169 | 81 | TCP |
**Note:** 76.53.10.38 must be assigned on the UDM Pro.
**Verify (from LAN):**
```bash
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.169:80/
curl -s -o /dev/null -w "%{http_code}" -k https://192.168.11.169:81/
```
After port forward (from internet): `curl -s -o /dev/null -w "%{http_code}" http://76.53.10.38:80/`
**Doc:** [UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md](UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md)
---
### 2. Alltra/HYBX 502 failures (required)
**Observed (E2E 2026-02-07):** RPC and HTTPS return 502 for:
- `rpc-alltra.d-bis.org`, `rpc-alltra-2.d-bis.org`, `rpc-alltra-3.d-bis.org`
- `rpc-hybx.d-bis.org`, `rpc-hybx-2.d-bis.org`, `rpc-hybx-3.d-bis.org`
- `cacti-alltra.d-bis.org`, `cacti-hybx.d-bis.org`
**Traffic path:** Cloudflare DNS (CNAME to tunnel) → Cloudflare Tunnel → **primary NPMplus 192.168.11.167:443** → proxy hosts → backends.
**Root cause (choose one or both):**
1. **Backends not running** — Alltra/HYBX RPC (25002502, 25032505) and Cacti (5201, 5202) containers not deployed or stopped.
2. **NPMplus proxy target wrong** — Proxy hosts on 192.168.11.167 point to wrong IP/port (see [NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md) for correct backends).
**Expected backends (from master plan):**
| Domain type | Backend IP(s) | Port |
|-------------|---------------|------|
| rpc-alltra* | 192.168.11.172, .173, .174 (VMID 25002502) | 8545 |
| rpc-hybx* | 192.168.11.246, .247, .248 (VMID 25032505) | 8545 |
| cacti-alltra | 192.168.11.177 (VMID 5201) | 80 |
| cacti-hybx | 192.168.11.251 (VMID 5202) | 80 |
**Fix steps:**
1. **Verify backends from LAN (Proxmox or jump host):**
```bash
# Alltra RPC
curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.172:8545
# HYBX RPC
curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.246:8545
# Cacti
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.177:80/
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.251:80/
```
2. **If backends respond:** In NPMplus (https://192.168.11.167:81) check Proxy Hosts for each Alltra/HYBX hostname: Forward hostname = backend IP, port = 8545 or 80 as above. Save and test.
3. **If backends do not respond:** Deploy or start the Alltra/HYBX containers (25002502, 25032505, 5201, 5202) per [NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md) and [MISSING_CONTAINERS_LIST.md](../03-deployment/MISSING_CONTAINERS_LIST.md). Then re-check NPMplus proxy targets.
---
## Optional fixes
### 3. NPMplus certificates (remaining Alltra/HYBX) (optional)
Request Let's Encrypt for any Alltra/HYBX proxy host that does not yet have a cert.
**From project root (LAN required; NPMplus API reachable):**
```bash
cd /path/to/proxmox
# First host only (verify before bulk)
FIRST_ONLY=1 NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh
# Then all remaining (no FIRST_ONLY)
NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh
```
**Via SSH to r630-01:**
```bash
bash scripts/run-via-proxmox-ssh.sh request-cert --host 192.168.11.11
```
**Reference:** CHECKS_AND_FIXES: *"For remaining hosts, run: NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh"*
---
### 4. Explorer SSL (optional)
If **https://explorer.d-bis.org** shows "Your connection isn't private":
1. Open NPMplus: **https://192.168.11.167:81** (use `.167` if `.166` refuses; credentials: `NPM_EMAIL`, `NPM_PASSWORD` from `.env`).
2. **SSL Certificates** → Add Let's Encrypt for `explorer.d-bis.org` (DNS Challenge + Cloudflare credential if needed).
3. **Proxy Hosts** → explorer.d-bis.org → **SSL** tab → assign cert, Force SSL, Save.
**Doc:** [EXPLORER_TROUBLESHOOTING.md](EXPLORER_TROUBLESHOOTING.md), [NEXT_STEPS_OPERATOR.md](../00-meta/NEXT_STEPS_OPERATOR.md) § Explorer SSL.
---
### 5. NPMplus cert 134 (optional)
If verification reports **"cert files missing"** for cert ID 134 (cross-all.defi-oracle.io):
1. Open NPMplus: **https://192.168.11.167:81****SSL Certificates**.
2. Find **cross-all.defi-oracle.io** → re-save or **Request** Let's Encrypt again to restore cert files on disk.
No automated script; UI only.
---
### 6. Shellcheck (optional)
Install and run optional shellcheck (no failure if not installed):
```bash
# Install (one of)
sudo apt install shellcheck # Debian/Ubuntu
brew install shellcheck # macOS
# Run (from project root)
cd /path/to/proxmox
bash scripts/verify/run-shellcheck.sh --optional
# Or without --optional to fail on issues:
bash scripts/verify/run-shellcheck.sh
```
---
### 7. Env permissions (optional)
Re-run if you added new `.env` files and want consistent permissions:
```bash
cd /path/to/proxmox
bash scripts/security/secure-env-permissions.sh
```
Applies `chmod 600` to `.env`, `unifi-api/.env`, `smom-dbis-138/.env`, `dbis_core/.env` where present.
---
### 8. Re-run full verification (optional)
Re-run the full 6-step verification and regenerate source-of-truth:
```bash
cd /path/to/proxmox
bash scripts/verify/run-full-verification.sh
```
Outputs under `docs/04-configuration/verification-evidence/` and updates `docs/04-configuration/INGRESS_SOURCE_OF_TRUTH.json`.
---
## Quick command index
| Goal | Command |
|------|---------|
| UDM Pro Alltra/HYBX | Manual: [UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md](UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md) |
| Request NPMplus certs (first only) | `FIRST_ONLY=1 NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh` |
| Request NPMplus certs (all remaining) | `NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh` |
| Explorer SSL | NPMplus UI → SSL Certificates → explorer.d-bis.org; Proxy Hosts → SSL tab |
| Cert 134 fix | NPMplus UI → SSL Certificates → cross-all.defi-oracle.io → re-save / re-request |
| Shellcheck | `bash scripts/verify/run-shellcheck.sh --optional` |
| Env permissions | `bash scripts/security/secure-env-permissions.sh` |
| Full verification | `bash scripts/verify/run-full-verification.sh` |
| Backup NPMplus | `bash scripts/verify/backup-npmplus.sh` |
---
## Execution order suggestion
1. **Required:** UDM Pro port forward (if you use direct 76.53.10.38 access).
2. **Required:** Diagnose Alltra/HYBX 502 (verify backends, then fix NPMplus or deploy containers).
3. **Optional:** NPMplus certs for remaining Alltra/HYBX hosts.
4. **Optional:** Explorer SSL, cert 134, shellcheck, env permissions, full verification re-run as needed.
Evidence and prior checks: [verification-evidence/CHECKS_AND_FIXES_20260206.md](verification-evidence/CHECKS_AND_FIXES_20260206.md).