docs: sync The Order routing (10210 HAProxy) and fix stale TBDs

- E2E, ALL_VMIDS, operator checklist, RPC_ENDPOINTS_MASTER, DNS/NPM architecture - PROXMOX deployment template: the-order wired via 10210 - Placeholders master + r630-02 incomplete summary for 10210 - CT 10210: chown /var/cache on host idmap (mandb clean) — applied on cluster Made-with: Cursor
2026-03-27 15:06:06 -07:00
parent 430431f2f6
commit a086c451c3
8 changed files with 201 additions and 12 deletions
--- a/docs/03-deployment/PROXMOX_VE_OPERATIONAL_DEPLOYMENT_TEMPLATE.md
+++ b/docs/03-deployment/PROXMOX_VE_OPERATIONAL_DEPLOYMENT_TEMPLATE.md
@@ -0,0 +1,143 @@
+# Proxmox VE — Operational deployment template
+
+**Last Updated:** 2026-03-25  
+**Status:** Active — ties hypervisors, LAN/WAN, cluster peering, Chain 138 Besu tiers, NPMplus ingress, FQDNs, and deployment gates into one place.
+
+**Machine-readable:** [`config/proxmox-operational-template.json`](../../config/proxmox-operational-template.json) (sync when you change VMIDs/IPs/FQDNs).
+
+**Authoritative detail (do not drift):**
+
+- VMID, port, status tables: [`docs/04-configuration/ALL_VMIDS_ENDPOINTS.md`](../04-configuration/ALL_VMIDS_ENDPOINTS.md)
+- Shell/env single source: [`config/ip-addresses.conf`](../../config/ip-addresses.conf)
+- Edge, port forwards, four NPMplus picture: [`docs/11-references/NETWORK_CONFIGURATION_MASTER.md`](../11-references/NETWORK_CONFIGURATION_MASTER.md)
+- Contract deploy order / gates: [`docs/03-deployment/DEPLOYMENT_ORDER_OF_OPERATIONS.md`](DEPLOYMENT_ORDER_OF_OPERATIONS.md)
+
+---
+
+## 1. Proxmox VE hosts (management)
+
+| Hostname | MGMT IP | Proxmox UI | Cluster | Role (target) |
+|----------|---------|------------|---------|----------------|
+| ml110 | 192.168.11.10 | https://192.168.11.10:8006 | h (legacy) | Planned WAN aggregator (OPNsense/pfSense); **migrate CT/VM off before repurpose** |
+| r630-01 | 192.168.11.11 | https://192.168.11.11:8006 | h | Primary: Chain 138 RPC/CCIP-adjacent workloads, Sankofa Phoenix stack, much of DBIS |
+| r630-02 | 192.168.11.12 | https://192.168.11.12:8006 | h | Firefly, MIM4U, Mifos LXC, extra NPMplus instances, supporting infra |
+
+**LAN:** 192.168.11.0/24, gateway **192.168.11.1** (UDM Pro), VLAN 11. Extended node IP plan (r630-03 …): `config/ip-addresses.conf` comments.
+
+---
+
+## 2. Cluster peering (Corosync / quorum)
+
+| Item | Value / note |
+|------|----------------|
+| Cluster name | **h** (verify live: `pvecm status`) |
+| Ring | Typically same L2/L3 as MGMT — **192.168.11.0/24** |
+| UDP ports | **5405–5412** between all nodes (+ SSH 22, API **8006** TCP) |
+| Quorum | Odd node count preferred; during ml110 removal use 2-node awareness (risk window) or add qdevice |
+
+Cluster and UDM: [`docs/04-configuration/UDM_PRO_PROXMOX_CLUSTER.md`](../04-configuration/UDM_PRO_PROXMOX_CLUSTER.md). **Live inventory:** [`docs/04-configuration/ALL_VMIDS_ENDPOINTS.md`](../04-configuration/ALL_VMIDS_ENDPOINTS.md), [`config/proxmox-operational-template.json`](../../config/proxmox-operational-template.json).
+
+---
+
+## 3. Chain 138 Besu — peering model (summary)
+
+| Layer | VMID range (typical) | IPv4 pattern | P2P |
+|--------|----------------------|--------------|-----|
+| Validators | 1000–1004 | 192.168.11.100–104 | 30303 — **to sentries**, not raw public |
+| Sentries | 1500–1506 | .150–.154, .213–.214 | Boundary / fan-out |
+| Core RPC (deploy) | 2101 | **192.168.11.211** | 8545/8546 + 30303 |
+| Core RPC (Nathan core-2) | 2102 | **192.168.11.212** | NPMplus **10235** / tunnel |
+| Public RPC | 2201 | **192.168.11.221** | Frontends / bridge / read-mostly |
+| Named RPC | 2303–2308 | .233–.238 | Partner-dedicated |
+| ThirdWeb stack | 2400–2403 | .240–.243 | Includes translator/nginx on 2400 |
+
+Canonical roles and adjacency rules: [`docs/02-architecture/CHAIN138_CANONICAL_NETWORK_ROLES_VALIDATORS_SENTRY_AND_RPC.md`](../02-architecture/CHAIN138_CANONICAL_NETWORK_ROLES_VALIDATORS_SENTRY_AND_RPC.md).
+
+---
+
+## 4. NPMplus and public ingress
+
+| VMID | Internal IP(s) | Public IP (typical) | Purpose |
+|------|----------------|---------------------|---------|
+| 10233 | 192.168.11.166 / **.167** | 76.53.10.36 | Main d-bis.org, explorer, Option B RPC, MIM4U |
+| 10234 | 192.168.11.168 | 76.53.10.37 | Secondary HA (confirm running) |
+| 10235 | 192.168.11.169 | 76.53.10.38 (alt **76.53.10.42**) | rpc-core-2, Alltra, HYBX |
+| 10236 | 192.168.11.170 | 76.53.10.40 | Dev / Codespaces tunnel, Gitea, Proxmox admin |
+| 10237 | 192.168.11.171 | (tunnel/Mifos) | mifos.d-bis.org → VMID 5800 |
+
+UDM Pro forwards **80 / 443** (and **81** where documented) to the matching internal IP. Detail: [`docs/04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md`](../04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md).
+
+---
+
+## 5. FQDN → backend (high level)
+
+Use the full table in **ALL_VMIDS_ENDPOINTS** (“NPMplus Endpoint Configuration Reference”). Critical correctness checks:
+
+- **explorer.d-bis.org** → VMID **5000**, **192.168.11.140** (not Sankofa IPs).
+- **sankofa.nexus** / **phoenix.sankofa.nexus** → VMID **7801** / **7800** at **.51:3000** / **.50:4000**.
+- **rpc-http-prv / rpc-ws-prv** → **2101** (.211); **rpc-http-pub / rpc-ws-pub** → **2201** (.221).
+- **rpc.public-0138.defi-oracle.io** → **2400** **192.168.11.240:443** (update NPM if still pointing at decommissioned IPs).
+
+**the-order.sankofa.nexus:** NPMplus → order HAProxy **10210** @ **192.168.11.39:80** (proxies to Sankofa portal **192.168.11.51:3000**). See `scripts/deployment/provision-order-haproxy-10210.sh`.
+
+### 5.1 Order stack (live VMIDs, r630-01 unless noted)
+
+| VMID | Hostname | IP | Role (short) |
+|------|----------|-----|----------------|
+| 10030 | order-identity | 192.168.11.40 | Identity |
+| 10040 | order-intake | 192.168.11.41 | Intake |
+| 10050 | order-finance | 192.168.11.49 | Finance |
+| 10060 | order-dataroom | 192.168.11.42 | Dataroom |
+| 10070 | order-legal | **192.168.11.87** | Legal — **moved off .54 2026-03-25** (`IP_ORDER_LEGAL`); .54 is **only** VMID 7804 gov-portals |
+| 10080 | order-eresidency | 192.168.11.43 | eResidency |
+| 10090 | order-portal-public | 192.168.11.36 | Public portal |
+| 10091 | order-portal-internal | 192.168.11.35 | Internal portal |
+| 10092 | order-mcp-legal | 192.168.11.37 | MCP legal |
+| 10200 | order-prometheus | 192.168.11.46 | Metrics |
+| 10201 | order-grafana | 192.168.11.47 | Dashboards |
+| 10202 | order-opensearch | 192.168.11.48 | Search |
+| 10210 | order-haproxy | 192.168.11.39 | Edge / HAProxy |
+
+**Redis:** `ORDER_REDIS_IP` = 192.168.11.38 in `ip-addresses.conf` — bind to live VMID via `pct list` / audit script.
+
+---
+
+## 6. Deployment requirements (cross-domain)
+
+### 6.1 Platform (Proxmox / network)
+
+- [ ] All cluster nodes **quorate**; storage sufficient for CT/VM disks (local-lvm / future Ceph per master plan).
+- [ ] **vmbr0** VLAN-aware; each workload IP **unique** on 192.168.11.0/24 (see ALL_VMIDS conflict section).
+- [ ] UDM Pro routes and port-forwards match **NETWORK_CONFIGURATION_MASTER**.
+- [ ] NPMplus proxy host rows match **ALL_VMIDS** (no Blockscout IP on Sankofa hostnames).
+
+### 6.2 Chain 138 (contracts / ops)
+
+- [ ] **Core RPC** 2101 reachable: `http://192.168.11.211:8545` for **deploy only** (not public RPC).
+- [ ] `smom-dbis-138/.env`: `PRIVATE_KEY`, `RPC_URL_138`, nonce discipline — **DEPLOYMENT_ORDER_OF_OPERATIONS** Phase 0.
+- [ ] Optional: `./scripts/deployment/preflight-chain138-deploy.sh` before any broadcast.
+
+### 6.3 Application / operator
+
+- [ ] Repo **`.env`** + **`smom-dbis-138/.env`** for operator scripts (`scripts/lib/load-project-env.sh`).
+- [ ] Blockscout / verify / NPM backup scripts per **OPERATOR_READY_CHECKLIST** when doing release ops.
+
+---
+
+## 7. Maintaining this template
+
+1. Change **ALL_VMIDS_ENDPOINTS** and/or **ip-addresses.conf** first (operator truth).
+2. Update **`config/proxmox-operational-template.json`** so automation (future CMDB, checks) stays aligned.
+3. Run **`./scripts/validation/validate-config-files.sh`** (includes JSON shape check for the template).
+4. **Live diff (read-only, SSH):** from repo root on a host with SSH to Proxmox nodes: **`bash scripts/verify/audit-proxmox-operational-template.sh`**. Compares template VMIDs to `pct`/`qm` lists on ML110 + R630s (override **`PROXMOX_HOSTS`** if needed).
+
+---
+
+## 8. Related runbooks
+
+| Topic | Doc |
+|-------|-----|
+| Operational runbooks index | [`OPERATIONAL_RUNBOOKS.md`](OPERATIONAL_RUNBOOKS.md) |
+| Phoenix / Sankofa deploy | [`PHOENIX_DEPLOYMENT_RUNBOOK.md`](PHOENIX_DEPLOYMENT_RUNBOOK.md) |
+| NPMplus health | [`docs/04-configuration/NPMPLUS_QUICK_REF.md`](../04-configuration/NPMPLUS_QUICK_REF.md) |
+| 13-node / HA roadmap | [`docs/02-architecture/R630_13_NODE_DOD_HA_MASTER_PLAN.md`](../02-architecture/R630_13_NODE_DOD_HA_MASTER_PLAN.md) |