d-bis/proxmox
4
0
Fork 0
Code Issues 1 Pull Requests 9 Actions Packages Projects Releases Wiki Activity
Files
f0f0a4bb844eefe6c19e9568c3af9b2d387e5a2e
proxmox/docs/03-deployment/PROXMOX_VE_OPERATIONAL_DEPLOYMENT_TEMPLATE.md
defiQUG a086c451c3 docs: sync The Order routing (10210 HAProxy) and fix stale TBDs 
- E2E, ALL_VMIDS, operator checklist, RPC_ENDPOINTS_MASTER, DNS/NPM architecture
- PROXMOX deployment template: the-order wired via 10210
- Placeholders master + r630-02 incomplete summary for 10210
- CT 10210: chown /var/cache on host idmap (mandb clean) — applied on cluster

Made-with: Cursor
2026-03-27 15:06:06 -07:00

144 lines
8.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Proxmox VE — Operational deployment template
**Last Updated:** 2026-03-25
**Status:** Active — ties hypervisors, LAN/WAN, cluster peering, Chain 138 Besu tiers, NPMplus ingress, FQDNs, and deployment gates into one place.
**Machine-readable:** [`config/proxmox-operational-template.json`](../../config/proxmox-operational-template.json) (sync when you change VMIDs/IPs/FQDNs).
**Authoritative detail (do not drift):**
- VMID, port, status tables: [`docs/04-configuration/ALL_VMIDS_ENDPOINTS.md`](../04-configuration/ALL_VMIDS_ENDPOINTS.md)
- Shell/env single source: [`config/ip-addresses.conf`](../../config/ip-addresses.conf)
- Edge, port forwards, four NPMplus picture: [`docs/11-references/NETWORK_CONFIGURATION_MASTER.md`](../11-references/NETWORK_CONFIGURATION_MASTER.md)
- Contract deploy order / gates: [`docs/03-deployment/DEPLOYMENT_ORDER_OF_OPERATIONS.md`](DEPLOYMENT_ORDER_OF_OPERATIONS.md)
---
## 1. Proxmox VE hosts (management)
| Hostname | MGMT IP | Proxmox UI | Cluster | Role (target) |
|----------|---------|------------|---------|----------------|
| ml110 | 192.168.11.10 | https://192.168.11.10:8006 | h (legacy) | Planned WAN aggregator (OPNsense/pfSense); **migrate CT/VM off before repurpose** |
| r630-01 | 192.168.11.11 | https://192.168.11.11:8006 | h | Primary: Chain 138 RPC/CCIP-adjacent workloads, Sankofa Phoenix stack, much of DBIS |
| r630-02 | 192.168.11.12 | https://192.168.11.12:8006 | h | Firefly, MIM4U, Mifos LXC, extra NPMplus instances, supporting infra |
**LAN:** 192.168.11.0/24, gateway **192.168.11.1** (UDM Pro), VLAN 11. Extended node IP plan (r630-03 …): `config/ip-addresses.conf` comments.
---
## 2. Cluster peering (Corosync / quorum)
| Item | Value / note |
|------|----------------|
| Cluster name | **h** (verify live: `pvecm status`) |
| Ring | Typically same L2/L3 as MGMT — **192.168.11.0/24** |
| UDP ports | **5405–5412** between all nodes (+ SSH 22, API **8006** TCP) |
| Quorum | Odd node count preferred; during ml110 removal use 2-node awareness (risk window) or add qdevice |
Cluster and UDM: [`docs/04-configuration/UDM_PRO_PROXMOX_CLUSTER.md`](../04-configuration/UDM_PRO_PROXMOX_CLUSTER.md). **Live inventory:** [`docs/04-configuration/ALL_VMIDS_ENDPOINTS.md`](../04-configuration/ALL_VMIDS_ENDPOINTS.md), [`config/proxmox-operational-template.json`](../../config/proxmox-operational-template.json).
---
## 3. Chain 138 Besu — peering model (summary)
| Layer | VMID range (typical) | IPv4 pattern | P2P |
|--------|----------------------|--------------|-----|
| Validators | 1000–1004 | 192.168.11.100–104 | 30303 — **to sentries**, not raw public |
| Sentries | 1500–1506 | .150–.154, .213–.214 | Boundary / fan-out |
| Core RPC (deploy) | 2101 | **192.168.11.211** | 8545/8546 + 30303 |
| Core RPC (Nathan core-2) | 2102 | **192.168.11.212** | NPMplus **10235** / tunnel |
| Public RPC | 2201 | **192.168.11.221** | Frontends / bridge / read-mostly |
| Named RPC | 2303–2308 | .233–.238 | Partner-dedicated |
| ThirdWeb stack | 2400–2403 | .240–.243 | Includes translator/nginx on 2400 |
Canonical roles and adjacency rules: [`docs/02-architecture/CHAIN138_CANONICAL_NETWORK_ROLES_VALIDATORS_SENTRY_AND_RPC.md`](../02-architecture/CHAIN138_CANONICAL_NETWORK_ROLES_VALIDATORS_SENTRY_AND_RPC.md).
---
## 4. NPMplus and public ingress
| VMID | Internal IP(s) | Public IP (typical) | Purpose |
|------|----------------|---------------------|---------|
| 10233 | 192.168.11.166 / **.167** | 76.53.10.36 | Main d-bis.org, explorer, Option B RPC, MIM4U |
| 10234 | 192.168.11.168 | 76.53.10.37 | Secondary HA (confirm running) |
| 10235 | 192.168.11.169 | 76.53.10.38 (alt **76.53.10.42**) | rpc-core-2, Alltra, HYBX |
| 10236 | 192.168.11.170 | 76.53.10.40 | Dev / Codespaces tunnel, Gitea, Proxmox admin |
| 10237 | 192.168.11.171 | (tunnel/Mifos) | mifos.d-bis.org → VMID 5800 |
UDM Pro forwards **80 / 443** (and **81** where documented) to the matching internal IP. Detail: [`docs/04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md`](../04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md).
---
## 5. FQDN → backend (high level)
Use the full table in **ALL_VMIDS_ENDPOINTS** (“NPMplus Endpoint Configuration Reference”). Critical correctness checks:
- **explorer.d-bis.org** → VMID **5000**, **192.168.11.140** (not Sankofa IPs).
- **sankofa.nexus** / **phoenix.sankofa.nexus** → VMID **7801** / **7800** at **.51:3000** / **.50:4000**.
- **rpc-http-prv / rpc-ws-prv** → **2101** (.211); **rpc-http-pub / rpc-ws-pub** → **2201** (.221).
- **rpc.public-0138.defi-oracle.io** → **2400** **192.168.11.240:443** (update NPM if still pointing at decommissioned IPs).
**the-order.sankofa.nexus:** NPMplus → order HAProxy **10210** @ **192.168.11.39:80** (proxies to Sankofa portal **192.168.11.51:3000**). See `scripts/deployment/provision-order-haproxy-10210.sh`.
### 5.1 Order stack (live VMIDs, r630-01 unless noted)
| VMID | Hostname | IP | Role (short) |
|------|----------|-----|----------------|
| 10030 | order-identity | 192.168.11.40 | Identity |
| 10040 | order-intake | 192.168.11.41 | Intake |
| 10050 | order-finance | 192.168.11.49 | Finance |
| 10060 | order-dataroom | 192.168.11.42 | Dataroom |
| 10070 | order-legal | **192.168.11.87** | Legal — **moved off .54 2026-03-25** (`IP_ORDER_LEGAL`); .54 is **only** VMID 7804 gov-portals |
| 10080 | order-eresidency | 192.168.11.43 | eResidency |
| 10090 | order-portal-public | 192.168.11.36 | Public portal |
| 10091 | order-portal-internal | 192.168.11.35 | Internal portal |
| 10092 | order-mcp-legal | 192.168.11.37 | MCP legal |
| 10200 | order-prometheus | 192.168.11.46 | Metrics |
| 10201 | order-grafana | 192.168.11.47 | Dashboards |
| 10202 | order-opensearch | 192.168.11.48 | Search |
| 10210 | order-haproxy | 192.168.11.39 | Edge / HAProxy |
**Redis:** `ORDER_REDIS_IP` = 192.168.11.38 in `ip-addresses.conf` — bind to live VMID via `pct list` / audit script.
---
## 6. Deployment requirements (cross-domain)
### 6.1 Platform (Proxmox / network)
- [ ] All cluster nodes **quorate**; storage sufficient for CT/VM disks (local-lvm / future Ceph per master plan).
- [ ] **vmbr0** VLAN-aware; each workload IP **unique** on 192.168.11.0/24 (see ALL_VMIDS conflict section).
- [ ] UDM Pro routes and port-forwards match **NETWORK_CONFIGURATION_MASTER**.
- [ ] NPMplus proxy host rows match **ALL_VMIDS** (no Blockscout IP on Sankofa hostnames).
### 6.2 Chain 138 (contracts / ops)
- [ ] **Core RPC** 2101 reachable: `http://192.168.11.211:8545` for **deploy only** (not public RPC).
- [ ] `smom-dbis-138/.env`: `PRIVATE_KEY`, `RPC_URL_138`, nonce discipline — **DEPLOYMENT_ORDER_OF_OPERATIONS** Phase 0.
- [ ] Optional: `./scripts/deployment/preflight-chain138-deploy.sh` before any broadcast.
### 6.3 Application / operator
- [ ] Repo **`.env`** + **`smom-dbis-138/.env`** for operator scripts (`scripts/lib/load-project-env.sh`).
- [ ] Blockscout / verify / NPM backup scripts per **OPERATOR_READY_CHECKLIST** when doing release ops.
---
## 7. Maintaining this template
1. Change **ALL_VMIDS_ENDPOINTS** and/or **ip-addresses.conf** first (operator truth).
2. Update **`config/proxmox-operational-template.json`** so automation (future CMDB, checks) stays aligned.
3. Run **`./scripts/validation/validate-config-files.sh`** (includes JSON shape check for the template).
4. **Live diff (read-only, SSH):** from repo root on a host with SSH to Proxmox nodes: **`bash scripts/verify/audit-proxmox-operational-template.sh`**. Compares template VMIDs to `pct`/`qm` lists on ML110 + R630s (override **`PROXMOX_HOSTS`** if needed).
---
## 8. Related runbooks
| Topic | Doc |
|-------|-----|
| Operational runbooks index | [`OPERATIONAL_RUNBOOKS.md`](OPERATIONAL_RUNBOOKS.md) |
| Phoenix / Sankofa deploy | [`PHOENIX_DEPLOYMENT_RUNBOOK.md`](PHOENIX_DEPLOYMENT_RUNBOOK.md) |
| NPMplus health | [`docs/04-configuration/NPMPLUS_QUICK_REF.md`](../04-configuration/NPMPLUS_QUICK_REF.md) |
| 13-node / HA roadmap | [`docs/02-architecture/R630_13_NODE_DOD_HA_MASTER_PLAN.md`](../02-architecture/R630_13_NODE_DOD_HA_MASTER_PLAN.md) |
Reference in New Issue View Git Blame Copy Permalink