d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c281c67a79b9113a3fd781c265544f90b0b50da2
proxmox/docs/11-references/NETWORK_CONFIGURATION_MASTER.md
defiQUG dbd517b279 Sync workspace: config, docs, scripts, CI, operator rules, and submodule pointers. 
- Update dbis_core, cross-chain-pmm-lps, explorer-monorepo, metamask-integration, pr-workspace/chains
- Omit embedded publish git dirs and empty placeholders from index

Made-with: Cursor
2026-04-12 06:12:20 -07:00

17 KiB
Raw Blame History

Network Configuration Master

Last Updated: 2026-04-03
Status: 🟢 Active Master Reference
Purpose: Single source of truth for all network configurations (UDM Pro edge, Proxmox hosts, NPMplus, port forwarding)

Recent: Option B (RPC via Cloudflare Tunnel) active for 6 RPC hostnames. E2E: 05-network/E2E_CLOUDFLARE_DOMAINS_RUNBOOK.md; Option B: 05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md.

Proxmox cluster (verified 2026-04-02): Five nodes, quorate (pvecm): ml110 192.168.11.10, r630-01 .11, r630-02 .12, r630-03 .13, r630-04 .14 (r630-04.sankofa.nexus). r630-03 / r630-04 remain empty of guests; workload stays on .10–.12. Template vs live (read-only): bash scripts/verify/audit-proxmox-operational-template.sh now SSHs all five IPs by default (config/ip-addresses.conf); ML110 may skip if SSH is down or host repurposed. 2026-04-08: config/proxmox-operational-template.json + ALL_VMIDS_ENDPOINTS.md include Order VMID 10000/10001/10020 (Postgres primary/replica + Redis on r630-01). Package baseline (operator run): all five nodes upgraded toward pve-manager 9.1.7 and kernel 6.17.13-2-pve (apt full-upgrade, one node at a time, reboot where a new kernel was installed). r630-03 and r630-04 had no-subscription apt sources applied first (they previously hit 401 on enterprise.proxmox.com without a subscription). Shared LVM thin storage: data / local-lvm in /etc/pve/storage.cfg include ml110,r630-01,r630-03,r630-04; r630-04 uses dual SSDs in VG pve (~467 GiB thin data) plus Ceph OSDs on four SSDs; r630-03 uses sda3+sdb in VG pve (~1 TiB thin data); r630-03 sdc–sdh are LVM thin pools thin1-r630-03 … thin6-r630-03 (~226 GiB each; provision script in repo). Other workstations: if SSH to r630-04 fails with host key changed, run bash scripts/verify/refresh-proxmox-host-key-r630-04.sh (or ssh-keygen -R 192.168.11.14) after confirming the new key out-of-band.

Network Overview

Primary Network

  • Subnet: 192.168.11.0/24
  • Gateway: 192.168.11.1
  • Netmask: 255.255.255.0
  • VLAN: 11 (MGMT-LAN)
  • DNS Servers: 8.8.8.8, 8.8.4.4

Proxmox Hosts (five-node cluster; ml110 still PVE until WAN-aggregator cutover)

Host (short) Canonical FQDN IP Address Role Status
ml110 ml110.sankofa.nexus 192.168.11.10 Besu validators/RPC (Chain 138); still Proxmox in cluster ✅ Active
r630-01 r630-01.sankofa.nexus 192.168.11.11 Infrastructure, RPC, Services, CCIP Relay ✅ Active
r630-02 r630-02.sankofa.nexus 192.168.11.12 Firefly, NPMplus secondary, MIM4U, Blockscout ✅ Active
r630-03 r630-03.sankofa.nexus 192.168.11.13 Spare (no LXCs/VMs); pve ~1 TiB + thin1-r630-03…thin6-r630-03 on 6×SSD ✅ Active
r630-04 r630-04.sankofa.nexus 192.168.11.14 Spare (no LXCs/VMs); pve thin ~467 GiB + Ceph OSDs ✅ Active

Naming: Proxmox hypervisor management DNS uses short-hostname.sankofa.nexus (same label as the Host column + .sankofa.nexus; see config/ip-addresses.conf PROXMOX_FQDN_*). Use FQDN for SSH, TLS cert SANs, and docs; IPs remain the wire target on VLAN 11. Verify / bootstrap: bash scripts/verify/check-proxmox-mgmt-fqdn.sh (--print-hosts for /etc/hosts); bash scripts/security/ensure-proxmox-ssh-access.sh (--fqdn when DNS exists).

ML110 (192.168.11.10) repurposed: ML110 Gen9 is being converted to OPNsense/pfSense with 8–12 GbE, acting as WAN aggregator between 6–10 Spectrum cable modems and the 2× UDM Pro gateways. After repurpose, .10 is the firewall appliance (not Proxmox). See ML110_OPNSENSE_PFSENSE_WAN_AGGREGATOR.md. Before repurpose: Migrate all containers/VMs off ml110 to r630-01/r630-02 (or other R630s). r630-03/04 are available as migration targets (no guests; local data/local-lvm storage live as of 2026-04-02).

ml110 LVM hygiene (2026-04-02): Stale thin LVs on ml110 named vm-2503-disk-0, vm-6201-disk-0, vm-9000-* were removed after cluster config check: 2503 / 6201 live disks are on r630-01 / r630-02 (/etc/pve/nodes/.../lxc/*.conf); 9000 had no vmlist entry. ml110 pve-guests.service: can stay activating (start) for days if startall wedges (historical cfs-lock / vzstart timeouts). That blocks apt during pve-manager postinst ( systemctl reload-or-restart pvescheduler waits on pve-guests ). Unblock: systemctl list-jobs, then systemctl cancel <jobid> for pve-guests.service and pvescheduler.service, then dpkg --configure -a if needed. After a host reboot, confirm validators 1003 / 1004 are running (pct start if not).

CCIP Relay (r630-01): Host service at /opt/smom-dbis-138/services/relay; relays Chain 138 → Mainnet; uses VMID 2201 (192.168.11.221) for RPC. See 07-ccip/CCIP_RELAY_DEPLOYMENT.md.

Four NPMplus instances (one per public IP): 76.53.10.36, 76.53.10.37, 76.53.10.38, 76.53.10.40. See 04-configuration/NPMPLUS_FOUR_INSTANCES_MASTER.md.

NPMplus #1 (76.53.10.36, LXC VMID 10233): 192.168.11.166 (eth0) and 192.168.11.167 (eth1). Only 192.168.11.167 is used in UDM Pro port forwarding: 76.53.10.36:80 → 192.168.11.167:80, 76.53.10.36:443 → 192.168.11.167:443. Main d-bis.org, explorer, Option B RPC (6 hostnames), MIM4U, etc.

NPMplus #3 (76.53.10.38, LXC VMID 10235): 192.168.11.169 (single NIC). Port forwarding: 76.53.10.38:80/81/443 → 192.168.11.169:80/81/443. Nathan's core-2 RPC, All Mainnet (Alltra), and HYBX nodes and services route here. Designated public IP: 76.53.10.42. Public service names are intended to use the Cloudflare tunnel / proxied CNAME path first, with the direct edge kept as management or fallback. See 04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md.

NPMplus #4 (76.53.10.40, LXC VMID 10236): 192.168.11.170. Port forwarding: 76.53.10.40:80/81/443 → 192.168.11.170:80/81/443; optional 22 → 192.168.11.59 (dev VM). Dev/Codespaces: Gitea, Cursor Remote SSH, Proxmox admin panels (pve.r630-01, pve.r630-02). Dedicated Cloudflare Tunnel. (ml110 repurposed to OPNsense/pfSense WAN aggregator; no longer Proxmox.) See 04-configuration/DEV_CODESPACES_76_53_10_40.md and 04-configuration/DEV_CODESPACES_NEXT_STEPS_CHECKLIST.md.

Dev VM (VMID 5700): 192.168.11.59. Shared Cursor dev environment, four users, Gitea (private GitOps). See 04-configuration/DEV_VM_GITOPS_PLAN.md.

IP reference format: Use IP (VMID) or VMID (IP) consistently. Full registry: 02-architecture/VMID_ALLOCATION_FINAL.md.

Fixed Permanent VMID → IP (Do Not Change)

VMID Hostname IP Address Purpose
2101 besu-rpc-core-1 192.168.11.211 Admin, contract deployment (RPC_CORE_1)
2102 besu-rpc-core-2 192.168.11.212 Nathan RPC, SFValley2 tunnel (RPC_CORE_2)
2201 besu-rpc-public-1 192.168.11.221 Bridge, monitoring, public-facing (RPC_PUBLIC_1)
5000 blockscout-1 192.168.11.140 Explorer (IP_BLOCKSCOUT); web:80, API:4000

These IPs are fixed and permanent. Scripts and configs must use these values. Source: config/ip-addresses.conf.

IP Address Ranges by Service Type

Infrastructure Services (192.168.11.20-39)

  • Range: 192.168.11.20 - 192.168.11.39
  • Purpose: Proxmox infrastructure, monitoring, gateways
  • VMIDs: 100-130, 3500-3501

MIM4U Services (192.168.11.36-37)

  • Range: 192.168.11.36 - 192.168.11.37
  • Purpose: MIM4U web and API services
  • VMIDs: 7810-7811

Sankofa/Phoenix Services (192.168.11.50-59)

  • Range: 192.168.11.50 - 192.168.11.59
  • Purpose: Sankofa and Phoenix services
  • VMIDs: 7800-7803

Machine Learning (192.168.11.60-69)

  • Range: 192.168.11.60 - 192.168.11.69
  • Purpose: ML nodes, Hyperledger services
  • VMIDs: 3000-3003, 6000, 6400

Monitoring (192.168.11.80-89)

  • Range: 192.168.11.80 - 192.168.11.89
  • Purpose: Monitoring and telemetry
  • VMIDs: 5200

RPC Translator Services (192.168.11.110-112)

  • Range: 192.168.11.110 - 192.168.11.112
  • Purpose: RPC translator supporting services
  • VMIDs: 106-108

Besu Validators (192.168.11.100-109)

  • Range: 192.168.11.100 - 192.168.11.109
  • Purpose: Besu validator nodes
  • VMIDs: 1000-1004, 10100-10101

Besu Sentries (192.168.11.150-159, 192.168.11.213-214)

  • Range: 192.168.11.150 - 192.168.11.159, 192.168.11.213 - 192.168.11.214
  • Purpose: Besu sentry nodes (1505-1506 moved from .170/.171 for CCIP range 2026-02-01)
  • VMIDs: 1500-1506

DBIS Services (192.168.11.120-159)

  • Range: 192.168.11.120 - 192.168.11.159
  • Purpose: DBIS Core services
  • VMIDs: 10120, 10130, 10150-10151
  • 10120 dbis-redis: live/static IP 192.168.11.125 (DBIS_REDIS_IP in config/ip-addresses.conf); older docs may still say .120.

RPC Nodes & Phoenix Vault (192.168.11.200-243)

  • Range: 192.168.11.200 - 192.168.11.243 (excl. 192.168.11.170-212 reserved for CCIP interim)
  • Purpose: Besu RPC nodes, Phoenix Vault (8641 at .215 as of 2026-02-01)
  • VMIDs: 2101, 2201, 2301-2308, 2400-2403, 2500-2505 (Besu RPC; 2506-2508 destroyed 2026-02-08), 8640, 8641, 8642

Explorer & Public (192.168.11.140-149)

  • Range: 192.168.11.140 - 192.168.11.149
  • Purpose: Public-facing services
  • VMIDs: 5000

NPMplus & Order (192.168.11.160-170)

  • Range: 192.168.11.160 - 192.168.11.170
  • Purpose: NPMplus proxy (10233: .166/.167), NPMplus secondary (10234: .168), NPMplus Alltra/HYBX (10235: .169), NPMplus Fourth (10236: .170 — dev/Codespaces)
  • VMIDs: 10233-10236

Dev VM (192.168.11.59)

  • VMID: 5700 (dev-vm)
  • Purpose: Shared Cursor dev, four users, Gitea (private GitOps). Access via fourth NPMplus and 76.53.10.40.

CCIP Interim (192.168.11.171-212) - Reserved for CCIP Fleet

  • Range: 192.168.11.171 - 192.168.11.212 (170 = NPMplus Fourth)
  • Purpose: CCIP Ops/Admin, Monitoring, Commit, Execute, RMN
  • Status: ✅ Cleared 2026-02-01 (1505, 1506, 8641 relocated)

Order Services (192.168.11.40-49)

  • Range: 192.168.11.40 - 192.168.11.49
  • Purpose: Order services
  • VMIDs: 10000-10001

VLAN Configuration

Current (Flat Network)

  • VLAN 11: All services (192.168.11.0/24)
  • Status: Active, all services on single VLAN

Planned (Future Migration)

  • VLAN 110: BESU-VAL (10.110.0.0/24) - Validators
  • VLAN 111: BESU-SEN (10.111.0.0/24) - Sentries
  • VLAN 112: BESU-RPC (10.112.0.0/24) - RPC nodes
  • VLAN 120: BLOCKSCOUT (10.120.0.0/24) - Explorer
  • VLAN 160: SANKOFA-SVC (10.160.0.0/22) - Sankofa services
  • VLAN 200-203: Sovereign tenants (10.200.0.0/20 each)

Port Assignments

Standard Besu Ports

  • 8545: HTTP JSON-RPC
  • 8546: WebSocket JSON-RPC
  • 30303: P2P networking (TCP/UDP)
  • 9545: Prometheus metrics

Standard Application Ports

  • 80: HTTP
  • 443: HTTPS
  • 3000: Node.js API
  • 4000: Blockscout API (VMID 5000 @ 192.168.11.140)
  • 3080: Forge Verification Proxy (for Blockscout contract verification)
  • 5432: PostgreSQL
  • 6379: Redis
  • 8006: Proxmox Web UI
  • 8080: Keycloak
  • 8200: Vault
  • 9000: Web3Signer

Public IP Configuration

Block #1 (Spectrum) - 76.53.10.32/28

  • Gateway: 76.53.10.33 (Spectrum CPE; nmap shows 21, 22, 23, 80, 110, 143, 443, 3389 filtered on .33)
  • UDM Pro: 76.53.10.34 (replaced ER605; edge router)
  • Port forwarding: 76.53.10.36:80/443 → 192.168.11.167:80/443 (NPMplus). Origin for public traffic = 76.53.10.36. Verify 76.53.10.36:80 and :443 are open from the internet before using Fastly or direct; see 05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md.
  • NPMplus Alltra/HYBX: 76.53.10.38:80/81/443 → 192.168.11.169:80/81/443 (port forward); 76.53.10.42 designated public IP. Public DNS for Alltra/HYBX services should prefer proxied Cloudflare tunnel CNAMEs rather than direct A records to the designated IP. See 04-configuration/NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md.
  • NPMplus Fourth (dev/Codespaces): 76.53.10.40:80/81/443 → 192.168.11.170; optional 22 → 192.168.11.59. See 04-configuration/UDM_PRO_DEV_CODESPACES_PORT_FORWARD.md.
  • Usable: 76.53.10.35-46 (13 IPs)
  • Status: ✅ Active

Blocks #2-#6

  • Status: To be configured
  • Purpose: Role-based egress NAT pools

Network Access Patterns

Public Internet Access

Primary path (web/api): DNS (Cloudflare) → Fastly or A 76.53.10.36 → UDM Pro (76.53.10.36:80/443) → NPMplus (192.168.11.167) → internal services. Option B (RPC): The 6 RPC HTTP hostnames use Cloudflare Tunnel (CNAME to cfargotunnel.com); cloudflared (e.g. VMID 102) → NPMplus https://192.168.11.167:443. See 05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md. Verify 76.53.10.36:80/443 for direct/Fastly: 05-network/EDGE_PORT_VERIFICATION_RUNBOOK.md.

Internet
  ↓
Cloudflare DNS (optional proxy) → Fastly or 76.53.10.36
  ↓
UDM Pro (76.53.10.36:80/443 port forward)
  ↓
NPMplus (VMID 10233: 192.168.11.167:443)
  ↓
Internal Services

Internal RPC Access

Internal Network (192.168.11.0/24)
  ↓
Direct to RPC Nodes (192.168.11.211-243:8545/8546)

Firewall Rules

P2P Communication

  • Port: 30303 (TCP/UDP)
  • Allowed: Between Besu nodes
  • Status: ✅ Enabled

RPC Access

  • Ports: 8545 (HTTP), 8546 (WebSocket)
  • Allowed IPs: 0.0.0.0/0 (public access)
  • Status: ✅ Enabled

Metrics Scraping

  • Port: 9545
  • Allowed: Monitoring systems
  • Status: ✅ Enabled

DNS Configuration

Internal DNS

  • Primary: 8.8.8.8
  • Secondary: 8.8.4.4
  • Internal Domains: sankofa.nexus (internal)

Public DNS

  • Provider: Cloudflare (retained for all public hostnames)
  • Domains: d-bis.org, mim4u.org, defi-oracle.io, etc.
  • Public path: Web/api: CNAME to Fastly (Option A) or A to 76.53.10.36 (Option C). RPC (Option B): The 6 RPC HTTP hostnames use CNAME to <tunnel-id>.cfargotunnel.com (Proxied); tunnel connector → NPMplus https://192.168.11.167:443. See 05-network/OPTION_B_RPC_VIA_TUNNEL_RUNBOOK.md.

Centralized IP Configuration

Configuration File: config/ip-addresses.conf
Purpose: Centralized IP address definitions for all scripts
Status: ✅ Active - 8+ scripts updated to use centralized config
Automation: scripts/centralize-ip-addresses.sh - Automated IP centralization

Related Documents

Last Updated: 2026-02-06
Maintainer: System Administrator
Update Frequency: On network configuration changes
Current Status: ✅ Up to date - Option B (RPC via tunnel) documented; Blockscout API :4000, Forge Verification Proxy :3080

Reference in New Issue View Git Blame Copy Permalink