# Complete VM Inventory Review & Recommendations **Date:** 2026-01-20 **Review Status:** Comprehensive Analysis Complete **Reviewer:** Automated Inventory Comparison --- ## Executive Summary **Total VMIDs Found:** 77 LXC Containers **Documentation Status:** ⚠️ **OUTDATED** - Significant discrepancies identified **Action Required:** Update ALL_VMIDS_ENDPOINTS.md with current state --- ## Complete VM Inventory by Host ### r630-01 (192.168.11.11) - 47 LXC Containers | VMID | Name | Status | IP Address | Documentation Status | |------|------|--------|------------|---------------------| | 100 | proxmox-mail-gateway | ✅ Running | 192.168.11.32 | ✅ Documented | | 101 | proxmox-datacenter-manager | ✅ Running | 192.168.11.33 | ✅ Documented | | 102 | cloudflared | ✅ Running | 192.168.11.34 | ⚠️ Partially documented | | 103 | omada | ✅ Running | 192.168.11.30 | ✅ Documented | | 104 | gitea | ✅ Running | 192.168.11.31 | ✅ Documented | | 105 | nginxproxymanager | ✅ Running | 192.168.11.26 | ✅ Documented | | 106 | redis-rpc-translator | ✅ Running | 192.168.11.110 | ✅ Documented | | 107 | web3signer-rpc-translator | ✅ Running | 192.168.11.111 | ✅ Documented | | 108 | vault-rpc-translator | ✅ Running | 192.168.11.112 | ✅ Documented | | 130 | monitoring-1 | ✅ Running | 192.168.11.27 | ✅ Documented | | 3000 | ml110 | ✅ Running | 192.168.11.60 | ✅ Documented | | 3001 | ml110 | ✅ Running | 192.168.11.61 | ✅ Documented | | 3002 | ml110 | ✅ Running | 192.168.11.62 | ✅ Documented | | 3003 | ml110 | ✅ Running | 192.168.11.63 | ✅ Documented | | 3500 | oracle-publisher-1 | ✅ Running | 192.168.11.29 | ✅ Documented | | 3501 | ccip-monitor-1 | ✅ Running | 192.168.11.28 | ✅ Documented | | 5200 | cacti-1 | ✅ Running | 192.168.11.80 | ✅ Documented | | 6000 | fabric-1 | ✅ Running | 192.168.11.65 | ✅ Documented | | 6400 | indy-1 | ✅ Running | 192.168.11.64 | ✅ Documented | | **7800** | **sankofa-api-1** | **✅ Running** | **192.168.11.50** | **❌ CRITICAL: Documentation says NOT deployed** | | **7801** | **sankofa-portal-1** | **✅ Running** | **192.168.11.51** | **❌ CRITICAL: Documentation says NOT deployed** | | **7802** | **sankofa-keycloak-1** | **✅ Running** | **192.168.11.52** | **❌ CRITICAL: Documentation says NOT deployed** | | **7803** | **sankofa-postgres-1** | **✅ Running** | **192.168.11.53** | **❌ CRITICAL: Documentation says NOT deployed** | | 8640 | vault-phoenix-1 | ✅ Running | 192.168.11.200 | ❌ Missing from documentation | | 8642 | vault-phoenix-3 | ✅ Running | 192.168.11.202 | ❌ Missing from documentation | | **10000** | **order-postgres-primary** | **✅ Running** | **192.168.11.44** | **❌ Missing from documentation** | | **10001** | **order-postgres-replica** | **✅ Running** | **192.168.11.45** | **❌ Missing from documentation** | | **10020** | **order-redis** | **✅ Running** | **192.168.11.38** | **❌ Missing from documentation** | | **10030** | **order-identity** | **✅ Running** | **192.168.11.40** | **❌ Missing from documentation** | | **10040** | **order-intake** | **✅ Running** | **192.168.11.41** | **❌ Missing from documentation** | | **10050** | **order-finance** | **✅ Running** | **192.168.11.49** | **❌ Missing from documentation** | | **10060** | **order-dataroom** | **✅ Running** | **192.168.11.42** | **❌ Missing from documentation** | | **10070** | **order-legal** | **✅ Running** | **192.168.11.50** | **❌ Missing from documentation** | | **10080** | **order-eresidency** | **✅ Running** | **192.168.11.43** | **❌ Missing from documentation** | | **10090** | **order-portal-public** | **✅ Running** | **192.168.11.36** | **❌ Missing from documentation** | | **10091** | **order-portal-internal** | **✅ Running** | **192.168.11.35** | **❌ Missing from documentation** | | **10092** | **order-mcp-legal** | **✅ Running** | **192.168.11.37** | **❌ Missing from documentation** | | 10100 | dbis-postgres-primary | ✅ Running | 192.168.11.105 | ✅ Documented | | 10101 | dbis-postgres-replica-1 | ✅ Running | 192.168.11.106 | ✅ Documented | | 10120 | dbis-redis | ✅ Running | 192.168.11.125 | ✅ Documented | | 10130 | dbis-frontend | ✅ Running | 192.168.11.130 | ✅ Documented | | 10150 | dbis-api-primary | ✅ Running | 192.168.11.155 | ✅ Documented | | 10151 | dbis-api-secondary | ✅ Running | 192.168.11.156 | ✅ Documented | | **10200** | **order-prometheus** | **✅ Running** | **192.168.11.46** | **❌ Missing from documentation** | | **10201** | **order-grafana** | **✅ Running** | **192.168.11.47** | **❌ Missing from documentation** | | **10202** | **order-opensearch** | **✅ Running** | **192.168.11.48** | **❌ Missing from documentation** | | **10210** | **order-haproxy** | **✅ Running** | **192.168.11.39** | **❌ Missing from documentation** | | **10230** | **order-vault** | **✅ Running** | **192.168.11.51** | **❌ Missing from documentation** | | 10232 | CT10232 | ✅ Running | 192.168.11.52 | ❌ Missing from documentation | | 10233 | npmplus | ✅ Running | 192.168.11.166 | ✅ Documented (IP discrepancy: doc says 192.168.0.166) | ### r630-02 (192.168.11.12) - 7 LXC Containers | VMID | Name | Status | IP Address | Documentation Status | |------|------|--------|------------|---------------------| | 5000 | blockscout-1 | ✅ Running | 192.168.11.140 | ✅ Documented | | 6200 | firefly-1 | ✅ Running | 192.168.11.35 | ✅ Documented | | 6201 | firefly-ali-1 | ✅ Running | 192.168.11.57 | ✅ Documented (but doc says stopped) | | 7810 | mim-web-1 | ✅ Running | 192.168.11.37 | ❌ Missing from documentation | | 7811 | mim-api-1 | ✅ Running | 192.168.11.36 | ✅ Documented | | 8641 | vault-phoenix-2 | ✅ Running | 192.168.11.201 | ❌ Missing from documentation | | 10234 | npmplus-secondary | ✅ Running | - | ❌ Missing from documentation | ### ml110 (192.168.11.10) - 23 LXC Containers | VMID | Name | Status | IP Address | Documentation Status | |------|------|--------|------------|---------------------| | 1000 | besu-validator-1 | ✅ Running | 192.168.11.100 | ✅ Documented | | 1001 | besu-validator-2 | ✅ Running | 192.168.11.101 | ✅ Documented | | 1002 | besu-validator-3 | ✅ Running | 192.168.11.102 | ✅ Documented | | 1003 | besu-validator-4 | ✅ Running | 192.168.11.103 | ✅ Documented | | 1004 | besu-validator-5 | ✅ Running | 192.168.11.104 | ✅ Documented | | 1500 | besu-sentry-1 | ✅ Running | 192.168.11.150 | ✅ Documented | | 1501 | besu-sentry-2 | ✅ Running | 192.168.11.151 | ✅ Documented | | 1502 | besu-sentry-3 | ✅ Running | 192.168.11.152 | ✅ Documented | | 1503 | besu-sentry-4 | ✅ Running | 192.168.11.153 | ✅ Documented | | **1504** | **besu-sentry-ali** | **✅ Running** | **192.168.11.154** | **❌ Documentation says stopped** | | 2101 | besu-rpc-core-1 | ✅ Running | 192.168.11.211 | ✅ Documented | | 2201 | besu-rpc-public-1 | ✅ Running | 192.168.11.221 | ✅ Documented | | 2301 | besu-rpc-private-1 | ⏸️ Stopped | - | ✅ Documented (correctly shown as stopped) | | 2303 | besu-rpc-ali-0x8a | ✅ Running | 192.168.11.233 | ✅ Documented | | 2304 | besu-rpc-ali-0x1 | ✅ Running | 192.168.11.234 | ✅ Documented | | 2305 | besu-rpc-luis-0x8a | ✅ Running | 192.168.11.235 | ✅ Documented | | 2306 | besu-rpc-luis-0x1 | ✅ Running | 192.168.11.236 | ✅ Documented | | 2307 | besu-rpc-putu-0x8a | ✅ Running | 192.168.11.237 | ✅ Documented | | 2308 | besu-rpc-putu-0x1 | ✅ Running | 192.168.11.238 | ✅ Documented | | 2400 | thirdweb-rpc-1 | ✅ Running | 192.168.11.240 | ✅ Documented | | 2401 | besu-rpc-thirdweb-0x8a-1 | ✅ Running | 192.168.11.241 | ✅ Documented | | 2402 | besu-rpc-thirdweb-0x8a-2 | ✅ Running | 192.168.11.242 | ✅ Documented | | 2403 | besu-rpc-thirdweb-0x8a-3 | ✅ Running | 192.168.11.243 | ✅ Documented | --- ## Critical Discrepancies Identified ### 1. ❌ Sankofa Services Status - CRITICAL **Documentation Says:** > "Sankofa services are not currently deployed as separate VMIDs. No VMIDs in the 7800-8999 or 8600-8699 ranges exist on Proxmox hosts." **Actual State:** - ✅ VMID 7800: sankofa-api-1 (192.168.11.50) - Running - ✅ VMID 7801: sankofa-portal-1 (192.168.11.51) - Running - ✅ VMID 7802: sankofa-keycloak-1 (192.168.11.52) - Running - ✅ VMID 7803: sankofa-postgres-1 (192.168.11.53) - Running **Impact:** Documentation is completely incorrect about Sankofa deployment status. **Recommendation:** ⚠️ **URGENT** - Update documentation immediately to reflect deployed Sankofa services. --- ### 2. ❌ Missing "The Order" Services Documentation **22 VMIDs completely missing from documentation:** - Order Infrastructure (10000-10092): 13 services - Order Monitoring (10200-10202): 3 services - Order Support (10210, 10230): 2 services - Phoenix Vault (8640, 8641, 8642): 3 services (partial) - Other services: 1 service (10232, 7810, 10234) **Impact:** No documentation for significant production services. **Recommendation:** ⚠️ **HIGH PRIORITY** - Document all Order services and Phoenix Vault instances. --- ### 3. ⚠️ IP Address Conflicts - **VERIFIED** ⚠️ **CRITICAL** **Verified Conflicts (2026-01-20):** - ⚠️ **192.168.11.50:** VMID 7800 (sankofa-api-1) **AND** VMID 10070 (order-legal) - **CONFLICT VERIFIED** - ⚠️ **192.168.11.51:** VMID 7801 (sankofa-portal-1) **AND** VMID 10230 (order-vault) - **CONFLICT VERIFIED** - ⚠️ **192.168.11.52:** VMID 7802 (sankofa-keycloak-1) **AND** VMID 10232 (CT10232) - **CONFLICT VERIFIED** **Impact:** Network routing conflicts will occur. Services may not be accessible via expected IPs. **Verification:** All conflicts verified from running containers on r630-01. **Recommendation:** ⚠️ **CRITICAL** - Resolve IP conflicts immediately. See `IP_CONFLICTS_VERIFIED.md` for detailed resolution steps. **Recommended Resolution:** - VMID 10070 (order-legal): Reassign to 192.168.11.54 - VMID 10230 (order-vault): Reassign to 192.168.11.55 - VMID 10232 (CT10232): Reassign to 192.168.11.56 --- ### 4. ⚠️ Status Discrepancies **Documentation vs Actual:** - VMID 1504 (besu-sentry-ali): Doc says ⏸️ Stopped, Actually ✅ Running - VMID 6201 (firefly-ali-1): Doc says ⏸️ Stopped, Actually ✅ Running **Recommendation:** ⚠️ Update status in documentation. --- ### 5. ⚠️ NPMplus IP Address Discrepancy **Documentation Says:** > "VMID 10233: 192.168.0.166 (npmplus)" **Actual State:** - VMID 10233: 192.168.11.166 (npmplus) **Impact:** Network documentation incorrect. **Recommendation:** Verify correct IP and update documentation. --- ### 6. ⚠️ Decommissioned VMIDs Still Referenced **Documentation references destroyed VMIDs:** - VMID 2500-2508 (all destroyed) - Still referenced in NPMplus configuration notes **Impact:** Confusion about which RPC endpoints to use. **Recommendation:** Remove all references to decommissioned VMIDs. --- ## Endpoints Documentation Review ### Current Endpoints Documentation Status **Primary Document:** `docs/04-configuration/ALL_VMIDS_ENDPOINTS.md` - **Last Updated:** 2026-01-18 - **Status:** ⚠️ **OUTDATED** - Major discrepancies identified - **Accuracy:** ~60% (missing 22+ VMIDs, incorrect Sankofa status) **Key Issues:** 1. Sankofa services marked as "not deployed" (FALSE) 2. Missing all "The Order" services (10000-10092 range) 3. Missing Phoenix Vault services (8640-8642) 4. Status discrepancies for multiple services 5. IP conflicts not documented --- ## Recommendations & Actions ### Priority 1: CRITICAL - Immediate Actions Required #### 1.1 Resolve IP Address Conflicts ⚠️ **CRITICAL** **Action Required:** ```bash # Identify which services actually use these IPs # VMID 7800 vs VMID 10070 (both claim 192.168.11.50) # VMID 7802 vs VMID 10232 (both claim 192.168.11.52) # VMID 7801 vs VMID 10230 (both claim 192.168.11.51) # Recommended: Reassign Order service IPs # - order-legal (10070): Change to 192.168.11.54 # - order-vault (10230): Change to 192.168.11.55 # - CT10232 (10232): Change to 192.168.11.56 ``` **Priority:** ⚠️ **CRITICAL** - Network conflicts will cause routing issues. --- #### 1.2 Update Sankofa Services Documentation ⚠️ **URGENT** **Action Required:** 1. Update `ALL_VMIDS_ENDPOINTS.md` to reflect deployed Sankofa services: - VMID 7800: sankofa-api-1 (192.168.11.50:4000) - VMID 7801: sankofa-portal-1 (192.168.11.51:3000) - VMID 7802: sankofa-keycloak-1 (192.168.11.52:8080) - VMID 7803: sankofa-postgres-1 (192.168.11.53:5432) 2. Update NPMplus routing notes (Sankofa domains now correctly route to actual services) 3. Remove false statement: "Sankofa services are not currently deployed" **Priority:** ⚠️ **URGENT** - Documentation contradicts reality. --- ### Priority 2: HIGH - Important Documentation Updates #### 2.1 Document "The Order" Services ⚠️ **HIGH PRIORITY** **Action Required:** Add complete documentation for all Order services (VMID 10000-10092): **Order Infrastructure:** - VMID 10000: order-postgres-primary (192.168.11.44:5432) - VMID 10001: order-postgres-replica (192.168.11.45:5432) - VMID 10020: order-redis (192.168.11.38:6379) - VMID 10030: order-identity (192.168.11.40) - VMID 10040: order-intake (192.168.11.41) - VMID 10050: order-finance (192.168.11.49) - VMID 10060: order-dataroom (192.168.11.42) - VMID 10070: order-legal (192.168.11.50 - **CONFLICT**) - VMID 10080: order-eresidency (192.168.11.43) - VMID 10090: order-portal-public (192.168.11.36) - VMID 10091: order-portal-internal (192.168.11.35) - VMID 10092: order-mcp-legal (192.168.11.37) **Order Monitoring:** - VMID 10200: order-prometheus (192.168.11.46:9090) - VMID 10201: order-grafana (192.168.11.47:3000) - VMID 10202: order-opensearch (192.168.11.48:9200) **Order Support:** - VMID 10210: order-haproxy (192.168.11.39:80/443) - VMID 10230: order-vault (192.168.11.51 - **CONFLICT**) **Priority:** ⚠️ **HIGH** - Production services need documentation. --- #### 2.2 Document Phoenix Vault Services **Action Required:** Document all Phoenix Vault instances: - VMID 8640: vault-phoenix-1 (192.168.11.200:8200) - r630-01 - VMID 8641: vault-phoenix-2 (192.168.11.201:8200) - r630-02 - VMID 8642: vault-phoenix-3 (192.168.11.202:8200) - r630-01 **Note:** These appear to be HA Vault cluster nodes. **Priority:** ⚠️ **HIGH** - Critical infrastructure services. --- #### 2.3 Update Service Status Information **Action Required:** Correct status for: - VMID 1504: besu-sentry-ali - Update to ✅ Running - VMID 6201: firefly-ali-1 - Update to ✅ Running **Priority:** ⚠️ **MEDIUM** - Accuracy improvement. --- #### 2.4 Document Missing Services **Action Required:** Add documentation for: - VMID 10232: CT10232 (192.168.11.52 - **CONFLICT**) - VMID 7810: mim-web-1 (192.168.11.37) - r630-02 - VMID 10234: npmplus-secondary - r630-02 (HA setup?) **Priority:** ⚠️ **MEDIUM** - Complete documentation. --- ### Priority 3: MEDIUM - Documentation Improvements #### 3.1 Clean Up Decommissioned VMID References **Action Required:** - Remove all references to VMID 2500-2508 from active documentation - Archive old references to `docs/archive/` - Update NPMplus endpoint references to use new VMIDs **Priority:** ⚠️ **MEDIUM** - Reduce confusion. --- #### 3.2 Update Quick Summary Statistics **Action Required:** Update summary statistics in `ALL_VMIDS_ENDPOINTS.md`: - **Total VMIDs**: 77 (not 50+) - **Running**: 75 (not 45+) - **Stopped**: 2 (not 5) - **Infrastructure Services**: 10+ (may need recount) - **Blockchain Nodes**: 23 (not 22) - **Application Services**: 44+ (not 22) **Priority:** ⚠️ **LOW** - Accuracy improvement. --- #### 3.3 Verify NPMplus IP Address **Action Required:** - Verify actual IP for VMID 10233 (npmplus) - Update documentation if incorrect - Check if 192.168.0.166 is a different interface or routing **Priority:** ⚠️ **LOW** - Verification needed. --- ## Summary of Missing Services ### Completely Missing from Documentation (22+ VMIDs): **The Order Services (13):** - 10000, 10001, 10020, 10030, 10040, 10050, 10060, 10070, 10080, 10090, 10091, 10092 **Order Monitoring (3):** - 10200, 10201, 10202 **Order Support (2):** - 10210, 10230 **Phoenix Vault (3):** - 8640, 8641, 8642 **Other Services (4):** - 10232, 7810, 10234, 102 (cloudflared - partially documented) --- ## Documentation Update Checklist ### Immediate (Priority 1) - [ ] Resolve IP address conflicts (7800/10070, 7802/10232, 7801/10230) - [ ] Update Sankofa services documentation (7800-7803) - [ ] Remove false "Sankofa not deployed" statement ### High Priority (Priority 2) - [ ] Document all Order services (10000-10092) - [ ] Document Phoenix Vault services (8640-8642) - [ ] Update service status (1504, 6201) - [ ] Document missing services (10232, 7810, 10234) ### Medium Priority (Priority 3) - [ ] Remove decommissioned VMID references (2500-2508) - [ ] Update summary statistics - [ ] Verify NPMplus IP address --- ## Endpoints Documentation Status ### Current Documentation Accuracy: ~60% **Strengths:** - ✅ Blockchain nodes well documented - ✅ RPC nodes accurately documented - ✅ Most infrastructure services documented - ✅ DBIS Core services documented **Weaknesses:** - ❌ Sankofa services incorrectly marked as not deployed - ❌ Missing 22+ VMIDs from documentation - ❌ IP conflicts not identified - ❌ Status discrepancies for multiple services - ❌ Decommissioned VMIDs still referenced --- ## Recommendations Summary ### Critical Actions (Do First) 1. **Resolve IP conflicts** - Network routing issues will occur 2. **Update Sankofa documentation** - Completely incorrect status 3. **Document Order services** - Production services need docs ### Important Actions (Do Next) 4. **Document Phoenix Vault** - Critical infrastructure 5. **Update service statuses** - Ensure accuracy 6. **Clean up old references** - Reduce confusion ### Optional Actions (Improvement) 7. **Update statistics** - Better summary information 8. **Verify IP addresses** - Ensure accuracy 9. **Add endpoint details** - For all missing services --- **Last Updated:** 2026-01-20 **Next Review:** After Priority 1 actions completed