Files
proxmox/docs/archive/root-status-reports/VM_INVENTORY_REVIEW_AND_RECOMMENDATIONS.md

455 lines
18 KiB
Markdown
Raw Normal View History

# Complete VM Inventory Review & Recommendations
**Date:** 2026-01-20
**Review Status:** Comprehensive Analysis Complete
**Reviewer:** Automated Inventory Comparison
---
## Executive Summary
**Total VMIDs Found:** 77 LXC Containers
**Documentation Status:** ⚠️ **OUTDATED** - Significant discrepancies identified
**Action Required:** Update ALL_VMIDS_ENDPOINTS.md with current state
---
## Complete VM Inventory by Host
### r630-01 (192.168.11.11) - 47 LXC Containers
| VMID | Name | Status | IP Address | Documentation Status |
|------|------|--------|------------|---------------------|
| 100 | proxmox-mail-gateway | ✅ Running | 192.168.11.32 | ✅ Documented |
| 101 | proxmox-datacenter-manager | ✅ Running | 192.168.11.33 | ✅ Documented |
| 102 | cloudflared | ✅ Running | 192.168.11.34 | ⚠️ Partially documented |
| 103 | omada | ✅ Running | 192.168.11.30 | ✅ Documented |
| 104 | gitea | ✅ Running | 192.168.11.31 | ✅ Documented |
| 105 | nginxproxymanager | ✅ Running | 192.168.11.26 | ✅ Documented |
| 106 | redis-rpc-translator | ✅ Running | 192.168.11.110 | ✅ Documented |
| 107 | web3signer-rpc-translator | ✅ Running | 192.168.11.111 | ✅ Documented |
| 108 | vault-rpc-translator | ✅ Running | 192.168.11.112 | ✅ Documented |
| 130 | monitoring-1 | ✅ Running | 192.168.11.27 | ✅ Documented |
| 3000 | ml110 | ✅ Running | 192.168.11.60 | ✅ Documented |
| 3001 | ml110 | ✅ Running | 192.168.11.61 | ✅ Documented |
| 3002 | ml110 | ✅ Running | 192.168.11.62 | ✅ Documented |
| 3003 | ml110 | ✅ Running | 192.168.11.63 | ✅ Documented |
| 3500 | oracle-publisher-1 | ✅ Running | 192.168.11.29 | ✅ Documented |
| 3501 | ccip-monitor-1 | ✅ Running | 192.168.11.28 | ✅ Documented |
| 5200 | cacti-1 | ✅ Running | 192.168.11.80 | ✅ Documented |
| 6000 | fabric-1 | ✅ Running | 192.168.11.65 | ✅ Documented |
| 6400 | indy-1 | ✅ Running | 192.168.11.64 | ✅ Documented |
| **7800** | **sankofa-api-1** | **✅ Running** | **192.168.11.50** | **❌ CRITICAL: Documentation says NOT deployed** |
| **7801** | **sankofa-portal-1** | **✅ Running** | **192.168.11.51** | **❌ CRITICAL: Documentation says NOT deployed** |
| **7802** | **sankofa-keycloak-1** | **✅ Running** | **192.168.11.52** | **❌ CRITICAL: Documentation says NOT deployed** |
| **7803** | **sankofa-postgres-1** | **✅ Running** | **192.168.11.53** | **❌ CRITICAL: Documentation says NOT deployed** |
| 8640 | vault-phoenix-1 | ✅ Running | 192.168.11.200 | ❌ Missing from documentation |
| 8642 | vault-phoenix-3 | ✅ Running | 192.168.11.202 | ❌ Missing from documentation |
| **10000** | **order-postgres-primary** | **✅ Running** | **192.168.11.44** | **❌ Missing from documentation** |
| **10001** | **order-postgres-replica** | **✅ Running** | **192.168.11.45** | **❌ Missing from documentation** |
| **10020** | **order-redis** | **✅ Running** | **192.168.11.38** | **❌ Missing from documentation** |
| **10030** | **order-identity** | **✅ Running** | **192.168.11.40** | **❌ Missing from documentation** |
| **10040** | **order-intake** | **✅ Running** | **192.168.11.41** | **❌ Missing from documentation** |
| **10050** | **order-finance** | **✅ Running** | **192.168.11.49** | **❌ Missing from documentation** |
| **10060** | **order-dataroom** | **✅ Running** | **192.168.11.42** | **❌ Missing from documentation** |
| **10070** | **order-legal** | **✅ Running** | **192.168.11.50** | **❌ Missing from documentation** |
| **10080** | **order-eresidency** | **✅ Running** | **192.168.11.43** | **❌ Missing from documentation** |
| **10090** | **order-portal-public** | **✅ Running** | **192.168.11.36** | **❌ Missing from documentation** |
| **10091** | **order-portal-internal** | **✅ Running** | **192.168.11.35** | **❌ Missing from documentation** |
| **10092** | **order-mcp-legal** | **✅ Running** | **192.168.11.37** | **❌ Missing from documentation** |
| 10100 | dbis-postgres-primary | ✅ Running | 192.168.11.105 | ✅ Documented |
| 10101 | dbis-postgres-replica-1 | ✅ Running | 192.168.11.106 | ✅ Documented |
| 10120 | dbis-redis | ✅ Running | 192.168.11.125 | ✅ Documented |
| 10130 | dbis-frontend | ✅ Running | 192.168.11.130 | ✅ Documented |
| 10150 | dbis-api-primary | ✅ Running | 192.168.11.155 | ✅ Documented |
| 10151 | dbis-api-secondary | ✅ Running | 192.168.11.156 | ✅ Documented |
| **10200** | **order-prometheus** | **✅ Running** | **192.168.11.46** | **❌ Missing from documentation** |
| **10201** | **order-grafana** | **✅ Running** | **192.168.11.47** | **❌ Missing from documentation** |
| **10202** | **order-opensearch** | **✅ Running** | **192.168.11.48** | **❌ Missing from documentation** |
| **10210** | **order-haproxy** | **✅ Running** | **192.168.11.39** | **❌ Missing from documentation** |
| **10230** | **order-vault** | **✅ Running** | **192.168.11.51** | **❌ Missing from documentation** |
| 10232 | CT10232 | ✅ Running | 192.168.11.52 | ❌ Missing from documentation |
| 10233 | npmplus | ✅ Running | 192.168.11.166 | ✅ Documented (IP discrepancy: doc says 192.168.0.166) |
### r630-02 (192.168.11.12) - 7 LXC Containers
| VMID | Name | Status | IP Address | Documentation Status |
|------|------|--------|------------|---------------------|
| 5000 | blockscout-1 | ✅ Running | 192.168.11.140 | ✅ Documented |
| 6200 | firefly-1 | ✅ Running | 192.168.11.35 | ✅ Documented |
| 6201 | firefly-ali-1 | ✅ Running | 192.168.11.57 | ✅ Documented (but doc says stopped) |
| 7810 | mim-web-1 | ✅ Running | 192.168.11.37 | ❌ Missing from documentation |
| 7811 | mim-api-1 | ✅ Running | 192.168.11.36 | ✅ Documented |
| 8641 | vault-phoenix-2 | ✅ Running | 192.168.11.201 | ❌ Missing from documentation |
| 10234 | npmplus-secondary | ✅ Running | - | ❌ Missing from documentation |
### ml110 (192.168.11.10) - 23 LXC Containers
| VMID | Name | Status | IP Address | Documentation Status |
|------|------|--------|------------|---------------------|
| 1000 | besu-validator-1 | ✅ Running | 192.168.11.100 | ✅ Documented |
| 1001 | besu-validator-2 | ✅ Running | 192.168.11.101 | ✅ Documented |
| 1002 | besu-validator-3 | ✅ Running | 192.168.11.102 | ✅ Documented |
| 1003 | besu-validator-4 | ✅ Running | 192.168.11.103 | ✅ Documented |
| 1004 | besu-validator-5 | ✅ Running | 192.168.11.104 | ✅ Documented |
| 1500 | besu-sentry-1 | ✅ Running | 192.168.11.150 | ✅ Documented |
| 1501 | besu-sentry-2 | ✅ Running | 192.168.11.151 | ✅ Documented |
| 1502 | besu-sentry-3 | ✅ Running | 192.168.11.152 | ✅ Documented |
| 1503 | besu-sentry-4 | ✅ Running | 192.168.11.153 | ✅ Documented |
| **1504** | **besu-sentry-ali** | **✅ Running** | **192.168.11.154** | **❌ Documentation says stopped** |
| 2101 | besu-rpc-core-1 | ✅ Running | 192.168.11.211 | ✅ Documented |
| 2201 | besu-rpc-public-1 | ✅ Running | 192.168.11.221 | ✅ Documented |
| 2301 | besu-rpc-private-1 | ⏸️ Stopped | - | ✅ Documented (correctly shown as stopped) |
| 2303 | besu-rpc-ali-0x8a | ✅ Running | 192.168.11.233 | ✅ Documented |
| 2304 | besu-rpc-ali-0x1 | ✅ Running | 192.168.11.234 | ✅ Documented |
| 2305 | besu-rpc-luis-0x8a | ✅ Running | 192.168.11.235 | ✅ Documented |
| 2306 | besu-rpc-luis-0x1 | ✅ Running | 192.168.11.236 | ✅ Documented |
| 2307 | besu-rpc-putu-0x8a | ✅ Running | 192.168.11.237 | ✅ Documented |
| 2308 | besu-rpc-putu-0x1 | ✅ Running | 192.168.11.238 | ✅ Documented |
| 2400 | thirdweb-rpc-1 | ✅ Running | 192.168.11.240 | ✅ Documented |
| 2401 | besu-rpc-thirdweb-0x8a-1 | ✅ Running | 192.168.11.241 | ✅ Documented |
| 2402 | besu-rpc-thirdweb-0x8a-2 | ✅ Running | 192.168.11.242 | ✅ Documented |
| 2403 | besu-rpc-thirdweb-0x8a-3 | ✅ Running | 192.168.11.243 | ✅ Documented |
---
## Critical Discrepancies Identified
### 1. ❌ Sankofa Services Status - CRITICAL
**Documentation Says:**
> "Sankofa services are not currently deployed as separate VMIDs. No VMIDs in the 7800-8999 or 8600-8699 ranges exist on Proxmox hosts."
**Actual State:**
- ✅ VMID 7800: sankofa-api-1 (192.168.11.50) - Running
- ✅ VMID 7801: sankofa-portal-1 (192.168.11.51) - Running
- ✅ VMID 7802: sankofa-keycloak-1 (192.168.11.52) - Running
- ✅ VMID 7803: sankofa-postgres-1 (192.168.11.53) - Running
**Impact:** Documentation is completely incorrect about Sankofa deployment status.
**Recommendation:** ⚠️ **URGENT** - Update documentation immediately to reflect deployed Sankofa services.
---
### 2. ❌ Missing "The Order" Services Documentation
**22 VMIDs completely missing from documentation:**
- Order Infrastructure (10000-10092): 13 services
- Order Monitoring (10200-10202): 3 services
- Order Support (10210, 10230): 2 services
- Phoenix Vault (8640, 8641, 8642): 3 services (partial)
- Other services: 1 service (10232, 7810, 10234)
**Impact:** No documentation for significant production services.
**Recommendation:** ⚠️ **HIGH PRIORITY** - Document all Order services and Phoenix Vault instances.
---
### 3. ⚠️ IP Address Conflicts - **VERIFIED** ⚠️ **CRITICAL**
**Verified Conflicts (2026-01-20):**
- ⚠️ **192.168.11.50:** VMID 7800 (sankofa-api-1) **AND** VMID 10070 (order-legal) - **CONFLICT VERIFIED**
- ⚠️ **192.168.11.51:** VMID 7801 (sankofa-portal-1) **AND** VMID 10230 (order-vault) - **CONFLICT VERIFIED**
- ⚠️ **192.168.11.52:** VMID 7802 (sankofa-keycloak-1) **AND** VMID 10232 (CT10232) - **CONFLICT VERIFIED**
**Impact:** Network routing conflicts will occur. Services may not be accessible via expected IPs.
**Verification:** All conflicts verified from running containers on r630-01.
**Recommendation:** ⚠️ **CRITICAL** - Resolve IP conflicts immediately. See `IP_CONFLICTS_VERIFIED.md` for detailed resolution steps.
**Recommended Resolution:**
- VMID 10070 (order-legal): Reassign to 192.168.11.54
- VMID 10230 (order-vault): Reassign to 192.168.11.55
- VMID 10232 (CT10232): Reassign to 192.168.11.56
---
### 4. ⚠️ Status Discrepancies
**Documentation vs Actual:**
- VMID 1504 (besu-sentry-ali): Doc says ⏸️ Stopped, Actually ✅ Running
- VMID 6201 (firefly-ali-1): Doc says ⏸️ Stopped, Actually ✅ Running
**Recommendation:** ⚠️ Update status in documentation.
---
### 5. ⚠️ NPMplus IP Address Discrepancy
**Documentation Says:**
> "VMID 10233: 192.168.0.166 (npmplus)"
**Actual State:**
- VMID 10233: 192.168.11.166 (npmplus)
**Impact:** Network documentation incorrect.
**Recommendation:** Verify correct IP and update documentation.
---
### 6. ⚠️ Decommissioned VMIDs Still Referenced
**Documentation references destroyed VMIDs:**
- VMID 2500-2508 (all destroyed)
- Still referenced in NPMplus configuration notes
**Impact:** Confusion about which RPC endpoints to use.
**Recommendation:** Remove all references to decommissioned VMIDs.
---
## Endpoints Documentation Review
### Current Endpoints Documentation Status
**Primary Document:** `docs/04-configuration/ALL_VMIDS_ENDPOINTS.md`
- **Last Updated:** 2026-01-18
- **Status:** ⚠️ **OUTDATED** - Major discrepancies identified
- **Accuracy:** ~60% (missing 22+ VMIDs, incorrect Sankofa status)
**Key Issues:**
1. Sankofa services marked as "not deployed" (FALSE)
2. Missing all "The Order" services (10000-10092 range)
3. Missing Phoenix Vault services (8640-8642)
4. Status discrepancies for multiple services
5. IP conflicts not documented
---
## Recommendations & Actions
### Priority 1: CRITICAL - Immediate Actions Required
#### 1.1 Resolve IP Address Conflicts ⚠️ **CRITICAL**
**Action Required:**
```bash
# Identify which services actually use these IPs
# VMID 7800 vs VMID 10070 (both claim 192.168.11.50)
# VMID 7802 vs VMID 10232 (both claim 192.168.11.52)
# VMID 7801 vs VMID 10230 (both claim 192.168.11.51)
# Recommended: Reassign Order service IPs
# - order-legal (10070): Change to 192.168.11.54
# - order-vault (10230): Change to 192.168.11.55
# - CT10232 (10232): Change to 192.168.11.56
```
**Priority:** ⚠️ **CRITICAL** - Network conflicts will cause routing issues.
---
#### 1.2 Update Sankofa Services Documentation ⚠️ **URGENT**
**Action Required:**
1. Update `ALL_VMIDS_ENDPOINTS.md` to reflect deployed Sankofa services:
- VMID 7800: sankofa-api-1 (192.168.11.50:4000)
- VMID 7801: sankofa-portal-1 (192.168.11.51:3000)
- VMID 7802: sankofa-keycloak-1 (192.168.11.52:8080)
- VMID 7803: sankofa-postgres-1 (192.168.11.53:5432)
2. Update NPMplus routing notes (Sankofa domains now correctly route to actual services)
3. Remove false statement: "Sankofa services are not currently deployed"
**Priority:** ⚠️ **URGENT** - Documentation contradicts reality.
---
### Priority 2: HIGH - Important Documentation Updates
#### 2.1 Document "The Order" Services ⚠️ **HIGH PRIORITY**
**Action Required:**
Add complete documentation for all Order services (VMID 10000-10092):
**Order Infrastructure:**
- VMID 10000: order-postgres-primary (192.168.11.44:5432)
- VMID 10001: order-postgres-replica (192.168.11.45:5432)
- VMID 10020: order-redis (192.168.11.38:6379)
- VMID 10030: order-identity (192.168.11.40)
- VMID 10040: order-intake (192.168.11.41)
- VMID 10050: order-finance (192.168.11.49)
- VMID 10060: order-dataroom (192.168.11.42)
- VMID 10070: order-legal (192.168.11.50 - **CONFLICT**)
- VMID 10080: order-eresidency (192.168.11.43)
- VMID 10090: order-portal-public (192.168.11.36)
- VMID 10091: order-portal-internal (192.168.11.35)
- VMID 10092: order-mcp-legal (192.168.11.37)
**Order Monitoring:**
- VMID 10200: order-prometheus (192.168.11.46:9090)
- VMID 10201: order-grafana (192.168.11.47:3000)
- VMID 10202: order-opensearch (192.168.11.48:9200)
**Order Support:**
- VMID 10210: order-haproxy (192.168.11.39:80/443)
- VMID 10230: order-vault (192.168.11.51 - **CONFLICT**)
**Priority:** ⚠️ **HIGH** - Production services need documentation.
---
#### 2.2 Document Phoenix Vault Services
**Action Required:**
Document all Phoenix Vault instances:
- VMID 8640: vault-phoenix-1 (192.168.11.200:8200) - r630-01
- VMID 8641: vault-phoenix-2 (192.168.11.201:8200) - r630-02
- VMID 8642: vault-phoenix-3 (192.168.11.202:8200) - r630-01
**Note:** These appear to be HA Vault cluster nodes.
**Priority:** ⚠️ **HIGH** - Critical infrastructure services.
---
#### 2.3 Update Service Status Information
**Action Required:**
Correct status for:
- VMID 1504: besu-sentry-ali - Update to ✅ Running
- VMID 6201: firefly-ali-1 - Update to ✅ Running
**Priority:** ⚠️ **MEDIUM** - Accuracy improvement.
---
#### 2.4 Document Missing Services
**Action Required:**
Add documentation for:
- VMID 10232: CT10232 (192.168.11.52 - **CONFLICT**)
- VMID 7810: mim-web-1 (192.168.11.37) - r630-02
- VMID 10234: npmplus-secondary - r630-02 (HA setup?)
**Priority:** ⚠️ **MEDIUM** - Complete documentation.
---
### Priority 3: MEDIUM - Documentation Improvements
#### 3.1 Clean Up Decommissioned VMID References
**Action Required:**
- Remove all references to VMID 2500-2508 from active documentation
- Archive old references to `docs/archive/`
- Update NPMplus endpoint references to use new VMIDs
**Priority:** ⚠️ **MEDIUM** - Reduce confusion.
---
#### 3.2 Update Quick Summary Statistics
**Action Required:**
Update summary statistics in `ALL_VMIDS_ENDPOINTS.md`:
- **Total VMIDs**: 77 (not 50+)
- **Running**: 75 (not 45+)
- **Stopped**: 2 (not 5)
- **Infrastructure Services**: 10+ (may need recount)
- **Blockchain Nodes**: 23 (not 22)
- **Application Services**: 44+ (not 22)
**Priority:** ⚠️ **LOW** - Accuracy improvement.
---
#### 3.3 Verify NPMplus IP Address
**Action Required:**
- Verify actual IP for VMID 10233 (npmplus)
- Update documentation if incorrect
- Check if 192.168.0.166 is a different interface or routing
**Priority:** ⚠️ **LOW** - Verification needed.
---
## Summary of Missing Services
### Completely Missing from Documentation (22+ VMIDs):
**The Order Services (13):**
- 10000, 10001, 10020, 10030, 10040, 10050, 10060, 10070, 10080, 10090, 10091, 10092
**Order Monitoring (3):**
- 10200, 10201, 10202
**Order Support (2):**
- 10210, 10230
**Phoenix Vault (3):**
- 8640, 8641, 8642
**Other Services (4):**
- 10232, 7810, 10234, 102 (cloudflared - partially documented)
---
## Documentation Update Checklist
### Immediate (Priority 1)
- [ ] Resolve IP address conflicts (7800/10070, 7802/10232, 7801/10230)
- [ ] Update Sankofa services documentation (7800-7803)
- [ ] Remove false "Sankofa not deployed" statement
### High Priority (Priority 2)
- [ ] Document all Order services (10000-10092)
- [ ] Document Phoenix Vault services (8640-8642)
- [ ] Update service status (1504, 6201)
- [ ] Document missing services (10232, 7810, 10234)
### Medium Priority (Priority 3)
- [ ] Remove decommissioned VMID references (2500-2508)
- [ ] Update summary statistics
- [ ] Verify NPMplus IP address
---
## Endpoints Documentation Status
### Current Documentation Accuracy: ~60%
**Strengths:**
- ✅ Blockchain nodes well documented
- ✅ RPC nodes accurately documented
- ✅ Most infrastructure services documented
- ✅ DBIS Core services documented
**Weaknesses:**
- ❌ Sankofa services incorrectly marked as not deployed
- ❌ Missing 22+ VMIDs from documentation
- ❌ IP conflicts not identified
- ❌ Status discrepancies for multiple services
- ❌ Decommissioned VMIDs still referenced
---
## Recommendations Summary
### Critical Actions (Do First)
1. **Resolve IP conflicts** - Network routing issues will occur
2. **Update Sankofa documentation** - Completely incorrect status
3. **Document Order services** - Production services need docs
### Important Actions (Do Next)
4. **Document Phoenix Vault** - Critical infrastructure
5. **Update service statuses** - Ensure accuracy
6. **Clean up old references** - Reduce confusion
### Optional Actions (Improvement)
7. **Update statistics** - Better summary information
8. **Verify IP addresses** - Ensure accuracy
9. **Add endpoint details** - For all missing services
---
**Last Updated:** 2026-01-20
**Next Review:** After Priority 1 actions completed