# Proxmox Cluster Architecture **Last Updated:** 2025-01-20 **Document Version:** 1.0 **Status:** Active Documentation --- ## Overview This document describes the Proxmox cluster architecture, including node configuration, storage setup, network bridges, and VM/container distribution. --- ## Cluster Architecture Diagram ```mermaid graph TB Cluster[Proxmox Cluster
Name: h] ML110[ML110 Management Node
192.168.11.10
6 cores, 125GB RAM] R6301[R630-01
192.168.11.11
32 cores, 503GB RAM] R6302[R630-02
192.168.11.12
32 cores, 503GB RAM] R6303[R630-03
192.168.11.13
32 cores, 512GB RAM] R6304[R630-04
192.168.11.14
32 cores, 512GB RAM] Cluster --> ML110 Cluster --> R6301 Cluster --> R6302 Cluster --> R6303 Cluster --> R6304 ML110 --> Storage1[local: 94GB
local-lvm: 813GB] R6301 --> Storage2[local: 536GB
local-lvm: Available] R6302 --> Storage3[local: Available
local-lvm: Available] R6303 --> Storage4[Storage: Available] R6304 --> Storage5[Storage: Available] ML110 --> Bridge1[vmbr0
VLAN-aware] R6301 --> Bridge2[vmbr0
VLAN-aware] R6302 --> Bridge3[vmbr0
VLAN-aware] R6303 --> Bridge4[vmbr0
VLAN-aware] R6304 --> Bridge5[vmbr0
VLAN-aware] ``` --- ## Cluster Nodes ### Node Summary | Hostname | IP Address | CPU | RAM | Storage | VMs/Containers | Status | |----------|------------|-----|-----|---------|----------------|--------| | ml110 | 192.168.11.10 | 6 cores @ 1.60GHz | 125GB | local (94GB), local-lvm (813GB) | 34 | ✅ Active | | r630-01 | 192.168.11.11 | 32 cores @ 2.40GHz | 503GB | local (536GB), local-lvm (available) | 0 | ✅ Active | | r630-02 | 192.168.11.12 | 32 cores @ 2.40GHz | 503GB | local (available), local-lvm (available) | 0 | ✅ Active | | r630-03 | 192.168.11.13 | 32 cores | 512GB | Available | 0 | ✅ Active | | r630-04 | 192.168.11.14 | 32 cores | 512GB | Available | 0 | ✅ Active | --- ## Storage Configuration ### Storage Types **local (Directory Storage):** - Type: Directory-based storage - Used for: ISO images, container templates, backups - Location: `/var/lib/vz` **local-lvm (LVM Thin Storage):** - Type: LVM thin provisioning - Used for: VM/container disk images - Benefits: Thin provisioning, snapshots, efficient space usage ### Storage by Node **ml110:** - `local`: 94GB total, 7.4GB used (7.87%) - `local-lvm`: 813GB total, 214GB used (26.29%) - Status: ✅ Active and operational **r630-01:** - `local`: 536GB total, 0% used - `local-lvm`: Available (needs activation) - Status: ⏳ Storage available, ready for use **r630-02:** - `local`: Available - `local-lvm`: Available (needs activation) - Status: ⏳ Storage available, ready for use **r630-03/r630-04:** - Storage: Available - Status: ⏳ Ready for configuration --- ## Network Configuration ### Network Bridge (vmbr0) **All nodes use VLAN-aware bridge:** ```bash # Bridge configuration (all nodes) auto vmbr0 iface vmbr0 inet static address 192.168.11./24 gateway 192.168.11.1 bridge-ports bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 11 110 111 112 120 121 130 132 133 134 140 141 150 160 200 201 202 203 ``` **Bridge Features:** - **VLAN-aware:** Supports multiple VLANs on single bridge - **Native VLAN:** 11 (MGMT-LAN) - **Tagged VLANs:** All service VLANs (110-203) - **802.1Q Trunking:** Enabled for VLAN support --- ## VM/Container Distribution ### Current Distribution **ml110 (192.168.11.10):** - **Total:** 34 containers/VMs - **Services:** All current services running here - **Breakdown:** - Besu validators: 5 (VMIDs 1000-1004) - Besu sentries: 4 (VMIDs 1500-1503) - Besu RPC: 3+ (VMIDs 2500-2502+) - Blockscout: 1 (VMID 5000) - DBIS services: Multiple - Other services: Various **r630-01, r630-02, r630-03, r630-04:** - **Total:** 0 containers/VMs - **Status:** Ready for VM migration/deployment --- ## High Availability ### Current Setup - **Cluster Name:** "h" - **HA Mode:** Active/Standby (manual) - **Quorum:** 3+ nodes required for quorum - **Storage:** Local storage (not shared) ### HA Considerations **Current Limitations:** - No shared storage (each node has local storage) - Manual VM migration required - No automatic failover **Future Enhancements:** - Consider shared storage (NFS, Ceph, etc.) for true HA - Implement automatic VM migration - Configure HA groups for critical services --- ## Resource Allocation ### CPU Resources | Node | CPU Cores | CPU Usage | Available | |------|-----------|-----------|-----------| | ml110 | 6 @ 1.60GHz | High | Limited | | r630-01 | 32 @ 2.40GHz | Low | Excellent | | r630-02 | 32 @ 2.40GHz | Low | Excellent | | r630-03 | 32 cores | Low | Excellent | | r630-04 | 32 cores | Low | Excellent | ### Memory Resources | Node | Total RAM | Used | Available | Usage % | |------|-----------|------|-----------|---------| | ml110 | 125GB | 94GB | 31GB | 75% ⚠️ | | r630-01 | 503GB | ~5GB | ~498GB | 1% ✅ | | r630-02 | 503GB | ~5GB | ~498GB | 1% ✅ | | r630-03 | 512GB | Low | High | Low ✅ | | r630-04 | 512GB | Low | High | Low ✅ | --- ## Storage Recommendations ### For R630 Nodes **Boot Drives (2×600GB):** - **Recommended:** ZFS mirror or hardware RAID1 - **Purpose:** Proxmox OS and boot files - **Benefits:** Redundancy, data integrity **Data SSDs (6×250GB):** - **Option 1:** ZFS striped mirrors (3 pairs) - Capacity: ~750GB usable - Performance: High - Redundancy: Good - **Option 2:** ZFS RAIDZ1 (5 drives + 1 parity) - Capacity: ~1.25TB usable - Performance: Good - Redundancy: Single drive failure tolerance - **Option 3:** ZFS RAIDZ2 (4 drives + 2 parity) - Capacity: ~1TB usable - Performance: Good - Redundancy: Dual drive failure tolerance --- ## Network Recommendations ### VLAN Configuration **All Proxmox hosts should:** - Use VLAN-aware bridge (vmbr0) - Support all 19 VLANs - Maintain native VLAN 11 for management - Enable 802.1Q trunking on physical interfaces ### Network Performance - **Link Speed:** Ensure 1Gbps or higher for trunk ports - **Jumbo Frames:** Consider enabling if supported - **Bonding:** Consider link aggregation for redundancy --- ## Related Documentation - **[NETWORK_ARCHITECTURE.md](NETWORK_ARCHITECTURE.md)** ⭐⭐⭐ - Network architecture with VLAN plan - **[PHYSICAL_HARDWARE_INVENTORY.md](PHYSICAL_HARDWARE_INVENTORY.md)** ⭐⭐⭐ - Physical hardware inventory - **[PROXMOX_COMPREHENSIVE_REVIEW.md](PROXMOX_COMPREHENSIVE_REVIEW.md)** ⭐⭐ - Comprehensive Proxmox review - **[ORCHESTRATION_DEPLOYMENT_GUIDE.md](ORCHESTRATION_DEPLOYMENT_GUIDE.md)** ⭐⭐⭐ - Deployment orchestration --- **Last Updated:** 2025-01-20 **Document Version:** 1.0 **Review Cycle:** Quarterly