# Deployment Comparison: LXC Containers vs Single VM This document compares the two deployment options for SMOM-DBIS-138 Besu network nodes. ## Overview ### Deployment 1: Individual LXC Containers (VMID 1000-2502) Each Besu node runs in its own LXC container on Proxmox VE. This is the **production-ready** deployment method. ### Deployment 2: Single VM with Docker (VMID 9000) All Besu nodes run as Docker containers within a single VM. This is the **temporary/testing** deployment method. --- ## Deployment 1: LXC Containers (1000-2502) ### Architecture ``` ┌──────────────────────────────────────────────────────────────┐ │ Proxmox VE Host │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ LXC 1000 │ │ LXC 1001 │ │ LXC 1002 │ │ │ │ validator-1 │ │ validator-2 │ │ validator-3 │ │ │ │ 8GB, 4 cores │ │ 8GB, 4 cores │ │ 8GB, 4 cores │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ │ │ │ LXC 1003 │ │ LXC 1004 │ │ │ │ validator-4 │ │ validator-5 │ │ │ │ 8GB, 4 cores │ │ 8GB, 4 cores │ │ │ └──────────────┘ └──────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ LXC 1500 │ │ LXC 1501 │ │ LXC 1502 │ │ │ │ sentry-1 │ │ sentry-2 │ │ sentry-3 │ │ │ │ 4GB, 2 cores │ │ 4GB, 2 cores │ │ 4GB, 2 cores │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ ┌──────────────┐ │ │ │ LXC 1503 │ │ │ │ sentry-4 │ │ │ │ 4GB, 2 cores │ │ │ └──────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ LXC 2500 │ │ LXC 2501 │ │ LXC 2502 │ │ │ │ rpc-1 │ │ rpc-2 │ │ rpc-3 │ │ │ │ 16GB, 4 cores│ │ 16GB, 4 cores│ │ 16GB, 4 cores│ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ └──────────────────────────────────────────────────────────────┘ ``` ### VMID Allocation | Category | VMID Range | Count | Hostnames | IP Range | |----------|-----------|-------|-----------|----------| | **Validators** | 1000-1004 | 5 | besu-validator-1 through besu-validator-5 | 192.168.11.100-104 | | **Sentries** | 1500-1503 | 4 | besu-sentry-1 through besu-sentry-4 | 192.168.11.150-153 | | **RPC Nodes** | 2500-2502 | 3 | besu-rpc-1 through besu-rpc-3 | 192.168.11.250-252 | **Total Containers**: 12 ### Resource Allocation #### Validators (1000-1004) - **Memory**: 8GB each (40GB total) - **CPU Cores**: 4 each (20 cores total) - **Disk**: 100GB each (500GB total) - **IP Addresses**: 192.168.11.100-104 - **Purpose**: QBFT consensus nodes #### Sentries (1500-1503) - **Memory**: 4GB each (16GB total) - **CPU Cores**: 2 each (8 cores total) - **Disk**: 100GB each (400GB total) - **IP Addresses**: 192.168.11.150-153 - **Purpose**: P2P relay and protection nodes #### RPC Nodes (2500-2502) - **Memory**: 16GB each (48GB total) - **CPU Cores**: 4 each (12 cores total) - **Disk**: 200GB each (600GB total) - **IP Addresses**: 192.168.11.250-252 - **Purpose**: Public RPC endpoints **Total Resources**: - **Total Memory**: 104GB - **Total CPU Cores**: 40 cores - **Total Disk**: 1.5TB ### Deployment Script ```bash cd /opt/smom-dbis-138-proxmox sudo ./scripts/deployment/deploy-besu-nodes.sh ``` ### Configuration Files Each container has: - `/etc/besu/config-{type}.toml` - Besu configuration - `/opt/besu/data/` - Blockchain data directory - `/opt/besu/keys/` - Validator keys (validators only) - Systemd service: `besu-validator`, `besu-sentry`, or `besu-rpc` ### Service Management ```bash # Start/stop individual container services pct exec 1000 -- systemctl start besu-validator pct exec 1500 -- systemctl start besu-sentry pct exec 2500 -- systemctl start besu-rpc # Check status pct exec 1000 -- systemctl status besu-validator # View logs pct exec 1000 -- journalctl -u besu-validator -f ``` ### Advantages ✅ **Resource Isolation**: Each node has dedicated resources ✅ **Individual Scaling**: Scale nodes independently ✅ **Production Ready**: Suitable for production deployments ✅ **Security Isolation**: Better security boundaries ✅ **Independent Management**: Start/stop nodes individually ✅ **Better Monitoring**: Monitor each node separately ✅ **Fault Isolation**: Failure of one node doesn't affect others ### Disadvantages ❌ **Longer Deployment Time**: ~30-45 minutes ❌ **More Complex Setup**: Requires configuration per container ❌ **Higher Resource Usage**: More overhead per node ❌ **More Management Overhead**: Manage 12 containers --- ## Deployment 2: Single VM with Docker (VMID 9000) ### Architecture ``` ┌──────────────────────────────────────────────────────────────┐ │ Proxmox VE Host │ │ │ │ ┌────────────────────────────────────────────────────────┐ │ │ │ VM 9000 │ │ │ │ IP: 192.168.11.90 │ │ │ │ Memory: 32GB, CPU: 8 cores, Disk: 500GB │ │ │ │ │ │ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ │ │ Docker Network │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ │ │validator│ │validator│ │validator│ │ │ │ │ │ │ │ -1 │ │ -2 │ │ -3 │ │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┐ ┌─────────┐ │ │ │ │ │ │ │validator│ │validator│ │ │ │ │ │ │ │ -4 │ │ -5 │ │ │ │ │ │ │ └─────────┘ └─────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ │ │ sentry │ │ sentry │ │ sentry │ │ │ │ │ │ │ │ -1 │ │ -2 │ │ -3 │ │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┐ │ │ │ │ │ │ │ sentry │ │ │ │ │ │ │ │ -4 │ │ │ │ │ │ │ └─────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ │ │ rpc-1 │ │ rpc-2 │ │ rpc-3 │ │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ │ └──────────────────────────────────────────────────┘ │ │ │ └────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘ ``` ### VMID Allocation | Category | VMID | Description | IP Address | |----------|------|-------------|------------| | **Temporary VM** | 9000 | All 12 Besu nodes in Docker | 192.168.11.90 | **Total VMs**: 1 **Total Docker Containers**: 12 (5 validators + 4 sentries + 3 RPC) ### Resource Allocation - **VM Memory**: 32GB (shared by all containers) - **VM CPU Cores**: 8 cores (shared by all containers) - **VM Disk**: 500GB (shared by all containers) - **VM IP Address**: 192.168.11.90 - **OS**: Ubuntu 22.04 #### Container Resource Allocation (within VM) ##### Validators (5 containers) - **Memory**: 4GB each (20GB total) - **P2P Ports**: 30303-30307 - **Metrics Ports**: 9545-9549 ##### Sentries (4 containers) - **Memory**: 2GB each (8GB total) - **P2P Ports**: 30308-30311 - **Metrics Ports**: 9550-9553 ##### RPC Nodes (3 containers) - **Memory**: 8GB each (24GB total) - **HTTP RPC Ports**: 8545, 8547, 8549 - **WS RPC Ports**: 8546, 8548, 8550 - **P2P Ports**: 30312-30314 - **Metrics Ports**: 9554-9556 **Total Container Memory**: 52GB (with some headroom in 32GB VM) ### Deployment Script ```bash cd /opt/smom-dbis-138-proxmox sudo ./scripts/deployment/deploy-besu-temp-vm-complete.sh /opt/smom-dbis-138 ``` Or step-by-step: ```bash # Step 1: Create VM sudo ./scripts/deployment/deploy-besu-temp-vm.sh # Step 2: SSH into VM and setup ssh root@192.168.11.90 cd /opt/smom-dbis-138-proxmox ./scripts/deployment/setup-docker-besu.sh /opt/smom-dbis-138 # Step 3: Start containers cd /opt/besu docker compose up -d ``` ### Configuration Files All containers share: - `/opt/besu/docker-compose.yml` - Docker Compose configuration - `/opt/besu/validators/validator-{N}/config/config-validator.toml` - Validator configs - `/opt/besu/sentries/sentry-{N}/config/config-sentry.toml` - Sentry configs - `/opt/besu/rpc/rpc-{N}/config/config-rpc.toml` - RPC configs - `/opt/besu/shared/genesis/` - Shared genesis.json - `/opt/besu/shared/permissions/` - Shared permissions files ### Service Management ```bash # SSH into VM ssh root@192.168.11.90 # Start/stop all containers cd /opt/besu docker compose up -d docker compose stop docker compose restart # Start/stop specific container docker compose start besu-validator-1 docker compose stop besu-validator-1 # View logs docker compose logs -f docker compose logs -f besu-validator-1 # Check status docker compose ps docker stats ``` ### RPC Endpoints - **RPC-1**: `http://192.168.11.90:8545` - **RPC-2**: `http://192.168.11.90:8547` - **RPC-3**: `http://192.168.11.90:8549` ### Metrics Endpoints - **Validators**: `http://192.168.11.90:9545-9549/metrics` - **Sentries**: `http://192.168.11.90:9550-9553/metrics` - **RPC Nodes**: `http://192.168.11.90:9554-9556/metrics` ### Advantages ✅ **Faster Deployment**: ~15-30 minutes ✅ **Easier Initial Testing**: Single point of management ✅ **Simplified Troubleshooting**: All nodes in one place ✅ **Lower Resource Overhead**: Shared VM resources ✅ **Easy Migration Path**: Can migrate to LXC later ✅ **Single Management Point**: Manage all nodes together ### Disadvantages ❌ **Resource Sharing**: All nodes share VM resources ❌ **Less Isolation**: No resource boundaries between nodes ❌ **Not Production Ready**: Temporary/testing solution ❌ **Single Point of Failure**: VM failure affects all nodes ❌ **Resource Contention**: Nodes compete for resources --- ## Side-by-Side Comparison | Feature | LXC Containers (1000-2502) | Single VM (9000) | |---------|---------------------------|------------------| | **VMID Range** | 1000-2502 (12 containers) | 9000 (1 VM) | | **Architecture** | Individual LXC containers | Docker containers in single VM | | **Deployment Time** | 30-45 minutes | 15-30 minutes | | **Total Memory** | 104GB (dedicated) | 32GB (shared) | | **Total CPU Cores** | 40 cores (dedicated) | 8 cores (shared) | | **Total Disk** | 1.5TB (distributed) | 500GB (shared) | | **Resource Isolation** | High (per container) | Low (shared VM) | | **Scalability** | Individual per node | Limited (VM limits) | | **Management** | Per container (12 instances) | Single VM (1 instance) | | **Production Ready** | ✅ Yes | ❌ No (temporary) | | **Security Isolation** | High | Medium | | **Fault Isolation** | High (independent failures) | Low (VM is SPOF) | | **Monitoring** | Per container | Per container (within VM) | | **Migration Path** | N/A (final state) | ✅ To LXC available | --- ## Network Configuration Comparison ### LXC Containers | Node Type | VMID Range | IP Range | VLAN | |-----------|-----------|----------|------| | Validators | 1000-1004 | 192.168.11.100-104 | 100 | | Sentries | 1500-1503 | 192.168.11.150-153 | 101 | | RPC Nodes | 2500-2502 | 192.168.11.250-252 | 102 | **Network Isolation**: Separate VLANs per node type ### Single VM | Node Type | Container Count | Port Mappings | Network | |-----------|----------------|---------------|---------| | Validators | 5 | P2P: 30303-30307, Metrics: 9545-9549 | Docker bridge | | Sentries | 4 | P2P: 30308-30311, Metrics: 9550-9553 | Docker bridge | | RPC Nodes | 3 | HTTP: 8545/8547/8549, WS: 8546/8548/8550 | Docker bridge | **Network Isolation**: Docker bridge network, port-based separation --- ## When to Use Each Deployment ### Use LXC Containers (1000-2502) When: ✅ Production deployment ✅ Need resource isolation ✅ Individual node scaling required ✅ Long-term deployment ✅ Maximum security needed ✅ Need independent node management ✅ Want better fault isolation ### Use Single VM (9000) When: ✅ Quick testing and validation ✅ Development environment ✅ Proof of concept ✅ Limited initial resources ✅ Planning to migrate to LXC later ✅ Need faster initial deployment --- ## Migration Path ### From VM (9000) to LXC Containers (1000-2502) ```bash # 1. Deploy LXC containers cd /opt/smom-dbis-138-proxmox sudo ./scripts/deployment/deploy-besu-nodes.sh # 2. Migrate data from VM to LXC sudo ./scripts/migration/migrate-vm-to-lxc.sh /opt/smom-dbis-138 # 3. Verify and start services in LXC containers for vmid in 1000 1001 1002 1003 1004; do pct exec $vmid -- systemctl start besu-validator done for vmid in 1500 1501 1502 1503; do pct exec $vmid -- systemctl start besu-sentry done for vmid in 2500 2501 2502; do pct exec $vmid -- systemctl start besu-rpc done # 4. Validate deployment ./scripts/validation/validate-deployment-comprehensive.sh # 5. Shut down temporary VM qm stop 9000 ``` --- ## Deployment Scripts Reference ### LXC Containers Deployment **Main Script**: `scripts/deployment/deploy-besu-nodes.sh` **Key Functions**: - Creates 12 LXC containers (5 validators, 4 sentries, 3 RPC) - Configures network (static IPs, VLANs) - Installs Besu in each container - Sets up systemd services - Generates inventory file ### Single VM Deployment **Main Scripts**: - `scripts/deployment/deploy-besu-temp-vm.sh` - Creates VM only - `scripts/deployment/deploy-besu-temp-vm-complete.sh` - Complete deployment - `scripts/deployment/setup-docker-besu.sh` - Sets up Docker and configs **Key Functions**: - Creates single VM (VMID 9000) - Installs Docker and Docker Compose - Copies configuration files - Sets up Docker Compose file - Starts all 12 containers --- ## Validation and Monitoring ### LXC Containers ```bash # Validate all containers ./scripts/validation/validate-deployment-comprehensive.sh # Check individual container status pct exec 1000 -- systemctl status besu-validator pct exec 1500 -- systemctl status besu-sentry pct exec 2500 -- systemctl status besu-rpc # View logs pct exec 1000 -- journalctl -u besu-validator -f ``` ### Single VM ```bash # Validate VM deployment ./scripts/validation/validate-besu-temp-vm.sh # Check container status (from within VM) ssh root@192.168.11.90 docker compose ps docker stats # View logs docker compose logs -f ``` --- ## Related Documentation - [Validated Set Deployment Guide](../03-deployment/VALIDATED_SET_DEPLOYMENT_GUIDE.md) - [Deployment Status Consolidated](../03-deployment/DEPLOYMENT_STATUS_CONSOLIDATED.md) - [Troubleshooting FAQ](../09-troubleshooting/TROUBLESHOOTING_FAQ.md) --- **Last Updated**: $(date)