d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
proxmox/docs/archive/DEPLOYMENT_COMPARISON.md
defiQUG fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
Details
docs: Ledger Live integration, contract deploy learnings, NEXT_STEPS updates 
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands
- CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround
- CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check
- NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere
- MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates
- LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-12 15:46:57 -08:00

490 lines
18 KiB
Markdown
Raw Permalink Blame History

# Deployment Comparison: LXC Containers vs Single VM
This document compares the two deployment options for SMOM-DBIS-138 Besu network nodes.
## Overview
### Deployment 1: Individual LXC Containers (VMID 1000-2502)
Each Besu node runs in its own LXC container on Proxmox VE. This is the **production-ready** deployment method.
### Deployment 2: Single VM with Docker (VMID 9000)
All Besu nodes run as Docker containers within a single VM. This is the **temporary/testing** deployment method.
---
## Deployment 1: LXC Containers (1000-2502)
### Architecture
```
┌──────────────────────────────────────────────────────────────┐
│ Proxmox VE Host │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ LXC 1000 │ │ LXC 1001 │ │ LXC 1002 │ │
│ │ validator-1 │ │ validator-2 │ │ validator-3 │ │
│ │ 8GB, 4 cores │ │ 8GB, 4 cores │ │ 8GB, 4 cores │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ LXC 1003 │ │ LXC 1004 │ │
│ │ validator-4 │ │ validator-5 │ │
│ │ 8GB, 4 cores │ │ 8GB, 4 cores │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ LXC 1500 │ │ LXC 1501 │ │ LXC 1502 │ │
│ │ sentry-1 │ │ sentry-2 │ │ sentry-3 │ │
│ │ 4GB, 2 cores │ │ 4GB, 2 cores │ │ 4GB, 2 cores │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────┐ │
│ │ LXC 1503 │ │
│ │ sentry-4 │ │
│ │ 4GB, 2 cores │ │
│ └──────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ LXC 2500 │ │ LXC 2501 │ │ LXC 2502 │ │
│ │ rpc-1 │ │ rpc-2 │ │ rpc-3 │ │
│ │ 16GB, 4 cores│ │ 16GB, 4 cores│ │ 16GB, 4 cores│ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└──────────────────────────────────────────────────────────────┘
```
### VMID Allocation
| Category | VMID Range | Count | Hostnames | IP Range |
|----------|-----------|-------|-----------|----------|
| **Validators** | 1000-1004 | 5 | besu-validator-1 through besu-validator-5 | 192.168.11.100-104 |
| **Sentries** | 1500-1503 | 4 | besu-sentry-1 through besu-sentry-4 | 192.168.11.150-153 |
| **RPC Nodes** | 2500-2502 | 3 | besu-rpc-1 through besu-rpc-3 | 192.168.11.250-252 |
**Total Containers**: 12
### Resource Allocation
#### Validators (1000-1004)
- **Memory**: 8GB each (40GB total)
- **CPU Cores**: 4 each (20 cores total)
- **Disk**: 100GB each (500GB total)
- **IP Addresses**: 192.168.11.100-104
- **Purpose**: QBFT consensus nodes
#### Sentries (1500-1503)
- **Memory**: 4GB each (16GB total)
- **CPU Cores**: 2 each (8 cores total)
- **Disk**: 100GB each (400GB total)
- **IP Addresses**: 192.168.11.150-153
- **Purpose**: P2P relay and protection nodes
#### RPC Nodes (2500-2502)
- **Memory**: 16GB each (48GB total)
- **CPU Cores**: 4 each (12 cores total)
- **Disk**: 200GB each (600GB total)
- **IP Addresses**: 192.168.11.250-252
- **Purpose**: Public RPC endpoints
**Total Resources**:
- **Total Memory**: 104GB
- **Total CPU Cores**: 40 cores
- **Total Disk**: 1.5TB
### Deployment Script
```bash
cd /opt/smom-dbis-138-proxmox
sudo ./scripts/deployment/deploy-besu-nodes.sh
```
### Configuration Files
Each container has:
- `/etc/besu/config-{type}.toml` - Besu configuration
- `/opt/besu/data/` - Blockchain data directory
- `/opt/besu/keys/` - Validator keys (validators only)
- Systemd service: `besu-validator`, `besu-sentry`, or `besu-rpc`
### Service Management
```bash
# Start/stop individual container services
pct exec 1000 -- systemctl start besu-validator
pct exec 1500 -- systemctl start besu-sentry
pct exec 2500 -- systemctl start besu-rpc
# Check status
pct exec 1000 -- systemctl status besu-validator
# View logs
pct exec 1000 -- journalctl -u besu-validator -f
```
### Advantages
✅ **Resource Isolation**: Each node has dedicated resources
✅ **Individual Scaling**: Scale nodes independently
✅ **Production Ready**: Suitable for production deployments
✅ **Security Isolation**: Better security boundaries
✅ **Independent Management**: Start/stop nodes individually
✅ **Better Monitoring**: Monitor each node separately
✅ **Fault Isolation**: Failure of one node doesn't affect others
### Disadvantages
❌ **Longer Deployment Time**: ~30-45 minutes
❌ **More Complex Setup**: Requires configuration per container
❌ **Higher Resource Usage**: More overhead per node
❌ **More Management Overhead**: Manage 12 containers
---
## Deployment 2: Single VM with Docker (VMID 9000)
### Architecture
```
┌──────────────────────────────────────────────────────────────┐
│ Proxmox VE Host │
│ │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ VM 9000 │ │
│ │ IP: 192.168.11.90 │ │
│ │ Memory: 32GB, CPU: 8 cores, Disk: 500GB │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────────────┐ │ │
│ │ │ Docker Network │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │
│ │ │ │validator│ │validator│ │validator│ │ │ │
│ │ │ │ -1 │ │ -2 │ │ -3 │ │ │ │
│ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ │ │ │
│ │ │ │validator│ │validator│ │ │ │
│ │ │ │ -4 │ │ -5 │ │ │ │
│ │ │ └─────────┘ └─────────┘ │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │
│ │ │ │ sentry │ │ sentry │ │ sentry │ │ │ │
│ │ │ │ -1 │ │ -2 │ │ -3 │ │ │ │
│ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ │ │ │
│ │ │ │ sentry │ │ │ │
│ │ │ │ -4 │ │ │ │
│ │ │ └─────────┘ │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │
│ │ │ │ rpc-1 │ │ rpc-2 │ │ rpc-3 │ │ │ │
│ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
```
### VMID Allocation
| Category | VMID | Description | IP Address |
|----------|------|-------------|------------|
| **Temporary VM** | 9000 | All 12 Besu nodes in Docker | 192.168.11.90 |
**Total VMs**: 1
**Total Docker Containers**: 12 (5 validators + 4 sentries + 3 RPC)
### Resource Allocation
- **VM Memory**: 32GB (shared by all containers)
- **VM CPU Cores**: 8 cores (shared by all containers)
- **VM Disk**: 500GB (shared by all containers)
- **VM IP Address**: 192.168.11.90
- **OS**: Ubuntu 22.04
#### Container Resource Allocation (within VM)
##### Validators (5 containers)
- **Memory**: 4GB each (20GB total)
- **P2P Ports**: 30303-30307
- **Metrics Ports**: 9545-9549
##### Sentries (4 containers)
- **Memory**: 2GB each (8GB total)
- **P2P Ports**: 30308-30311
- **Metrics Ports**: 9550-9553
##### RPC Nodes (3 containers)
- **Memory**: 8GB each (24GB total)
- **HTTP RPC Ports**: 8545, 8547, 8549
- **WS RPC Ports**: 8546, 8548, 8550
- **P2P Ports**: 30312-30314
- **Metrics Ports**: 9554-9556
**Total Container Memory**: 52GB (with some headroom in 32GB VM)
### Deployment Script
```bash
cd /opt/smom-dbis-138-proxmox
sudo ./scripts/deployment/deploy-besu-temp-vm-complete.sh /opt/smom-dbis-138
```
Or step-by-step:
```bash
# Step 1: Create VM
sudo ./scripts/deployment/deploy-besu-temp-vm.sh
# Step 2: SSH into VM and setup
ssh root@192.168.11.90
cd /opt/smom-dbis-138-proxmox
./scripts/deployment/setup-docker-besu.sh /opt/smom-dbis-138
# Step 3: Start containers
cd /opt/besu
docker compose up -d
```
### Configuration Files
All containers share:
- `/opt/besu/docker-compose.yml` - Docker Compose configuration
- `/opt/besu/validators/validator-{N}/config/config-validator.toml` - Validator configs
- `/opt/besu/sentries/sentry-{N}/config/config-sentry.toml` - Sentry configs
- `/opt/besu/rpc/rpc-{N}/config/config-rpc.toml` - RPC configs
- `/opt/besu/shared/genesis/` - Shared genesis.json
- `/opt/besu/shared/permissions/` - Shared permissions files
### Service Management
```bash
# SSH into VM
ssh root@192.168.11.90
# Start/stop all containers
cd /opt/besu
docker compose up -d
docker compose stop
docker compose restart
# Start/stop specific container
docker compose start besu-validator-1
docker compose stop besu-validator-1
# View logs
docker compose logs -f
docker compose logs -f besu-validator-1
# Check status
docker compose ps
docker stats
```
### RPC Endpoints
- **RPC-1**: `http://192.168.11.90:8545`
- **RPC-2**: `http://192.168.11.90:8547`
- **RPC-3**: `http://192.168.11.90:8549`
### Metrics Endpoints
- **Validators**: `http://192.168.11.90:9545-9549/metrics`
- **Sentries**: `http://192.168.11.90:9550-9553/metrics`
- **RPC Nodes**: `http://192.168.11.90:9554-9556/metrics`
### Advantages
✅ **Faster Deployment**: ~15-30 minutes
✅ **Easier Initial Testing**: Single point of management
✅ **Simplified Troubleshooting**: All nodes in one place
✅ **Lower Resource Overhead**: Shared VM resources
✅ **Easy Migration Path**: Can migrate to LXC later
✅ **Single Management Point**: Manage all nodes together
### Disadvantages
❌ **Resource Sharing**: All nodes share VM resources
❌ **Less Isolation**: No resource boundaries between nodes
❌ **Not Production Ready**: Temporary/testing solution
❌ **Single Point of Failure**: VM failure affects all nodes
❌ **Resource Contention**: Nodes compete for resources
---
## Side-by-Side Comparison
| Feature | LXC Containers (1000-2502) | Single VM (9000) |
|---------|---------------------------|------------------|
| **VMID Range** | 1000-2502 (12 containers) | 9000 (1 VM) |
| **Architecture** | Individual LXC containers | Docker containers in single VM |
| **Deployment Time** | 30-45 minutes | 15-30 minutes |
| **Total Memory** | 104GB (dedicated) | 32GB (shared) |
| **Total CPU Cores** | 40 cores (dedicated) | 8 cores (shared) |
| **Total Disk** | 1.5TB (distributed) | 500GB (shared) |
| **Resource Isolation** | High (per container) | Low (shared VM) |
| **Scalability** | Individual per node | Limited (VM limits) |
| **Management** | Per container (12 instances) | Single VM (1 instance) |
| **Production Ready** | ✅ Yes | ❌ No (temporary) |
| **Security Isolation** | High | Medium |
| **Fault Isolation** | High (independent failures) | Low (VM is SPOF) |
| **Monitoring** | Per container | Per container (within VM) |
| **Migration Path** | N/A (final state) | ✅ To LXC available |
---
## Network Configuration Comparison
### LXC Containers
| Node Type | VMID Range | IP Range | VLAN |
|-----------|-----------|----------|------|
| Validators | 1000-1004 | 192.168.11.100-104 | 100 |
| Sentries | 1500-1503 | 192.168.11.150-153 | 101 |
| RPC Nodes | 2500-2502 | 192.168.11.250-252 | 102 |
**Network Isolation**: Separate VLANs per node type
### Single VM
| Node Type | Container Count | Port Mappings | Network |
|-----------|----------------|---------------|---------|
| Validators | 5 | P2P: 30303-30307, Metrics: 9545-9549 | Docker bridge |
| Sentries | 4 | P2P: 30308-30311, Metrics: 9550-9553 | Docker bridge |
| RPC Nodes | 3 | HTTP: 8545/8547/8549, WS: 8546/8548/8550 | Docker bridge |
**Network Isolation**: Docker bridge network, port-based separation
---
## When to Use Each Deployment
### Use LXC Containers (1000-2502) When:
✅ Production deployment
✅ Need resource isolation
✅ Individual node scaling required
✅ Long-term deployment
✅ Maximum security needed
✅ Need independent node management
✅ Want better fault isolation
### Use Single VM (9000) When:
✅ Quick testing and validation
✅ Development environment
✅ Proof of concept
✅ Limited initial resources
✅ Planning to migrate to LXC later
✅ Need faster initial deployment
---
## Migration Path
### From VM (9000) to LXC Containers (1000-2502)
```bash
# 1. Deploy LXC containers
cd /opt/smom-dbis-138-proxmox
sudo ./scripts/deployment/deploy-besu-nodes.sh
# 2. Migrate data from VM to LXC
sudo ./scripts/migration/migrate-vm-to-lxc.sh /opt/smom-dbis-138
# 3. Verify and start services in LXC containers
for vmid in 1000 1001 1002 1003 1004; do
pct exec $vmid -- systemctl start besu-validator
done
for vmid in 1500 1501 1502 1503; do
pct exec $vmid -- systemctl start besu-sentry
done
for vmid in 2500 2501 2502; do
pct exec $vmid -- systemctl start besu-rpc
done
# 4. Validate deployment
./scripts/validation/validate-deployment-comprehensive.sh
# 5. Shut down temporary VM
qm stop 9000
```
---
## Deployment Scripts Reference
### LXC Containers Deployment
**Main Script**: `scripts/deployment/deploy-besu-nodes.sh`
**Key Functions**:
- Creates 12 LXC containers (5 validators, 4 sentries, 3 RPC)
- Configures network (static IPs, VLANs)
- Installs Besu in each container
- Sets up systemd services
- Generates inventory file
### Single VM Deployment
**Main Scripts**:
- `scripts/deployment/deploy-besu-temp-vm.sh` - Creates VM only
- `scripts/deployment/deploy-besu-temp-vm-complete.sh` - Complete deployment
- `scripts/deployment/setup-docker-besu.sh` - Sets up Docker and configs
**Key Functions**:
- Creates single VM (VMID 9000)
- Installs Docker and Docker Compose
- Copies configuration files
- Sets up Docker Compose file
- Starts all 12 containers
---
## Validation and Monitoring
### LXC Containers
```bash
# Validate all containers
./scripts/validation/validate-deployment-comprehensive.sh
# Check individual container status
pct exec 1000 -- systemctl status besu-validator
pct exec 1500 -- systemctl status besu-sentry
pct exec 2500 -- systemctl status besu-rpc
# View logs
pct exec 1000 -- journalctl -u besu-validator -f
```
### Single VM
```bash
# Validate VM deployment
./scripts/validation/validate-besu-temp-vm.sh
# Check container status (from within VM)
ssh root@192.168.11.90
docker compose ps
docker stats
# View logs
docker compose logs -f
```
---
## Related Documentation
- [Validated Set Deployment Guide](../03-deployment/VALIDATED_SET_DEPLOYMENT_GUIDE.md)
- [Deployment Status Consolidated](../03-deployment/DEPLOYMENT_STATUS_CONSOLIDATED.md)
- [Troubleshooting FAQ](../09-troubleshooting/TROUBLESHOOTING_FAQ.md)
---
**Last Updated**: $(date)
Reference in New Issue View Git Blame Copy Permalink