proxmox/docs/archive/DEPLOYMENT_STEPS_COMPLETE.md

Complete Deployment Steps - Besu Network

Date: 2025-12-20
Deployment Type: Complete Validated Deployment
Total Containers: 12 Besu nodes (5 validators, 4 sentries, 3 RPC)

Quick Command

ssh root@192.168.11.10
cd /opt/smom-dbis-138-proxmox
./scripts/deployment/deploy-validated-set.sh --source-project /opt/smom-dbis-138

---

Complete Deployment Steps

Pre-Deployment Requirements

✅ 1. Verify Prerequisites

• Source project exists: /opt/smom-dbis-138
• Validator keys generated (5 validators)
• Genesis.json updated with correct extraData
• All files synced to ml110
• Scripts have executable permissions
• OS template available: ubuntu-22.04-standard

Check Command:

ls -la /opt/smom-dbis-138/keys/validators/
ls -la /opt/smom-dbis-138/config/genesis.json
pveam list local | grep ubuntu-22.04

---

Phase 1: Deploy Containers ⏱️ ~30-45 minutes

Timeout: 3600 seconds (1 hour)
Script: scripts/deployment/deploy-besu-nodes.sh

Step 1.1: Deploy Validator Containers

Creates 5 validator containers (VMIDs 1000-1004):

For each validator (1-5):

1. Pre-deployment validation

   • Check VMID availability
   • Verify resources (memory, CPU, disk)
   • Validate OS template exists

2. Create container with DHCP

   • Container: besu-validator-{N}
   • VMID: 1000 + (N-1)
   • IP: 192.168.11.10{N} (configured after creation)
   • Memory: 8192 MB
   • CPU: 4 cores
   • Disk: 100 GB
   • Network: DHCP initially

3. Configure static IP address

   • Apply static IP: 192.168.11.10{N}/24
   • Gateway: 192.168.11.1
   • DNS: 8.8.8.8 8.8.4.4

4. Start container

   • Wait for container to be ready
   • Verify container is running

5. Configure container

   • Enable features: nesting, keyctl
   • Configure locale settings
   • Set up environment variables

6. Install Besu

   • Push install script: install/besu-validator-install.sh
   • Execute installation
   • Verify Besu installation

Output: Container running with Besu installed

Step 1.2: Deploy Sentry Containers

Creates 4 sentry containers (VMIDs 1500-1503):

For each sentry (1-4):

1. Pre-deployment validation

2. Create container with DHCP

   • Container: besu-sentry-{N}
   • VMID: 1500 + (N-1)
   • IP: 192.168.11.15{N}
   • Memory: 4096 MB
   • CPU: 2 cores
   • Disk: 100 GB

3. Configure static IP: 192.168.11.15{N}/24

4. Start container

5. Configure container

6. Install Besu (sentry variant)

Output: 4 sentry containers running

Step 1.3: Deploy RPC Containers

Creates 3 RPC containers (VMIDs 2500-2502):

For each RPC node (1-3):

1. Pre-deployment validation

2. Create container with DHCP

   • Container: besu-rpc-{N}
   • VMID: 2500 + (N-1)
   • IP: 192.168.11.25{N}
   • Memory: 16384 MB
   • CPU: 4 cores
   • Disk: 200 GB

3. Configure static IP: 192.168.11.25{N}/24

4. Start container

5. Configure container

6. Install Besu (RPC variant)

Output: 3 RPC containers running

Step 1.4: Save Deployment Inventory

• Generate config/inventory.conf
• Record VMID, hostname, IP for each container
• Used for subsequent operations

Phase 1 Complete: 12 containers created and running

---

Phase 2: Copy Configuration Files ⏱️ ~5-10 minutes

Timeout: 1800 seconds (30 minutes)
Script: scripts/copy-besu-config.sh

Step 2.1: Prerequisites Check

1. Verify source project exists
2. Check required directories:
   • config/ - Configuration files
   • keys/validators/ - Validator keys
3. Check required files:
   • genesis.json
   • config-validator.toml
   • config-sentry.toml
   • config-rpc-*.toml
   • permissions-nodes.toml
   • permissions-accounts.toml

Step 2.2: Copy Genesis File

For all containers (1000-1004, 1500-1503, 2500-2502):

• Copy genesis.json → /etc/besu/genesis.json
• Set ownership: besu:besu
• Set permissions: 644

Step 2.3: Copy Validator Configuration

For validator containers (1000-1004):

• Copy config-validator.toml → /etc/besu/config.toml
• Update paths if needed
• Set ownership and permissions

Step 2.4: Copy Sentry Configuration

For sentry containers (1500-1503):

• Copy config-sentry.toml → /etc/besu/config.toml
• Set ownership and permissions

Step 2.5: Copy RPC Configuration

For RPC containers (2500-2502):

• Copy type-specific config:
  • 2500: config-rpc-core.toml
  • 2501: config-rpc-perm.toml
  • 2502: config-rpc-public.toml
• Copy to /etc/besu/config.toml
• Update systemd service files

Step 2.6: Copy Permissions Files

For all containers:

• Copy permissions-nodes.toml → /etc/besu/permissions-nodes.toml
• Copy permissions-accounts.toml → /etc/besu/permissions-accounts.toml
• Set ownership and permissions

Step 2.7: Copy Validator Keys

For validator containers (1000-1004):

• Copy all validator key directories:
  • validator-1/ → /keys/validators/validator-1/
  • validator-2/ → /keys/validators/validator-2/
  • validator-3/ → /keys/validators/validator-3/
  • validator-4/ → /keys/validators/validator-4/
  • validator-5/ → /keys/validators/validator-5/
• Set ownership: besu:besu
• Set permissions: 600 for private keys

Phase 2 Complete: All configuration files and keys copied

---

Phase 3: Bootstrap Network ⏱️ ~2-5 minutes

Timeout: 300 seconds (5 minutes)
Script: scripts/network/bootstrap-network.sh

Step 3.1: Collect Enode URLs from Validators

For each validator container (1000-1004):

1. Start Besu service (if not running)
2. Wait for node to be ready
3. Extract enode URL from node info
   • Read from /data/besu/nodekey or node info
   • Format: enode://{node_id}@{ip}:30303
4. Verify enode URL is valid

Output: Array of 5 validator enode URLs

Step 3.2: Generate static-nodes.json

1. Create JSON array with all validator enodes
2. Include sentry enodes if available
3. Format: ["enode://...", "enode://...", ...]

Step 3.3: Deploy static-nodes.json

For all containers (1000-1004, 1500-1503, 2500-2502):

1. Copy static-nodes.json → /etc/besu/static-nodes.json
2. Set ownership: besu:besu
3. Set permissions: 644
4. Verify file exists and is valid JSON

Phase 3 Complete: Network bootstrapped, all nodes can discover each other

---

Phase 4: Validate Deployment ⏱️ ~2-5 minutes

Script: scripts/validation/validate-deployment-comprehensive.sh

Step 4.1: Container Status Validation

For all containers:

• Check container exists
• Check container is running
• Verify container responds to commands

Step 4.2: Service Status Validation

For validator containers (1000-1004):

• Check besu-validator.service status
• Verify service is enabled
• Check service is running

For sentry containers (1500-1503):

• Check besu-sentry.service status
• Verify service is enabled and running

For RPC containers (2500-2502):

• Check besu-rpc.service status
• Verify service is enabled and running

Step 4.3: Configuration File Validation

For all containers:

• Verify genesis.json exists and is valid
• Verify config.toml exists and is valid
• Verify static-nodes.json exists and is valid
• Verify permissions files exist

Step 4.4: Key File Validation

For validator containers (1000-1004):

• Verify validator keys exist: /keys/validators/validator-{N}/
• Check key files: key.priv, key.pub, address.txt
• Verify key file permissions and ownership
• Verify keys match genesis.json extraData

Step 4.5: Network Connectivity Validation

For all containers:

• Verify IP addresses are configured correctly
• Check network connectivity (ping gateway)
• Verify Besu ports are listening (30303, 8545, 8546)

Step 4.6: Besu Node Validation

For validator containers:

• Check Besu is running and responsive
• Verify RPC endpoint responds
• Check node is connected to network
• Verify validator is participating in consensus

Phase 4 Complete: Deployment validated and verified

---

Post-Deployment Steps

Step 5: Start All Services

If services are not already running:

# Validators
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- systemctl enable besu-validator.service
    pct exec $vmid -- systemctl start besu-validator.service
done

# Sentries
for vmid in 1500 1501 1502 1503; do
    pct exec $vmid -- systemctl enable besu-sentry.service
    pct exec $vmid -- systemctl start besu-sentry.service
done

# RPC Nodes
for vmid in 2500 2501 2502; do
    pct exec $vmid -- systemctl enable besu-rpc.service
    pct exec $vmid -- systemctl start besu-rpc.service
done

Step 6: Monitor Network Status

Check node connectivity and consensus:

# Check peer count
pct exec 1000 -- curl -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
    http://localhost:8545

# Check block number
pct exec 1000 -- curl -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    http://localhost:8545

# Check validators
pct exec 1000 -- curl -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"qbft_getValidatorsByBlockNumber","params":["latest"],"id":1}' \
    http://localhost:8545

Step 7: Verify Consensus

Ensure all validators are participating:

# Check logs for consensus activity
for vmid in 1000 1001 1002 1003 1004; do
    echo "=== Validator $vmid ==="
    pct exec $vmid -- journalctl -u besu-validator.service -n 20 --no-pager
done

---

Container Summary

Type	Count	VMIDs	IP Range	Memory	CPU	Disk
Validators	5	1000-1004	192.168.11.100-104	8 GB	4	100 GB
Sentries	4	1500-1503	192.168.11.150-153	4 GB	2	100 GB
RPC Nodes	3	2500-2502	192.168.11.250-252	16 GB	4	200 GB
Total	12	-	-	136 GB	38	1.4 TB

---

Estimated Duration

• Phase 1 (Deploy Containers): 30-45 minutes
• Phase 2 (Copy Configuration): 5-10 minutes
• Phase 3 (Bootstrap Network): 2-5 minutes
• Phase 4 (Validate): 2-5 minutes
• Total: 40-65 minutes

---

Skip Options

The deployment script supports skipping phases:

# Skip container deployment (containers already exist)
./scripts/deployment/deploy-validated-set.sh \
    --skip-deployment \
    --source-project /opt/smom-dbis-138

# Skip configuration copy
./scripts/deployment/deploy-validated-set.sh \
    --skip-config \
    --source-project /opt/smom-dbis-138

# Skip network bootstrap
./scripts/deployment/deploy-validated-set.sh \
    --skip-bootstrap \
    --source-project /opt/smom-dbis-138

# Skip validation
./scripts/deployment/deploy-validated-set.sh \
    --skip-validation \
    --source-project /opt/smom-dbis-138

---

Troubleshooting

Containers Not Created

• Check logs: /opt/smom-dbis-138-proxmox/logs/deploy-validated-set-*.log
• Verify OS template exists
• Check Proxmox resources (storage, memory)
• Review network configuration

Configuration Files Missing

• Verify source project path is correct
• Check files exist in source project
• Review copy script logs

Network Bootstrap Fails

• Ensure containers are running
• Check Besu services are started
• Verify static-nodes.json format

Validation Fails

• Review validation output for specific failures
• Check container logs: pct exec <vmid> -- journalctl -u besu-validator.service
• Verify configuration files are correct

---

Status: Complete deployment steps documented
Last Updated: 2025-12-20