proxmox/docs/archive/DEPLOYMENT_STEPS_COMPLETE.md

# Complete Deployment Steps - Besu Network

**Date**: 2025-12-20  
**Deployment Type**: Complete Validated Deployment  
**Total Containers**: 12 Besu nodes (5 validators, 4 sentries, 3 RPC)

## Quick Command

```bash
ssh root@192.168.11.10
cd /opt/smom-dbis-138-proxmox
./scripts/deployment/deploy-validated-set.sh --source-project /opt/smom-dbis-138
```

---

## Complete Deployment Steps

### Pre-Deployment Requirements

#### ✅ 1. Verify Prerequisites
- [ ] Source project exists: `/opt/smom-dbis-138`
- [ ] Validator keys generated (5 validators)
- [ ] Genesis.json updated with correct `extraData`
- [ ] All files synced to ml110
- [ ] Scripts have executable permissions
- [ ] OS template available: `ubuntu-22.04-standard`

**Check Command**:
```bash
ls -la /opt/smom-dbis-138/keys/validators/
ls -la /opt/smom-dbis-138/config/genesis.json
pveam list local | grep ubuntu-22.04
```

---

## Phase 1: Deploy Containers ⏱️ ~30-45 minutes

**Timeout**: 3600 seconds (1 hour)  
**Script**: `scripts/deployment/deploy-besu-nodes.sh`

### Step 1.1: Deploy Validator Containers
Creates 5 validator containers (VMIDs 1000-1004):

**For each validator (1-5):**
1. Pre-deployment validation
   - Check VMID availability
   - Verify resources (memory, CPU, disk)
   - Validate OS template exists

2. Create container with DHCP
   - Container: `besu-validator-{N}`
   - VMID: `1000 + (N-1)`
   - IP: `192.168.11.10{N}` (configured after creation)
   - Memory: 8192 MB
   - CPU: 4 cores
   - Disk: 100 GB
   - Network: DHCP initially

3. Configure static IP address
   - Apply static IP: `192.168.11.10{N}/24`
   - Gateway: `192.168.11.1`
   - DNS: `8.8.8.8 8.8.4.4`

4. Start container
   - Wait for container to be ready
   - Verify container is running

5. Configure container
   - Enable features: nesting, keyctl
   - Configure locale settings
   - Set up environment variables

6. Install Besu
   - Push install script: `install/besu-validator-install.sh`
   - Execute installation
   - Verify Besu installation

**Output**: Container running with Besu installed

### Step 1.2: Deploy Sentry Containers
Creates 4 sentry containers (VMIDs 1500-1503):

**For each sentry (1-4):**
1. Pre-deployment validation
2. Create container with DHCP
   - Container: `besu-sentry-{N}`
   - VMID: `1500 + (N-1)`
   - IP: `192.168.11.15{N}`
   - Memory: 4096 MB
   - CPU: 2 cores
   - Disk: 100 GB

3. Configure static IP: `192.168.11.15{N}/24`
4. Start container
5. Configure container
6. Install Besu (sentry variant)

**Output**: 4 sentry containers running

### Step 1.3: Deploy RPC Containers
Creates 3 RPC containers (VMIDs 2500-2502):

**For each RPC node (1-3):**
1. Pre-deployment validation
2. Create container with DHCP
   - Container: `besu-rpc-{N}`
   - VMID: `2500 + (N-1)`
   - IP: `192.168.11.25{N}`
   - Memory: 16384 MB
   - CPU: 4 cores
   - Disk: 200 GB

3. Configure static IP: `192.168.11.25{N}/24`
4. Start container
5. Configure container
6. Install Besu (RPC variant)

**Output**: 3 RPC containers running

### Step 1.4: Save Deployment Inventory
- Generate `config/inventory.conf`
- Record VMID, hostname, IP for each container
- Used for subsequent operations

**Phase 1 Complete**: 12 containers created and running

---

## Phase 2: Copy Configuration Files ⏱️ ~5-10 minutes

**Timeout**: 1800 seconds (30 minutes)  
**Script**: `scripts/copy-besu-config.sh`

### Step 2.1: Prerequisites Check
1. Verify source project exists
2. Check required directories:
   - `config/` - Configuration files
   - `keys/validators/` - Validator keys
3. Check required files:
   - `genesis.json`
   - `config-validator.toml`
   - `config-sentry.toml`
   - `config-rpc-*.toml`
   - `permissions-nodes.toml`
   - `permissions-accounts.toml`

### Step 2.2: Copy Genesis File
**For all containers (1000-1004, 1500-1503, 2500-2502):**
- Copy `genesis.json` → `/etc/besu/genesis.json`
- Set ownership: `besu:besu`
- Set permissions: `644`

### Step 2.3: Copy Validator Configuration
**For validator containers (1000-1004):**
- Copy `config-validator.toml` → `/etc/besu/config.toml`
- Update paths if needed
- Set ownership and permissions

### Step 2.4: Copy Sentry Configuration
**For sentry containers (1500-1503):**
- Copy `config-sentry.toml` → `/etc/besu/config.toml`
- Set ownership and permissions

### Step 2.5: Copy RPC Configuration
**For RPC containers (2500-2502):**
- Copy type-specific config:
  - 2500: `config-rpc-core.toml`
  - 2501: `config-rpc-perm.toml`
  - 2502: `config-rpc-public.toml`
- Copy to `/etc/besu/config.toml`
- Update systemd service files

### Step 2.6: Copy Permissions Files
**For all containers:**
- Copy `permissions-nodes.toml` → `/etc/besu/permissions-nodes.toml`
- Copy `permissions-accounts.toml` → `/etc/besu/permissions-accounts.toml`
- Set ownership and permissions

### Step 2.7: Copy Validator Keys
**For validator containers (1000-1004):**
- Copy all validator key directories:
  - `validator-1/` → `/keys/validators/validator-1/`
  - `validator-2/` → `/keys/validators/validator-2/`
  - `validator-3/` → `/keys/validators/validator-3/`
  - `validator-4/` → `/keys/validators/validator-4/`
  - `validator-5/` → `/keys/validators/validator-5/`
- Set ownership: `besu:besu`
- Set permissions: `600` for private keys

**Phase 2 Complete**: All configuration files and keys copied

---

## Phase 3: Bootstrap Network ⏱️ ~2-5 minutes

**Timeout**: 300 seconds (5 minutes)  
**Script**: `scripts/network/bootstrap-network.sh`

### Step 3.1: Collect Enode URLs from Validators
**For each validator container (1000-1004):**
1. Start Besu service (if not running)
2. Wait for node to be ready
3. Extract enode URL from node info
   - Read from `/data/besu/nodekey` or node info
   - Format: `enode://{node_id}@{ip}:30303`
4. Verify enode URL is valid

**Output**: Array of 5 validator enode URLs

### Step 3.2: Generate static-nodes.json
1. Create JSON array with all validator enodes
2. Include sentry enodes if available
3. Format: `["enode://...", "enode://...", ...]`

### Step 3.3: Deploy static-nodes.json
**For all containers (1000-1004, 1500-1503, 2500-2502):**
1. Copy `static-nodes.json` → `/etc/besu/static-nodes.json`
2. Set ownership: `besu:besu`
3. Set permissions: `644`
4. Verify file exists and is valid JSON

**Phase 3 Complete**: Network bootstrapped, all nodes can discover each other

---

## Phase 4: Validate Deployment ⏱️ ~2-5 minutes

**Script**: `scripts/validation/validate-deployment-comprehensive.sh`

### Step 4.1: Container Status Validation
**For all containers:**
- Check container exists
- Check container is running
- Verify container responds to commands

### Step 4.2: Service Status Validation
**For validator containers (1000-1004):**
- Check `besu-validator.service` status
- Verify service is enabled
- Check service is running

**For sentry containers (1500-1503):**
- Check `besu-sentry.service` status
- Verify service is enabled and running

**For RPC containers (2500-2502):**
- Check `besu-rpc.service` status
- Verify service is enabled and running

### Step 4.3: Configuration File Validation
**For all containers:**
- Verify `genesis.json` exists and is valid
- Verify `config.toml` exists and is valid
- Verify `static-nodes.json` exists and is valid
- Verify permissions files exist

### Step 4.4: Key File Validation
**For validator containers (1000-1004):**
- Verify validator keys exist: `/keys/validators/validator-{N}/`
- Check key files: `key.priv`, `key.pub`, `address.txt`
- Verify key file permissions and ownership
- Verify keys match genesis.json extraData

### Step 4.5: Network Connectivity Validation
**For all containers:**
- Verify IP addresses are configured correctly
- Check network connectivity (ping gateway)
- Verify Besu ports are listening (30303, 8545, 8546)

### Step 4.6: Besu Node Validation
**For validator containers:**
- Check Besu is running and responsive
- Verify RPC endpoint responds
- Check node is connected to network
- Verify validator is participating in consensus

**Phase 4 Complete**: Deployment validated and verified

---

## Post-Deployment Steps

### Step 5: Start All Services
**If services are not already running:**

```bash
# Validators
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- systemctl enable besu-validator.service
    pct exec $vmid -- systemctl start besu-validator.service
done

# Sentries
for vmid in 1500 1501 1502 1503; do
    pct exec $vmid -- systemctl enable besu-sentry.service
    pct exec $vmid -- systemctl start besu-sentry.service
done

# RPC Nodes
for vmid in 2500 2501 2502; do
    pct exec $vmid -- systemctl enable besu-rpc.service
    pct exec $vmid -- systemctl start besu-rpc.service
done
```

### Step 6: Monitor Network Status
Check node connectivity and consensus:

```bash
# Check peer count
pct exec 1000 -- curl -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
    http://localhost:8545

# Check block number
pct exec 1000 -- curl -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    http://localhost:8545

# Check validators
pct exec 1000 -- curl -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"qbft_getValidatorsByBlockNumber","params":["latest"],"id":1}' \
    http://localhost:8545
```

### Step 7: Verify Consensus
Ensure all validators are participating:

```bash
# Check logs for consensus activity
for vmid in 1000 1001 1002 1003 1004; do
    echo "=== Validator $vmid ==="
    pct exec $vmid -- journalctl -u besu-validator.service -n 20 --no-pager
done
```

---

## Container Summary

| Type | Count | VMIDs | IP Range | Memory | CPU | Disk |
|------|-------|-------|----------|--------|-----|------|
| Validators | 5 | 1000-1004 | 192.168.11.100-104 | 8 GB | 4 | 100 GB |
| Sentries | 4 | 1500-1503 | 192.168.11.150-153 | 4 GB | 2 | 100 GB |
| RPC Nodes | 3 | 2500-2502 | 192.168.11.250-252 | 16 GB | 4 | 200 GB |
| **Total** | **12** | - | - | **136 GB** | **38** | **1.4 TB** |

---

## Estimated Duration

- **Phase 1 (Deploy Containers)**: 30-45 minutes
- **Phase 2 (Copy Configuration)**: 5-10 minutes
- **Phase 3 (Bootstrap Network)**: 2-5 minutes
- **Phase 4 (Validate)**: 2-5 minutes
- **Total**: **40-65 minutes**

---

## Skip Options

The deployment script supports skipping phases:

```bash
# Skip container deployment (containers already exist)
./scripts/deployment/deploy-validated-set.sh \
    --skip-deployment \
    --source-project /opt/smom-dbis-138

# Skip configuration copy
./scripts/deployment/deploy-validated-set.sh \
    --skip-config \
    --source-project /opt/smom-dbis-138

# Skip network bootstrap
./scripts/deployment/deploy-validated-set.sh \
    --skip-bootstrap \
    --source-project /opt/smom-dbis-138

# Skip validation
./scripts/deployment/deploy-validated-set.sh \
    --skip-validation \
    --source-project /opt/smom-dbis-138
```

---

## Troubleshooting

### Containers Not Created
- Check logs: `/opt/smom-dbis-138-proxmox/logs/deploy-validated-set-*.log`
- Verify OS template exists
- Check Proxmox resources (storage, memory)
- Review network configuration

### Configuration Files Missing
- Verify source project path is correct
- Check files exist in source project
- Review copy script logs

### Network Bootstrap Fails
- Ensure containers are running
- Check Besu services are started
- Verify static-nodes.json format

### Validation Fails
- Review validation output for specific failures
- Check container logs: `pct exec <vmid> -- journalctl -u besu-validator.service`
- Verify configuration files are correct

---

**Status**: Complete deployment steps documented  
**Last Updated**: 2025-12-20