Sankofa/docs/vm/DETAILED_VM_DEPLOYMENT_STEPS.md

Detailed VM Deployment Steps Review

Date: 2025-12-13
Status: Comprehensive Review of All VM Deployment Steps

---

Executive Summary

This document provides a detailed step-by-step review of the deployment process for each of the 30 VMs in the Sankofa Phoenix infrastructure. Each VM goes through a standardized deployment process with specific steps that are executed by the Crossplane Proxmox provider.

---

VM Deployment Process Overview

Standard Deployment Flow

For each VM, the following steps are executed in order:

1. Reconciliation Trigger: Kubernetes controller detects new/updated ProxmoxVM resource
2. Validation: Validate VM specification (name, CPU, memory, disk, network, image)
3. Authentication: Authenticate to Proxmox API using token or username/password
4. Site Resolution: Resolve site configuration from ProviderConfig
5. Node Health Check: Verify target Proxmox node is healthy and reachable
6. Network Validation: Verify network bridge exists on target node
7. VM Creation: Execute VM creation on Proxmox
8. Status Update: Update Kubernetes resource with VMID and status
9. Monitoring: Continue monitoring VM state and IP address

---

Detailed Deployment Steps by VM

Phase 1: Core Infrastructure

1.1 Nginx Proxy VM

File: examples/production/nginx-proxy-vm.yaml
Node: ml110-01
Site: site-1

Configuration:

• CPU: 2 cores
• Memory: 4 GiB
• Disk: 20 GiB
• Storage: local-lvm
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps:

1. Reconciliation:

   • Controller receives ProxmoxVM resource
   • Validates ProviderConfigReference exists

2. Authentication:

   • Reads credentials from proxmox-credentials secret
   • Detects token authentication (tokenid + token)
   • Creates Proxmox client with token: root@pam!sankofa-instance-1-api-token=token
   • Uses Cookie header: PVEAuthCookie=root@pam!sankofa-instance-1-api-token=token

3. Site Resolution:

   • Finds site-1 in ProviderConfig
   • Endpoint: https://192.168.11.10:8006
   • Node: ml110-01

4. Node Health Check:

   • Calls: GET /nodes/ml110-01/status
   • Verifies node is reachable and healthy
   • ⚠️ Potential Issue: If health check fails, VM creation is delayed

5. Network Validation:

   • Calls: GET /nodes/ml110-01/network
   • Verifies vmbr0 bridge exists
   • ⚠️ Potential Issue: If bridge doesn't exist, VM creation fails

6. VM Specification Validation:

   • Validates VM name: nginx-proxy-vm (1-100 chars, alphanumeric)
   • Validates CPU: 2 (1-1024 range)
   • Validates Memory: 4Gi (128MB-2TB range)
   • Validates Disk: 20Gi (1GB-100TB range)
   • Validates Network: vmbr0 (valid bridge name)
   • Validates Image: local:iso/ubuntu-22.04-cloud.img (must exist in storage)

7. Image Resolution:

   • Searches for image in Proxmox storage
   • Looks for ubuntu-22.04-cloud.img in storage pools
   • Expected volid: local:iso/ubuntu-22.04-cloud.img
   • ⚠️ Critical: If image not found, VM creation fails

8. VMID Assignment:

   • Calls: GET /cluster/nextid
   • Gets next available VMID (e.g., 100, 101, 102...)
   • VMID assigned: <next-available-id>

9. VM Configuration Creation:

   • Creates VM config with:
     • vmid: Assigned VMID
     • name: nginx-proxy-vm
     • cores: 2
     • memory: 4096 (4Gi in MB)
     • net0: virtio,bridge=vmbr0
     • scsi0: local-lvm:20,format=qcow2 (for cloud image import)
     • ostype: l26 (Linux 2.6+)
     • agent: 1 (QEMU guest agent enabled)
     • boot: order=scsi0

10. Cloud-init Configuration:

    • Adds cloud-init drive: ide2=local-lvm:cloudinit
    • Sets cloud-init user: ciuser=admin
    • Sets network config: ipconfig0=ip=dhcp
    • Writes userData to cloud-init drive

11. VM Creation:

    • Calls: POST /nodes/ml110-01/qemu
    • Creates VM with configuration
    • Returns: VMID if successful

12. Image Import (if needed):

    • If image is .img or .qcow2:
      • Stops VM (if running)
      • Calls: POST /nodes/ml110-01/qemu/{vmid}/importdisk
      • Imports image from local:iso/ubuntu-22.04-cloud.img
      • Waits for import to complete (with timeout)
      • ⚠️ Critical: If importdisk fails, VM is cleaned up

13. Status Update:

    • Updates ProxmoxVM status:
      • vmId: Assigned VMID
      • state: "created"
      • conditions: Ready=True
    • Requeues for status monitoring (10 seconds)

14. VM State Monitoring:

    • Subsequent reconciles check VM state
    • Calls: GET /nodes/ml110-01/qemu/{vmid}/status/current
    • Updates status.state (running, stopped, etc.)
    • Retrieves IP address from QEMU guest agent

Expected Timeline: 2-5 minutes

Dependencies: None

Potential Issues:

• Image not found in storage
• Network bridge doesn't exist
• Node health check fails
• Importdisk API not supported (Proxmox version)

---

1.2 Cloudflare Tunnel VM

File: examples/production/cloudflare-tunnel-vm.yaml
Node: r630-01
Site: site-2

Configuration:

• CPU: 2 cores
• Memory: 4 GiB
• Disk: 10 GiB
• Storage: local-lvm
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Same as Nginx Proxy VM, but:

• Target node: r630-01
• Endpoint: https://192.168.11.11:8006
• Disk: 10 GiB (smaller)
• ⚠️ Note: Uses local-lvm on R630-01 (limited to 171.3 GB)

Expected Timeline: 2-5 minutes

Dependencies: None

---

Phase 2: Phoenix Infrastructure

2.1 DNS Primary

File: examples/production/phoenix/dns-primary.yaml
Node: ml110-01
Site: site-1

Configuration:

• CPU: 2 cores (reduced from 4)
• Memory: 4 GiB (reduced from 8Gi)
• Disk: 50 GiB
• Storage: local-lvm
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 2-5 minutes

Dependencies: None (but other services may depend on DNS)

---

2.2 Git Server

File: examples/production/phoenix/git-server.yaml
Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration:

• CPU: 4 cores (reduced from 8)
• Memory: 16 GiB
• Disk: 500 GiB (large disk)
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process, but:

• ⚠️ Large Disk: 500 GiB requires Ceph storage
• ⚠️ Storage Check: Verifies ceph-fs storage exists
• ⚠️ Image Import: Large image import may take longer

Expected Timeline: 5-10 minutes (due to large disk)

Dependencies: None

Optimization Notes:

• Moved to R630-01 for better CPU resources
• Using ceph-fs for large disk (avoids local-lvm constraint)

---

2.3 Email Server

File: examples/production/phoenix/email-server.yaml
Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration:

• CPU: 4 cores (reduced from 8)
• Memory: 16 GiB
• Disk: 200 GiB
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 3-7 minutes

Dependencies: None

---

2.4 DevOps Runner

File: examples/production/phoenix/devops-runner.yaml
Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration:

• CPU: 4 cores (reduced from 8)
• Memory: 16 GiB
• Disk: 200 GiB
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 3-7 minutes

Dependencies: None

---

2.5 Codespaces IDE

File: examples/production/phoenix/codespaces-ide.yaml
Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration:

• CPU: 4 cores (reduced from 8)
• Memory: 32 GiB (reduced from 32Gi, unchanged)
• Disk: 200 GiB
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 3-7 minutes

Dependencies: None

Note: High memory requirement (32 GiB)

---

2.6 AS4 Gateway

File: examples/production/phoenix/as4-gateway.yaml
Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration:

• CPU: 4 cores (reduced from 8)
• Memory: 16 GiB
• Disk: 500 GiB (large disk)
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 5-10 minutes (due to large disk)

Dependencies: None

---

2.7 Business Integration Gateway

File: examples/production/phoenix/business-integration-gateway.yaml
Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration:

• CPU: 4 cores (reduced from 8)
• Memory: 16 GiB
• Disk: 200 GiB
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 3-7 minutes

Dependencies: None

---

2.8 Financial Messaging Gateway

File: examples/production/phoenix/financial-messaging-gateway.yaml
Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration:

• CPU: 4 cores (reduced from 8)
• Memory: 16 GiB
• Disk: 500 GiB (large disk)
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 5-10 minutes (due to large disk)

Dependencies: None

---

Phase 3: Blockchain Infrastructure

3.1 Validators (4x)

Files:

• examples/production/smom-dbis-138/validator-01.yaml
• examples/production/smom-dbis-138/validator-02.yaml
• examples/production/smom-dbis-138/validator-03.yaml
• examples/production/smom-dbis-138/validator-04.yaml

Node: r630-01 (moved from ml110-01)
Site: site-2

Configuration (per validator):

• CPU: 3 cores (reduced from 6)
• Memory: 12 GiB (reduced from 12Gi, unchanged)
• Disk: 20 GiB
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process for each validator

Expected Timeline: 2-5 minutes per validator (8-20 minutes total)

Dependencies: None (but validators should be deployed together)

Total Resources:

• CPU: 12 cores (4 × 3)
• Memory: 48 GiB (4 × 12 GiB)
• Disk: 80 GiB (4 × 20 GiB)

---

3.2 Sentries (4x)

Files:

• examples/production/smom-dbis-138/sentry-01.yaml (ml110-01)
• examples/production/smom-dbis-138/sentry-02.yaml (ml110-01)
• examples/production/smom-dbis-138/sentry-03.yaml (r630-01)
• examples/production/smom-dbis-138/sentry-04.yaml (r630-01)

Configuration (per sentry):

• CPU: 2 cores (reduced from 4)
• Memory: 4 GiB (reduced from 8Gi)
• Disk: 20 GiB
• Storage: local-lvm (sentries 01-02), ceph-fs (sentries 03-04)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 2-5 minutes per sentry

Dependencies: None

Total Resources:

• ML110-01: 4 CPU, 8 GiB RAM, 40 GiB disk
• R630-01: 4 CPU, 8 GiB RAM, 40 GiB disk

---

3.3 RPC Nodes (4x)

Files:

• examples/production/smom-dbis-138/rpc-node-01.yaml
• examples/production/smom-dbis-138/rpc-node-02.yaml
• examples/production/smom-dbis-138/rpc-node-03.yaml
• examples/production/smom-dbis-138/rpc-node-04.yaml

Node: r630-01
Site: site-2

Configuration (per RPC node):

• CPU: 2 cores (reduced from 4)
• Memory: 4 GiB (reduced from 8Gi)
• Disk: 20 GiB
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 2-5 minutes per node (8-20 minutes total)

Dependencies: None

Total Resources:

• CPU: 8 cores (4 × 2)
• Memory: 16 GiB (4 × 4 GiB)
• Disk: 80 GiB (4 × 20 GiB)

---

3.4 Services (4x)

Files:

• examples/production/smom-dbis-138/management.yaml
• examples/production/smom-dbis-138/monitoring.yaml
• examples/production/smom-dbis-138/services.yaml
• examples/production/smom-dbis-138/blockscout.yaml

Node: r630-01
Site: site-2

Configuration (per service):

• CPU: 2 cores (reduced from 4)
• Memory: 4 GiB (reduced from 8Gi)
• Disk: 20 GiB
• Storage: ceph-fs (changed from local-lvm)
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 2-5 minutes per service (8-20 minutes total)

Dependencies: None

Total Resources:

• CPU: 8 cores (4 × 2)
• Memory: 16 GiB (4 × 4 GiB)
• Disk: 80 GiB (4 × 20 GiB)

---

Phase 4: Test/Example VMs

4.1 Basic VM

File: examples/production/basic-vm.yaml
Node: ml110-01
Site: site-1

Configuration:

• CPU: 2 cores
• Memory: 4 GiB
• Disk: 50 GiB
• Storage: local-lvm
• Network: vmbr0
• Image: local:iso/ubuntu-22.04-cloud.img
• Cloud-init: ✅ Configured

Deployment Steps: Standard process

Expected Timeline: 2-5 minutes

Priority: LOW (test VM)

---

4.2 VM-100

File: examples/production/vm-100.yaml
Node: ml110-01
Site: site-1

Configuration: Similar to basic-vm

Deployment Steps: Standard process

Expected Timeline: 2-5 minutes

Priority: LOW (test VM)

---

4.3 Medium VM

File: examples/production/medium-vm.yaml
Status: ⚠️ Error reading configuration

Issue: YAML file may have syntax errors or missing fields

Action Required: Review and fix YAML file

---

4.4 Large VM

File: examples/production/large-vm.yaml
Status: ⚠️ Error reading configuration

Issue: YAML file may have syntax errors or missing fields

Action Required: Review and fix YAML file

---

4.5 Blockscout

File: examples/production/smom-dbis-138/blockscout.yaml
Status: ⚠️ Error reading configuration

Issue: YAML file may have syntax errors or missing fields

Action Required: Review and fix YAML file

---

Detailed Deployment Process Flow

Step-by-Step Execution

For each VM, the controller executes these steps in sequence:

Step 1: Resource Validation

// Validate VM name (1-100 chars, alphanumeric + hyphens/underscores)
utils.ValidateVMName(vm.Spec.ForProvider.Name)

// Validate CPU (1-1024)
utils.ValidateCPU(vm.Spec.ForProvider.CPU)

// Validate Memory (128MB-2TB, supports Gi/Mi/Ki)
utils.ValidateMemory(vm.Spec.ForProvider.Memory)

// Validate Disk (1GB-100TB, supports Ti/Gi/Mi)
utils.ValidateDisk(vm.Spec.ForProvider.Disk)

// Validate Network bridge name
utils.ValidateNetworkBridge(vm.Spec.ForProvider.Network)

// Validate Image specification
utils.ValidateImageSpec(vm.Spec.ForProvider.Image)

Step 2: Authentication

// Read credentials from secret
creds := getCredentials(providerConfig)

// Create Proxmox client
if creds.Token != "" {
    // Token authentication
    token := fmt.Sprintf("%s=%s", creds.Username, creds.Token)
    client := NewClientWithToken(endpoint, token, insecureSkipTLS)
} else {
    // Username/password authentication
    client := NewClient(endpoint, username, password, insecureSkipTLS)
}

Step 3: Node Health Check

// Check node is reachable and healthy
client.CheckNodeHealth(ctx, node)
// Calls: GET /nodes/{node}/status

Step 4: Network Validation

// Verify network bridge exists
client.NetworkExists(ctx, node, network)
// Calls: GET /nodes/{node}/network
// Searches for bridge in network list

Step 5: VMID Assignment

// Get next available VMID
client.Get(ctx, "/cluster/nextid")
// Returns: Next available VMID (e.g., "100", "101")

Step 6: Image Resolution

// Search for image in storage
if image contains ":" {
    // Already a volid (storage:path)
    imageVolid = image
} else {
    // Search storage pools
    client.findImageInStorage(ctx, node, image)
    // Searches all storages for matching image
}

Step 7: VM Configuration

vmConfig := {
    "vmid": vmID,
    "name": vmName,
    "cores": cpu,
    "memory": parseMemory(memory), // Convert to MB
    "net0": fmt.Sprintf("virtio,bridge=%s", network),
    "scsi0": diskConfig, // Format: storage:size,format=qcow2
    "ostype": "l26",
    "agent": "1",
    "boot": "order=scsi0"
}

// Add cloud-init if userData provided
if userData != "" {
    vmConfig["ide2"] = fmt.Sprintf("%s:cloudinit", storage)
    vmConfig["ciuser"] = "admin"
    vmConfig["ipconfig0"] = "ip=dhcp"
}

Step 8: VM Creation

// Create VM on Proxmox
client.Post(ctx, fmt.Sprintf("/nodes/%s/qemu", node), vmConfig)
// Calls: POST /nodes/{node}/qemu
// Returns: VMID if successful

Step 9: Image Import (if needed)

if needsImageImport && imageVolid != "" {
    // Stop VM if running
    client.Post(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/status/stop", node, vmID))
    
    // Wait for VM to stop (up to 30 seconds)
    waitForVMStop(ctx, node, vmID, 30*time.Second)
    
    // Import image
    client.Post(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/importdisk", node, vmID), {
        "storage": storageName,
        "filename": imagePath,
        "format": "qcow2"
    })
    
    // Monitor import task until completion
    monitorTask(ctx, taskID, timeout)
}

Step 10: Status Update

// Update ProxmoxVM status
vm.Status.VMID = createdVM.ID
vm.Status.State = "created"
vm.Status.Conditions = append(conditions, {
    Type: "Ready",
    Status: "True",
    Reason: "Created",
    Message: fmt.Sprintf("VM %d created successfully", vmID)
})

// Update Kubernetes resource
r.Status().Update(ctx, &vm)

// Requeue for monitoring (10 seconds)
return ctrl.Result{RequeueAfter: 10 * time.Second}, nil

Step 11: State Monitoring

// Subsequent reconciles check VM state
client.Get(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/status/current", node, vmID))
// Updates: vm.Status.State (running, stopped, etc.)

// Get IP address from QEMU guest agent
client.Get(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/agent/network-get-interfaces", node, vmID))
// Updates: vm.Status.IPAddress

---

Potential Issues and Mitigations

Issue 1: Image Not Found

Symptom: Error "image 'X' not found in storage"
Cause: Image not uploaded to Proxmox storage
Mitigation:

• Verify image exists: ./scripts/verify-image-availability.sh
• Upload image if missing
• Provider will retry automatically

Issue 2: Network Bridge Missing

Symptom: Error "network bridge 'X' does not exist"
Cause: Network bridge not configured on Proxmox node
Mitigation:

• Verify bridge exists: Check Proxmox network configuration
• Create bridge if missing
• Provider will fail fast (no retry)

Issue 3: Node Health Check Fails

Symptom: Error "node X is not reachable or unhealthy"
Cause: Node down, network issue, or authentication failure
Mitigation:

• Check node status on Proxmox
• Verify network connectivity
• Check authentication credentials
• Provider retries with exponential backoff

Issue 4: Importdisk Not Supported

Symptom: Error "importdisk API not supported"
Cause: Proxmox version too old (< 6.2)
Mitigation:

• Upgrade Proxmox to 6.2+
• Or use template-based deployment instead
• Provider detects and cleans up VM

Issue 5: Resource Exhaustion

Symptom: Error "insufficient resources"
Cause: Node out of CPU/RAM/disk
Mitigation:

• Check resource usage: ./scripts/check-proxmox-quota-ssh.sh
• Free up resources or move VMs
• Provider will retry (may succeed if resources freed)

Issue 6: Storage Full

Symptom: Error "storage full" or "disk creation failed"
Cause: Storage pool out of space
Mitigation:

• Check storage usage on Proxmox
• Free up space or expand storage
• Consider using Ceph for large disks

---

Deployment Order Recommendations

Critical Path (Must Deploy First)

1. Nginx Proxy VM (ML110-01)
2. Cloudflare Tunnel VM (R630-01)
3. DNS Primary (ML110-01)

High Priority (Deploy After Core)

4. Git Server (R630-01)
5. Email Server (R630-01)
6. DevOps Runner (R630-01)

Medium Priority

7. Codespaces IDE (R630-01)
8. AS4 Gateway (R630-01)
9. Business Integration Gateway (R630-01)
10. Financial Messaging Gateway (R630-01)

Blockchain Infrastructure

11. Validators (4x) - Deploy together
12. Sentries (4x) - Deploy together
13. RPC Nodes (4x) - Deploy together
14. Services (4x) - Deploy together

Low Priority (Test VMs)

15. Test/Example VMs (deploy only if resources allow)

---

Resource Allocation Summary

ML110-01 (Site-1)

• VMs: 6
• Total CPU: 6 cores (slightly exceeds 5 available)
• Total RAM: 12 GiB
• Total Disk: 90 GiB (local-lvm)

R630-01 (Site-2)

• VMs: 21
• Total CPU: 54 cores (slightly exceeds 50 available)
• Total RAM: 212 GiB
• Total Disk: 2,440 GiB (using ceph-fs)

---

Monitoring and Verification

Real-Time Monitoring

# Watch all VMs
kubectl get proxmoxvm -A -w

# Check specific VM
kubectl describe proxmoxvm <vm-name>

# View provider logs
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox -f

Verification Commands

# Check VM status
kubectl get proxmoxvm -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.vmId}{"\t"}{.status.state}{"\n"}{end}'

# Verify on Proxmox
./scripts/ssh-ml110-01.sh 'qm list'
./scripts/ssh-r630-01.sh 'qm list'

---

Summary

Total VMs: 30
Production VMs: 25
Test VMs: 5

Deployment Method: Crossplane Proxmox Provider
Authentication: Token-based (Cookie header)
Expected Timeline: 30-60 minutes for all VMs

Status: ✅ All configurations reviewed, deployment ready

---

Last Updated: 2025-12-13
Status: ✅ COMPREHENSIVE REVIEW COMPLETE