Sankofa/docs/vm/DETAILED_VM_DEPLOYMENT_STEPS.md

# Detailed VM Deployment Steps Review

**Date**: 2025-12-13  
**Status**: Comprehensive Review of All VM Deployment Steps

---

## Executive Summary

This document provides a detailed step-by-step review of the deployment process for each of the 30 VMs in the Sankofa Phoenix infrastructure. Each VM goes through a standardized deployment process with specific steps that are executed by the Crossplane Proxmox provider.

---

## VM Deployment Process Overview

### Standard Deployment Flow

For each VM, the following steps are executed in order:

1. **Reconciliation Trigger**: Kubernetes controller detects new/updated ProxmoxVM resource
2. **Validation**: Validate VM specification (name, CPU, memory, disk, network, image)
3. **Authentication**: Authenticate to Proxmox API using token or username/password
4. **Site Resolution**: Resolve site configuration from ProviderConfig
5. **Node Health Check**: Verify target Proxmox node is healthy and reachable
6. **Network Validation**: Verify network bridge exists on target node
7. **VM Creation**: Execute VM creation on Proxmox
8. **Status Update**: Update Kubernetes resource with VMID and status
9. **Monitoring**: Continue monitoring VM state and IP address

---

## Detailed Deployment Steps by VM

### Phase 1: Core Infrastructure

#### 1.1 Nginx Proxy VM

**File**: `examples/production/nginx-proxy-vm.yaml`  
**Node**: ml110-01  
**Site**: site-1

**Configuration**:
- CPU: 2 cores
- Memory: 4 GiB
- Disk: 20 GiB
- Storage: local-lvm
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**:

1. **Reconciliation**:
   - Controller receives ProxmoxVM resource
   - Validates ProviderConfigReference exists

2. **Authentication**:
   - Reads credentials from `proxmox-credentials` secret
   - Detects token authentication (tokenid + token)
   - Creates Proxmox client with token: `root@pam!sankofa-instance-1-api-token=token`
   - Uses Cookie header: `PVEAuthCookie=root@pam!sankofa-instance-1-api-token=token`

3. **Site Resolution**:
   - Finds site-1 in ProviderConfig
   - Endpoint: `https://192.168.11.10:8006`
   - Node: `ml110-01`

4. **Node Health Check**:
   - Calls: `GET /nodes/ml110-01/status`
   - Verifies node is reachable and healthy
   - ⚠️ **Potential Issue**: If health check fails, VM creation is delayed

5. **Network Validation**:
   - Calls: `GET /nodes/ml110-01/network`
   - Verifies `vmbr0` bridge exists
   - ⚠️ **Potential Issue**: If bridge doesn't exist, VM creation fails

6. **VM Specification Validation**:
   - Validates VM name: `nginx-proxy-vm` (1-100 chars, alphanumeric)
   - Validates CPU: 2 (1-1024 range)
   - Validates Memory: 4Gi (128MB-2TB range)
   - Validates Disk: 20Gi (1GB-100TB range)
   - Validates Network: vmbr0 (valid bridge name)
   - Validates Image: `local:iso/ubuntu-22.04-cloud.img` (must exist in storage)

7. **Image Resolution**:
   - Searches for image in Proxmox storage
   - Looks for `ubuntu-22.04-cloud.img` in storage pools
   - Expected volid: `local:iso/ubuntu-22.04-cloud.img`
   - ⚠️ **Critical**: If image not found, VM creation fails

8. **VMID Assignment**:
   - Calls: `GET /cluster/nextid`
   - Gets next available VMID (e.g., 100, 101, 102...)
   - VMID assigned: `<next-available-id>`

9. **VM Configuration Creation**:
   - Creates VM config with:
     - `vmid`: Assigned VMID
     - `name`: nginx-proxy-vm
     - `cores`: 2
     - `memory`: 4096 (4Gi in MB)
     - `net0`: `virtio,bridge=vmbr0`
     - `scsi0`: `local-lvm:20,format=qcow2` (for cloud image import)
     - `ostype`: l26 (Linux 2.6+)
     - `agent`: 1 (QEMU guest agent enabled)
     - `boot`: `order=scsi0`

10. **Cloud-init Configuration**:
    - Adds cloud-init drive: `ide2=local-lvm:cloudinit`
    - Sets cloud-init user: `ciuser=admin`
    - Sets network config: `ipconfig0=ip=dhcp`
    - Writes userData to cloud-init drive

11. **VM Creation**:
    - Calls: `POST /nodes/ml110-01/qemu`
    - Creates VM with configuration
    - Returns: VMID if successful

12. **Image Import** (if needed):
    - If image is `.img` or `.qcow2`:
      - Stops VM (if running)
      - Calls: `POST /nodes/ml110-01/qemu/{vmid}/importdisk`
      - Imports image from `local:iso/ubuntu-22.04-cloud.img`
      - Waits for import to complete (with timeout)
      - ⚠️ **Critical**: If importdisk fails, VM is cleaned up

13. **Status Update**:
    - Updates ProxmoxVM status:
      - `vmId`: Assigned VMID
      - `state`: "created"
      - `conditions`: Ready=True
    - Requeues for status monitoring (10 seconds)

14. **VM State Monitoring**:
    - Subsequent reconciles check VM state
    - Calls: `GET /nodes/ml110-01/qemu/{vmid}/status/current`
    - Updates status.state (running, stopped, etc.)
    - Retrieves IP address from QEMU guest agent

**Expected Timeline**: 2-5 minutes

**Dependencies**: None

**Potential Issues**:
- Image not found in storage
- Network bridge doesn't exist
- Node health check fails
- Importdisk API not supported (Proxmox version)

---

#### 1.2 Cloudflare Tunnel VM

**File**: `examples/production/cloudflare-tunnel-vm.yaml`  
**Node**: r630-01  
**Site**: site-2

**Configuration**:
- CPU: 2 cores
- Memory: 4 GiB
- Disk: 10 GiB
- Storage: local-lvm
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Same as Nginx Proxy VM, but:
- Target node: r630-01
- Endpoint: `https://192.168.11.11:8006`
- Disk: 10 GiB (smaller)
- ⚠️ **Note**: Uses local-lvm on R630-01 (limited to 171.3 GB)

**Expected Timeline**: 2-5 minutes

**Dependencies**: None

---

### Phase 2: Phoenix Infrastructure

#### 2.1 DNS Primary

**File**: `examples/production/phoenix/dns-primary.yaml`  
**Node**: ml110-01  
**Site**: site-1

**Configuration**:
- CPU: 2 cores (reduced from 4)
- Memory: 4 GiB (reduced from 8Gi)
- Disk: 50 GiB
- Storage: local-lvm
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 2-5 minutes

**Dependencies**: None (but other services may depend on DNS)

---

#### 2.2 Git Server

**File**: `examples/production/phoenix/git-server.yaml`  
**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration**:
- CPU: 4 cores (reduced from 8)
- Memory: 16 GiB
- Disk: 500 GiB (large disk)
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process, but:
- ⚠️ **Large Disk**: 500 GiB requires Ceph storage
- ⚠️ **Storage Check**: Verifies ceph-fs storage exists
- ⚠️ **Image Import**: Large image import may take longer

**Expected Timeline**: 5-10 minutes (due to large disk)

**Dependencies**: None

**Optimization Notes**:
- Moved to R630-01 for better CPU resources
- Using ceph-fs for large disk (avoids local-lvm constraint)

---

#### 2.3 Email Server

**File**: `examples/production/phoenix/email-server.yaml`  
**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration**:
- CPU: 4 cores (reduced from 8)
- Memory: 16 GiB
- Disk: 200 GiB
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 3-7 minutes

**Dependencies**: None

---

#### 2.4 DevOps Runner

**File**: `examples/production/phoenix/devops-runner.yaml`  
**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration**:
- CPU: 4 cores (reduced from 8)
- Memory: 16 GiB
- Disk: 200 GiB
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 3-7 minutes

**Dependencies**: None

---

#### 2.5 Codespaces IDE

**File**: `examples/production/phoenix/codespaces-ide.yaml`  
**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration**:
- CPU: 4 cores (reduced from 8)
- Memory: 32 GiB (reduced from 32Gi, unchanged)
- Disk: 200 GiB
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 3-7 minutes

**Dependencies**: None

**Note**: High memory requirement (32 GiB)

---

#### 2.6 AS4 Gateway

**File**: `examples/production/phoenix/as4-gateway.yaml`  
**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration**:
- CPU: 4 cores (reduced from 8)
- Memory: 16 GiB
- Disk: 500 GiB (large disk)
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 5-10 minutes (due to large disk)

**Dependencies**: None

---

#### 2.7 Business Integration Gateway

**File**: `examples/production/phoenix/business-integration-gateway.yaml`  
**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration**:
- CPU: 4 cores (reduced from 8)
- Memory: 16 GiB
- Disk: 200 GiB
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 3-7 minutes

**Dependencies**: None

---

#### 2.8 Financial Messaging Gateway

**File**: `examples/production/phoenix/financial-messaging-gateway.yaml`  
**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration**:
- CPU: 4 cores (reduced from 8)
- Memory: 16 GiB
- Disk: 500 GiB (large disk)
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 5-10 minutes (due to large disk)

**Dependencies**: None

---

### Phase 3: Blockchain Infrastructure

#### 3.1 Validators (4x)

**Files**: 
- `examples/production/smom-dbis-138/validator-01.yaml`
- `examples/production/smom-dbis-138/validator-02.yaml`
- `examples/production/smom-dbis-138/validator-03.yaml`
- `examples/production/smom-dbis-138/validator-04.yaml`

**Node**: r630-01 (moved from ml110-01)  
**Site**: site-2

**Configuration** (per validator):
- CPU: 3 cores (reduced from 6)
- Memory: 12 GiB (reduced from 12Gi, unchanged)
- Disk: 20 GiB
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process for each validator

**Expected Timeline**: 2-5 minutes per validator (8-20 minutes total)

**Dependencies**: None (but validators should be deployed together)

**Total Resources**:
- CPU: 12 cores (4 × 3)
- Memory: 48 GiB (4 × 12 GiB)
- Disk: 80 GiB (4 × 20 GiB)

---

#### 3.2 Sentries (4x)

**Files**:
- `examples/production/smom-dbis-138/sentry-01.yaml` (ml110-01)
- `examples/production/smom-dbis-138/sentry-02.yaml` (ml110-01)
- `examples/production/smom-dbis-138/sentry-03.yaml` (r630-01)
- `examples/production/smom-dbis-138/sentry-04.yaml` (r630-01)

**Configuration** (per sentry):
- CPU: 2 cores (reduced from 4)
- Memory: 4 GiB (reduced from 8Gi)
- Disk: 20 GiB
- Storage: local-lvm (sentries 01-02), ceph-fs (sentries 03-04)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 2-5 minutes per sentry

**Dependencies**: None

**Total Resources**:
- ML110-01: 4 CPU, 8 GiB RAM, 40 GiB disk
- R630-01: 4 CPU, 8 GiB RAM, 40 GiB disk

---

#### 3.3 RPC Nodes (4x)

**Files**:
- `examples/production/smom-dbis-138/rpc-node-01.yaml`
- `examples/production/smom-dbis-138/rpc-node-02.yaml`
- `examples/production/smom-dbis-138/rpc-node-03.yaml`
- `examples/production/smom-dbis-138/rpc-node-04.yaml`

**Node**: r630-01  
**Site**: site-2

**Configuration** (per RPC node):
- CPU: 2 cores (reduced from 4)
- Memory: 4 GiB (reduced from 8Gi)
- Disk: 20 GiB
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 2-5 minutes per node (8-20 minutes total)

**Dependencies**: None

**Total Resources**:
- CPU: 8 cores (4 × 2)
- Memory: 16 GiB (4 × 4 GiB)
- Disk: 80 GiB (4 × 20 GiB)

---

#### 3.4 Services (4x)

**Files**:
- `examples/production/smom-dbis-138/management.yaml`
- `examples/production/smom-dbis-138/monitoring.yaml`
- `examples/production/smom-dbis-138/services.yaml`
- `examples/production/smom-dbis-138/blockscout.yaml`

**Node**: r630-01  
**Site**: site-2

**Configuration** (per service):
- CPU: 2 cores (reduced from 4)
- Memory: 4 GiB (reduced from 8Gi)
- Disk: 20 GiB
- Storage: ceph-fs (changed from local-lvm)
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 2-5 minutes per service (8-20 minutes total)

**Dependencies**: None

**Total Resources**:
- CPU: 8 cores (4 × 2)
- Memory: 16 GiB (4 × 4 GiB)
- Disk: 80 GiB (4 × 20 GiB)

---

### Phase 4: Test/Example VMs

#### 4.1 Basic VM

**File**: `examples/production/basic-vm.yaml`  
**Node**: ml110-01  
**Site**: site-1

**Configuration**:
- CPU: 2 cores
- Memory: 4 GiB
- Disk: 50 GiB
- Storage: local-lvm
- Network: vmbr0
- Image: local:iso/ubuntu-22.04-cloud.img
- Cloud-init: ✅ Configured

**Deployment Steps**: Standard process

**Expected Timeline**: 2-5 minutes

**Priority**: LOW (test VM)

---

#### 4.2 VM-100

**File**: `examples/production/vm-100.yaml`  
**Node**: ml110-01  
**Site**: site-1

**Configuration**: Similar to basic-vm

**Deployment Steps**: Standard process

**Expected Timeline**: 2-5 minutes

**Priority**: LOW (test VM)

---

#### 4.3 Medium VM

**File**: `examples/production/medium-vm.yaml`  
**Status**: ⚠️ Error reading configuration

**Issue**: YAML file may have syntax errors or missing fields

**Action Required**: Review and fix YAML file

---

#### 4.4 Large VM

**File**: `examples/production/large-vm.yaml`  
**Status**: ⚠️ Error reading configuration

**Issue**: YAML file may have syntax errors or missing fields

**Action Required**: Review and fix YAML file

---

#### 4.5 Blockscout

**File**: `examples/production/smom-dbis-138/blockscout.yaml`  
**Status**: ⚠️ Error reading configuration

**Issue**: YAML file may have syntax errors or missing fields

**Action Required**: Review and fix YAML file

---

## Detailed Deployment Process Flow

### Step-by-Step Execution

For each VM, the controller executes these steps in sequence:

#### Step 1: Resource Validation
```go
// Validate VM name (1-100 chars, alphanumeric + hyphens/underscores)
utils.ValidateVMName(vm.Spec.ForProvider.Name)

// Validate CPU (1-1024)
utils.ValidateCPU(vm.Spec.ForProvider.CPU)

// Validate Memory (128MB-2TB, supports Gi/Mi/Ki)
utils.ValidateMemory(vm.Spec.ForProvider.Memory)

// Validate Disk (1GB-100TB, supports Ti/Gi/Mi)
utils.ValidateDisk(vm.Spec.ForProvider.Disk)

// Validate Network bridge name
utils.ValidateNetworkBridge(vm.Spec.ForProvider.Network)

// Validate Image specification
utils.ValidateImageSpec(vm.Spec.ForProvider.Image)
```

#### Step 2: Authentication
```go
// Read credentials from secret
creds := getCredentials(providerConfig)

// Create Proxmox client
if creds.Token != "" {
    // Token authentication
    token := fmt.Sprintf("%s=%s", creds.Username, creds.Token)
    client := NewClientWithToken(endpoint, token, insecureSkipTLS)
} else {
    // Username/password authentication
    client := NewClient(endpoint, username, password, insecureSkipTLS)
}
```

#### Step 3: Node Health Check
```go
// Check node is reachable and healthy
client.CheckNodeHealth(ctx, node)
// Calls: GET /nodes/{node}/status
```

#### Step 4: Network Validation
```go
// Verify network bridge exists
client.NetworkExists(ctx, node, network)
// Calls: GET /nodes/{node}/network
// Searches for bridge in network list
```

#### Step 5: VMID Assignment
```go
// Get next available VMID
client.Get(ctx, "/cluster/nextid")
// Returns: Next available VMID (e.g., "100", "101")
```

#### Step 6: Image Resolution
```go
// Search for image in storage
if image contains ":" {
    // Already a volid (storage:path)
    imageVolid = image
} else {
    // Search storage pools
    client.findImageInStorage(ctx, node, image)
    // Searches all storages for matching image
}
```

#### Step 7: VM Configuration
```go
vmConfig := {
    "vmid": vmID,
    "name": vmName,
    "cores": cpu,
    "memory": parseMemory(memory), // Convert to MB
    "net0": fmt.Sprintf("virtio,bridge=%s", network),
    "scsi0": diskConfig, // Format: storage:size,format=qcow2
    "ostype": "l26",
    "agent": "1",
    "boot": "order=scsi0"
}

// Add cloud-init if userData provided
if userData != "" {
    vmConfig["ide2"] = fmt.Sprintf("%s:cloudinit", storage)
    vmConfig["ciuser"] = "admin"
    vmConfig["ipconfig0"] = "ip=dhcp"
}
```

#### Step 8: VM Creation
```go
// Create VM on Proxmox
client.Post(ctx, fmt.Sprintf("/nodes/%s/qemu", node), vmConfig)
// Calls: POST /nodes/{node}/qemu
// Returns: VMID if successful
```

#### Step 9: Image Import (if needed)
```go
if needsImageImport && imageVolid != "" {
    // Stop VM if running
    client.Post(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/status/stop", node, vmID))
    
    // Wait for VM to stop (up to 30 seconds)
    waitForVMStop(ctx, node, vmID, 30*time.Second)
    
    // Import image
    client.Post(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/importdisk", node, vmID), {
        "storage": storageName,
        "filename": imagePath,
        "format": "qcow2"
    })
    
    // Monitor import task until completion
    monitorTask(ctx, taskID, timeout)
}
```

#### Step 10: Status Update
```go
// Update ProxmoxVM status
vm.Status.VMID = createdVM.ID
vm.Status.State = "created"
vm.Status.Conditions = append(conditions, {
    Type: "Ready",
    Status: "True",
    Reason: "Created",
    Message: fmt.Sprintf("VM %d created successfully", vmID)
})

// Update Kubernetes resource
r.Status().Update(ctx, &vm)

// Requeue for monitoring (10 seconds)
return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
```

#### Step 11: State Monitoring
```go
// Subsequent reconciles check VM state
client.Get(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/status/current", node, vmID))
// Updates: vm.Status.State (running, stopped, etc.)

// Get IP address from QEMU guest agent
client.Get(ctx, fmt.Sprintf("/nodes/%s/qemu/%d/agent/network-get-interfaces", node, vmID))
// Updates: vm.Status.IPAddress
```

---

## Potential Issues and Mitigations

### Issue 1: Image Not Found
**Symptom**: Error "image 'X' not found in storage"  
**Cause**: Image not uploaded to Proxmox storage  
**Mitigation**: 
- Verify image exists: `./scripts/verify-image-availability.sh`
- Upload image if missing
- Provider will retry automatically

### Issue 2: Network Bridge Missing
**Symptom**: Error "network bridge 'X' does not exist"  
**Cause**: Network bridge not configured on Proxmox node  
**Mitigation**:
- Verify bridge exists: Check Proxmox network configuration
- Create bridge if missing
- Provider will fail fast (no retry)

### Issue 3: Node Health Check Fails
**Symptom**: Error "node X is not reachable or unhealthy"  
**Cause**: Node down, network issue, or authentication failure  
**Mitigation**:
- Check node status on Proxmox
- Verify network connectivity
- Check authentication credentials
- Provider retries with exponential backoff

### Issue 4: Importdisk Not Supported
**Symptom**: Error "importdisk API not supported"  
**Cause**: Proxmox version too old (< 6.2)  
**Mitigation**:
- Upgrade Proxmox to 6.2+
- Or use template-based deployment instead
- Provider detects and cleans up VM

### Issue 5: Resource Exhaustion
**Symptom**: Error "insufficient resources"  
**Cause**: Node out of CPU/RAM/disk  
**Mitigation**:
- Check resource usage: `./scripts/check-proxmox-quota-ssh.sh`
- Free up resources or move VMs
- Provider will retry (may succeed if resources freed)

### Issue 6: Storage Full
**Symptom**: Error "storage full" or "disk creation failed"  
**Cause**: Storage pool out of space  
**Mitigation**:
- Check storage usage on Proxmox
- Free up space or expand storage
- Consider using Ceph for large disks

---

## Deployment Order Recommendations

### Critical Path (Must Deploy First)
1. Nginx Proxy VM (ML110-01)
2. Cloudflare Tunnel VM (R630-01)
3. DNS Primary (ML110-01)

### High Priority (Deploy After Core)
4. Git Server (R630-01)
5. Email Server (R630-01)
6. DevOps Runner (R630-01)

### Medium Priority
7. Codespaces IDE (R630-01)
8. AS4 Gateway (R630-01)
9. Business Integration Gateway (R630-01)
10. Financial Messaging Gateway (R630-01)

### Blockchain Infrastructure
11. Validators (4x) - Deploy together
12. Sentries (4x) - Deploy together
13. RPC Nodes (4x) - Deploy together
14. Services (4x) - Deploy together

### Low Priority (Test VMs)
15. Test/Example VMs (deploy only if resources allow)

---

## Resource Allocation Summary

### ML110-01 (Site-1)
- **VMs**: 6
- **Total CPU**: 6 cores (slightly exceeds 5 available)
- **Total RAM**: 12 GiB
- **Total Disk**: 90 GiB (local-lvm)

### R630-01 (Site-2)
- **VMs**: 21
- **Total CPU**: 54 cores (slightly exceeds 50 available)
- **Total RAM**: 212 GiB
- **Total Disk**: 2,440 GiB (using ceph-fs)

---

## Monitoring and Verification

### Real-Time Monitoring
```bash
# Watch all VMs
kubectl get proxmoxvm -A -w

# Check specific VM
kubectl describe proxmoxvm <vm-name>

# View provider logs
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox -f
```

### Verification Commands
```bash
# Check VM status
kubectl get proxmoxvm -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.vmId}{"\t"}{.status.state}{"\n"}{end}'

# Verify on Proxmox
./scripts/ssh-ml110-01.sh 'qm list'
./scripts/ssh-r630-01.sh 'qm list'
```

---

## Summary

**Total VMs**: 30  
**Production VMs**: 25  
**Test VMs**: 5  

**Deployment Method**: Crossplane Proxmox Provider  
**Authentication**: Token-based (Cookie header)  
**Expected Timeline**: 30-60 minutes for all VMs  

**Status**: ✅ All configurations reviewed, deployment ready

---

**Last Updated**: 2025-12-13  
**Status**: ✅ **COMPREHENSIVE REVIEW COMPLETE**