# Remaining Blockers - Complete Step-by-Step Guide

**Last Updated**: 2024-12-19

## Overview

This document provides detailed step-by-step instructions to resolve all remaining blockers for deployment.

## Priority Order

**Important**: Resolve blockers in this order for optimal workflow:

1. **SSH Access** (Blocker 2) - Required first to verify/download images
2. **Image Verification** (Blocker 3) - Depends on SSH, needed before VM deployment
3. **Kubernetes Cluster** (Blocker 1) - Needed for provider deployment and VM creation

**Rationale**: SSH access is needed to verify and download images, which must be ready before deploying VMs via Crossplane. Kubernetes can be set up in parallel, but images should be verified first.

## Blocker 2: SSH Access to Proxmox Nodes (PRIORITY 1)

**Resolve this first** - Required for image verification and download

### Required For
- TASK-009: Build and test Crossplane provider
- TASK-010: Deploy Crossplane provider to Kubernetes
- TASK-011: Create ProviderConfig resource
- TASK-014: Set up monitoring dashboards
- TASK-015: Deploy test VMs via Crossplane

### Step-by-Step Instructions

#### Option A: Using kind (Kubernetes in Docker) - Recommended for Local Development

**Step 1: Install kind**
```bash
# On Linux
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

# Verify installation
kind version
```

**Step 2: Install kubectl**
```bash
# On Linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

# Verify installation
kubectl version --client
```

**Step 3: Create kind cluster**
```bash
# Create cluster configuration
cat > kind-config.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: sankofa
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 30080
  - containerPort: 30443
    hostPort: 30443
EOF

# Create cluster
kind create cluster --name sankofa --config kind-config.yaml

# Verify cluster
kubectl cluster-info --context kind-sankofa
kubectl get nodes
```

**Step 4: Configure kubeconfig**
```bash
# Set context
kubectl config use-context kind-sankofa

# Verify
kubectl config current-context
```

**Step 5: Install Crossplane**
```bash
# Add Crossplane Helm repository
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

# Install Crossplane
helm install crossplane \
  crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace \
  --wait

# Verify installation
kubectl get pods -n crossplane-system
```

**Step 6: Verify cluster is ready**
```bash
# Check all pods are running
kubectl get pods --all-namespaces

# Test cluster connectivity
kubectl get nodes
```

#### Option B: Using minikube

**Step 1: Install minikube**
```bash
# On Linux
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Verify installation
minikube version
```

**Step 2: Start minikube**
```bash
# Start cluster
minikube start --driver=docker

# Verify
minikube status
kubectl get nodes
```

**Step 3: Install Crossplane** (same as kind, Step 5 above)

#### Option C: Using Existing Kubernetes Cluster

**Step 1: Verify cluster access**
```bash
# Check current context
kubectl config current-context

# Verify connectivity
kubectl cluster-info
kubectl get nodes
```

**Step 2: Install Crossplane** (same as kind, Step 5 above)

### Verification Checklist
- [ ] kind/minikube installed OR existing cluster accessible
- [ ] kubectl installed and configured
- [ ] Cluster created and nodes ready
- [ ] Crossplane installed in crossplane-system namespace
- [ ] All Crossplane pods running

---

## Blocker 2: SSH Access to Proxmox Nodes

### Required For
- TASK-012: Deploy Prometheus exporters
- TASK-013: Configure Cloudflare tunnels
- TASK-030: Generate Cloudflare tunnel credentials
- Image verification and download

### Step-by-Step Instructions

#### Step 1: Generate SSH Key Pair (if not exists)

```bash
# Generate SSH key
ssh-keygen -t ed25519 -C "sankofa-proxmox" -f ~/.ssh/sankofa_proxmox

# Or use existing key
# Skip this step if you already have an SSH key
```

#### Step 2: Copy Public Key to ML110-01

**Option A: Using ssh-copy-id**
```bash
# Copy key to ML110-01
ssh-copy-id -i ~/.ssh/sankofa_proxmox.pub root@192.168.11.10

# Test connection
ssh -i ~/.ssh/sankofa_proxmox root@192.168.11.10 'echo "SSH working"'
```

**Option B: Manual Copy**
```bash
# Display public key
cat ~/.ssh/sankofa_proxmox.pub

# SSH to ML110-01 with password
ssh root@192.168.11.10

# On ML110-01, add key to authorized_keys
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "YOUR_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
exit
```

#### Step 3: Copy Public Key to R630-01

```bash
# Copy key to R630-01
ssh-copy-id -i ~/.ssh/sankofa_proxmox.pub root@192.168.11.11

# Test connection
ssh -i ~/.ssh/sankofa_proxmox root@192.168.11.11 'echo "SSH working"'
```

#### Step 4: Configure SSH Config (Optional but Recommended)

```bash
# Add to ~/.ssh/config
cat >> ~/.ssh/config <<EOF

Host ml110-01
    HostName 192.168.11.10
    User root
    IdentityFile ~/.ssh/sankofa_proxmox
    StrictHostKeyChecking no

Host r630-01
    HostName 192.168.11.11
    User root
    IdentityFile ~/.ssh/sankofa_proxmox
    StrictHostKeyChecking no
EOF

# Test connections
ssh ml110-01 'hostname'
ssh r630-01 'hostname'
```

#### Step 5: Update .env with SSH Key Path (Optional)

```bash
# Add to .env file
echo "SSH_KEY=~/.ssh/sankofa_proxmox" >> .env
echo "SSH_USER=root" >> .env
```

### Verification Checklist
- [ ] SSH key pair generated
- [ ] Public key copied to ML110-01
- [ ] Public key copied to R630-01
- [ ] SSH connection works to ML110-01 (passwordless)
- [ ] SSH connection works to R630-01 (passwordless)
- [ ] SSH config file updated (optional)

---

## Blocker 1: Kubernetes Cluster Setup (PRIORITY 3)

**Can be done in parallel** - Needed for provider deployment and VM creation

### Required For
- TASK-015: Deploy test VMs via Crossplane
- All VM deployment tasks

### Step-by-Step Instructions

#### Step 1: Verify Images via SSH

```bash
# Check ML110-01
ssh ml110-01 'pveam list local | grep ubuntu'

# Check R630-01
ssh r630-01 'pveam list local | grep ubuntu'
```

#### Step 2: Download Images if Missing

**Option A: Using pveam (Proxmox Template Downloader)**

```bash
# On ML110-01
ssh ml110-01 <<EOF
# List available templates
pveam available | grep ubuntu-22.04

# Download Ubuntu 22.04 template
pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz

# Verify download
pveam list local | grep ubuntu-22.04
EOF

# On R630-01
ssh r630-01 <<EOF
# Download Ubuntu 22.04 template
pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz

# Verify download
pveam list local | grep ubuntu-22.04
EOF
```

**Option B: Download Cloud Image and Upload**

```bash
# Download Ubuntu 22.04 Cloud Image
wget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img

# Upload to ML110-01 via SCP
scp ubuntu-22.04-server-cloudimg-amd64.img ml110-01:/var/lib/vz/template/iso/

# Upload to R630-01 via SCP
scp ubuntu-22.04-server-cloudimg-amd64.img r630-01:/var/lib/vz/template/iso/

# On each node, rename if needed
ssh ml110-01 'mv /var/lib/vz/template/iso/ubuntu-22.04-server-cloudimg-amd64.img /var/lib/vz/template/iso/ubuntu-22.04-cloud.img'
ssh r630-01 'mv /var/lib/vz/template/iso/ubuntu-22.04-server-cloudimg-amd64.img /var/lib/vz/template/iso/ubuntu-22.04-cloud.img'
```

**Option C: Using Proxmox Web UI**

1. Log in to ML110-01: https://ml110-01.sankofa.nexus:8006
2. Go to: **Datacenter** → **Storage** → **local** → **Content**
3. Click **Templates** → **Download**
4. Search for: `ubuntu-22.04-standard`
5. Click **Download**
6. Repeat for R630-01

#### Step 3: Verify Image Names Match Manifests

```bash
# Check actual image names on ML110-01
ssh ml110-01 'pveam list local'

# Check actual image names on R630-01
ssh r630-01 'pveam list local'

# If image name differs from "ubuntu-22.04-cloud", update manifests:
# - crossplane-provider-proxmox/examples/test-vm-instance-1.yaml
# - crossplane-provider-proxmox/examples/test-vm-instance-2.yaml
# - crossplane-provider-proxmox/examples/vm-example.yaml
```

#### Step 4: Test Image (Optional)

```bash
# Create a test VM via Proxmox Web UI or API to verify image works
# This ensures the image is valid before using it with Crossplane
```

### Verification Checklist
- [ ] Images verified on ML110-01
- [ ] Images verified on R630-01
- [ ] Image names match manifest references (ubuntu-22.04-cloud)
- [ ] Images accessible from storage pools
- [ ] (Optional) Test VM created successfully

---

## Complete Deployment Sequence

Once all blockers are resolved, follow this sequence:

### Phase 1: SSH Setup (Blocker 2) - DO THIS FIRST
1. ✅ Generate SSH keys
2. ✅ Copy keys to both Proxmox nodes
3. ✅ Test passwordless SSH
4. ✅ Configure SSH config

### Phase 2: Image Preparation (Blocker 3) - DO THIS SECOND
1. ✅ Verify images exist
2. ✅ Download missing images
3. ✅ Verify image names match manifests
4. ✅ Test image (optional)

### Phase 3: Kubernetes Setup (Blocker 1) - CAN BE DONE IN PARALLEL
1. ✅ Install kind/minikube or access existing cluster
2. ✅ Install kubectl
3. ✅ Create/verify cluster
4. ✅ Install Crossplane
5. ✅ Verify Crossplane pods running

### Phase 2: SSH Setup (Blocker 2)
1. ✅ Generate SSH keys
2. ✅ Copy keys to both Proxmox nodes
3. ✅ Test passwordless SSH
4. ✅ Configure SSH config

### Phase 3: Image Preparation (Blocker 3)
1. ✅ Verify images exist
2. ✅ Download missing images
3. ✅ Verify image names match manifests
4. ✅ Test image (optional)

### Phase 4: Provider Deployment
1. Build Crossplane provider
2. Deploy provider to Kubernetes
3. Create ProviderConfig secret
4. Apply ProviderConfig
5. Verify provider connectivity

### Phase 5: Test Deployment
1. Deploy test VM on ML110-01
2. Deploy test VM on R630-01
3. Verify VM lifecycle operations

### Phase 6: Monitoring Setup
1. Deploy Prometheus exporters (via SSH)
2. Configure Grafana dashboards
3. Set up alerts

### Phase 7: Cloudflare Tunnels
1. Generate tunnel credentials
2. Deploy tunnels to nodes (via SSH)
3. Verify tunnel connectivity

## Quick Reference Commands

### Kubernetes
```bash
# Create cluster
kind create cluster --name sankofa

# Install Crossplane
helm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace

# Verify
kubectl get pods -n crossplane-system
```

### SSH
```bash
# Test connections
ssh ml110-01 'hostname'
ssh r630-01 'hostname'
```

### Images
```bash
# Check images
ssh ml110-01 'pveam list local | grep ubuntu'
ssh r630-01 'pveam list local | grep ubuntu'

# Download images
ssh ml110-01 'pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz'
ssh r630-01 'pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.gz'
```

## Troubleshooting

### Kubernetes Issues
- **Cluster won't start**: Check Docker is running (for kind/minikube)
- **Crossplane won't install**: Check cluster has sufficient resources
- **Pods not running**: Check `kubectl describe pod` for errors

### SSH Issues
- **Permission denied**: Verify public key is in `~/.ssh/authorized_keys`
- **Connection timeout**: Check firewall rules and network connectivity
- **Host key verification failed**: Add `StrictHostKeyChecking no` to SSH config

### Image Issues
- **Image not found**: Verify storage pool name and image location
- **Download fails**: Check internet connectivity and Proxmox template repository
- **Name mismatch**: Update manifests or rename image files

## Related Documentation

- [Deployment Readiness Final](./DEPLOYMENT_READINESS_FINAL.md)
- [Kubernetes Deployment Status](./KUBERNETES_DEPLOYMENT_STATUS.md)
- [Image Inventory](./IMAGE_INVENTORY.md)
- [Task List](./TASK_LIST.md)