proxmox/docs/03-deployment/PHOENIX_DEPLOYMENT_RUNBOOK.md

# Phoenix Deployment Runbook

**Target System:** Phoenix Core (VLAN 160)  
**Target Host:** r630-01 (192.168.11.11)  
**VMID Range:** 8600-8699  
**Version:** 1.0.0  
**Last Updated:** 2026-01-09
**Status:** Active Documentation

---

## Decision Summary

Phoenix Core uses **VMID range 8600-8699** (not 7800-7803) to avoid conflicts with existing legacy containers. This enables parallel deployment with DNS-based cutover.

**Phoenix Core Components:**
- VMID 8600: Phoenix API (10.160.0.10)
- VMID 8601: Phoenix Portal (10.160.0.11)
- VMID 8602: Phoenix Keycloak (10.160.0.12)
- VMID 8603: Phoenix PostgreSQL (10.160.0.13)

---

## Table of Contents

1. [Pre-Flight Checks](#pre-flight-checks)
2. [Network Readiness Verification](#network-readiness-verification)
3. [Phase 1: PostgreSQL Deployment (VMID 8603)](#phase-1-postgresql-deployment-vmid-8603)
4. [Phase 2: Keycloak Deployment (VMID 8602)](#phase-2-keycloak-deployment-vmid-8602)
5. [Phase 3: Phoenix API Deployment (VMID 8600)](#phase-3-phoenix-api-deployment-vmid-8600)
6. [Phase 4: Phoenix Portal Deployment (VMID 8601)](#phase-4-phoenix-portal-deployment-vmid-8601)
7. [Validation Gates](#validation-gates)
8. [Troubleshooting](#troubleshooting)
9. [Rollback Procedures](#rollback-procedures)

---

## Pre-Flight Checks

Before starting deployment, verify the following prerequisites:

### 1. SSH Access to r630-01

```bash
ssh root@192.168.11.11
```

**Verification:**
```bash
ssh -o StrictHostKeyChecking=no root@192.168.11.11 "pvecm status >/dev/null 2>&1 && echo '✓ Connected' || echo '✗ Connection failed'"
```

### 2. Storage Availability

```bash
ssh root@192.168.11.11 "pvesm status | grep thin1"
```

**Expected:** thin1 storage available with sufficient space (minimum 180GB free for all 4 containers).

### 3. Source Project Availability

```bash
ls -la /home/intlc/projects/Sankofa/api
ls -la /home/intlc/projects/Sankofa/portal
```

**Required:** Both `api/` and `portal/` directories must exist.

### 4. VMID Availability

```bash
ssh root@192.168.11.11 "pct list | grep -E '^860[0-3]'"
```

**Expected:** No containers with VMIDs 8600-8603 should exist.

### 5. IP Address Availability

```bash
ssh root@192.168.11.11 "pct list | grep -E '10\.160\.0\.(10|11|12|13)'"
```

**Expected:** IPs 10.160.0.10-13 should not be in use.

---

## Network Readiness Verification

### Step 1: Verify VLAN 160 Configuration

```bash
# Check if VLAN 160 exists on the switch/router
ssh root@192.168.11.1 "ip addr show | grep '160' || echo 'VLAN 160 not configured'"
```

**Expected:** VLAN 160 interface should exist on the gateway/router.

### Step 2: Verify Proxmox Bridge Configuration

```bash
# Check bridge configuration
ssh root@192.168.11.11 "cat /etc/network/interfaces | grep -A 5 vmbr0"
```

**Expected:** Bridge should support VLAN tagging.

**If VLAN-aware bridge needed:**
```bash
ssh root@192.168.11.11 "cat /etc/network/interfaces.d/vmbr0"
# Should contain: bridge-vlan-aware yes
```

### Step 3: Verify Gateway Accessibility

```bash
# Test gateway connectivity
ping -c 3 10.160.0.1
```

**Expected:** Gateway (10.160.0.1) should respond to ping.

### Step 4: Verify IP Addresses Not in Use

```bash
# Test each IP
for ip in 10.160.0.10 10.160.0.11 10.160.0.12 10.160.0.13; do
    ping -c 1 -W 1 $ip 2>&1 | grep -q "100% packet loss" && echo "$ip: Available" || echo "$ip: In use"
done
```

**Expected:** All IPs should show "Available".

---

## Phase 1: PostgreSQL Deployment (VMID 8603)

**Order:** Must be deployed first (database is required by other services)

### Step 1: Create Container

```bash
# On r630-01, create PostgreSQL container
ssh root@192.168.11.11 "pct create 8603 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-postgres-1 \
    --memory 2048 \
    --cores 2 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.13/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"
```

### Step 2: Start Container

```bash
ssh root@192.168.11.11 "pct start 8603"
sleep 10
```

### Step 3: Verify Container Status

```bash
ssh root@192.168.11.11 "pct status 8603"
```

**Expected:** Status should be "running".

### Step 4: Install PostgreSQL 16

```bash
ssh root@192.168.11.11 "pct exec 8603 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    apt-get update -qq && \
    apt-get install -y -qq wget ca-certificates gnupg lsb-release curl git build-essential sudo'"

ssh root@192.168.11.11 "pct exec 8603 -- bash -c 'wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - && \
    echo \"deb http://apt.postgresql.org/pub/repos/apt \$(lsb_release -cs)-pgdg main\" > /etc/apt/sources.list.d/pgdg.list && \
    apt-get update -qq && \
    apt-get install -y -qq postgresql-16 postgresql-contrib-16'"
```

### Step 5: Configure PostgreSQL

```bash
# Generate secure password
DB_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-24)
echo "Generated DB_PASSWORD: $DB_PASSWORD"
# Save this password - you'll need it for the next steps!

# Enable and start PostgreSQL
ssh root@192.168.11.11 "pct exec 8603 -- systemctl enable postgresql && \
    pct exec 8603 -- systemctl start postgresql"

# Wait for PostgreSQL to start
sleep 5

# Create database and user
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"sudo -u postgres psql << 'EOF'
CREATE USER phoenix WITH PASSWORD '$DB_PASSWORD';
CREATE DATABASE phoenix OWNER phoenix ENCODING 'UTF8';
GRANT ALL PRIVILEGES ON DATABASE phoenix TO phoenix;
\\c phoenix
GRANT ALL ON SCHEMA public TO phoenix;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO phoenix;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO phoenix;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO phoenix;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
CREATE EXTENSION IF NOT EXISTS \"pg_stat_statements\";
EOF\""
```

### Step 6: Configure Network Access

```bash
# Allow connections from VLAN 160 subnet
ssh root@192.168.11.11 "pct exec 8603 -- bash -c 'echo \"host    all             all             10.160.0.0/22           md5\" >> /etc/postgresql/16/main/pg_hba.conf'"

# Enable network listening
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"sed -i \\\"s/#listen_addresses = 'localhost'/listen_addresses = '*'/\\\" /etc/postgresql/16/main/postgresql.conf\""

# Restart PostgreSQL
ssh root@192.168.11.11 "pct exec 8603 -- systemctl restart postgresql"
sleep 3
```

### Step 7: Verify PostgreSQL

```bash
# Test connection
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"PGPASSWORD='$DB_PASSWORD' psql -h localhost -U phoenix -d phoenix -c 'SELECT version();'\""
```

**Expected:** Should return PostgreSQL version information.

---

## Phase 2: Keycloak Deployment (VMID 8602)

**Order:** Deploy after PostgreSQL (requires database)

### Step 1: Create Container

```bash
ssh root@192.168.11.11 "pct create 8602 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-keycloak-1 \
    --memory 2048 \
    --cores 2 \
    --rootfs thin1:30 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.12/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"
```

### Step 2: Start Container and Install Dependencies

```bash
ssh root@192.168.11.11 "pct start 8602"
sleep 10

# Install Java 21 and dependencies
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    apt-get update -qq && \
    apt-get install -y -qq openjdk-21-jdk wget curl unzip'"

# Set JAVA_HOME
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'echo \"export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64\" >> /etc/profile'"
```

### Step 3: Create Keycloak Database

```bash
# Generate Keycloak database password
KEYCLOAK_DB_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-24)
echo "Generated KEYCLOAK_DB_PASSWORD: $KEYCLOAK_DB_PASSWORD"

# Create database on PostgreSQL container (8603)
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"sudo -u postgres psql << 'EOF'
CREATE USER keycloak WITH PASSWORD '$KEYCLOAK_DB_PASSWORD';
CREATE DATABASE keycloak OWNER keycloak ENCODING 'UTF8';
GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak;
EOF\""
```

### Step 4: Download and Install Keycloak

```bash
# Download Keycloak 24.0.0
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'cd /opt && \
    wget -q https://github.com/keycloak/keycloak/releases/download/24.0.0/keycloak-24.0.0.tar.gz && \
    tar -xzf keycloak-24.0.0.tar.gz && \
    mv keycloak-24.0.0 keycloak && \
    rm keycloak-24.0.0.tar.gz && \
    chmod +x keycloak/bin/kc.sh'"

# Build Keycloak (may take several minutes)
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'cd /opt/keycloak && \
    export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 && \
    ./bin/kc.sh build --db postgres'"
```

### Step 5: Configure Keycloak Service

```bash
# Generate admin password
KEYCLOAK_ADMIN_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-24)
echo "Generated KEYCLOAK_ADMIN_PASSWORD: $KEYCLOAK_ADMIN_PASSWORD"

# Generate client secrets
KEYCLOAK_CLIENT_SECRET_API=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-32)
KEYCLOAK_CLIENT_SECRET_PORTAL=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | cut -c1-32)

# Create systemd service
ssh root@192.168.11.11 "pct exec 8602 -- bash -c \"cat > /etc/systemd/system/keycloak.service << 'EOF'
[Unit]
Description=Keycloak Authorization Server
After=network.target

[Service]
Type=idle
User=root
WorkingDirectory=/opt/keycloak
Environment=\\\"JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64\\\"
Environment=\\\"KC_DB=postgres\\\"
Environment=\\\"KC_DB_URL_HOST=10.160.0.13\\\"
Environment=\\\"KC_DB_URL_DATABASE=keycloak\\\"
Environment=\\\"KC_DB_USERNAME=keycloak\\\"
Environment=\\\"KC_DB_PASSWORD=$KEYCLOAK_DB_PASSWORD\\\"
Environment=\\\"KC_HTTP_ENABLED=true\\\"
Environment=\\\"KC_HOSTNAME_STRICT=false\\\"
Environment=\\\"KC_HOSTNAME_PORT=8080\\\"
Environment=\\\"KC_HTTP_PORT=8080\\\"
ExecStart=/opt/keycloak/bin/kc.sh start --optimized
ExecStop=/bin/kill -TERM \\\$MAINPID
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF\""

# Start Keycloak
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'systemctl daemon-reload && \
    systemctl enable keycloak && \
    systemctl start keycloak'"

# Wait for Keycloak to start (may take 1-2 minutes)
echo "Waiting for Keycloak to start..."
sleep 60

# Check if Keycloak is ready
for i in {1..30}; do
    if ssh root@192.168.11.11 "pct exec 8602 -- curl -s -f http://localhost:8080/health/ready >/dev/null 2>&1"; then
        echo "✓ Keycloak is ready"
        break
    fi
    echo "Waiting for Keycloak... ($i/30)"
    sleep 5
done
```

### Step 6: Create Admin User and Clients

```bash
# Create admin user (first-time setup only)
ssh root@192.168.11.11 "pct exec 8602 -- bash -c 'cd /opt/keycloak && \
    export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 && \
    ./bin/kc.sh config credentials --server http://localhost:8080 --realm master --user admin --password admin 2>/dev/null || \
    ./bin/kc.sh add-user-keycloak --realm master --username admin --password $KEYCLOAK_ADMIN_PASSWORD'"

# Wait for Keycloak to fully start
sleep 30

# Get admin token and create clients
ssh root@192.168.11.11 "pct exec 8602 -- bash -c \"
TOKEN=\\\$(curl -s -X POST \\\"http://localhost:8080/realms/master/protocol/openid-connect/token\\\" \\
  -H \\\"Content-Type: application/x-www-form-urlencoded\\\" \\
  -d \\\"username=admin\\\" \\
  -d \\\"password=\\\$KEYCLOAK_ADMIN_PASSWORD\\\" \\
  -d \\\"grant_type=password\\\" \\
  -d \\\"client_id=admin-cli\\\" | jq -r '.access_token')

# Create phoenix-api client
curl -s -X POST \\\"http://localhost:8080/admin/realms/master/clients\\\" \\
  -H \\\"Authorization: Bearer \\\$TOKEN\\\" \\
  -H \\\"Content-Type: application/json\\\" \\
  -d '{
    \\\"clientId\\\": \\\"phoenix-api\\\",
    \\\"enabled\\\": true,
    \\\"clientAuthenticatorType\\\": \\\"client-secret\\\",
    \\\"secret\\\": \\\"$KEYCLOAK_CLIENT_SECRET_API\\\",
    \\\"protocol\\\": \\\"openid-connect\\\",
    \\\"publicClient\\\": false,
    \\\"standardFlowEnabled\\\": true,
    \\\"directAccessGrantsEnabled\\\": true,
    \\\"serviceAccountsEnabled\\\": true
  }'

# Create portal-client
curl -s -X POST \\\"http://localhost:8080/admin/realms/master/clients\\\" \\
  -H \\\"Authorization: Bearer \\\$TOKEN\\\" \\
  -H \\\"Content-Type: application/json\\\" \\
  -d '{
    \\\"clientId\\\": \\\"portal-client\\\",
    \\\"enabled\\\": true,
    \\\"clientAuthenticatorType\\\": \\\"client-secret\\\",
    \\\"secret\\\": \\\"$KEYCLOAK_CLIENT_SECRET_PORTAL\\\",
    \\\"protocol\\\": \\\"openid-connect\\\",
    \\\"publicClient\\\": false,
    \\\"standardFlowEnabled\\\": true,
    \\\"directAccessGrantsEnabled\\\": true
  }'
\""
```

**Note:** Save these passwords and secrets:
- `KEYCLOAK_ADMIN_PASSWORD`
- `KEYCLOAK_CLIENT_SECRET_API`
- `KEYCLOAK_CLIENT_SECRET_PORTAL`
- `KEYCLOAK_DB_PASSWORD`

---

## Phase 3: Phoenix API Deployment (VMID 8600)

**Order:** Deploy after PostgreSQL and Keycloak

### Step 1: Create Container

```bash
ssh root@192.168.11.11 "pct create 8600 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-api-1 \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.10/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"
```

### Step 2: Start Container and Install Node.js

```bash
ssh root@192.168.11.11 "pct start 8600"
sleep 10

# Install Node.js 18
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
    apt-get install -y -qq nodejs'"

# Install pnpm
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'npm install -g pnpm'"
```

### Step 3: Copy API Project Files

```bash
# Create app directory
ssh root@192.168.11.11 "pct exec 8600 -- mkdir -p /opt/phoenix-api"

# Copy API directory (assuming source is on deployment machine)
# If source is on r630-01, adjust path accordingly
# If source is remote, use rsync or scp
rsync -avz --exclude node_modules --exclude .git \
    /home/intlc/projects/Sankofa/api/ \
    root@192.168.11.11:/tmp/phoenix-api-source/

ssh root@192.168.11.11 "pct push 8600 /tmp/phoenix-api-source /opt/phoenix-api --recursive"
```

### Step 4: Install Dependencies and Configure

```bash
# Install dependencies
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'cd /opt/phoenix-api && pnpm install --frozen-lockfile'"

# Create environment file (use the passwords/secrets generated earlier)
ssh root@192.168.11.11 "pct exec 8600 -- bash -c \"cat > /opt/phoenix-api/.env << 'EOF'
# Database
DB_HOST=10.160.0.13
DB_PORT=5432
DB_NAME=phoenix
DB_USER=phoenix
DB_PASSWORD=$DB_PASSWORD

# Keycloak
KEYCLOAK_URL=http://10.160.0.12:8080
KEYCLOAK_REALM=master
KEYCLOAK_CLIENT_ID=phoenix-api
KEYCLOAK_CLIENT_SECRET=$KEYCLOAK_CLIENT_SECRET_API
KEYCLOAK_MULTI_REALM=false

# API
API_PORT=4000
JWT_SECRET=$(openssl rand -base64 32)
NODE_ENV=production

# Multi-Tenancy
ENABLE_MULTI_TENANT=true
EOF\""
```

### Step 5: Run Migrations and Build

```bash
# Run database migrations
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'cd /opt/phoenix-api && pnpm db:migrate'"

# Build API
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'cd /opt/phoenix-api && pnpm build'"
```

### Step 6: Create Systemd Service

```bash
ssh root@192.168.11.11 "pct exec 8600 -- bash -c \"cat > /etc/systemd/system/phoenix-api.service << 'EOF'
[Unit]
Description=Phoenix API Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/phoenix-api
Environment=\\\"NODE_ENV=production\\\"
EnvironmentFile=/opt/phoenix-api/.env
ExecStart=/usr/bin/node /opt/phoenix-api/dist/server.js
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF\""

# Start service
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'systemctl daemon-reload && \
    systemctl enable phoenix-api && \
    systemctl start phoenix-api'"

sleep 10

# Verify service is running
ssh root@192.168.11.11 "pct exec 8600 -- systemctl status phoenix-api --no-pager | head -10"
```

---

## Phase 4: Phoenix Portal Deployment (VMID 8601)

**Order:** Deploy last (depends on API)

### Step 1: Create Container

```bash
ssh root@192.168.11.11 "pct create 8601 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname phoenix-portal-1 \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.11/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --swap 512 \
    --onboot 1 \
    --timezone America/Los_Angeles \
    --features nesting=1,keyctl=1"
```

### Step 2: Start Container and Install Node.js

```bash
ssh root@192.168.11.11 "pct start 8601"
sleep 10

# Install Node.js 18
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'export DEBIAN_FRONTEND=noninteractive && \
    curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
    apt-get install -y -qq nodejs'"

# Install pnpm
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'npm install -g pnpm'"
```

### Step 3: Copy Portal Project Files

```bash
# Copy portal directory
rsync -avz --exclude node_modules --exclude .git --exclude .next \
    /home/intlc/projects/Sankofa/portal/ \
    root@192.168.11.11:/tmp/phoenix-portal-source/

ssh root@192.168.11.11 "pct push 8601 /tmp/phoenix-portal-source /opt/phoenix-portal --recursive"
```

### Step 4: Install Dependencies and Configure

```bash
# Install dependencies
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'cd /opt/phoenix-portal && pnpm install --frozen-lockfile'"

# Create environment file
ssh root@192.168.11.11 "pct exec 8601 -- bash -c \"cat > /opt/phoenix-portal/.env.local << 'EOF'
# Keycloak
KEYCLOAK_URL=http://10.160.0.12:8080
KEYCLOAK_REALM=master
KEYCLOAK_CLIENT_ID=portal-client
KEYCLOAK_CLIENT_SECRET=$KEYCLOAK_CLIENT_SECRET_PORTAL

# API
NEXT_PUBLIC_GRAPHQL_ENDPOINT=http://10.160.0.10:4000/graphql
NEXT_PUBLIC_GRAPHQL_WS_ENDPOINT=ws://10.160.0.10:4000/graphql-ws

# NextAuth
NEXTAUTH_URL=http://10.160.0.11:3000
NEXTAUTH_SECRET=$(openssl rand -base64 32)

# App
NEXT_PUBLIC_APP_URL=http://10.160.0.11:3000
NODE_ENV=production
EOF\""
```

### Step 5: Build Portal

```bash
# Build Portal (may take several minutes)
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'cd /opt/phoenix-portal && pnpm build'"
```

### Step 6: Create Systemd Service

```bash
ssh root@192.168.11.11 "pct exec 8601 -- bash -c \"cat > /etc/systemd/system/phoenix-portal.service << 'EOF'
[Unit]
Description=Phoenix Portal
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/phoenix-portal
Environment=\\\"NODE_ENV=production\\\"
EnvironmentFile=/opt/phoenix-portal/.env.local
ExecStart=/usr/bin/node /opt/phoenix-portal/node_modules/.bin/next start
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF\""

# Start service
ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'systemctl daemon-reload && \
    systemctl enable phoenix-portal && \
    systemctl start phoenix-portal'"

sleep 15

# Verify service is running
ssh root@192.168.11.11 "pct exec 8601 -- systemctl status phoenix-portal --no-pager | head -10"
```

---

## Validation Gates

Phoenix is **NOT "live"** until all validation gates pass:

### Gate 1: Container Status

```bash
for vmid in 8600 8601 8602 8603; do
    status=$(ssh root@192.168.11.11 "pct status $vmid" 2>/dev/null | awk '{print $2}')
    echo "VMID $vmid: $status"
done
```

**Expected:** All containers should show "running".

### Gate 2: PostgreSQL Database

```bash
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"PGPASSWORD='$DB_PASSWORD' psql -h localhost -U phoenix -d phoenix -c 'SELECT 1;'\""
```

**Expected:** Should return "1" without errors.

### Gate 3: Keycloak Health

```bash
ssh root@192.168.11.11 "pct exec 8602 -- curl -s http://localhost:8080/health/ready"
```

**Expected:** Should return JSON with status "UP".

### Gate 4: Keycloak Token Issuance

```bash
ssh root@192.168.11.11 "pct exec 8602 -- curl -s -X POST 'http://localhost:8080/realms/master/protocol/openid-connect/token' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=admin' \
  -d 'password=$KEYCLOAK_ADMIN_PASSWORD' \
  -d 'grant_type=password' \
  -d 'client_id=admin-cli' | jq -r '.access_token' | head -c 50"
```

**Expected:** Should return an access token (JWT string).

### Gate 5: API Health Endpoint

```bash
curl -s http://10.160.0.10:4000/health
```

**Expected:** Should return healthy status (may be JSON or plain text).

### Gate 6: API Token Validation

```bash
# Get token from Keycloak
TOKEN=$(ssh root@192.168.11.11 "pct exec 8602 -- curl -s -X POST 'http://localhost:8080/realms/master/protocol/openid-connect/token' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=admin' \
  -d 'password=$KEYCLOAK_ADMIN_PASSWORD' \
  -d 'grant_type=password' \
  -d 'client_id=admin-cli' | jq -r '.access_token'")

# Test API with token
curl -s -H "Authorization: Bearer $TOKEN" http://10.160.0.10:4000/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ __typename }"}'
```

**Expected:** Should return GraphQL response.

### Gate 7: Portal Accessibility

```bash
curl -s -I http://10.160.0.11:3000 | head -1
```

**Expected:** Should return HTTP 200 or 302 (redirect).

### Gate 8: Database Persistence

```bash
# Restart PostgreSQL container
ssh root@192.168.11.11 "pct reboot 8603"
sleep 30

# Test database after restart
ssh root@192.168.11.11 "pct exec 8603 -- bash -c \"PGPASSWORD='$DB_PASSWORD' psql -h localhost -U phoenix -d phoenix -c 'SELECT 1;'\""
```

**Expected:** Database should be accessible after restart.

### Gate 9: Service Survivability

```bash
# Reboot host (if in maintenance window)
# ssh root@192.168.11.11 "reboot"
# Wait for host to come back up, then verify all services start automatically

# Check all services are active
for vmid in 8600 8601 8602 8603; do
    ssh root@192.168.11.11 "pct status $vmid"
done
```

**Expected:** All containers should auto-start and services should be active.

### Gate 10: No Dependency on 192.168.11.x

```bash
# Verify no hardcoded references to management network
ssh root@192.168.11.11 "pct exec 8600 -- env | grep -i '192.168.11' || echo 'No 192.168.11.x dependencies'"
ssh root@192.168.11.11 "pct exec 8601 -- env | grep -i '192.168.11' || echo 'No 192.168.11.x dependencies'"
```

**Expected:** Should show "No 192.168.11.x dependencies".

---

## Troubleshooting

### Container Won't Start

**Symptoms:** Container status shows "stopped" after `pct start`.

**Diagnosis:**
```bash
ssh root@192.168.11.11 "pct status 8600"
ssh root@192.168.11.11 "journalctl -u pve-container@8600 -n 50"
```

**Common Causes:**
- Network configuration error
- Storage full
- Template not available

**Solution:**
- Check network config: `ssh root@192.168.11.11 "pct config 8600"`
- Check storage: `ssh root@192.168.11.11 "pvesm status"`
- Check template: `ssh root@192.168.11.11 "pvesm list local"`

### PostgreSQL Connection Issues

**Symptoms:** API cannot connect to database.

**Diagnosis:**
```bash
# From API container
ssh root@192.168.11.11 "pct exec 8600 -- bash -c 'PGPASSWORD=password psql -h 10.160.0.13 -U phoenix -d phoenix -c \"SELECT 1;\"'"
```

**Common Causes:**
- Firewall blocking port 5432
- PostgreSQL not listening on network interface
- Wrong password

**Solution:**
- Check pg_hba.conf: `ssh root@192.168.11.11 "pct exec 8603 -- cat /etc/postgresql/16/main/pg_hba.conf | grep 10.160.0.0/22"`
- Check postgresql.conf: `ssh root@192.168.11.11 "pct exec 8603 -- grep listen_addresses /etc/postgresql/16/main/postgresql.conf"`
- Verify password matches

### Keycloak Not Starting

**Symptoms:** Keycloak service fails to start or health check fails.

**Diagnosis:**
```bash
ssh root@192.168.11.11 "pct exec 8602 -- journalctl -u keycloak -n 100 --no-pager"
ssh root@192.168.11.11 "pct exec 8602 -- ps aux | grep keycloak"
```

**Common Causes:**
- Java not found
- Database connection failed
- Port 8080 already in use

**Solution:**
- Check Java: `ssh root@192.168.11.11 "pct exec 8602 -- java -version"`
- Check database connectivity from Keycloak container
- Check port: `ssh root@192.168.11.11 "pct exec 8602 -- netstat -tlnp | grep 8080"`

### API Service Issues

**Symptoms:** API service fails to start or health check fails.

**Diagnosis:**
```bash
ssh root@192.168.11.11 "pct exec 8600 -- journalctl -u phoenix-api -n 100 --no-pager"
ssh root@192.168.11.11 "pct exec 8600 -- systemctl status phoenix-api --no-pager"
```

**Common Causes:**
- Database connection failed
- Keycloak connection failed
- Build errors
- Missing environment variables

**Solution:**
- Check environment file: `ssh root@192.168.11.11 "pct exec 8600 -- cat /opt/phoenix-api/.env"`
- Verify database connection
- Verify Keycloak is accessible: `curl http://10.160.0.12:8080/health/ready`

### Portal Build Failures

**Symptoms:** Portal build fails or service won't start.

**Diagnosis:**
```bash
ssh root@192.168.11.11 "pct exec 8601 -- journalctl -u phoenix-portal -n 100 --no-pager"
# Check build logs (if available)
ssh root@192.168.11.11 "pct exec 8601 -- cat /opt/phoenix-portal/.next/build.log 2>/dev/null || echo 'No build log'"
```

**Common Causes:**
- Build errors
- Missing environment variables
- API endpoint unreachable

**Solution:**
- Rebuild: `ssh root@192.168.11.11 "pct exec 8601 -- bash -c 'cd /opt/phoenix-portal && pnpm build'"`
- Check environment: `ssh root@192.168.11.11 "pct exec 8601 -- cat /opt/phoenix-portal/.env.local"`
- Verify API is accessible: `curl http://10.160.0.10:4000/health`

---

## Rollback Procedures

### Scenario 1: Rollback Before DNS Cutover

If issues are discovered before DNS cutover, rollback is simple:

1. **Stop all Phoenix containers:**
```bash
for vmid in 8600 8601 8602 8603; do
    ssh root@192.168.11.11 "pct stop $vmid"
done
```

2. **Do NOT delete containers** (they may contain valuable debugging information)

3. **Legacy services (7800-series) remain operational** - no action needed

### Scenario 2: Rollback After DNS Cutover

If issues are discovered after DNS cutover:

1. **Revert DNS records** (see DNS template document for exact records)
2. **Stop Phoenix containers** (as above)
3. **Legacy services become active again** via DNS

### Scenario 3: Partial Rollback

If only one service has issues:

1. **Stop only the problematic container**
2. **Other services continue running**
3. **Re-deploy the problematic service** after fixing issues

### Data Preservation

**Important:** Database data is preserved in VMID 8603. If rolling back:

- **Option 1:** Keep container stopped (data preserved)
- **Option 2:** Export data before deletion: `pg_dump -h 10.160.0.13 -U phoenix phoenix > backup.sql`
- **Option 3:** Backup entire container: `vzdump 8603`

---

## Post-Deployment Checklist

- [ ] All validation gates passed
- [ ] All services running and accessible
- [ ] Database backups configured
- [ ] Log rotation configured (prevent disk growth)
- [ ] Monitoring configured (optional)
- [ ] Firewall rules applied (see firewall rules document)
- [ ] DNS records ready (see DNS template document)
- [ ] Documentation updated
- [ ] Team notified of deployment

---

## Next Steps

After successful deployment:

1. **Configure DNS** (see `PHOENIX_DNS_ZONE_TEMPLATE.md`)
2. **Configure Firewall Rules** (see `PHOENIX_VLAN160_FIREWALL_RULES.md`)
3. **Set up monitoring** (optional)
4. **Configure backups** for database
5. **Document credentials** securely
6. **Plan DNS cutover** (when ready to go live)

---

**Last Updated:** 2026-01-09  
**Status:** Ready for Deployment