proxmox/docs/04-configuration/SANKOFA_CUTOVER_PLAN.md

# Sankofa Cutover Plan

**Last Updated:** 2026-03-27  
**Document Version:** 1.1  
**Status:** Active Documentation (historical procedure + live targets)

**Live NPM routing (2026-03-27):** Sankofa / Phoenix / The Order / Studio are on production backends. Canonical: [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md), [RPC_ENDPOINTS_MASTER.md](RPC_ENDPOINTS_MASTER.md). **The Order:** NPM → **192.168.11.39:80** (VMID **10210** HAProxy) → **192.168.11.51:3000** (portal 7801). Fleet updater: `scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`. NPM proxy host numeric IDs below may differ from your DB—verify in NPM UI.

---

**Date**: 2026-01-20  
**Status**: Cutover Plan for Sankofa Services Deployment  
**Purpose**: Step-by-step plan to cutover Sankofa domains from temporary Blockscout routing to actual Sankofa services

---

## Current state (post-cutover)

### Sankofa zone domains (authoritative backends)

| Domain | NPMplus forwards to (HTTP) | Origin stack | Notes |
|--------|----------------------------|--------------|--------|
| `sankofa.nexus`, `www.sankofa.nexus` | `192.168.11.51:3000` | VMID 7801 portal | `www` → 301 apex in NPM |
| `phoenix.sankofa.nexus`, `www.phoenix.sankofa.nexus` | `192.168.11.50:4000` | VMID 7800 API | `www` → 301 apex |
| `the-order.sankofa.nexus`, `www.the-order.sankofa.nexus` | `192.168.11.39:80` | VMID 10210 → `.51:3000` | `www` → 301 apex; HAProxy: `provision-order-haproxy-10210.sh` |
| `studio.sankofa.nexus` | `192.168.11.72:8000` | VMID 7805 | — |

**SSL:** Terminated at NPMplus (Let’s Encrypt). **Do not** point these hostnames at Blockscout (`192.168.11.140`) except for explorer domains.

### Historical note (pre-2026 cutover)

Previously these hostnames temporarily targeted Blockscout. The step-by-step below documents that migration; IDs (SSL cert / proxy host #) were examples—confirm in your NPMplus instance.

---

## Pre-Cutover Checklist

Before proceeding with cutover, verify:

### 1. Sankofa Services Deployed and Validated

- [ ] Sankofa services deployed on Proxmox VMs
- [ ] VMIDs assigned and documented
- [ ] IP addresses assigned and documented
- [ ] Services running and healthy
- [ ] Health endpoints responding

**Verification Commands**:
```bash
# Check if Sankofa VMs exist
for vmid in <SANKOFA_VMIDS>; do
    ssh root@<PROXMOX_HOST> "pct status $vmid"
done

# Test health endpoints
for ip in <SANKOFA_IPS>; do
    curl -I http://$ip:80
done
```

### 2. VMIDs and IPs Assigned

**Document the following before cutover**:

| Domain | Target VMID | Target IP | Target Port | Service Type | Notes |
|--------|-------------|-----------|-------------|--------------|-------|
| `sankofa.nexus` | 7801 | 192.168.11.51 | 3000 | Portal | Sankofa Portal (Microsoft Website) |
| `www.sankofa.nexus` | 7801 | 192.168.11.51 | 3000 | Portal | Sankofa Portal (Microsoft Website) |
| `phoenix.sankofa.nexus` | 7800 | 192.168.11.50 | 4000 | API | Phoenix API (Azure-like Portal) |
| `www.phoenix.sankofa.nexus` | 7800 | 192.168.11.50 | 4000 | API | Phoenix API (Azure-like Portal) |
| `the-order.sankofa.nexus` | 10210 | 192.168.11.39 | 80 | HAProxy edge | Proxies to portal 7801 `:3000`; app **the_order** |

**Note:** `www.the-order.sankofa.nexus` uses the same NPM upstream as apex; NPM `advanced_config` 301 → `https://the-order.sankofa.nexus`.

### 3. Health Endpoints Verified

- [ ] All Sankofa services have health endpoints
- [ ] Health endpoints return HTTP 200 or appropriate status
- [ ] Health endpoints respond within acceptable timeout

**Example Health Check**:
```bash
# Test health endpoint
curl -I http://<SANKOFA_IP>:<PORT>/health
# Expected: HTTP 200
```

### 4. Internal Connectivity Tested

- [ ] NPMplus can reach Sankofa services internally
- [ ] Test from NPMplus container: `curl -I http://<SANKOFA_IP>:<PORT>`
- [ ] Verify no firewall rules blocking internal traffic

**Verification Command**:
```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- curl -I http://<SANKOFA_IP>:<PORT>"
```

### 5. SSL Certificates Valid

- [ ] Verify certificates exist for all 5 Sankofa domains
- [ ] Check certificate expiration dates
- [ ] Renew certificates if expired (or allow auto-renewal)

**Verification Command**:
```bash
bash scripts/verify/export-npmplus-config.sh
# Check certificates.json for Sankofa domains
```

---

## Cutover Steps

### Step 1: Verify Current State

**Before making changes, document current state**:

1. **Export Current NPMplus Configuration**:
```bash
bash scripts/verify/export-npmplus-config.sh
```

2. **Verify Current Routing**:
```bash
# Check current proxy hosts for Sankofa domains
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Get current proxy host for sankofa.nexus (example)
curl -s -k -X GET "$NPM_URL/api/nginx/proxy-hosts" \
    -H "Authorization: Bearer $TOKEN" | \
    jq '.[] | select(.domain_names[] == "sankofa.nexus")'
```

3. **Document state (historical pre-cutover)**:
   - Before cutover, these domains pointed at `192.168.11.140:80` (Blockscout)
   - SSL certificates existed (example IDs: 51, 57, 60, 63, 64); proxy hosts (example 21–25)—**confirm in your NPM DB**

---

### Step 2: Deploy Sankofa Services

**Complete Sankofa service deployment**:

1. **Deploy Services**:
   - Deploy Sankofa services on Proxmox VMs
   - Assign VMIDs and IP addresses
   - Configure services (nginx if needed, application config, etc.)

2. **Verify Services Running**:
```bash
for vmid in <SANKOFA_VMIDS>; do
    ssh root@<PROXMOX_HOST> "pct status $vmid"
    ssh root@<PROXMOX_HOST> "pct exec $vmid -- curl -I http://localhost:<PORT>"
done
```

3. **Document actual IPs/ports** (✅ filled in **Current state** section and [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md))

---

### Step 3: Update NPMplus Proxy Hosts

**Update all 5 Sankofa proxy hosts to point to actual services**:

#### Option A: Update via NPMplus API

```bash
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Update sankofa.nexus (example)
# Replace <TARGET_IP> and <TARGET_PORT> with actual values
HOST_ID=21
curl -s -k -X PUT "$NPM_URL/api/nginx/proxy-hosts/$HOST_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "forward_scheme": "http",
        "forward_host": "<TARGET_IP>",
        "forward_port": <TARGET_PORT>,
        "allow_websocket_upgrade": false
    }'
```

#### Option B: Update via NPMplus Web UI

1. Access NPMplus: `https://192.168.11.166:81`
2. Navigate to: Proxy Hosts
3. For each Sankofa domain:
   - Click Edit
   - Update Forward Hostname/IP: `<TARGET_IP>`
   - Update Forward Port: `<TARGET_PORT>`
   - Update Forward Scheme: `http` (or `https` if needed)
   - Save

**Domains to Update**:

| Domain | Proxy Host ID | Current Backend | New Backend |
|--------|---------------|-----------------|-------------|
| `sankofa.nexus` | 21 | 192.168.11.140:80 | 192.168.11.51:3000 |
| `www.sankofa.nexus` | 22 | 192.168.11.140:80 | 192.168.11.51:3000 |
| `phoenix.sankofa.nexus` | 23 | 192.168.11.140:80 | 192.168.11.50:4000 |
| `www.phoenix.sankofa.nexus` | 24 | 192.168.11.140:80 | 192.168.11.50:4000 |
| `the-order.sankofa.nexus` | 25 (example) | 192.168.11.140:80 (old) | `192.168.11.39:80` (10210 HAProxy) |

**Note:** Use `update-npmplus-proxy-hosts-api.sh` for domain-based updates; proxy host IDs vary.

---

### Step 4: Verify SSL Certificates

**Check certificate validity**:

1. **Verify Certificates Exist**:
```bash
bash scripts/verify/export-npmplus-config.sh
# Check certificates.json for Sankofa domains
```

2. **Check Certificate Expiration**:
```bash
# For each Sankofa domain
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo | openssl s_client -connect "$domain:443" -servername "$domain" 2>/dev/null | \
        openssl x509 -noout -dates
done
```

3. **Renew if Expired** (if needed):
   - Access NPMplus UI
   - Navigate to SSL Certificates
   - Select certificate and click Renew
   - Or wait for auto-renewal (if enabled)

**Expected**: All certificates valid until 2026-04-16 (auto-renewal enabled).

---

### Step 5: Test End-to-End Routing

**Verify complete routing flow**:

1. **Test DNS Resolution**:
```bash
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo -n "$domain: "
    dig +short "$domain" @8.8.8.8
done
```

2. **Test HTTPS Requests**:
```bash
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo -n "$domain: "
    curl -s -o /dev/null -w "%{http_code}" "https://$domain"
    echo ""
done
```

3. **Test Internal Connectivity**:
```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

for ip in <SANKOFA_IPS>; do
    ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- curl -I http://$ip:<PORT>"
done
```

**Expected Results**:
- DNS resolves to `76.53.10.36`
- HTTPS requests return HTTP 200 (or appropriate status)
- SSL certificates valid and match domains
- Internal connectivity succeeds

---

### Step 6: Update Source-of-Truth Data

**Update canonical data model**:

1. **Run Verification**:
```bash
bash scripts/verify/run-full-verification.sh
```

2. **Review Generated JSON**:
```bash
cat docs/04-configuration/INGRESS_SOURCE_OF_TRUTH.json | jq '.backend_vms[] | select(.public_domains[] | contains("sankofa"))'
```

3. **Update Status Fields**:
   - Change `status: "documented"` to `status: "verified"` for Sankofa domains
   - Update `backend_vms` array with actual Sankofa VM details
   - Remove Sankofa from `issues` array (or mark as resolved)

---

### Step 7: Update baseline documentation

**Status 2026-03-27:** Comprehensive and streamlined DNS/NPM tables, RPC_ENDPOINTS_MASTER, and ALL_VMIDS_ENDPOINTS list live backends (including The Order via 10210). Re-open this step only if VMIDs or IPs change.

---

## Rollback Plan

**If cutover fails or issues occur**:

### Immediate Rollback (Within 5 Minutes)

1. **Restore NPMplus Proxy Hosts**:
```bash
# Restore sankofa.nexus to Blockscout (example)
HOST_ID=21
curl -s -k -X PUT "$NPM_URL/api/nginx/proxy-hosts/$HOST_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "forward_scheme": "http",
        "forward_host": "192.168.11.140",
        "forward_port": 80,
        "allow_websocket_upgrade": false
    }'
```

2. **Repeat for All 5 Domains**:
   - Update all 5 proxy hosts back to `192.168.11.140:80`

3. **Verify Rollback**:
```bash
# Test domains
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    curl -I "https://$domain"
done
```

### Complete Rollback (If Needed)

1. **Restore from Backup**:
   - If NPMplus backup exists, restore database
   - Or manually update all proxy hosts via UI

2. **Verify Services**:
   - Verify Blockscout still responds on `192.168.11.140:80`
   - Test all Sankofa domains route to Blockscout

---

## Post-Cutover Validation Checklist

After cutover, verify:

### 1. DNS Resolution
- [ ] All Sankofa domains resolve to `76.53.10.36`
- [ ] DNS propagation complete (test from external DNS servers)

### 2. SSL Certificates
- [ ] All certificates valid and match domains
- [ ] No certificate errors in browser
- [ ] Certificate chain valid

### 3. HTTPS Requests
- [ ] All Sankofa domains respond with HTTP 200 (or appropriate status)
- [ ] No 502 Bad Gateway errors
- [ ] No 503 Service Unavailable errors

### 4. Service Functionality
- [ ] Sankofa services accessible via HTTPS
- [ ] All features working correctly
- [ ] Health endpoints responding

### 5. Internal Connectivity
- [ ] NPMplus can reach all Sankofa services
- [ ] Internal HTTP requests succeed
- [ ] No firewall issues

### 6. Monitoring
- [ ] Set up monitoring for Sankofa services
- [ ] Configure alerts for service downtime
- [ ] Monitor certificate expiration

---

## Verification Scripts

**Run full verification after cutover**:

```bash
# Full verification
bash scripts/verify/run-full-verification.sh

# Specific tests
bash scripts/verify/verify-end-to-end-routing.sh
bash scripts/verify/verify-backend-vms.sh
bash scripts/verify/export-npmplus-config.sh
```

---

## Timeline Estimate

| Task | Estimated Time | Notes |
|------|----------------|-------|
| Pre-cutover verification | 30 minutes | Verify services deployed |
| Update NPMplus proxy hosts | 15 minutes | 5 domains to update |
| Verify SSL certificates | 10 minutes | Check expiration |
| Test end-to-end routing | 20 minutes | Test all domains |
| Update documentation | 15 minutes | Update source-of-truth and baseline docs |
| **Total** | **90 minutes** | Without rollback |

---

## Success Criteria

Cutover is successful if:

1. ✅ All 5 Sankofa domains route to actual Sankofa services (not Blockscout)
2. ✅ All SSL certificates valid and working
3. ✅ All HTTPS requests return HTTP 200 (or appropriate status)
4. ✅ Internal connectivity verified
5. ✅ Source-of-truth JSON updated
6. ✅ Baseline documentation updated

---

## Related Documentation

- **Verification Runbook**: `docs/04-configuration/INGRESS_VERIFICATION_RUNBOOK.md`
- **Comprehensive Architecture**: `docs/04-configuration/DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md`
- **Streamlined Table**: `docs/04-configuration/DNS_NPMPLUS_VM_STREAMLINED_TABLE.md`
- **NPMplus API Scripts**: `scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`

---

**Last Updated**: 2026-01-20  
**Maintained By**: Infrastructure Team  
**Status**: Cutover Plan Ready - Awaiting Sankofa Service Deployment