proxmox/docs/04-configuration/SANKOFA_CUTOVER_PLAN.md

Sankofa Cutover Plan

Last Updated: 2026-03-27
Document Version: 1.1
Status: Active Documentation (historical procedure + live targets)

Live NPM routing (2026-03-27): Sankofa / Phoenix / The Order / Studio are on production backends. Canonical: ALL_VMIDS_ENDPOINTS.md, RPC_ENDPOINTS_MASTER.md. The Order: NPM → 192.168.11.39:80 (VMID 10210 HAProxy) → 192.168.11.51:3000 (portal 7801). Fleet updater: scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh. NPM proxy host numeric IDs below may differ from your DB—verify in NPM UI.

---

Date: 2026-01-20
Status: Cutover Plan for Sankofa Services Deployment
Purpose: Step-by-step plan to cutover Sankofa domains from temporary Blockscout routing to actual Sankofa services

---

Current state (post-cutover)

Sankofa zone domains (authoritative backends)

Domain	NPMplus forwards to (HTTP)	Origin stack	Notes
sankofa.nexus, www.sankofa.nexus	192.168.11.51:3000	VMID 7801 portal	www → 301 apex in NPM
phoenix.sankofa.nexus, www.phoenix.sankofa.nexus	192.168.11.50:4000	VMID 7800 API	www → 301 apex
the-order.sankofa.nexus, www.the-order.sankofa.nexus	192.168.11.39:80	VMID 10210 → .51:3000	www → 301 apex; HAProxy: provision-order-haproxy-10210.sh
studio.sankofa.nexus	192.168.11.72:8000	VMID 7805	—

SSL: Terminated at NPMplus (Let’s Encrypt). Do not point these hostnames at Blockscout (192.168.11.140) except for explorer domains.

Historical note (pre-2026 cutover)

Previously these hostnames temporarily targeted Blockscout. The step-by-step below documents that migration; IDs (SSL cert / proxy host #) were examples—confirm in your NPMplus instance.

---

Pre-Cutover Checklist

Before proceeding with cutover, verify:

1. Sankofa Services Deployed and Validated

• Sankofa services deployed on Proxmox VMs
• VMIDs assigned and documented
• IP addresses assigned and documented
• Services running and healthy
• Health endpoints responding

Verification Commands:

# Check if Sankofa VMs exist
for vmid in <SANKOFA_VMIDS>; do
    ssh root@<PROXMOX_HOST> "pct status $vmid"
done

# Test health endpoints
for ip in <SANKOFA_IPS>; do
    curl -I http://$ip:80
done

2. VMIDs and IPs Assigned

Document the following before cutover:

Domain	Target VMID	Target IP	Target Port	Service Type	Notes
sankofa.nexus	7801	192.168.11.51	3000	Portal	Sankofa Portal (Microsoft Website)
www.sankofa.nexus	7801	192.168.11.51	3000	Portal	Sankofa Portal (Microsoft Website)
phoenix.sankofa.nexus	7800	192.168.11.50	4000	API	Phoenix API (Azure-like Portal)
www.phoenix.sankofa.nexus	7800	192.168.11.50	4000	API	Phoenix API (Azure-like Portal)
the-order.sankofa.nexus	10210	192.168.11.39	80	HAProxy edge	Proxies to portal 7801 :3000; app the_order

Note: www.the-order.sankofa.nexus uses the same NPM upstream as apex; NPM advanced_config 301 → https://the-order.sankofa.nexus.

3. Health Endpoints Verified

• All Sankofa services have health endpoints
• Health endpoints return HTTP 200 or appropriate status
• Health endpoints respond within acceptable timeout

Example Health Check:

# Test health endpoint
curl -I http://<SANKOFA_IP>:<PORT>/health
# Expected: HTTP 200

4. Internal Connectivity Tested

• NPMplus can reach Sankofa services internally
• Test from NPMplus container: curl -I http://<SANKOFA_IP>:<PORT>
• Verify no firewall rules blocking internal traffic

Verification Command:

NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- curl -I http://<SANKOFA_IP>:<PORT>"

5. SSL Certificates Valid

• Verify certificates exist for all 5 Sankofa domains
• Check certificate expiration dates
• Renew certificates if expired (or allow auto-renewal)

Verification Command:

bash scripts/verify/export-npmplus-config.sh
# Check certificates.json for Sankofa domains

---

Cutover Steps

Step 1: Verify Current State

Before making changes, document current state:

1. Export Current NPMplus Configuration:

bash scripts/verify/export-npmplus-config.sh

2. Verify Current Routing:

# Check current proxy hosts for Sankofa domains
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Get current proxy host for sankofa.nexus (example)
curl -s -k -X GET "$NPM_URL/api/nginx/proxy-hosts" \
    -H "Authorization: Bearer $TOKEN" | \
    jq '.[] | select(.domain_names[] == "sankofa.nexus")'

3. Document state (historical pre-cutover):
   • Before cutover, these domains pointed at 192.168.11.140:80 (Blockscout)
   • SSL certificates existed (example IDs: 51, 57, 60, 63, 64); proxy hosts (example 21–25)—confirm in your NPM DB

---

Step 2: Deploy Sankofa Services

Complete Sankofa service deployment:

1. Deploy Services:

   • Deploy Sankofa services on Proxmox VMs
   • Assign VMIDs and IP addresses
   • Configure services (nginx if needed, application config, etc.)

2. Verify Services Running:

for vmid in <SANKOFA_VMIDS>; do
    ssh root@<PROXMOX_HOST> "pct status $vmid"
    ssh root@<PROXMOX_HOST> "pct exec $vmid -- curl -I http://localhost:<PORT>"
done

3. Document actual IPs/ports (✅ filled in Current state section and ALL_VMIDS_ENDPOINTS.md)

---

Step 3: Update NPMplus Proxy Hosts

Update all 5 Sankofa proxy hosts to point to actual services:

Option A: Update via NPMplus API

NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file

TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Update sankofa.nexus (example)
# Replace <TARGET_IP> and <TARGET_PORT> with actual values
HOST_ID=21
curl -s -k -X PUT "$NPM_URL/api/nginx/proxy-hosts/$HOST_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "forward_scheme": "http",
        "forward_host": "<TARGET_IP>",
        "forward_port": <TARGET_PORT>,
        "allow_websocket_upgrade": false
    }'

Option B: Update via NPMplus Web UI

1. Access NPMplus: https://192.168.11.166:81
2. Navigate to: Proxy Hosts
3. For each Sankofa domain:
   • Click Edit
   • Update Forward Hostname/IP: <TARGET_IP>
   • Update Forward Port: <TARGET_PORT>
   • Update Forward Scheme: http (or https if needed)
   • Save

Domains to Update:

Domain	Proxy Host ID	Current Backend	New Backend
sankofa.nexus	21	192.168.11.140:80	192.168.11.51:3000
www.sankofa.nexus	22	192.168.11.140:80	192.168.11.51:3000
phoenix.sankofa.nexus	23	192.168.11.140:80	192.168.11.50:4000
www.phoenix.sankofa.nexus	24	192.168.11.140:80	192.168.11.50:4000
the-order.sankofa.nexus	25 (example)	192.168.11.140:80 (old)	192.168.11.39:80 (10210 HAProxy)

Note: Use update-npmplus-proxy-hosts-api.sh for domain-based updates; proxy host IDs vary.

---

Step 4: Verify SSL Certificates

Check certificate validity:

1. Verify Certificates Exist:

bash scripts/verify/export-npmplus-config.sh
# Check certificates.json for Sankofa domains

2. Check Certificate Expiration:

# For each Sankofa domain
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo | openssl s_client -connect "$domain:443" -servername "$domain" 2>/dev/null | \
        openssl x509 -noout -dates
done

3. Renew if Expired (if needed):
   • Access NPMplus UI
   • Navigate to SSL Certificates
   • Select certificate and click Renew
   • Or wait for auto-renewal (if enabled)

Expected: All certificates valid until 2026-04-16 (auto-renewal enabled).

---

Step 5: Test End-to-End Routing

Verify complete routing flow:

1. Test DNS Resolution:

for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo -n "$domain: "
    dig +short "$domain" @8.8.8.8
done

2. Test HTTPS Requests:

for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    echo -n "$domain: "
    curl -s -o /dev/null -w "%{http_code}" "https://$domain"
    echo ""
done

3. Test Internal Connectivity:

NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11

for ip in <SANKOFA_IPS>; do
    ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- curl -I http://$ip:<PORT>"
done

Expected Results:

• DNS resolves to 76.53.10.36
• HTTPS requests return HTTP 200 (or appropriate status)
• SSL certificates valid and match domains
• Internal connectivity succeeds

---

Step 6: Update Source-of-Truth Data

Update canonical data model:

1. Run Verification:

bash scripts/verify/run-full-verification.sh

2. Review Generated JSON:

cat docs/04-configuration/INGRESS_SOURCE_OF_TRUTH.json | jq '.backend_vms[] | select(.public_domains[] | contains("sankofa"))'

3. Update Status Fields:
   • Change status: "documented" to status: "verified" for Sankofa domains
   • Update backend_vms array with actual Sankofa VM details
   • Remove Sankofa from issues array (or mark as resolved)

---

Step 7: Update baseline documentation

Status 2026-03-27: Comprehensive and streamlined DNS/NPM tables, RPC_ENDPOINTS_MASTER, and ALL_VMIDS_ENDPOINTS list live backends (including The Order via 10210). Re-open this step only if VMIDs or IPs change.

---

Rollback Plan

If cutover fails or issues occur:

Immediate Rollback (Within 5 Minutes)

1. Restore NPMplus Proxy Hosts:

# Restore sankofa.nexus to Blockscout (example)
HOST_ID=21
curl -s -k -X PUT "$NPM_URL/api/nginx/proxy-hosts/$HOST_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "forward_scheme": "http",
        "forward_host": "192.168.11.140",
        "forward_port": 80,
        "allow_websocket_upgrade": false
    }'

2. Repeat for All 5 Domains:

   • Update all 5 proxy hosts back to 192.168.11.140:80

3. Verify Rollback:

# Test domains
for domain in sankofa.nexus www.sankofa.nexus phoenix.sankofa.nexus www.phoenix.sankofa.nexus the-order.sankofa.nexus; do
    curl -I "https://$domain"
done

Complete Rollback (If Needed)

1. Restore from Backup:

   • If NPMplus backup exists, restore database
   • Or manually update all proxy hosts via UI

2. Verify Services:

   • Verify Blockscout still responds on 192.168.11.140:80
   • Test all Sankofa domains route to Blockscout

---

Post-Cutover Validation Checklist

After cutover, verify:

1. DNS Resolution

• All Sankofa domains resolve to 76.53.10.36
• DNS propagation complete (test from external DNS servers)

2. SSL Certificates

• All certificates valid and match domains
• No certificate errors in browser
• Certificate chain valid

3. HTTPS Requests

• All Sankofa domains respond with HTTP 200 (or appropriate status)
• No 502 Bad Gateway errors
• No 503 Service Unavailable errors

4. Service Functionality

• Sankofa services accessible via HTTPS
• All features working correctly
• Health endpoints responding

5. Internal Connectivity

• NPMplus can reach all Sankofa services
• Internal HTTP requests succeed
• No firewall issues

6. Monitoring

• Set up monitoring for Sankofa services
• Configure alerts for service downtime
• Monitor certificate expiration

---

Verification Scripts

Run full verification after cutover:

# Full verification
bash scripts/verify/run-full-verification.sh

# Specific tests
bash scripts/verify/verify-end-to-end-routing.sh
bash scripts/verify/verify-backend-vms.sh
bash scripts/verify/export-npmplus-config.sh

---

Timeline Estimate

Task	Estimated Time	Notes
Pre-cutover verification	30 minutes	Verify services deployed
Update NPMplus proxy hosts	15 minutes	5 domains to update
Verify SSL certificates	10 minutes	Check expiration
Test end-to-end routing	20 minutes	Test all domains
Update documentation	15 minutes	Update source-of-truth and baseline docs
Total	90 minutes	Without rollback

---

Success Criteria

Cutover is successful if:

1. ✅ All 5 Sankofa domains route to actual Sankofa services (not Blockscout)
2. ✅ All SSL certificates valid and working
3. ✅ All HTTPS requests return HTTP 200 (or appropriate status)
4. ✅ Internal connectivity verified
5. ✅ Source-of-truth JSON updated
6. ✅ Baseline documentation updated

---

Related Documentation

• Verification Runbook: docs/04-configuration/INGRESS_VERIFICATION_RUNBOOK.md
• Comprehensive Architecture: docs/04-configuration/DNS_NPMPLUS_VM_COMPREHENSIVE_ARCHITECTURE.md
• Streamlined Table: docs/04-configuration/DNS_NPMPLUS_VM_STREAMLINED_TABLE.md
• NPMplus API Scripts: scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh

---

Last Updated: 2026-01-20
Maintained By: Infrastructure Team
Status: Cutover Plan Ready - Awaiting Sankofa Service Deployment