Sankofa/docs/phoenix/OPERATIONAL_RUNBOOKS.md

Phoenix Operating Model - Operational Runbooks

Operational procedures, troubleshooting guides, and incident response for Phoenix

This document provides operational runbooks for managing Phoenix operating model deployments, including troubleshooting, incident response, and operational procedures.

---

Table of Contents

1. Daily Operations
2. Troubleshooting
3. Incident Response
4. Maintenance Procedures
5. Monitoring and Alerting
6. Backup and Recovery

---

Daily Operations

Health Checks

Check Control Plane Health

#!/bin/bash
# Check all control plane services

PHOENIX_API="https://api.phoenix.sankofa.nexus"
TOKEN="${PHOENIX_TOKEN}"

echo "Checking Commercial Plane..."
curl -s "$PHOENIX_API/api/v1/commercial/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Tenancy Plane..."
curl -s "$PHOENIX_API/api/v1/tenancy/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Subscription Plane..."
curl -s "$PHOENIX_API/api/v1/subscription/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Environment Plane..."
curl -s "$PHOENIX_API/api/v1/environment/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Content Plane..."
curl -s "$PHOENIX_API/api/v1/content/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

Check Tenant Status

#!/bin/bash
# Check tenant status and Keycloak realm

TENANT_ID="${1}"

# Get tenant status
TENANT=$(curl -s "$PHOENIX_API/api/v1/tenancy/tenants/$TENANT_ID" \
  -H "Authorization: Bearer $TOKEN")

echo "Tenant Status:"
echo "$TENANT" | jq '{id, name, status, keycloakRealmId}'

# Check Keycloak realm
REALM_ID=$(echo "$TENANT" | jq -r '.keycloakRealmId')
if [ "$REALM_ID" != "null" ]; then
  echo "Checking Keycloak realm $REALM_ID..."
  curl -s "$KEYCLOAK_URL/admin/realms/$REALM_ID" \
    -H "Authorization: Bearer $KEYCLOAK_TOKEN" | jq '{realm, enabled}'
fi

Quota Monitoring

#!/bin/bash
# Monitor subscription quotas

SUBSCRIPTION_ID="${1}"

QUOTAS=$(curl -s "$PHOENIX_API/api/v1/subscription/subscriptions/$SUBSCRIPTION_ID/quotas" \
  -H "Authorization: Bearer $TOKEN")

echo "Quota Status:"
echo "$QUOTAS" | jq '{
  compute: {
    vcpu: {used: .compute.vcpu.used, limit: .compute.vcpu.limit, percentage: (.compute.vcpu.used / .compute.vcpu.limit * 100)},
    memory: {used: .compute.memory.used, limit: .compute.memory.limit, percentage: (.compute.memory.used / .compute.memory.limit * 100)},
    instances: {used: .compute.instances.used, limit: .compute.instances.limit, percentage: (.compute.instances.used / .compute.instances.limit * 100)}
  },
  storage: {
    total: {used: .storage.total.used, limit: .storage.total.limit, percentage: (.storage.total.used / .storage.total.limit * 100)}
  }
}'

# Check for quota warnings
echo "$QUOTAS" | jq -r '.warnings[]?' | while read warning; do
  echo "WARNING: $warning"
done

---

Troubleshooting

Issue 1: Tenant Creation Fails

Symptoms:

• Tenant creation returns error
• Keycloak realm not created
• Identity provider configuration fails

Diagnosis:

# Check tenant creation logs
kubectl logs -n phoenix deployment/tenancy-service --tail=100 | grep -i "tenant.*create"

# Check Keycloak connectivity
curl -s "$KEYCLOAK_URL/health" | jq '.'

# Check Keycloak admin access
curl -s "$KEYCLOAK_URL/admin/realms" \
  -H "Authorization: Bearer $KEYCLOAK_TOKEN" | jq '.'

Resolution:

1. Keycloak Connectivity Issue:

   # Verify Keycloak is accessible
   kubectl get pods -n keycloak
   kubectl get svc -n keycloak
   
   # Check network connectivity
   kubectl exec -n phoenix deployment/tenancy-service -- \
     curl -s "$KEYCLOAK_URL/health"
   

2. Keycloak Admin Access Issue:

   # Verify Keycloak admin token
   TOKEN=$(curl -s -X POST "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" \
     -d "client_id=admin-cli" \
     -d "username=$KEYCLOAK_ADMIN" \
     -d "password=$KEYCLOAK_PASSWORD" \
     -d "grant_type=password" | jq -r '.access_token')
   
   # Test admin access
   curl -s "$KEYCLOAK_URL/admin/realms" \
     -H "Authorization: Bearer $TOKEN"
   

3. Retry Tenant Creation:

   # Retry with verbose logging
   curl -v -X POST "$PHOENIX_API/api/v1/tenancy/tenants" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d @tenant-input.json
   

Issue 2: Promotion Fails

Symptoms:

• Promotion request fails
• Approval not received
• Deployment fails after approval

Diagnosis:

# Check promotion status
PROMOTION_ID="${1}"
curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Check promotion logs
kubectl logs -n phoenix deployment/environment-service --tail=100 | \
  grep -i "promotion.*$PROMOTION_ID"

# Check policy validation
curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/policies" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

Resolution:

1. Policy Validation Failure:

   # Review policy violations
   curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/policies" \
     -H "Authorization: Bearer $TOKEN" | jq '.violations[]'
   
   # Fix policy violations and retry
   

2. Approval Timeout:

   # Check approval status
   curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/approval" \
     -H "Authorization: Bearer $TOKEN" | jq '.'
   
   # Manually approve if needed (with proper authorization)
   curl -X POST "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/approve" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"approved": true, "reason": "Manual approval"}'
   

3. Deployment Failure:

   # Check deployment logs
   DEPLOYMENT_ID=$(curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID" \
     -H "Authorization: Bearer $TOKEN" | jq -r '.deploymentId')
   
   kubectl logs -n phoenix deployment/environment-service --tail=100 | \
     grep -i "deployment.*$DEPLOYMENT_ID"
   

Issue 3: Billing Aggregation Fails

Symptoms:

• Billing data not aggregated
• Invoice generation fails
• Cost tracking inaccurate

Diagnosis:

# Check billing aggregation status
CLIENT_ID="${1}"
curl -s "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/billing/status" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Check billing service logs
kubectl logs -n phoenix deployment/commercial-service --tail=100 | \
  grep -i "billing.*aggregation"

# Check subscription cost data
curl -s "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/subscriptions" \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {id, name, costTracking}'

Resolution:

1. Aggregation Job Failure:

   # Check aggregation job status
   kubectl get jobs -n phoenix | grep billing-aggregation
   
   # Restart aggregation job
   kubectl delete job -n phoenix billing-aggregation-$(date +%Y%m%d)
   kubectl apply -f billing-aggregation-job.yaml
   

2. Missing Subscription Data:

   # Verify all subscriptions have cost tracking
   curl -s "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/subscriptions" \
     -H "Authorization: Bearer $TOKEN" | jq '.[] | select(.costTracking == null)'
   
   # Enable cost tracking for missing subscriptions
   

3. Manual Aggregation:

   # Trigger manual aggregation
   curl -X POST "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/billing/aggregate" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"timeRange": {"start": "2025-01-01T00:00:00Z", "end": "2025-01-31T23:59:59Z"}}'
   

Issue 4: Cross-Region Connectivity Fails

Symptoms:

• Cannot connect between landing zones
• Federated identity fails
• Cross-region governance fails

Diagnosis:

# Check landing zone connectivity
LANDING_ZONE_1="${1}"
LANDING_ZONE_2="${2}"

curl -s "$PHOENIX_API/api/v1/landing-zones/$LANDING_ZONE_1/connectivity" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Test network connectivity
kubectl exec -n phoenix deployment/environment-service -- \
  ping -c 3 $(curl -s "$PHOENIX_API/api/v1/landing-zones/$LANDING_ZONE_2" \
    -H "Authorization: Bearer $TOKEN" | jq -r '.networkEndpoint')

Resolution:

1. Network Connectivity Issue:

   # Check network policies
   kubectl get networkpolicies -n phoenix
   
   # Check firewall rules
   kubectl get firewallrules -n phoenix
   
   # Update network policies if needed
   

2. Federated Identity Issue:

   # Check Keycloak federation
   curl -s "$KEYCLOAK_URL/admin/realms/$REALM_1/identity-provider/instances" \
     -H "Authorization: Bearer $KEYCLOAK_TOKEN" | jq '.'
   
   # Test federation
   curl -s "$KEYCLOAK_URL/realms/$REALM_1/protocol/openid-connect/token" \
     -d "client_id=test-client" \
     -d "grant_type=client_credentials"
   

---

Incident Response

Severity Levels

P0 - Critical:

• Complete service outage
• Data loss or corruption
• Security breach
• Billing system failure

P1 - High:

• Partial service outage
• Performance degradation
• Quota exhaustion
• Promotion failures

P2 - Medium:

• Non-critical service issues
• Minor performance issues
• Configuration issues

P3 - Low:

• Documentation issues
• Feature requests
• Minor bugs

Incident Response Process

Step 1: Detection

# Check monitoring alerts
kubectl get events -n phoenix --sort-by='.lastTimestamp' | tail -20

# Check service health
./health-check.sh

# Check error rates
curl -s "$PHOENIX_API/api/v1/metrics/errors" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

Step 2: Assessment

# Gather diagnostic information
./collect-diagnostics.sh

# Check logs
kubectl logs -n phoenix --all-containers --tail=1000 > incident-logs.txt

# Check resource usage
kubectl top pods -n phoenix
kubectl top nodes

Step 3: Containment

# Isolate affected services if needed
kubectl scale deployment/environment-service -n phoenix --replicas=0

# Block affected tenants if needed
curl -X POST "$PHOENIX_API/api/v1/tenancy/tenants/$TENANT_ID/suspend" \
  -H "Authorization: Bearer $TOKEN"

Step 4: Resolution

# Apply fix
# (specific to incident)

# Verify resolution
./health-check.sh

# Monitor for recurrence
watch -n 5 './health-check.sh'

Step 5: Post-Incident

# Document incident
# Update runbooks
# Review and improve

Common Incident Scenarios

Scenario 1: Keycloak Outage

Impact: All tenant authentication fails

Response:

1. Check Keycloak pod status
2. Check database connectivity
3. Restart Keycloak if needed
4. Verify realm synchronization

# Check Keycloak status
kubectl get pods -n keycloak
kubectl logs -n keycloak deployment/keycloak --tail=100

# Restart Keycloak
kubectl rollout restart deployment/keycloak -n keycloak

# Verify recovery
curl -s "$KEYCLOAK_URL/health" | jq '.'

Scenario 2: Quota Exhaustion

Impact: New resource provisioning fails

Response:

1. Identify exhausted quotas
2. Notify subscription owners
3. Increase quotas if authorized
4. Clean up unused resources

# Find exhausted quotas
curl -s "$PHOENIX_API/api/v1/subscription/subscriptions" \
  -H "Authorization: Bearer $TOKEN" | \
  jq '.[] | select(.quotas.compute.vcpu.used >= .quotas.compute.vcpu.limit)'

# Increase quota (with authorization)
curl -X PUT "$PHOENIX_API/api/v1/subscription/subscriptions/$SUBSCRIPTION_ID/quotas" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"compute": {"vcpu": {"limit": 200}}}'

---

Maintenance Procedures

Regular Maintenance Tasks

Daily

• Health checks
• Quota monitoring
• Error log review
• Performance monitoring

Weekly

• Billing aggregation verification
• Compliance audit review
• Security scan review
• Capacity planning review

Monthly

• Cost optimization review
• Compliance report generation
• Security audit
• Performance optimization

Backup Procedures

Database Backup

#!/bin/bash
# Backup Phoenix databases

BACKUP_DIR="/backups/phoenix/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

# Backup PostgreSQL
kubectl exec -n phoenix postgres-0 -- \
  pg_dump -U phoenix phoenix_db > "$BACKUP_DIR/phoenix_db.sql"

# Backup Keycloak database
kubectl exec -n keycloak postgres-keycloak-0 -- \
  pg_dump -U keycloak keycloak_db > "$BACKUP_DIR/keycloak_db.sql"

# Compress backups
tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR"
rm -rf "$BACKUP_DIR"

# Upload to backup storage
aws s3 cp "$BACKUP_DIR.tar.gz" s3://phoenix-backups/

Configuration Backup

#!/bin/bash
# Backup Phoenix configurations

BACKUP_DIR="/backups/phoenix-config/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

# Export all entities
curl -s "$PHOENIX_API/api/v1/commercial/clients" \
  -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/clients.json"

curl -s "$PHOENIX_API/api/v1/tenancy/tenants" \
  -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/tenants.json"

curl -s "$PHOENIX_API/api/v1/subscription/subscriptions" \
  -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/subscriptions.json"

# Compress and upload
tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR"
aws s3 cp "$BACKUP_DIR.tar.gz" s3://phoenix-backups/config/

Recovery Procedures

Database Recovery

#!/bin/bash
# Restore Phoenix database

BACKUP_FILE="${1}"
NAMESPACE="${2:-phoenix}"

# Stop services
kubectl scale deployment --all -n "$NAMESPACE" --replicas=0

# Restore database
kubectl exec -n "$NAMESPACE" postgres-0 -- \
  psql -U phoenix phoenix_db < "$BACKUP_FILE"

# Restart services
kubectl scale deployment --all -n "$NAMESPACE" --replicas=1

# Verify recovery
./health-check.sh

---

Monitoring and Alerting

Key Metrics

Commercial Plane:

• Billing aggregation latency
• Invoice generation success rate
• Payment processing success rate

Tenancy Plane:

• Tenant creation success rate
• Keycloak realm sync success rate
• Identity provider connectivity

Subscription Plane:

• Quota utilization
• Quota exhaustion alerts
• Policy pack enforcement

Environment Plane:

• Promotion success rate
• Deployment success rate
• Environment health

Content & DevOps Plane:

• Git repository sync
• CI/CD pipeline success rate
• Artifact registry availability

Alerting Rules

# prometheus/alerts/phoenix.yml

groups:
  - name: phoenix
    rules:
      - alert: KeycloakDown
        expr: up{job="keycloak"} == 0
        for: 5m
        annotations:
          summary: "Keycloak is down"
          
      - alert: QuotaExhausted
        expr: phoenix_quota_utilization > 0.95
        for: 10m
        annotations:
          summary: "Quota nearly exhausted"
          
      - alert: PromotionFailure
        expr: rate(phoenix_promotions_failed_total[5m]) > 0.1
        for: 5m
        annotations:
          summary: "High promotion failure rate"
          
      - alert: BillingAggregationFailure
        expr: phoenix_billing_aggregation_failed_total > 0
        for: 1h
        annotations:
          summary: "Billing aggregation failed"

---

Backup and Recovery

Backup Strategy

Database Backups:

• Daily full backups
• Hourly incremental backups
• 30-day retention
• Off-site storage

Configuration Backups:

• Daily configuration exports
• Version-controlled configurations
• 90-day retention

Disaster Recovery:

• RTO: 4 hours
• RPO: 1 hour
• Multi-region backups

Recovery Testing

#!/bin/bash
# Test recovery procedures

# Test database recovery
./test-database-recovery.sh

# Test configuration recovery
./test-configuration-recovery.sh

# Test service recovery
./test-service-recovery.sh

# Verify recovery
./health-check.sh

---

References

• Operating Model - Complete operating model
• API Specification - API reference
• Troubleshooting Guide - Detailed troubleshooting

---

Last Updated: 2025-01-09
Version: 1.0
Status: Complete Operational Runbooks