Sankofa/docs/phoenix/OPERATIONAL_RUNBOOKS.md

# Phoenix Operating Model - Operational Runbooks

**Operational procedures, troubleshooting guides, and incident response for Phoenix**

This document provides operational runbooks for managing Phoenix operating model deployments, including troubleshooting, incident response, and operational procedures.

---

## Table of Contents

1. [Daily Operations](#daily-operations)
2. [Troubleshooting](#troubleshooting)
3. [Incident Response](#incident-response)
4. [Maintenance Procedures](#maintenance-procedures)
5. [Monitoring and Alerting](#monitoring-and-alerting)
6. [Backup and Recovery](#backup-and-recovery)

---

## Daily Operations

### Health Checks

#### Check Control Plane Health

```bash
#!/bin/bash
# Check all control plane services

PHOENIX_API="https://api.phoenix.sankofa.nexus"
TOKEN="${PHOENIX_TOKEN}"

echo "Checking Commercial Plane..."
curl -s "$PHOENIX_API/api/v1/commercial/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Tenancy Plane..."
curl -s "$PHOENIX_API/api/v1/tenancy/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Subscription Plane..."
curl -s "$PHOENIX_API/api/v1/subscription/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Environment Plane..."
curl -s "$PHOENIX_API/api/v1/environment/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

echo "Checking Content Plane..."
curl -s "$PHOENIX_API/api/v1/content/health" \
  -H "Authorization: Bearer $TOKEN" | jq '.'
```

#### Check Tenant Status

```bash
#!/bin/bash
# Check tenant status and Keycloak realm

TENANT_ID="${1}"

# Get tenant status
TENANT=$(curl -s "$PHOENIX_API/api/v1/tenancy/tenants/$TENANT_ID" \
  -H "Authorization: Bearer $TOKEN")

echo "Tenant Status:"
echo "$TENANT" | jq '{id, name, status, keycloakRealmId}'

# Check Keycloak realm
REALM_ID=$(echo "$TENANT" | jq -r '.keycloakRealmId')
if [ "$REALM_ID" != "null" ]; then
  echo "Checking Keycloak realm $REALM_ID..."
  curl -s "$KEYCLOAK_URL/admin/realms/$REALM_ID" \
    -H "Authorization: Bearer $KEYCLOAK_TOKEN" | jq '{realm, enabled}'
fi
```

### Quota Monitoring

```bash
#!/bin/bash
# Monitor subscription quotas

SUBSCRIPTION_ID="${1}"

QUOTAS=$(curl -s "$PHOENIX_API/api/v1/subscription/subscriptions/$SUBSCRIPTION_ID/quotas" \
  -H "Authorization: Bearer $TOKEN")

echo "Quota Status:"
echo "$QUOTAS" | jq '{
  compute: {
    vcpu: {used: .compute.vcpu.used, limit: .compute.vcpu.limit, percentage: (.compute.vcpu.used / .compute.vcpu.limit * 100)},
    memory: {used: .compute.memory.used, limit: .compute.memory.limit, percentage: (.compute.memory.used / .compute.memory.limit * 100)},
    instances: {used: .compute.instances.used, limit: .compute.instances.limit, percentage: (.compute.instances.used / .compute.instances.limit * 100)}
  },
  storage: {
    total: {used: .storage.total.used, limit: .storage.total.limit, percentage: (.storage.total.used / .storage.total.limit * 100)}
  }
}'

# Check for quota warnings
echo "$QUOTAS" | jq -r '.warnings[]?' | while read warning; do
  echo "WARNING: $warning"
done
```

---

## Troubleshooting

### Issue 1: Tenant Creation Fails

**Symptoms:**
- Tenant creation returns error
- Keycloak realm not created
- Identity provider configuration fails

**Diagnosis:**

```bash
# Check tenant creation logs
kubectl logs -n phoenix deployment/tenancy-service --tail=100 | grep -i "tenant.*create"

# Check Keycloak connectivity
curl -s "$KEYCLOAK_URL/health" | jq '.'

# Check Keycloak admin access
curl -s "$KEYCLOAK_URL/admin/realms" \
  -H "Authorization: Bearer $KEYCLOAK_TOKEN" | jq '.'
```

**Resolution:**

1. **Keycloak Connectivity Issue:**
   ```bash
   # Verify Keycloak is accessible
   kubectl get pods -n keycloak
   kubectl get svc -n keycloak
   
   # Check network connectivity
   kubectl exec -n phoenix deployment/tenancy-service -- \
     curl -s "$KEYCLOAK_URL/health"
   ```

2. **Keycloak Admin Access Issue:**
   ```bash
   # Verify Keycloak admin token
   TOKEN=$(curl -s -X POST "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" \
     -d "client_id=admin-cli" \
     -d "username=$KEYCLOAK_ADMIN" \
     -d "password=$KEYCLOAK_PASSWORD" \
     -d "grant_type=password" | jq -r '.access_token')
   
   # Test admin access
   curl -s "$KEYCLOAK_URL/admin/realms" \
     -H "Authorization: Bearer $TOKEN"
   ```

3. **Retry Tenant Creation:**
   ```bash
   # Retry with verbose logging
   curl -v -X POST "$PHOENIX_API/api/v1/tenancy/tenants" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d @tenant-input.json
   ```

### Issue 2: Promotion Fails

**Symptoms:**
- Promotion request fails
- Approval not received
- Deployment fails after approval

**Diagnosis:**

```bash
# Check promotion status
PROMOTION_ID="${1}"
curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Check promotion logs
kubectl logs -n phoenix deployment/environment-service --tail=100 | \
  grep -i "promotion.*$PROMOTION_ID"

# Check policy validation
curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/policies" \
  -H "Authorization: Bearer $TOKEN" | jq '.'
```

**Resolution:**

1. **Policy Validation Failure:**
   ```bash
   # Review policy violations
   curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/policies" \
     -H "Authorization: Bearer $TOKEN" | jq '.violations[]'
   
   # Fix policy violations and retry
   ```

2. **Approval Timeout:**
   ```bash
   # Check approval status
   curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/approval" \
     -H "Authorization: Bearer $TOKEN" | jq '.'
   
   # Manually approve if needed (with proper authorization)
   curl -X POST "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID/approve" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"approved": true, "reason": "Manual approval"}'
   ```

3. **Deployment Failure:**
   ```bash
   # Check deployment logs
   DEPLOYMENT_ID=$(curl -s "$PHOENIX_API/api/v1/environment/promotions/$PROMOTION_ID" \
     -H "Authorization: Bearer $TOKEN" | jq -r '.deploymentId')
   
   kubectl logs -n phoenix deployment/environment-service --tail=100 | \
     grep -i "deployment.*$DEPLOYMENT_ID"
   ```

### Issue 3: Billing Aggregation Fails

**Symptoms:**
- Billing data not aggregated
- Invoice generation fails
- Cost tracking inaccurate

**Diagnosis:**

```bash
# Check billing aggregation status
CLIENT_ID="${1}"
curl -s "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/billing/status" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Check billing service logs
kubectl logs -n phoenix deployment/commercial-service --tail=100 | \
  grep -i "billing.*aggregation"

# Check subscription cost data
curl -s "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/subscriptions" \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {id, name, costTracking}'
```

**Resolution:**

1. **Aggregation Job Failure:**
   ```bash
   # Check aggregation job status
   kubectl get jobs -n phoenix | grep billing-aggregation
   
   # Restart aggregation job
   kubectl delete job -n phoenix billing-aggregation-$(date +%Y%m%d)
   kubectl apply -f billing-aggregation-job.yaml
   ```

2. **Missing Subscription Data:**
   ```bash
   # Verify all subscriptions have cost tracking
   curl -s "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/subscriptions" \
     -H "Authorization: Bearer $TOKEN" | jq '.[] | select(.costTracking == null)'
   
   # Enable cost tracking for missing subscriptions
   ```

3. **Manual Aggregation:**
   ```bash
   # Trigger manual aggregation
   curl -X POST "$PHOENIX_API/api/v1/commercial/clients/$CLIENT_ID/billing/aggregate" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"timeRange": {"start": "2025-01-01T00:00:00Z", "end": "2025-01-31T23:59:59Z"}}'
   ```

### Issue 4: Cross-Region Connectivity Fails

**Symptoms:**
- Cannot connect between landing zones
- Federated identity fails
- Cross-region governance fails

**Diagnosis:**

```bash
# Check landing zone connectivity
LANDING_ZONE_1="${1}"
LANDING_ZONE_2="${2}"

curl -s "$PHOENIX_API/api/v1/landing-zones/$LANDING_ZONE_1/connectivity" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Test network connectivity
kubectl exec -n phoenix deployment/environment-service -- \
  ping -c 3 $(curl -s "$PHOENIX_API/api/v1/landing-zones/$LANDING_ZONE_2" \
    -H "Authorization: Bearer $TOKEN" | jq -r '.networkEndpoint')
```

**Resolution:**

1. **Network Connectivity Issue:**
   ```bash
   # Check network policies
   kubectl get networkpolicies -n phoenix
   
   # Check firewall rules
   kubectl get firewallrules -n phoenix
   
   # Update network policies if needed
   ```

2. **Federated Identity Issue:**
   ```bash
   # Check Keycloak federation
   curl -s "$KEYCLOAK_URL/admin/realms/$REALM_1/identity-provider/instances" \
     -H "Authorization: Bearer $KEYCLOAK_TOKEN" | jq '.'
   
   # Test federation
   curl -s "$KEYCLOAK_URL/realms/$REALM_1/protocol/openid-connect/token" \
     -d "client_id=test-client" \
     -d "grant_type=client_credentials"
   ```

---

## Incident Response

### Severity Levels

**P0 - Critical:**
- Complete service outage
- Data loss or corruption
- Security breach
- Billing system failure

**P1 - High:**
- Partial service outage
- Performance degradation
- Quota exhaustion
- Promotion failures

**P2 - Medium:**
- Non-critical service issues
- Minor performance issues
- Configuration issues

**P3 - Low:**
- Documentation issues
- Feature requests
- Minor bugs

### Incident Response Process

#### Step 1: Detection

```bash
# Check monitoring alerts
kubectl get events -n phoenix --sort-by='.lastTimestamp' | tail -20

# Check service health
./health-check.sh

# Check error rates
curl -s "$PHOENIX_API/api/v1/metrics/errors" \
  -H "Authorization: Bearer $TOKEN" | jq '.'
```

#### Step 2: Assessment

```bash
# Gather diagnostic information
./collect-diagnostics.sh

# Check logs
kubectl logs -n phoenix --all-containers --tail=1000 > incident-logs.txt

# Check resource usage
kubectl top pods -n phoenix
kubectl top nodes
```

#### Step 3: Containment

```bash
# Isolate affected services if needed
kubectl scale deployment/environment-service -n phoenix --replicas=0

# Block affected tenants if needed
curl -X POST "$PHOENIX_API/api/v1/tenancy/tenants/$TENANT_ID/suspend" \
  -H "Authorization: Bearer $TOKEN"
```

#### Step 4: Resolution

```bash
# Apply fix
# (specific to incident)

# Verify resolution
./health-check.sh

# Monitor for recurrence
watch -n 5 './health-check.sh'
```

#### Step 5: Post-Incident

```bash
# Document incident
# Update runbooks
# Review and improve
```

### Common Incident Scenarios

#### Scenario 1: Keycloak Outage

**Impact:** All tenant authentication fails

**Response:**
1. Check Keycloak pod status
2. Check database connectivity
3. Restart Keycloak if needed
4. Verify realm synchronization

```bash
# Check Keycloak status
kubectl get pods -n keycloak
kubectl logs -n keycloak deployment/keycloak --tail=100

# Restart Keycloak
kubectl rollout restart deployment/keycloak -n keycloak

# Verify recovery
curl -s "$KEYCLOAK_URL/health" | jq '.'
```

#### Scenario 2: Quota Exhaustion

**Impact:** New resource provisioning fails

**Response:**
1. Identify exhausted quotas
2. Notify subscription owners
3. Increase quotas if authorized
4. Clean up unused resources

```bash
# Find exhausted quotas
curl -s "$PHOENIX_API/api/v1/subscription/subscriptions" \
  -H "Authorization: Bearer $TOKEN" | \
  jq '.[] | select(.quotas.compute.vcpu.used >= .quotas.compute.vcpu.limit)'

# Increase quota (with authorization)
curl -X PUT "$PHOENIX_API/api/v1/subscription/subscriptions/$SUBSCRIPTION_ID/quotas" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"compute": {"vcpu": {"limit": 200}}}'
```

---

## Maintenance Procedures

### Regular Maintenance Tasks

#### Daily

- Health checks
- Quota monitoring
- Error log review
- Performance monitoring

#### Weekly

- Billing aggregation verification
- Compliance audit review
- Security scan review
- Capacity planning review

#### Monthly

- Cost optimization review
- Compliance report generation
- Security audit
- Performance optimization

### Backup Procedures

#### Database Backup

```bash
#!/bin/bash
# Backup Phoenix databases

BACKUP_DIR="/backups/phoenix/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

# Backup PostgreSQL
kubectl exec -n phoenix postgres-0 -- \
  pg_dump -U phoenix phoenix_db > "$BACKUP_DIR/phoenix_db.sql"

# Backup Keycloak database
kubectl exec -n keycloak postgres-keycloak-0 -- \
  pg_dump -U keycloak keycloak_db > "$BACKUP_DIR/keycloak_db.sql"

# Compress backups
tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR"
rm -rf "$BACKUP_DIR"

# Upload to backup storage
aws s3 cp "$BACKUP_DIR.tar.gz" s3://phoenix-backups/
```

#### Configuration Backup

```bash
#!/bin/bash
# Backup Phoenix configurations

BACKUP_DIR="/backups/phoenix-config/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

# Export all entities
curl -s "$PHOENIX_API/api/v1/commercial/clients" \
  -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/clients.json"

curl -s "$PHOENIX_API/api/v1/tenancy/tenants" \
  -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/tenants.json"

curl -s "$PHOENIX_API/api/v1/subscription/subscriptions" \
  -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/subscriptions.json"

# Compress and upload
tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR"
aws s3 cp "$BACKUP_DIR.tar.gz" s3://phoenix-backups/config/
```

### Recovery Procedures

#### Database Recovery

```bash
#!/bin/bash
# Restore Phoenix database

BACKUP_FILE="${1}"
NAMESPACE="${2:-phoenix}"

# Stop services
kubectl scale deployment --all -n "$NAMESPACE" --replicas=0

# Restore database
kubectl exec -n "$NAMESPACE" postgres-0 -- \
  psql -U phoenix phoenix_db < "$BACKUP_FILE"

# Restart services
kubectl scale deployment --all -n "$NAMESPACE" --replicas=1

# Verify recovery
./health-check.sh
```

---

## Monitoring and Alerting

### Key Metrics

**Commercial Plane:**
- Billing aggregation latency
- Invoice generation success rate
- Payment processing success rate

**Tenancy Plane:**
- Tenant creation success rate
- Keycloak realm sync success rate
- Identity provider connectivity

**Subscription Plane:**
- Quota utilization
- Quota exhaustion alerts
- Policy pack enforcement

**Environment Plane:**
- Promotion success rate
- Deployment success rate
- Environment health

**Content & DevOps Plane:**
- Git repository sync
- CI/CD pipeline success rate
- Artifact registry availability

### Alerting Rules

```yaml
# prometheus/alerts/phoenix.yml

groups:
  - name: phoenix
    rules:
      - alert: KeycloakDown
        expr: up{job="keycloak"} == 0
        for: 5m
        annotations:
          summary: "Keycloak is down"
          
      - alert: QuotaExhausted
        expr: phoenix_quota_utilization > 0.95
        for: 10m
        annotations:
          summary: "Quota nearly exhausted"
          
      - alert: PromotionFailure
        expr: rate(phoenix_promotions_failed_total[5m]) > 0.1
        for: 5m
        annotations:
          summary: "High promotion failure rate"
          
      - alert: BillingAggregationFailure
        expr: phoenix_billing_aggregation_failed_total > 0
        for: 1h
        annotations:
          summary: "Billing aggregation failed"
```

---

## Backup and Recovery

### Backup Strategy

**Database Backups:**
- Daily full backups
- Hourly incremental backups
- 30-day retention
- Off-site storage

**Configuration Backups:**
- Daily configuration exports
- Version-controlled configurations
- 90-day retention

**Disaster Recovery:**
- RTO: 4 hours
- RPO: 1 hour
- Multi-region backups

### Recovery Testing

```bash
#!/bin/bash
# Test recovery procedures

# Test database recovery
./test-database-recovery.sh

# Test configuration recovery
./test-configuration-recovery.sh

# Test service recovery
./test-service-recovery.sh

# Verify recovery
./health-check.sh
```

---

## References

- **[Operating Model](./OPERATING_MODEL.md)** - Complete operating model
- **[API Specification](./API_SPECIFICATION.md)** - API reference
- **[Troubleshooting Guide](./TROUBLESHOOTING_GUIDE.md)** - Detailed troubleshooting

---

**Last Updated**: 2025-01-09  
**Version**: 1.0  
**Status**: Complete Operational Runbooks