proxmox/docs/02-architecture/SANKOFA_PHOENIX_PHASE4_MIGRATION_RUNBOOK.md

# Sankofa / Phoenix Phase 4 Migration Runbook

**Status:** Draft executable runbook for the additive Client / Subscription / Entitlement migration  
**Last Updated:** 2026-03-30  
**Related:** [SANKOFA_PHOENIX_COMPLETE_PHASED_EXECUTION_PLAN.md](./SANKOFA_PHOENIX_COMPLETE_PHASED_EXECUTION_PLAN.md), [SANKOFA_PHOENIX_REMAINING_TASKS.md](./SANKOFA_PHOENIX_REMAINING_TASKS.md), [SANKOFA_PHOENIX_CANONICAL_BOUNDARIES_AND_TAXONOMY.md](./SANKOFA_PHOENIX_CANONICAL_BOUNDARIES_AND_TAXONOMY.md)

## Purpose

This runbook describes how to apply, verify, and if needed roll back the additive Phoenix backend migration that introduces:

- `clients`
- `client_users`
- `service_subscriptions`
- `entitlements`
- `tenant.client_id`
- billing-table `client_id` and `subscription_id` foreign keys

This migration is intentionally additive-first. It does not remove the existing tenant-based reporting shape. It introduces the new commercial boundary while preserving tenant-scoped operations.

## Scope of the current tranche

Code already landed for this migration slice in the Sankofa API repo:

- migration `027_client_subscription_entitlements`
- operating-model types and service
- GraphQL queries for `clients`, `client`, `myClient`, `serviceSubscriptions`, `mySubscriptions`, `entitlements`, and `myEntitlements`
- tenant bootstrap hook that creates a default client/subscription/entitlement for new tenants
- identity propagation for `clientId` and `subscriptionId`

## Preconditions

- Confirm the target database already includes migrations through `026_api_keys`.
- Confirm a current backup or snapshot exists for the target Postgres instance.
- Confirm application deploy artifacts are available for the Sankofa API and portal.
- Confirm Keycloak tokens or local JWTs can carry `tenant_id`, and optionally `client_id` / `subscription_id`.
- Confirm the environment has a rollback window and operator coverage.

## Deployment order

1. Take a database backup or snapshot.
2. Deploy the API code that understands the new fields before applying the migration.
3. Apply migration `027_client_subscription_entitlements`.
4. Restart or reload the Sankofa API.
5. Deploy the updated portal build so the workspace can display `clientId` and `subscriptionId` context.
6. Run the verification checks below.

## Verification checklist

### Database verification

Run checks equivalent to the following:

```sql
SELECT to_regclass('public.clients');
SELECT to_regclass('public.client_users');
SELECT to_regclass('public.service_subscriptions');
SELECT to_regclass('public.entitlements');
```

```sql
SELECT COUNT(*) AS tenants_without_client
FROM tenants
WHERE client_id IS NULL;
```

```sql
SELECT COUNT(*) AS subscriptions_without_client
FROM service_subscriptions
WHERE client_id IS NULL;
```

```sql
SELECT COUNT(*) AS entitlements_without_subscription
FROM entitlements
WHERE subscription_id IS NULL;
```

Expected result:

- all four new tables exist
- `tenants_without_client = 0` after backfill completes
- `subscriptions_without_client = 0`
- `entitlements_without_subscription = 0`

### Data-shape verification

Spot-check a migrated tenant:

```sql
SELECT
  t.id AS tenant_id,
  t.name AS tenant_name,
  t.client_id,
  c.name AS client_name,
  s.id AS subscription_id,
  s.offer_code,
  s.commercial_model,
  e.id AS entitlement_id,
  e.entitlement_key
FROM tenants t
LEFT JOIN clients c ON c.id = t.client_id
LEFT JOIN service_subscriptions s ON s.tenant_id = t.id
LEFT JOIN entitlements e ON e.subscription_id = s.id
WHERE t.id = '<tenant-id>';
```

Expected result:

- one linked `client`
- at least one `tenant-workspace` subscription
- at least one `tenant.workspace` entitlement

### API verification

Verify GraphQL with an authenticated token that has tenant context:

```graphql
query VerifyOperatingModel {
  myTenant { id clientId name }
  myClient { id name status primaryDomain }
  mySubscriptions { id offerName commercialModel status fulfillmentMode }
  myEntitlements { id entitlementKey status }
}
```

Expected result:

- `myTenant.clientId` is populated
- `myClient` resolves for tenant-scoped users
- `mySubscriptions` and `myEntitlements` return workspace records

### Portal verification

- Sign into `portal.sankofa.nexus`.
- Confirm the dashboard renders:
  - client boundary card
  - active subscription card
  - entitlement summary card
- Confirm the session still works for users with only tenant-scoped claims.

## Post-deploy monitoring

Watch for:

- API errors referencing `client_id` or `subscription_id`
- onboarding failures during tenant creation
- billing queries returning null where tenant-linked records should have been backfilled
- portal sessions that include tenant context but fail to resolve `myClient`

## Rollback strategy

Use rollback only if the migration causes runtime or data-integrity issues that cannot be contained quickly.

### Application rollback

1. Roll back the API deployment to the previous artifact.
2. Roll back the portal deployment if the new workspace cards cause issues.

### Database rollback

If the additive schema itself must be removed, run the `down` path for migration `027_client_subscription_entitlements` only after the application is off the new model.

Rollback effects:

- removes `clients`, `client_users`, `service_subscriptions`, and `entitlements`
- drops `client_id` from `tenants`
- drops `client_id` / `subscription_id` additions from billing tables

### Safer partial rollback option

Prefer this if the issue is application logic rather than schema:

1. Keep the new tables in place.
2. Disable reads from the new GraphQL queries at the application layer.
3. Disable bootstrap of the operating-model service during tenant creation.
4. Leave the additive columns in place until a corrected deployment is ready.

This avoids destructive churn on newly backfilled data.

## Open follow-up work after this migration

- move billing and invoice views to client/subscription ownership
- add onboarding persistence and workflow states
- add subscription management UI and actions
- add entitlement-aware fulfillment and deployment mapping
- add production smoke tests for hostname plus operating-model GraphQL queries