Sankofa/docs/phoenix/OPERATING_MODEL.md

Phoenix Operating Model

Sankofa Phoenix Cloud Services — Enterprise-Grade Operating Model for Sovereign Governments

---

Executive Summary

Phoenix (Sankofa Cloud Services) is a competing cloud services offering purpose-built to service international and multi-national Sovereign Governments and their contractors. Phoenix competes directly with Azure, AWS, and other cloud service providers while offering superior capabilities for sovereign deployments.

This operating model separates commercial governance, technical tenancy, and content/devops control while enabling clean interoperability across distributed, multi-region deployments. The model is designed to support:

• International and multi-national sovereign governments requiring multi-region landing zones
• Decentralized architecture supporting distributed sovereignty
• Clouds for sovereignty with regional data residency and compliance
• Enterprise-scale multi-tenancy superior to Azure and AWS

This document is suitable for:

• Architecture decks
• Product specifications
• Client-facing enterprise offering memos
• Implementation guides

---

Core Management Layers (Separation of Concerns)

Phoenix is structured around five orthogonal but linked control planes, each with its own hierarchy and access model. Planes reference each other through IDs—not shared control.

The Five Control Planes

1. Commercial Plane – Who pays
2. Tenancy Plane – Who owns domains & identity
3. Subscription Plane – What is provisioned
4. Environment Plane – Where workloads run
5. Content & DevOps Plane – What is built, governed, and deployed

Design Principles

• Orthogonal Design: Each plane operates independently
• ID-Based References: Planes reference each other through IDs, not shared control
• Separation of Concerns: Commercial, technical, and content concerns are separated
• Clean Interoperability: Planes interoperate cleanly without tight coupling

---

I. Commercial Plane — Clients (Billing Profiles)

Purpose: Financial ownership, invoicing, entitlements, and contractual scope.

Entity: Client (Billing Profile)

A Client represents a legal entity that contracts with Phoenix for cloud services. It is the financial and contractual boundary for billing and invoicing.

Attributes

• Legal Entity: Legal name, registration number, jurisdiction
• Contract & MSA: Master Service Agreement, contract terms, SLAs
• Invoicing Configuration: Invoice format, frequency, currency, payment terms
• Payment Instruments: Credit cards, bank accounts, purchase orders
• Cost Centers / Departments: Internal cost allocation structure
• Usage Aggregation & Chargeback: Aggregation of usage across all tenants, chargeback to internal departments

Entity Schema

type Client {
  id: ID!
  name: String!
  legalEntity: LegalEntity!
  contract: Contract
  msa: MSA
  invoicingConfig: InvoicingConfig!
  paymentInstruments: [PaymentInstrument!]!
  costCenters: [CostCenter!]!
  tenants: [Tenant!]!
  usageAggregation: UsageAggregation
  chargebackRules: [ChargebackRule!]!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type LegalEntity {
  name: String!
  registrationNumber: String
  jurisdiction: String!
  taxId: String
  address: Address!
}

type Contract {
  id: ID!
  client: Client!
  startDate: DateTime!
  endDate: DateTime
  terms: String!
  sla: SLA!
  status: ContractStatus!
}

type InvoicingConfig {
  format: InvoiceFormat!
  frequency: InvoiceFrequency!
  currency: String!
  paymentTerms: String!
  billingAddress: Address!
  emailRecipients: [String!]!
}

enum InvoiceFormat {
  PDF
  XML
  JSON
}

enum InvoiceFrequency {
  MONTHLY
  QUARTERLY
  ANNUAL
}

Key Rules

1. A Client can own multiple Tenants

   • Rationale: A single legal entity (e.g., a government agency) may operate multiple domains/identities
   • Example: A defense contractor may have separate tenants for classified and unclassified work
   • Enforcement: Database foreign key constraint (Tenant.clientId → Client.id)

2. A Tenant cannot span multiple Clients

   • Rationale: Billing and contractual boundaries must be clear
   • Example: A tenant cannot be shared between two different government agencies
   • Enforcement: Database constraint (Tenant.clientId is NOT NULL and UNIQUE per tenant)

3. Billing is never tied directly to environments or repos

   • Rationale: Billing operates at Client/Subscription level, not at operational level
   • Example: Costs are aggregated at Subscription level, not per environment or Git repository
   • Enforcement: Billing APIs only accept Client or Subscription IDs, not Environment or Content IDs

Relationship to Existing Billing System

The existing Phoenix billing system (documented in docs/tenants/BILLING_GUIDE.md) currently tracks billing at the Tenant level. The new operating model introduces Client as the billing boundary:

• Migration Path: Existing tenants will be assigned to a default Client (or Clients can be created and tenants assigned)
• Billing Aggregation: Client-level billing aggregates costs from all associated Tenants and Subscriptions
• Backward Compatibility: Tenant-level billing queries remain available but aggregate to Client level

Multi-National Client Structures

For international sovereign governments:

• Single Client, Multiple Tenants: One Client (the government) with multiple Tenants per nation/region
• Multiple Clients, Coordinated Tenants: Separate Clients per nation with coordinated Tenant structures
• Federated Billing: Cross-border billing aggregation while maintaining sovereignty

---

II. Tenancy Plane — Tenants (Domains)

Purpose: Identity, domain ownership, trust boundaries, and security isolation.

Entity: Tenant

A Tenant represents an identity and domain boundary. It is the security blast-radius boundary and owns all identity, domain, and security configuration.

Attributes

• Primary Domain(s): One or more domains owned by the tenant (e.g., agency.gov, agency.sankofa.nexus)
• Identity Provider: SSO configuration (Entra, Okta, Keycloak, etc.)
• Global RBAC Namespace: Root namespace for all RBAC within the tenant
• Data Residency / Sovereignty Flags: Regional data residency requirements, sovereignty flags
• Compliance Profile: Compliance requirements (ISO, SOC, HIPAA, government-specific standards)
• Multi-Region Support: Whether tenant spans multiple regions
• Regional Data Residency Requirements: Per-region data residency rules
• Cross-Border Governance Settings: Governance rules for cross-border operations

Entity Schema

type Tenant {
  id: ID!
  name: String!
  primaryDomains: [String!]!
  identityProvider: IdentityProvider!
  rbacNamespace: String!
  dataResidencyFlags: [DataResidencyFlag!]!
  complianceProfile: ComplianceProfile!
  client: Client!
  subscriptions: [Subscription!]!
  environments: [Environment!]!
  regions: [Region!]!
  keycloakRealmId: String
  multiRegionEnabled: Boolean!
  regionalDataResidency: [RegionalDataResidency!]!
  crossBorderGovernance: CrossBorderGovernance
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type IdentityProvider {
  type: IdentityProviderType!
  config: JSON!
  ssoEnabled: Boolean!
  mfaRequired: Boolean!
}

enum IdentityProviderType {
  KEYCLOAK
  AZURE_AD
  OKTA
  GOOGLE_WORKSPACE
  CUSTOM_SAML
  CUSTOM_OIDC
}

type ComplianceProfile {
  standards: [ComplianceStandard!]!
  certifications: [Certification!]!
  auditRequirements: [AuditRequirement!]!
}

enum ComplianceStandard {
  ISO_27001
  ISO_27017
  ISO_27018
  SOC_2
  SOC_3
  HIPAA
  PCI_DSS
  GDPR
  CCPA
  FEDRAMP
  ITAR
  CUSTOM
}

type DataResidencyFlag {
  region: Region!
  requirement: DataResidencyRequirement!
  enforcement: DataResidencyEnforcement!
}

enum DataResidencyRequirement {
  REQUIRED
  PREFERRED
  PROHIBITED
}

enum DataResidencyEnforcement {
  HARD
  SOFT
  ADVISORY
}

Key Rules

1. One Tenant → many Subscriptions

   • Rationale: A tenant can have multiple service subscriptions
   • Example: A tenant may have separate subscriptions for compute, data, and AI services
   • Enforcement: Database foreign key (Subscription.tenantId → Tenant.id)

2. One Tenant → many Environments

   • Rationale: Environments are scoped to tenants for security isolation
   • Example: A tenant may have DEV, STAGING, and PROD environments
   • Enforcement: Environment belongs to Subscription, which belongs to Tenant

3. Tenant is the security blast-radius boundary

   • Rationale: Security incidents are contained within tenant boundaries
   • Example: A compromised tenant cannot access another tenant's resources
   • Enforcement: Network isolation, RBAC namespace isolation, data isolation

Relationship to Existing Tenant Management

The existing Phoenix tenant management (documented in docs/tenants/TENANT_MANAGEMENT.md) aligns with this model:

• Tenant Tiers: FREE, STANDARD, ENTERPRISE, SOVEREIGN map to Subscription types
• Keycloak Integration: Each tenant gets a Keycloak realm (when KEYCLOAK_MULTI_REALM=true)
• Custom Domains: Tenant primary domains support custom domain configuration
• Quotas: Tenant quotas map to Subscription quotas and limits

Keycloak Realm Mapping

• One Tenant = One Keycloak Realm: Each tenant has its own Keycloak realm for complete identity isolation
• Realm Name: Typically matches Tenant ID or primary domain
• Federated Identity: Tenants can federate with external identity providers (Entra, Okta, etc.)

Multi-National Tenant Structures

For international sovereign governments:

• Per-Nation Tenants: Separate tenant per nation with coordinated governance
• Federated Tenants: Tenants that share identity federation but maintain isolation
• Cross-Border Tenants: Tenants that span multiple nations with regional data residency

---

III. Subscription Plane — Subscriptions

Purpose: Logical containers for services, quotas, and spend.

Entity: Subscription

A Subscription represents a service bundle provisioned to a tenant. It defines what services are available, quotas, limits, and cost tracking.

Attributes

• Service Bundles: Compute, data, AI, storage, networking, etc.
• Quotas & Limits: Resource quotas, rate limits, capacity limits
• Cost Tracking: Cost aggregation, budget tracking, spending alerts
• Policy Packs: Security policies, networking policies, data access policies
• Feature Entitlements: Enabled features, beta features, premium features
• Multi-Region Subscriptions: Whether subscription spans multiple regions

Entity Schema

type Subscription {
  id: ID!
  name: String!
  tenant: Tenant!
  client: Client!
  type: SubscriptionType!
  serviceBundles: [ServiceBundle!]!
  quotas: Quotas
  limits: Limits
  costTracking: CostTracking!
  policyPacks: [PolicyPack!]!
  featureEntitlements: [FeatureEntitlement!]!
  environments: [Environment!]!
  regions: [Region!]!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

enum SubscriptionType {
  SHARED_PLATFORM
  PRODUCT
  SANDBOX
  INNOVATION
}

type ServiceBundle {
  service: ServiceType!
  enabled: Boolean!
  quotas: ServiceQuotas
  limits: ServiceLimits
}

enum ServiceType {
  COMPUTE
  STORAGE
  NETWORKING
  DATABASE
  AI_ML
  ANALYTICS
  SECURITY
  MONITORING
  BACKUP
}

type Quotas {
  compute: ComputeQuotas
  storage: StorageQuotas
  network: NetworkQuotas
  custom: JSON
}

type ComputeQuotas {
  vcpu: Int
  memory: Int # GB
  instances: Int
  gpu: Int
}

type PolicyPack {
  name: String!
  type: PolicyPackType!
  policies: [Policy!]!
  enforcement: PolicyEnforcement!
}

enum PolicyPackType {
  SECURITY
  NETWORKING
  DATA_ACCESS
  COMPLIANCE
  CUSTOM
}

enum PolicyEnforcement {
  HARD
  SOFT
  ADVISORY
}

Subscription Types

1. Shared Platform Subscription

   • Purpose: Shared infrastructure and platform services
   • Use Cases: Common platform services, shared networking, centralized monitoring
   • Characteristics: Shared resources, cost-efficient, managed by platform team

2. Product Subscriptions

   • Purpose: Dedicated resources for specific products or applications
   • Use Cases: Production workloads, customer-facing applications
   • Characteristics: Dedicated resources, higher quotas, production SLAs

3. Sandbox / Innovation Subscriptions

   • Purpose: Experimental and development environments
   • Use Cases: Proof of concepts, experimentation, learning
   • Characteristics: Lower quotas, relaxed policies, cost-optimized

Key Rules

1. Subscriptions live inside a Tenant

   • Rationale: Subscriptions inherit tenant identity and security boundaries
   • Example: All subscriptions for a tenant share the same identity provider
   • Enforcement: Database foreign key (Subscription.tenantId → Tenant.id, NOT NULL)

2. Subscriptions are mapped to one Client billing profile

   • Rationale: Billing aggregation happens at Client level
   • Example: All subscription costs for a tenant roll up to the tenant's Client
   • Enforcement: Subscription.clientId → Client.id (via Tenant.clientId)

Multi-Region Subscription Patterns

• Regional Subscriptions: Separate subscription per region for regional data residency
• Global Subscriptions: Single subscription spanning multiple regions
• Hybrid Subscriptions: Mix of regional and global services

---

IV. Environment Plane — Environments

Purpose: Operational isolation for lifecycle stages.

Entity: Environment

An Environment represents a lifecycle stage where workloads run. It provides network isolation, data isolation, and deployment policies.

Attributes

• Network Isolation: Network boundaries, firewall rules, network policies
• Data Isolation: Data boundaries, encryption, access controls
• Deployment Policies: Deployment rules, approval workflows, promotion policies
• Runtime Secrets: Secrets management, key rotation, access controls
• Compliance Overlays: Compliance requirements specific to environment
• Regional Scope: Region where environment is deployed

Entity Schema

type Environment {
  id: ID!
  name: String!
  type: EnvironmentType!
  subscription: Subscription!
  networkIsolation: NetworkIsolation!
  dataIsolation: DataIsolation!
  deploymentPolicies: [DeploymentPolicy!]!
  runtimeSecrets: [Secret!]!
  complianceOverlays: [ComplianceOverlay!]!
  region: Region
  promotionFlow: PromotionFlow
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

enum EnvironmentType {
  DEV
  INT
  UAT
  STAGING
  PROD
  REGULATED
  SOVEREIGN
  AIR_GAPPED
}

type NetworkIsolation {
  vpcId: String
  subnetIds: [String!]!
  firewallRules: [FirewallRule!]!
  networkPolicies: [NetworkPolicy!]!
  allowedConnections: [NetworkConnection!]!
}

type DataIsolation {
  encryptionAtRest: Boolean!
  encryptionInTransit: Boolean!
  accessControls: [AccessControl!]!
  dataBoundaries: [DataBoundary!]!
}

type DeploymentPolicy {
  name: String!
  type: DeploymentPolicyType!
  rules: [DeploymentRule!]!
  approvalRequired: Boolean!
  approvers: [String!]!
}

enum DeploymentPolicyType {
  AUTOMATED
  MANUAL_APPROVAL
  POLICY_DRIVEN
  SCHEDULED
}

type PromotionFlow {
  fromEnvironment: Environment
  toEnvironment: Environment!
  policies: [PromotionPolicy!]!
  approvalRequired: Boolean!
  automated: Boolean!
}

Environment Types

Standard Environments

1. DEV (Development)

   • Purpose: Developer workstations and development workloads
   • Characteristics: Relaxed policies, high developer access, cost-optimized
   • Access: Developers, DevOps engineers

2. INT (Integration)

   • Purpose: Integration testing and component testing
   • Characteristics: Moderate policies, limited access, test data
   • Access: QA engineers, integration testers

3. UAT (User Acceptance Testing)

   • Purpose: User acceptance testing and validation
   • Characteristics: Production-like policies, business user access, production-like data
   • Access: Business users, QA engineers

4. STAGING

   • Purpose: Pre-production validation and final testing
   • Characteristics: Production-equivalent policies, limited access, production data copies
   • Access: Release managers, senior engineers

5. PROD (Production)

   • Purpose: Production workloads serving end users
   • Characteristics: Strictest policies, minimal access, production data
   • Access: Operators, on-call engineers (read-only for most)

Specialized Environments

6. REGULATED

   • Purpose: Regulated workloads requiring specific compliance
   • Characteristics: Enhanced compliance overlays, audit logging, restricted access
   • Use Cases: Healthcare (HIPAA), Finance (PCI-DSS), Government (FedRAMP)

7. SOVEREIGN

   • Purpose: Sovereign workloads requiring data residency
   • Characteristics: Regional data residency, sovereignty flags, cross-border restrictions
   • Use Cases: Government data, national security, sovereign cloud deployments

8. AIR-GAPPED

   • Purpose: Classified workloads with no external connectivity
   • Characteristics: Complete network isolation, no internet access, physical security
   • Use Cases: Classified government systems, critical infrastructure

Key Rules

1. Environments belong to Subscriptions

   • Rationale: Environments inherit subscription quotas and policies
   • Example: PROD environment uses Product Subscription quotas
   • Enforcement: Database foreign key (Environment.subscriptionId → Subscription.id, NOT NULL)

2. Promotion flows are policy-driven, not manual

   • Rationale: Automated, auditable promotion reduces human error
   • Example: Code promotion from DEV → STAGING → PROD follows defined policies
   • Enforcement: PromotionFlow policies are enforced by CI/CD pipelines

3. PROD access is always the most restricted

   • Rationale: Production environments require highest security
   • Example: PROD requires MFA, approval workflows, and audit logging
   • Enforcement: RBAC policies enforce stricter access controls for PROD environments

Multi-Region Environment Patterns

• Regional Environments: Separate environment per region for data residency
• Global Environments: Single environment spanning multiple regions
• Hybrid Environments: Mix of regional and global components

---

V. Content & DevOps Plane (Separate but Integrated)

Purpose: What is built, governed, and deployed—separate from billing and tenancy.

This plane is intentionally not embedded into billing or tenancy. It operates independently but integrates with the other planes through IDs.

A. Enterprise Content Management Hierarchy

Entity Model

Enterprise
 └── Portfolio
     └── Product / Program
         └── Application / Service
             └── Component / Module

Entity Schema

type Enterprise {
  id: ID!
  name: String!
  portfolios: [Portfolio!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Portfolio {
  id: ID!
  name: String!
  enterprise: Enterprise!
  products: [Product!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Product {
  id: ID!
  name: String!
  portfolio: Portfolio!
  applications: [Application!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Application {
  id: ID!
  name: String!
  product: Product!
  components: [Component!]!
  gitRepos: [GitRepo!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Component {
  id: ID!
  name: String!
  application: Application!
  contentType: ContentType!
  content: Content!
  ownership: Ownership!
  governance: Governance!
  version: String!
  lineage: [LineageEntry!]!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

enum ContentType {
  SOURCE_CODE
  IAC
  PIPELINE
  CONFIG_TEMPLATE
  DOCUMENTATION
  DATA_SCHEMA
  AI_MODEL
  PROMPT
}

type Ownership {
  owner: String!
  team: String
  department: String
  contact: String
}

type Governance {
  approvalWorkflows: [ApprovalWorkflow!]!
  complianceTags: [ComplianceTag!]!
  accessControls: [AccessControl!]!
  retentionPolicies: [RetentionPolicy!]!
}

type ApprovalWorkflow {
  name: String!
  steps: [ApprovalStep!]!
  required: Boolean!
}

type ApprovalStep {
  approver: String!
  role: String!
  timeout: Int
}

Content Types

1. Source Code: Application source code, libraries, dependencies
2. IaC (Infrastructure as Code): Terraform, Pulumi, Bicep, CloudFormation
3. Pipelines: CI/CD pipeline definitions, workflow configurations
4. Configuration Templates: Environment configs, deployment templates
5. Documentation: Technical docs, runbooks, architecture diagrams
6. Data Schemas: Database schemas, data models, API schemas
7. AI Models / Prompts: ML models, AI prompts, training data

Governance

• Ownership at Each Level: Clear ownership from Enterprise to Component
• Approval Workflows: Required approvals for changes at each level
• Compliance Tagging: Compliance tags for regulatory requirements
• Versioning & Lineage: Complete version history and lineage tracking

B. Git & DevOps Integration Model

Git Structure

Enterprise Git Organization:

• Repositories mapped to Product / Service level
• Branch strategy enforced by policy
• Protected branches for regulated environments
• Multi-region Git repository patterns

Repository Mapping:

Enterprise Git Org
 ├── portfolio-1/
 │   ├── product-a/
 │   │   ├── application-1/ (repo)
 │   │   └── application-2/ (repo)
 │   └── product-b/
 │       └── application-3/ (repo)
 └── portfolio-2/
     └── product-c/
         └── application-4/ (repo)

Branch Strategy:

• main/master: Production-ready code
• develop: Integration branch
• feature/*: Feature branches
• release/*: Release branches
• hotfix/*: Production hotfixes

Protected Branches:

• PROD environments: Require approval, no direct pushes
• REGULATED environments: Additional compliance checks
• SOVEREIGN environments: Regional approval requirements

CI/CD Integration

Environment-Aware Pipelines:

• Pipelines detect target environment from branch or configuration
• Environment-specific policies are applied automatically
• Deployment requires explicit environment selection

Deployment Requirements:

1. Subscription Authorization: Pipeline verifies subscription has required services
2. Environment Approval: Automated or manual approval based on environment type
3. Policy Validation: Security, compliance, and governance policies are validated

GitOps for Infrastructure:

• Infrastructure changes via Git commits
• ArgoCD syncs Git state to infrastructure
• Crossplane provisions infrastructure resources
• Integration with existing ArgoCD infrastructure (see gitops/README.md)

Promotion Flow

Standard Promotion Flow:

Code Commit
  ↓
CI (Test, Scan, Build)
  ↓
Artifact Registry
  ↓
Environment Promotion (Policy-Driven)
  ↓
Subscription Deployment

Policy-Driven Promotion:

• Automated: DEV → INT → UAT (automated if tests pass)
• Approval Required: UAT → STAGING (requires approval)
• Strict Approval: STAGING → PROD (requires multiple approvals, compliance checks)

Critical Principle:

Git never directly deploys to PROD without environment + subscription authorization.

Enforcement:

• CI/CD pipelines check environment type
• PROD deployments require:
  • Subscription authorization
  • Environment approval workflow
  • Policy validation (security, compliance)
  • Audit logging

Multi-Region Promotion:

• Regional promotion flows for data residency
• Cross-region promotion with governance approval
• Sovereign promotion with regional compliance checks

---

VI. Hierarchical Access Model (RBAC)

Access Roles by Plane

1. Commercial Access

• Finance Admin: Full access to Client billing, invoicing, payment instruments
• Billing Viewer: Read-only access to billing and cost data
• Cost Center Owner: Access to specific cost center data and chargeback

2. Tenant Access

• Tenant Owner: Full control over tenant, identity, domains, subscriptions
• Security Admin: Security configuration, compliance, audit access
• Identity Admin: Identity provider configuration, user management
• Compliance Officer: Compliance configuration, audit, reporting

3. Subscription Access

• Subscription Owner: Full control over subscription, quotas, policies
• Platform Admin: Platform services administration
• Service Operator: Service-specific operations
• Read-only Auditor: Read-only access for auditing

4. Environment Access

• Environment Owner: Full control over environment configuration
• Release Manager: Promotion approval, release management
• Operator: Runtime operations, monitoring, troubleshooting
• Observer: Read-only access for monitoring

5. Content & DevOps Access

• Enterprise Architect: Enterprise-level architecture decisions
• Portfolio Lead: Portfolio-level governance and decisions
• Product Owner: Product-level decisions and priorities
• Dev Lead: Development team leadership
• Contributor: Code contribution, development
• Reviewer: Code review, approval
• Release Approver: Release approval for production

Cross-Plane Access

Default Rule: No role crosses planes by default.

Explicit Delegation Required:

• Cross-plane access must be explicitly granted
• Delegation is audited and logged
• Delegation can be time-limited or permanent

Delegation Mechanisms:

• Role Delegation: Grant role from one plane to user in another plane
• Temporary Access: Time-limited cross-plane access
• Escalation: Emergency escalation procedures

Multi-Region RBAC

• Regional Roles: Roles scoped to specific regions
• Cross-Region Roles: Roles that span multiple regions (requires approval)
• Federated RBAC: RBAC across federated tenants

Integration with Keycloak

• Keycloak Roles: Phoenix roles map to Keycloak roles
• Realm-Level Roles: Tenant-specific roles in Keycloak realm
• Federated Roles: Roles from federated identity providers

---

VII. Key Rules and Constraints

Commercial Plane Rules

1. A Client can own multiple Tenants

   • Rationale: Single legal entity may operate multiple domains/identities
   • Enforcement: Database foreign key (Tenant.clientId → Client.id)

2. A Tenant cannot span multiple Clients

   • Rationale: Billing and contractual boundaries must be clear
   • Enforcement: Database constraint (Tenant.clientId is NOT NULL and UNIQUE per tenant)

3. Billing is never tied directly to environments or repos

   • Rationale: Billing operates at Client/Subscription level
   • Enforcement: Billing APIs only accept Client or Subscription IDs

Tenancy Plane Rules

4. One Tenant → many Subscriptions

   • Rationale: Tenant can have multiple service subscriptions
   • Enforcement: Database foreign key (Subscription.tenantId → Tenant.id)

5. One Tenant → many Environments

   • Rationale: Environments are scoped to tenants for security
   • Enforcement: Environment belongs to Subscription, which belongs to Tenant

6. Tenant is the security blast-radius boundary

   • Rationale: Security incidents contained within tenant
   • Enforcement: Network isolation, RBAC namespace isolation, data isolation

Subscription Plane Rules

7. Subscriptions live inside a Tenant

   • Rationale: Subscriptions inherit tenant identity and security
   • Enforcement: Database foreign key (Subscription.tenantId → Tenant.id, NOT NULL)

8. Subscriptions are mapped to one Client billing profile

   • Rationale: Billing aggregation at Client level
   • Enforcement: Subscription.clientId → Client.id (via Tenant.clientId)

Environment Plane Rules

9. Environments belong to Subscriptions

   • Rationale: Environments inherit subscription quotas and policies
   • Enforcement: Database foreign key (Environment.subscriptionId → Subscription.id, NOT NULL)

10. Promotion flows are policy-driven, not manual

    • Rationale: Automated, auditable promotion reduces error
    • Enforcement: PromotionFlow policies enforced by CI/CD pipelines

11. PROD access is always the most restricted

    • Rationale: Production requires highest security
    • Enforcement: RBAC policies enforce stricter access for PROD

Content & DevOps Plane Rules

12. Git never directly deploys to PROD without environment + subscription authorization

    • Rationale: Production deployments require explicit authorization
    • Enforcement: CI/CD pipelines check environment type and require approvals

13. Content hierarchy ownership is required at each level

    • Rationale: Clear ownership enables governance
    • Enforcement: Ownership fields are required in entity schemas

Cross-Plane Rules

14. No role crosses planes by default

    • Rationale: Separation of concerns requires explicit cross-plane access
    • Enforcement: RBAC system enforces plane boundaries

15. Cross-plane access requires explicit delegation

    • Rationale: Auditability and security require explicit delegation
    • Enforcement: Delegation must be recorded and audited

Violation Handling

• Prevention: Database constraints prevent invalid relationships
• Detection: Audit logs detect policy violations
• Response: Automated alerts and manual review for violations
• Remediation: Automated remediation where possible, manual intervention required for critical violations

---

VIII. Multi-Region and Multi-National Capabilities

Sovereign Cloud Deployments

Per-Region/Nation Sovereign Clouds:

• Each region/nation can have its own sovereign cloud deployment
• Complete data residency and sovereignty per region
• Regional compliance and governance

Cross-Region Governance

Federated Governance:

• Governance policies can span multiple regions
• Cross-region coordination for multi-national operations
• Regional autonomy with coordinated governance

Multi-National Tenant Structures

Per-Nation Tenants:

• Separate tenant per nation with coordinated governance
• Federated identity across nations
• Cross-border data sharing with governance

Regional Data Residency

Data Residency Enforcement:

• Hard enforcement: Data cannot leave region
• Soft enforcement: Data preferred in region, warnings if outside
• Advisory: Recommendations for data placement

Landing Zone Patterns

Regional Landing Zones:

• Landing zone per region for sovereign deployments
• Cross-region connectivity for coordination
• Regional compliance per landing zone

---

IX. Decentralized Architecture

How Decentralization Enables Sovereignty

Distributed Control:

• Control planes can be deployed per region
• Regional autonomy with coordinated governance
• No single point of control

Sovereignty Benefits:

• Complete control over regional infrastructure
• Data sovereignty per region
• Regulatory compliance per region

Distributed Control Planes

Regional Control Planes:

• Each region can have its own control plane deployment
• Coordinated but not centralized
• Eventual consistency across regions

Cross-Region Coordination

Coordination Mechanisms:

• Event-driven coordination
• API-based coordination
• Governance-based coordination

Conflict Resolution:

• Regional autonomy with escalation
• Governance policies for conflict resolution
• Audit trails for coordination decisions

Federated Identity and Governance

Federated Identity:

• Identity federation across regions
• SSO across regions with regional control
• Multi-national identity coordination

Federated Governance:

• Governance policies can be federated
• Regional governance with coordination
• Cross-border governance patterns

---

X. Integration with Existing Infrastructure

Entity Mapping to Existing Systems

Proxmox Infrastructure

Mapping:

• Region → Proxmox Site
• Cluster → Proxmox Cluster
• Node → Proxmox Node
• VM → Proxmox VM

Integration:

• Environments map to Proxmox resource pools
• Subscriptions map to Proxmox quotas
• Tenants map to Proxmox access controls

Kubernetes Clusters

Mapping:

• Environment → Kubernetes Namespace
• Subscription → Kubernetes ResourceQuota
• Tenant → Kubernetes RBAC namespace

Integration:

• Environments deploy to Kubernetes namespaces
• Subscriptions enforce Kubernetes resource quotas
• Tenants enforce Kubernetes RBAC boundaries

Cloudflare Tunnels and Zero Trust

Mapping:

• Tenant → Cloudflare Access Policy
• Environment → Cloudflare Tunnel Configuration
• Region → Cloudflare Tunnel Endpoint

Integration:

• Tenant identity maps to Cloudflare Access policies
• Environments use Cloudflare tunnels for connectivity
• Regions map to Cloudflare tunnel endpoints

Keycloak Realms

Mapping:

• Tenant → Keycloak Realm (1:1)
• Identity Provider → Keycloak Identity Provider
• RBAC Roles → Keycloak Roles

Integration:

• Each tenant gets a Keycloak realm
• Tenant identity provider maps to Keycloak identity provider
• Phoenix RBAC roles map to Keycloak roles

ArgoCD Applications

Mapping:

• Application → ArgoCD Application
• Environment → ArgoCD Target Environment
• Subscription → ArgoCD Resource Quota

Integration:

• Applications deploy via ArgoCD
• Environments map to ArgoCD target environments
• Subscriptions enforce ArgoCD resource quotas

Crossplane Resources

Mapping:

• Subscription → Crossplane Composite Resource
• Environment → Crossplane Claim
• Infrastructure → Crossplane Managed Resources

Integration:

• Subscriptions provision infrastructure via Crossplane
• Environments create Crossplane claims
• Infrastructure resources managed by Crossplane

Monitoring and Observability

Mapping:

• Tenant → Monitoring Namespace
• Environment → Monitoring Labels
• Subscription → Cost Metrics

Integration:

• Monitoring scoped by tenant and environment
• Cost metrics aggregated by subscription
• Alerts configured per environment

Resource Model Mapping

Existing Model (from docs/architecture/data-model.md):

Region → Site → Cluster → Node → VM/Pod/Service

Operating Model Mapping:

• Region: Maps to Phoenix Region (with Tenant/Subscription context)
• Site: Maps to Landing Zone or Environment
• Cluster: Maps to Subscription service bundle
• Node: Maps to Environment resources
• VM/Pod/Service: Maps to Environment workloads

API Integration Points

Control Plane APIs:

• Commercial Plane API: Client and billing operations
• Tenancy Plane API: Tenant and identity operations
• Subscription Plane API: Subscription and quota operations
• Environment Plane API: Environment and deployment operations
• Content & DevOps Plane API: Content and Git operations

Integration APIs:

• Proxmox API integration
• Kubernetes API integration
• Cloudflare API integration
• Keycloak API integration
• ArgoCD API integration
• Crossplane API integration

---

XI. Use Cases for Sovereign Governments

Use Case 1: Multi-National Defense Contractor

Scenario: Defense contractor with classified and unclassified workloads across multiple nations.

Entity Mapping:

• Client: Defense contractor (single Client)
• Tenants:
  • Tenant 1: Classified workloads (US)
  • Tenant 2: Unclassified workloads (US)
  • Tenant 3: Classified workloads (EU)
  • Tenant 4: Unclassified workloads (EU)
• Subscriptions:
  • Classified Subscription (AIR-GAPPED environments)
  • Unclassified Subscription (REGULATED environments)
• Environments:
  • AIR-GAPPED PROD (classified)
  • REGULATED PROD (unclassified)
• Landing Zones: Separate landing zones per nation

Compliance: ITAR, FedRAMP, regional data residency

Use Case 2: International Healthcare Agency

Scenario: Healthcare agency operating across multiple countries with HIPAA requirements.

Entity Mapping:

• Client: Healthcare agency (single Client)
• Tenants:
  • Tenant per country (for data residency)
• Subscriptions:
  • Healthcare Subscription (HIPAA-compliant)
• Environments:
  • REGULATED PROD (HIPAA)
  • REGULATED STAGING (HIPAA)
• Landing Zones: Per-country landing zones

Compliance: HIPAA, GDPR, regional healthcare regulations

Use Case 3: Cross-Border Financial Regulator

Scenario: Financial regulator coordinating across multiple nations.

Entity Mapping:

• Clients: One Client per nation (coordinated)
• Tenants:
  • Tenant per nation (federated)
• Subscriptions:
  • Regulatory Subscription (cross-border)
• Environments:
  • REGULATED PROD (financial regulations)
• Landing Zones: Per-nation landing zones with cross-border connectivity

Compliance: Financial regulations per nation, cross-border coordination

Use Case 4: Multi-Region Public Sector Agency

Scenario: Public sector agency with operations across multiple regions.

Entity Mapping:

• Client: Public sector agency (single Client)
• Tenants:
  • Tenant per region (for regional autonomy)
• Subscriptions:
  • Public Sector Subscription
• Environments:
  • PROD (public services)
  • STAGING (pre-production)
• Landing Zones: Per-region landing zones

Compliance: Government regulations, regional data residency

Use Case 5: Air-Gapped Deployment Per Nation

Scenario: Classified government system with complete isolation per nation.

Entity Mapping:

• Client: Government (one per nation)
• Tenants:
  • Single tenant per nation (complete isolation)
• Subscriptions:
  • Classified Subscription
• Environments:
  • AIR-GAPPED PROD (no external connectivity)
• Landing Zones: Air-gapped landing zone per nation

Compliance: Classified systems, national security regulations

---

XII. Glossary

Core Entities

• Client (Billing Profile): Legal entity that contracts with Phoenix for cloud services. Financial and contractual boundary.
• Tenant: Identity and domain boundary. Security blast-radius. Owns identity, domain, and security configuration.
• Subscription: Logical container for services, quotas, and spend. Defines what services are available.
• Environment: Lifecycle stage where workloads run. Provides network isolation, data isolation, and deployment policies.
• Landing Zone: Regional deployment pattern for sovereign cloud deployments.

Control Planes

• Commercial Plane: Financial ownership, invoicing, entitlements, contractual scope.
• Tenancy Plane: Identity, domain ownership, trust boundaries, security isolation.
• Subscription Plane: Logical containers for services, quotas, and spend.
• Environment Plane: Operational isolation for lifecycle stages.
• Content & DevOps Plane: What is built, governed, and deployed.

Environment Types

• DEV: Development environment
• INT: Integration testing environment
• UAT: User acceptance testing environment
• STAGING: Pre-production validation environment
• PROD: Production environment
• REGULATED: Regulated workloads requiring specific compliance
• SOVEREIGN: Sovereign workloads requiring data residency
• AIR-GAPPED: Classified workloads with no external connectivity

Subscription Types

• Shared Platform Subscription: Shared infrastructure and platform services
• Product Subscription: Dedicated resources for specific products
• Sandbox Subscription: Experimental and development environments
• Innovation Subscription: Innovation and learning environments

Multi-Region Terminology

• Sovereign Cloud: Cloud deployment with complete regional control and data residency
• Landing Zone: Regional deployment pattern for sovereign deployments
• Cross-Region Governance: Governance policies spanning multiple regions
• Regional Data Residency: Requirement that data remains in specific region
• Federated Identity: Identity federation across regions or tenants

Decentralized Architecture Terminology

• Distributed Control Plane: Control plane deployed per region
• Federated Governance: Governance policies federated across regions
• Eventual Consistency: Data consistency achieved over time across regions
• Regional Autonomy: Regional control with coordinated governance

Comparison to Azure/AWS

• Azure AD Tenant → Phoenix Tenant
• Azure Subscription → Phoenix Subscription
• Azure Resource Group → Phoenix Environment
• AWS Organization → Phoenix Client/Tenant
• AWS Account → Phoenix Subscription
• AWS Region → Phoenix Region/Landing Zone

---

XIII. Why This Works for Phoenix / Sankofa

This operating model provides:

1. Enterprise-Scale Client Separation: Support for large multi-tenant deployments
2. Strong Security & Sovereignty Boundaries: Tenant as security blast-radius, regional sovereignty
3. Clean DevOps Velocity: Content & DevOps separate from billing/tenancy
4. Clear Monetization and Cost Attribution: Client-level billing with subscription aggregation
5. Regulator-Friendly Audit Trails: Complete governance and compliance logging
6. Future Readiness: Sovereign, regulated, and AI workloads supported
7. Multi-Region Native: Designed for international/multi-national sovereign governments
8. Decentralized Architecture: Supports distributed governance and sovereignty

Alignment with Industry Standards

• Azure / AWS / GCP Mental Models: Familiar concepts with Phoenix enhancements
• Regulated Banking & Public-Sector Frameworks: Compliance-ready architecture
• Multi-Tenant SaaS at Scale: Enterprise-grade multi-tenancy
• Sovereign Cloud Requirements: Regional sovereignty and data residency

Competitive Advantages

• Superior Multi-Tenancy: Better than Azure with finer-grained control
• Superior Billing: Per-second granularity vs Azure's hourly
• Sovereign Identity: Keycloak-based, no Azure dependencies
• Multi-Region Native: Built for international/multi-national deployments
• Decentralized Architecture: Supports distributed sovereignty
• Landing Zone Patterns: Sovereign cloud deployments per region

---

References

Related Phoenix Documentation

• Architecture Diagrams - Visual diagrams of the operating model
• Cloud Provider Mapping - Azure/AWS mapping and competitive analysis
• MVP Control Plane - Minimum viable product specification
• Multi-Region Landing Zones - Landing zone patterns and deployment
• Migration Guide - Migration from existing systems and cloud providers
• Product Specification - Client-facing product specification

Existing Documentation (Tenant-Based Model)

Note

: The following documents describe the current tenant-based model. See Migration Guide for migration to the new operating model.

• Tenant Management - Current multi-tenant operations guide
• Billing Guide - Current billing and cost management
• Identity Setup - Current Keycloak configuration

Architecture Documentation

• Data Model - GraphQL schema and data model
• GitOps Infrastructure - GitOps infrastructure
• Architecture Diagrams - System architecture diagrams

---

Last Updated: 2025-01-09
Version: 1.0
Status: Comprehensive Operating Model Documentation