# Phoenix Operating Model **Sankofa Phoenix Cloud Services — Enterprise-Grade Operating Model for Sovereign Governments** --- ## Executive Summary **Phoenix (Sankofa Cloud Services)** is a competing cloud services offering purpose-built to service **international and multi-national Sovereign Governments** and their contractors. Phoenix competes directly with Azure, AWS, and other cloud service providers while offering superior capabilities for sovereign deployments. This operating model separates **commercial governance**, **technical tenancy**, and **content/devops control** while enabling clean interoperability across distributed, multi-region deployments. The model is designed to support: - **International and multi-national sovereign governments** requiring multi-region landing zones - **Decentralized architecture** supporting distributed sovereignty - **Clouds for sovereignty** with regional data residency and compliance - **Enterprise-scale multi-tenancy** superior to Azure and AWS This document is suitable for: - Architecture decks - Product specifications - Client-facing enterprise offering memos - Implementation guides --- ## Core Management Layers (Separation of Concerns) Phoenix is structured around **five orthogonal but linked control planes**, each with its own hierarchy and access model. Planes reference each other through IDs—not shared control. ### The Five Control Planes 1. **Commercial Plane** – *Who pays* 2. **Tenancy Plane** – *Who owns domains & identity* 3. **Subscription Plane** – *What is provisioned* 4. **Environment Plane** – *Where workloads run* 5. **Content & DevOps Plane** – *What is built, governed, and deployed* ### Design Principles - **Orthogonal Design**: Each plane operates independently - **ID-Based References**: Planes reference each other through IDs, not shared control - **Separation of Concerns**: Commercial, technical, and content concerns are separated - **Clean Interoperability**: Planes interoperate cleanly without tight coupling --- ## I. Commercial Plane — Clients (Billing Profiles) **Purpose:** Financial ownership, invoicing, entitlements, and contractual scope. ### Entity: Client (Billing Profile) A **Client** represents a legal entity that contracts with Phoenix for cloud services. It is the financial and contractual boundary for billing and invoicing. #### Attributes - **Legal Entity**: Legal name, registration number, jurisdiction - **Contract & MSA**: Master Service Agreement, contract terms, SLAs - **Invoicing Configuration**: Invoice format, frequency, currency, payment terms - **Payment Instruments**: Credit cards, bank accounts, purchase orders - **Cost Centers / Departments**: Internal cost allocation structure - **Usage Aggregation & Chargeback**: Aggregation of usage across all tenants, chargeback to internal departments #### Entity Schema ```graphql type Client { id: ID! name: String! legalEntity: LegalEntity! contract: Contract msa: MSA invoicingConfig: InvoicingConfig! paymentInstruments: [PaymentInstrument!]! costCenters: [CostCenter!]! tenants: [Tenant!]! usageAggregation: UsageAggregation chargebackRules: [ChargebackRule!]! createdAt: DateTime! updatedAt: DateTime! metadata: JSON } type LegalEntity { name: String! registrationNumber: String jurisdiction: String! taxId: String address: Address! } type Contract { id: ID! client: Client! startDate: DateTime! endDate: DateTime terms: String! sla: SLA! status: ContractStatus! } type InvoicingConfig { format: InvoiceFormat! frequency: InvoiceFrequency! currency: String! paymentTerms: String! billingAddress: Address! emailRecipients: [String!]! } enum InvoiceFormat { PDF XML JSON } enum InvoiceFrequency { MONTHLY QUARTERLY ANNUAL } ``` ### Key Rules 1. **A Client can own multiple Tenants** - Rationale: A single legal entity (e.g., a government agency) may operate multiple domains/identities - Example: A defense contractor may have separate tenants for classified and unclassified work - Enforcement: Database foreign key constraint (Tenant.clientId → Client.id) 2. **A Tenant cannot span multiple Clients** - Rationale: Billing and contractual boundaries must be clear - Example: A tenant cannot be shared between two different government agencies - Enforcement: Database constraint (Tenant.clientId is NOT NULL and UNIQUE per tenant) 3. **Billing is never tied directly to environments or repos** - Rationale: Billing operates at Client/Subscription level, not at operational level - Example: Costs are aggregated at Subscription level, not per environment or Git repository - Enforcement: Billing APIs only accept Client or Subscription IDs, not Environment or Content IDs ### Relationship to Existing Billing System The existing Phoenix billing system (documented in `docs/tenants/BILLING_GUIDE.md`) currently tracks billing at the Tenant level. The new operating model introduces Client as the billing boundary: - **Migration Path**: Existing tenants will be assigned to a default Client (or Clients can be created and tenants assigned) - **Billing Aggregation**: Client-level billing aggregates costs from all associated Tenants and Subscriptions - **Backward Compatibility**: Tenant-level billing queries remain available but aggregate to Client level ### Multi-National Client Structures For international sovereign governments: - **Single Client, Multiple Tenants**: One Client (the government) with multiple Tenants per nation/region - **Multiple Clients, Coordinated Tenants**: Separate Clients per nation with coordinated Tenant structures - **Federated Billing**: Cross-border billing aggregation while maintaining sovereignty --- ## II. Tenancy Plane — Tenants (Domains) **Purpose:** Identity, domain ownership, trust boundaries, and security isolation. ### Entity: Tenant A **Tenant** represents an identity and domain boundary. It is the security blast-radius boundary and owns all identity, domain, and security configuration. #### Attributes - **Primary Domain(s)**: One or more domains owned by the tenant (e.g., `agency.gov`, `agency.sankofa.nexus`) - **Identity Provider**: SSO configuration (Entra, Okta, Keycloak, etc.) - **Global RBAC Namespace**: Root namespace for all RBAC within the tenant - **Data Residency / Sovereignty Flags**: Regional data residency requirements, sovereignty flags - **Compliance Profile**: Compliance requirements (ISO, SOC, HIPAA, government-specific standards) - **Multi-Region Support**: Whether tenant spans multiple regions - **Regional Data Residency Requirements**: Per-region data residency rules - **Cross-Border Governance Settings**: Governance rules for cross-border operations #### Entity Schema ```graphql type Tenant { id: ID! name: String! primaryDomains: [String!]! identityProvider: IdentityProvider! rbacNamespace: String! dataResidencyFlags: [DataResidencyFlag!]! complianceProfile: ComplianceProfile! client: Client! subscriptions: [Subscription!]! environments: [Environment!]! regions: [Region!]! keycloakRealmId: String multiRegionEnabled: Boolean! regionalDataResidency: [RegionalDataResidency!]! crossBorderGovernance: CrossBorderGovernance createdAt: DateTime! updatedAt: DateTime! metadata: JSON } type IdentityProvider { type: IdentityProviderType! config: JSON! ssoEnabled: Boolean! mfaRequired: Boolean! } enum IdentityProviderType { KEYCLOAK AZURE_AD OKTA GOOGLE_WORKSPACE CUSTOM_SAML CUSTOM_OIDC } type ComplianceProfile { standards: [ComplianceStandard!]! certifications: [Certification!]! auditRequirements: [AuditRequirement!]! } enum ComplianceStandard { ISO_27001 ISO_27017 ISO_27018 SOC_2 SOC_3 HIPAA PCI_DSS GDPR CCPA FEDRAMP ITAR CUSTOM } type DataResidencyFlag { region: Region! requirement: DataResidencyRequirement! enforcement: DataResidencyEnforcement! } enum DataResidencyRequirement { REQUIRED PREFERRED PROHIBITED } enum DataResidencyEnforcement { HARD SOFT ADVISORY } ``` ### Key Rules 1. **One Tenant → many Subscriptions** - Rationale: A tenant can have multiple service subscriptions - Example: A tenant may have separate subscriptions for compute, data, and AI services - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id) 2. **One Tenant → many Environments** - Rationale: Environments are scoped to tenants for security isolation - Example: A tenant may have DEV, STAGING, and PROD environments - Enforcement: Environment belongs to Subscription, which belongs to Tenant 3. **Tenant is the security blast-radius boundary** - Rationale: Security incidents are contained within tenant boundaries - Example: A compromised tenant cannot access another tenant's resources - Enforcement: Network isolation, RBAC namespace isolation, data isolation ### Relationship to Existing Tenant Management The existing Phoenix tenant management (documented in `docs/tenants/TENANT_MANAGEMENT.md`) aligns with this model: - **Tenant Tiers**: FREE, STANDARD, ENTERPRISE, SOVEREIGN map to Subscription types - **Keycloak Integration**: Each tenant gets a Keycloak realm (when `KEYCLOAK_MULTI_REALM=true`) - **Custom Domains**: Tenant primary domains support custom domain configuration - **Quotas**: Tenant quotas map to Subscription quotas and limits ### Keycloak Realm Mapping - **One Tenant = One Keycloak Realm**: Each tenant has its own Keycloak realm for complete identity isolation - **Realm Name**: Typically matches Tenant ID or primary domain - **Federated Identity**: Tenants can federate with external identity providers (Entra, Okta, etc.) ### Multi-National Tenant Structures For international sovereign governments: - **Per-Nation Tenants**: Separate tenant per nation with coordinated governance - **Federated Tenants**: Tenants that share identity federation but maintain isolation - **Cross-Border Tenants**: Tenants that span multiple nations with regional data residency --- ## III. Subscription Plane — Subscriptions **Purpose:** Logical containers for services, quotas, and spend. ### Entity: Subscription A **Subscription** represents a service bundle provisioned to a tenant. It defines what services are available, quotas, limits, and cost tracking. #### Attributes - **Service Bundles**: Compute, data, AI, storage, networking, etc. - **Quotas & Limits**: Resource quotas, rate limits, capacity limits - **Cost Tracking**: Cost aggregation, budget tracking, spending alerts - **Policy Packs**: Security policies, networking policies, data access policies - **Feature Entitlements**: Enabled features, beta features, premium features - **Multi-Region Subscriptions**: Whether subscription spans multiple regions #### Entity Schema ```graphql type Subscription { id: ID! name: String! tenant: Tenant! client: Client! type: SubscriptionType! serviceBundles: [ServiceBundle!]! quotas: Quotas limits: Limits costTracking: CostTracking! policyPacks: [PolicyPack!]! featureEntitlements: [FeatureEntitlement!]! environments: [Environment!]! regions: [Region!]! createdAt: DateTime! updatedAt: DateTime! metadata: JSON } enum SubscriptionType { SHARED_PLATFORM PRODUCT SANDBOX INNOVATION } type ServiceBundle { service: ServiceType! enabled: Boolean! quotas: ServiceQuotas limits: ServiceLimits } enum ServiceType { COMPUTE STORAGE NETWORKING DATABASE AI_ML ANALYTICS SECURITY MONITORING BACKUP } type Quotas { compute: ComputeQuotas storage: StorageQuotas network: NetworkQuotas custom: JSON } type ComputeQuotas { vcpu: Int memory: Int # GB instances: Int gpu: Int } type PolicyPack { name: String! type: PolicyPackType! policies: [Policy!]! enforcement: PolicyEnforcement! } enum PolicyPackType { SECURITY NETWORKING DATA_ACCESS COMPLIANCE CUSTOM } enum PolicyEnforcement { HARD SOFT ADVISORY } ``` ### Subscription Types 1. **Shared Platform Subscription** - Purpose: Shared infrastructure and platform services - Use Cases: Common platform services, shared networking, centralized monitoring - Characteristics: Shared resources, cost-efficient, managed by platform team 2. **Product Subscriptions** - Purpose: Dedicated resources for specific products or applications - Use Cases: Production workloads, customer-facing applications - Characteristics: Dedicated resources, higher quotas, production SLAs 3. **Sandbox / Innovation Subscriptions** - Purpose: Experimental and development environments - Use Cases: Proof of concepts, experimentation, learning - Characteristics: Lower quotas, relaxed policies, cost-optimized ### Key Rules 1. **Subscriptions live inside a Tenant** - Rationale: Subscriptions inherit tenant identity and security boundaries - Example: All subscriptions for a tenant share the same identity provider - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id, NOT NULL) 2. **Subscriptions are mapped to one Client billing profile** - Rationale: Billing aggregation happens at Client level - Example: All subscription costs for a tenant roll up to the tenant's Client - Enforcement: Subscription.clientId → Client.id (via Tenant.clientId) ### Multi-Region Subscription Patterns - **Regional Subscriptions**: Separate subscription per region for regional data residency - **Global Subscriptions**: Single subscription spanning multiple regions - **Hybrid Subscriptions**: Mix of regional and global services --- ## IV. Environment Plane — Environments **Purpose:** Operational isolation for lifecycle stages. ### Entity: Environment An **Environment** represents a lifecycle stage where workloads run. It provides network isolation, data isolation, and deployment policies. #### Attributes - **Network Isolation**: Network boundaries, firewall rules, network policies - **Data Isolation**: Data boundaries, encryption, access controls - **Deployment Policies**: Deployment rules, approval workflows, promotion policies - **Runtime Secrets**: Secrets management, key rotation, access controls - **Compliance Overlays**: Compliance requirements specific to environment - **Regional Scope**: Region where environment is deployed #### Entity Schema ```graphql type Environment { id: ID! name: String! type: EnvironmentType! subscription: Subscription! networkIsolation: NetworkIsolation! dataIsolation: DataIsolation! deploymentPolicies: [DeploymentPolicy!]! runtimeSecrets: [Secret!]! complianceOverlays: [ComplianceOverlay!]! region: Region promotionFlow: PromotionFlow createdAt: DateTime! updatedAt: DateTime! metadata: JSON } enum EnvironmentType { DEV INT UAT STAGING PROD REGULATED SOVEREIGN AIR_GAPPED } type NetworkIsolation { vpcId: String subnetIds: [String!]! firewallRules: [FirewallRule!]! networkPolicies: [NetworkPolicy!]! allowedConnections: [NetworkConnection!]! } type DataIsolation { encryptionAtRest: Boolean! encryptionInTransit: Boolean! accessControls: [AccessControl!]! dataBoundaries: [DataBoundary!]! } type DeploymentPolicy { name: String! type: DeploymentPolicyType! rules: [DeploymentRule!]! approvalRequired: Boolean! approvers: [String!]! } enum DeploymentPolicyType { AUTOMATED MANUAL_APPROVAL POLICY_DRIVEN SCHEDULED } type PromotionFlow { fromEnvironment: Environment toEnvironment: Environment! policies: [PromotionPolicy!]! approvalRequired: Boolean! automated: Boolean! } ``` ### Environment Types #### Standard Environments 1. **DEV** (Development) - Purpose: Developer workstations and development workloads - Characteristics: Relaxed policies, high developer access, cost-optimized - Access: Developers, DevOps engineers 2. **INT** (Integration) - Purpose: Integration testing and component testing - Characteristics: Moderate policies, limited access, test data - Access: QA engineers, integration testers 3. **UAT** (User Acceptance Testing) - Purpose: User acceptance testing and validation - Characteristics: Production-like policies, business user access, production-like data - Access: Business users, QA engineers 4. **STAGING** - Purpose: Pre-production validation and final testing - Characteristics: Production-equivalent policies, limited access, production data copies - Access: Release managers, senior engineers 5. **PROD** (Production) - Purpose: Production workloads serving end users - Characteristics: Strictest policies, minimal access, production data - Access: Operators, on-call engineers (read-only for most) #### Specialized Environments 6. **REGULATED** - Purpose: Regulated workloads requiring specific compliance - Characteristics: Enhanced compliance overlays, audit logging, restricted access - Use Cases: Healthcare (HIPAA), Finance (PCI-DSS), Government (FedRAMP) 7. **SOVEREIGN** - Purpose: Sovereign workloads requiring data residency - Characteristics: Regional data residency, sovereignty flags, cross-border restrictions - Use Cases: Government data, national security, sovereign cloud deployments 8. **AIR-GAPPED** - Purpose: Classified workloads with no external connectivity - Characteristics: Complete network isolation, no internet access, physical security - Use Cases: Classified government systems, critical infrastructure ### Key Rules 1. **Environments belong to Subscriptions** - Rationale: Environments inherit subscription quotas and policies - Example: PROD environment uses Product Subscription quotas - Enforcement: Database foreign key (Environment.subscriptionId → Subscription.id, NOT NULL) 2. **Promotion flows are policy-driven, not manual** - Rationale: Automated, auditable promotion reduces human error - Example: Code promotion from DEV → STAGING → PROD follows defined policies - Enforcement: PromotionFlow policies are enforced by CI/CD pipelines 3. **PROD access is always the most restricted** - Rationale: Production environments require highest security - Example: PROD requires MFA, approval workflows, and audit logging - Enforcement: RBAC policies enforce stricter access controls for PROD environments ### Multi-Region Environment Patterns - **Regional Environments**: Separate environment per region for data residency - **Global Environments**: Single environment spanning multiple regions - **Hybrid Environments**: Mix of regional and global components --- ## V. Content & DevOps Plane (Separate but Integrated) **Purpose:** What is built, governed, and deployed—separate from billing and tenancy. This plane is intentionally **not** embedded into billing or tenancy. It operates independently but integrates with the other planes through IDs. ### A. Enterprise Content Management Hierarchy #### Entity Model ``` Enterprise └── Portfolio └── Product / Program └── Application / Service └── Component / Module ``` #### Entity Schema ```graphql type Enterprise { id: ID! name: String! portfolios: [Portfolio!]! ownership: Ownership! governance: Governance! createdAt: DateTime! updatedAt: DateTime! metadata: JSON } type Portfolio { id: ID! name: String! enterprise: Enterprise! products: [Product!]! ownership: Ownership! governance: Governance! createdAt: DateTime! updatedAt: DateTime! metadata: JSON } type Product { id: ID! name: String! portfolio: Portfolio! applications: [Application!]! ownership: Ownership! governance: Governance! createdAt: DateTime! updatedAt: DateTime! metadata: JSON } type Application { id: ID! name: String! product: Product! components: [Component!]! gitRepos: [GitRepo!]! ownership: Ownership! governance: Governance! createdAt: DateTime! updatedAt: DateTime! metadata: JSON } type Component { id: ID! name: String! application: Application! contentType: ContentType! content: Content! ownership: Ownership! governance: Governance! version: String! lineage: [LineageEntry!]! createdAt: DateTime! updatedAt: DateTime! metadata: JSON } enum ContentType { SOURCE_CODE IAC PIPELINE CONFIG_TEMPLATE DOCUMENTATION DATA_SCHEMA AI_MODEL PROMPT } type Ownership { owner: String! team: String department: String contact: String } type Governance { approvalWorkflows: [ApprovalWorkflow!]! complianceTags: [ComplianceTag!]! accessControls: [AccessControl!]! retentionPolicies: [RetentionPolicy!]! } type ApprovalWorkflow { name: String! steps: [ApprovalStep!]! required: Boolean! } type ApprovalStep { approver: String! role: String! timeout: Int } ``` #### Content Types 1. **Source Code**: Application source code, libraries, dependencies 2. **IaC (Infrastructure as Code)**: Terraform, Pulumi, Bicep, CloudFormation 3. **Pipelines**: CI/CD pipeline definitions, workflow configurations 4. **Configuration Templates**: Environment configs, deployment templates 5. **Documentation**: Technical docs, runbooks, architecture diagrams 6. **Data Schemas**: Database schemas, data models, API schemas 7. **AI Models / Prompts**: ML models, AI prompts, training data #### Governance - **Ownership at Each Level**: Clear ownership from Enterprise to Component - **Approval Workflows**: Required approvals for changes at each level - **Compliance Tagging**: Compliance tags for regulatory requirements - **Versioning & Lineage**: Complete version history and lineage tracking ### B. Git & DevOps Integration Model #### Git Structure **Enterprise Git Organization**: - Repositories mapped to **Product / Service** level - Branch strategy enforced by policy - Protected branches for regulated environments - Multi-region Git repository patterns **Repository Mapping**: ``` Enterprise Git Org ├── portfolio-1/ │ ├── product-a/ │ │ ├── application-1/ (repo) │ │ └── application-2/ (repo) │ └── product-b/ │ └── application-3/ (repo) └── portfolio-2/ └── product-c/ └── application-4/ (repo) ``` **Branch Strategy**: - **main/master**: Production-ready code - **develop**: Integration branch - **feature/***: Feature branches - **release/***: Release branches - **hotfix/***: Production hotfixes **Protected Branches**: - **PROD environments**: Require approval, no direct pushes - **REGULATED environments**: Additional compliance checks - **SOVEREIGN environments**: Regional approval requirements #### CI/CD Integration **Environment-Aware Pipelines**: - Pipelines detect target environment from branch or configuration - Environment-specific policies are applied automatically - Deployment requires explicit environment selection **Deployment Requirements**: 1. **Subscription Authorization**: Pipeline verifies subscription has required services 2. **Environment Approval**: Automated or manual approval based on environment type 3. **Policy Validation**: Security, compliance, and governance policies are validated **GitOps for Infrastructure**: - Infrastructure changes via Git commits - ArgoCD syncs Git state to infrastructure - Crossplane provisions infrastructure resources - Integration with existing ArgoCD infrastructure (see `gitops/README.md`) #### Promotion Flow **Standard Promotion Flow**: ``` Code Commit ↓ CI (Test, Scan, Build) ↓ Artifact Registry ↓ Environment Promotion (Policy-Driven) ↓ Subscription Deployment ``` **Policy-Driven Promotion**: - **Automated**: DEV → INT → UAT (automated if tests pass) - **Approval Required**: UAT → STAGING (requires approval) - **Strict Approval**: STAGING → PROD (requires multiple approvals, compliance checks) **Critical Principle**: > **Git never directly deploys to PROD without environment + subscription authorization.** **Enforcement**: - CI/CD pipelines check environment type - PROD deployments require: - Subscription authorization - Environment approval workflow - Policy validation (security, compliance) - Audit logging **Multi-Region Promotion**: - Regional promotion flows for data residency - Cross-region promotion with governance approval - Sovereign promotion with regional compliance checks --- ## VI. Hierarchical Access Model (RBAC) ### Access Roles by Plane #### 1. Commercial Access - **Finance Admin**: Full access to Client billing, invoicing, payment instruments - **Billing Viewer**: Read-only access to billing and cost data - **Cost Center Owner**: Access to specific cost center data and chargeback #### 2. Tenant Access - **Tenant Owner**: Full control over tenant, identity, domains, subscriptions - **Security Admin**: Security configuration, compliance, audit access - **Identity Admin**: Identity provider configuration, user management - **Compliance Officer**: Compliance configuration, audit, reporting #### 3. Subscription Access - **Subscription Owner**: Full control over subscription, quotas, policies - **Platform Admin**: Platform services administration - **Service Operator**: Service-specific operations - **Read-only Auditor**: Read-only access for auditing #### 4. Environment Access - **Environment Owner**: Full control over environment configuration - **Release Manager**: Promotion approval, release management - **Operator**: Runtime operations, monitoring, troubleshooting - **Observer**: Read-only access for monitoring #### 5. Content & DevOps Access - **Enterprise Architect**: Enterprise-level architecture decisions - **Portfolio Lead**: Portfolio-level governance and decisions - **Product Owner**: Product-level decisions and priorities - **Dev Lead**: Development team leadership - **Contributor**: Code contribution, development - **Reviewer**: Code review, approval - **Release Approver**: Release approval for production ### Cross-Plane Access **Default Rule**: **No role crosses planes by default.** **Explicit Delegation Required**: - Cross-plane access must be explicitly granted - Delegation is audited and logged - Delegation can be time-limited or permanent **Delegation Mechanisms**: - **Role Delegation**: Grant role from one plane to user in another plane - **Temporary Access**: Time-limited cross-plane access - **Escalation**: Emergency escalation procedures ### Multi-Region RBAC - **Regional Roles**: Roles scoped to specific regions - **Cross-Region Roles**: Roles that span multiple regions (requires approval) - **Federated RBAC**: RBAC across federated tenants ### Integration with Keycloak - **Keycloak Roles**: Phoenix roles map to Keycloak roles - **Realm-Level Roles**: Tenant-specific roles in Keycloak realm - **Federated Roles**: Roles from federated identity providers --- ## VII. Key Rules and Constraints ### Commercial Plane Rules 1. **A Client can own multiple Tenants** - Rationale: Single legal entity may operate multiple domains/identities - Enforcement: Database foreign key (Tenant.clientId → Client.id) 2. **A Tenant cannot span multiple Clients** - Rationale: Billing and contractual boundaries must be clear - Enforcement: Database constraint (Tenant.clientId is NOT NULL and UNIQUE per tenant) 3. **Billing is never tied directly to environments or repos** - Rationale: Billing operates at Client/Subscription level - Enforcement: Billing APIs only accept Client or Subscription IDs ### Tenancy Plane Rules 4. **One Tenant → many Subscriptions** - Rationale: Tenant can have multiple service subscriptions - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id) 5. **One Tenant → many Environments** - Rationale: Environments are scoped to tenants for security - Enforcement: Environment belongs to Subscription, which belongs to Tenant 6. **Tenant is the security blast-radius boundary** - Rationale: Security incidents contained within tenant - Enforcement: Network isolation, RBAC namespace isolation, data isolation ### Subscription Plane Rules 7. **Subscriptions live inside a Tenant** - Rationale: Subscriptions inherit tenant identity and security - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id, NOT NULL) 8. **Subscriptions are mapped to one Client billing profile** - Rationale: Billing aggregation at Client level - Enforcement: Subscription.clientId → Client.id (via Tenant.clientId) ### Environment Plane Rules 9. **Environments belong to Subscriptions** - Rationale: Environments inherit subscription quotas and policies - Enforcement: Database foreign key (Environment.subscriptionId → Subscription.id, NOT NULL) 10. **Promotion flows are policy-driven, not manual** - Rationale: Automated, auditable promotion reduces error - Enforcement: PromotionFlow policies enforced by CI/CD pipelines 11. **PROD access is always the most restricted** - Rationale: Production requires highest security - Enforcement: RBAC policies enforce stricter access for PROD ### Content & DevOps Plane Rules 12. **Git never directly deploys to PROD without environment + subscription authorization** - Rationale: Production deployments require explicit authorization - Enforcement: CI/CD pipelines check environment type and require approvals 13. **Content hierarchy ownership is required at each level** - Rationale: Clear ownership enables governance - Enforcement: Ownership fields are required in entity schemas ### Cross-Plane Rules 14. **No role crosses planes by default** - Rationale: Separation of concerns requires explicit cross-plane access - Enforcement: RBAC system enforces plane boundaries 15. **Cross-plane access requires explicit delegation** - Rationale: Auditability and security require explicit delegation - Enforcement: Delegation must be recorded and audited ### Violation Handling - **Prevention**: Database constraints prevent invalid relationships - **Detection**: Audit logs detect policy violations - **Response**: Automated alerts and manual review for violations - **Remediation**: Automated remediation where possible, manual intervention required for critical violations --- ## VIII. Multi-Region and Multi-National Capabilities ### Sovereign Cloud Deployments **Per-Region/Nation Sovereign Clouds**: - Each region/nation can have its own sovereign cloud deployment - Complete data residency and sovereignty per region - Regional compliance and governance ### Cross-Region Governance **Federated Governance**: - Governance policies can span multiple regions - Cross-region coordination for multi-national operations - Regional autonomy with coordinated governance ### Multi-National Tenant Structures **Per-Nation Tenants**: - Separate tenant per nation with coordinated governance - Federated identity across nations - Cross-border data sharing with governance ### Regional Data Residency **Data Residency Enforcement**: - Hard enforcement: Data cannot leave region - Soft enforcement: Data preferred in region, warnings if outside - Advisory: Recommendations for data placement ### Landing Zone Patterns **Regional Landing Zones**: - Landing zone per region for sovereign deployments - Cross-region connectivity for coordination - Regional compliance per landing zone --- ## IX. Decentralized Architecture ### How Decentralization Enables Sovereignty **Distributed Control**: - Control planes can be deployed per region - Regional autonomy with coordinated governance - No single point of control **Sovereignty Benefits**: - Complete control over regional infrastructure - Data sovereignty per region - Regulatory compliance per region ### Distributed Control Planes **Regional Control Planes**: - Each region can have its own control plane deployment - Coordinated but not centralized - Eventual consistency across regions ### Cross-Region Coordination **Coordination Mechanisms**: - Event-driven coordination - API-based coordination - Governance-based coordination **Conflict Resolution**: - Regional autonomy with escalation - Governance policies for conflict resolution - Audit trails for coordination decisions ### Federated Identity and Governance **Federated Identity**: - Identity federation across regions - SSO across regions with regional control - Multi-national identity coordination **Federated Governance**: - Governance policies can be federated - Regional governance with coordination - Cross-border governance patterns --- ## X. Integration with Existing Infrastructure ### Entity Mapping to Existing Systems #### Proxmox Infrastructure **Mapping**: - **Region** → Proxmox Site - **Cluster** → Proxmox Cluster - **Node** → Proxmox Node - **VM** → Proxmox VM **Integration**: - Environments map to Proxmox resource pools - Subscriptions map to Proxmox quotas - Tenants map to Proxmox access controls #### Kubernetes Clusters **Mapping**: - **Environment** → Kubernetes Namespace - **Subscription** → Kubernetes ResourceQuota - **Tenant** → Kubernetes RBAC namespace **Integration**: - Environments deploy to Kubernetes namespaces - Subscriptions enforce Kubernetes resource quotas - Tenants enforce Kubernetes RBAC boundaries #### Cloudflare Tunnels and Zero Trust **Mapping**: - **Tenant** → Cloudflare Access Policy - **Environment** → Cloudflare Tunnel Configuration - **Region** → Cloudflare Tunnel Endpoint **Integration**: - Tenant identity maps to Cloudflare Access policies - Environments use Cloudflare tunnels for connectivity - Regions map to Cloudflare tunnel endpoints #### Keycloak Realms **Mapping**: - **Tenant** → Keycloak Realm (1:1) - **Identity Provider** → Keycloak Identity Provider - **RBAC Roles** → Keycloak Roles **Integration**: - Each tenant gets a Keycloak realm - Tenant identity provider maps to Keycloak identity provider - Phoenix RBAC roles map to Keycloak roles #### ArgoCD Applications **Mapping**: - **Application** → ArgoCD Application - **Environment** → ArgoCD Target Environment - **Subscription** → ArgoCD Resource Quota **Integration**: - Applications deploy via ArgoCD - Environments map to ArgoCD target environments - Subscriptions enforce ArgoCD resource quotas #### Crossplane Resources **Mapping**: - **Subscription** → Crossplane Composite Resource - **Environment** → Crossplane Claim - **Infrastructure** → Crossplane Managed Resources **Integration**: - Subscriptions provision infrastructure via Crossplane - Environments create Crossplane claims - Infrastructure resources managed by Crossplane #### Monitoring and Observability **Mapping**: - **Tenant** → Monitoring Namespace - **Environment** → Monitoring Labels - **Subscription** → Cost Metrics **Integration**: - Monitoring scoped by tenant and environment - Cost metrics aggregated by subscription - Alerts configured per environment ### Resource Model Mapping **Existing Model** (from `docs/architecture/data-model.md`): ``` Region → Site → Cluster → Node → VM/Pod/Service ``` **Operating Model Mapping**: - **Region**: Maps to Phoenix Region (with Tenant/Subscription context) - **Site**: Maps to Landing Zone or Environment - **Cluster**: Maps to Subscription service bundle - **Node**: Maps to Environment resources - **VM/Pod/Service**: Maps to Environment workloads ### API Integration Points **Control Plane APIs**: - Commercial Plane API: Client and billing operations - Tenancy Plane API: Tenant and identity operations - Subscription Plane API: Subscription and quota operations - Environment Plane API: Environment and deployment operations - Content & DevOps Plane API: Content and Git operations **Integration APIs**: - Proxmox API integration - Kubernetes API integration - Cloudflare API integration - Keycloak API integration - ArgoCD API integration - Crossplane API integration --- ## XI. Use Cases for Sovereign Governments ### Use Case 1: Multi-National Defense Contractor **Scenario**: Defense contractor with classified and unclassified workloads across multiple nations. **Entity Mapping**: - **Client**: Defense contractor (single Client) - **Tenants**: - Tenant 1: Classified workloads (US) - Tenant 2: Unclassified workloads (US) - Tenant 3: Classified workloads (EU) - Tenant 4: Unclassified workloads (EU) - **Subscriptions**: - Classified Subscription (AIR-GAPPED environments) - Unclassified Subscription (REGULATED environments) - **Environments**: - AIR-GAPPED PROD (classified) - REGULATED PROD (unclassified) - **Landing Zones**: Separate landing zones per nation **Compliance**: ITAR, FedRAMP, regional data residency ### Use Case 2: International Healthcare Agency **Scenario**: Healthcare agency operating across multiple countries with HIPAA requirements. **Entity Mapping**: - **Client**: Healthcare agency (single Client) - **Tenants**: - Tenant per country (for data residency) - **Subscriptions**: - Healthcare Subscription (HIPAA-compliant) - **Environments**: - REGULATED PROD (HIPAA) - REGULATED STAGING (HIPAA) - **Landing Zones**: Per-country landing zones **Compliance**: HIPAA, GDPR, regional healthcare regulations ### Use Case 3: Cross-Border Financial Regulator **Scenario**: Financial regulator coordinating across multiple nations. **Entity Mapping**: - **Clients**: One Client per nation (coordinated) - **Tenants**: - Tenant per nation (federated) - **Subscriptions**: - Regulatory Subscription (cross-border) - **Environments**: - REGULATED PROD (financial regulations) - **Landing Zones**: Per-nation landing zones with cross-border connectivity **Compliance**: Financial regulations per nation, cross-border coordination ### Use Case 4: Multi-Region Public Sector Agency **Scenario**: Public sector agency with operations across multiple regions. **Entity Mapping**: - **Client**: Public sector agency (single Client) - **Tenants**: - Tenant per region (for regional autonomy) - **Subscriptions**: - Public Sector Subscription - **Environments**: - PROD (public services) - STAGING (pre-production) - **Landing Zones**: Per-region landing zones **Compliance**: Government regulations, regional data residency ### Use Case 5: Air-Gapped Deployment Per Nation **Scenario**: Classified government system with complete isolation per nation. **Entity Mapping**: - **Client**: Government (one per nation) - **Tenants**: - Single tenant per nation (complete isolation) - **Subscriptions**: - Classified Subscription - **Environments**: - AIR-GAPPED PROD (no external connectivity) - **Landing Zones**: Air-gapped landing zone per nation **Compliance**: Classified systems, national security regulations --- ## XII. Glossary ### Core Entities - **Client (Billing Profile)**: Legal entity that contracts with Phoenix for cloud services. Financial and contractual boundary. - **Tenant**: Identity and domain boundary. Security blast-radius. Owns identity, domain, and security configuration. - **Subscription**: Logical container for services, quotas, and spend. Defines what services are available. - **Environment**: Lifecycle stage where workloads run. Provides network isolation, data isolation, and deployment policies. - **Landing Zone**: Regional deployment pattern for sovereign cloud deployments. ### Control Planes - **Commercial Plane**: Financial ownership, invoicing, entitlements, contractual scope. - **Tenancy Plane**: Identity, domain ownership, trust boundaries, security isolation. - **Subscription Plane**: Logical containers for services, quotas, and spend. - **Environment Plane**: Operational isolation for lifecycle stages. - **Content & DevOps Plane**: What is built, governed, and deployed. ### Environment Types - **DEV**: Development environment - **INT**: Integration testing environment - **UAT**: User acceptance testing environment - **STAGING**: Pre-production validation environment - **PROD**: Production environment - **REGULATED**: Regulated workloads requiring specific compliance - **SOVEREIGN**: Sovereign workloads requiring data residency - **AIR-GAPPED**: Classified workloads with no external connectivity ### Subscription Types - **Shared Platform Subscription**: Shared infrastructure and platform services - **Product Subscription**: Dedicated resources for specific products - **Sandbox Subscription**: Experimental and development environments - **Innovation Subscription**: Innovation and learning environments ### Multi-Region Terminology - **Sovereign Cloud**: Cloud deployment with complete regional control and data residency - **Landing Zone**: Regional deployment pattern for sovereign deployments - **Cross-Region Governance**: Governance policies spanning multiple regions - **Regional Data Residency**: Requirement that data remains in specific region - **Federated Identity**: Identity federation across regions or tenants ### Decentralized Architecture Terminology - **Distributed Control Plane**: Control plane deployed per region - **Federated Governance**: Governance policies federated across regions - **Eventual Consistency**: Data consistency achieved over time across regions - **Regional Autonomy**: Regional control with coordinated governance ### Comparison to Azure/AWS - **Azure AD Tenant** → **Phoenix Tenant** - **Azure Subscription** → **Phoenix Subscription** - **Azure Resource Group** → **Phoenix Environment** - **AWS Organization** → **Phoenix Client/Tenant** - **AWS Account** → **Phoenix Subscription** - **AWS Region** → **Phoenix Region/Landing Zone** --- ## XIII. Why This Works for Phoenix / Sankofa This operating model provides: 1. **Enterprise-Scale Client Separation**: Support for large multi-tenant deployments 2. **Strong Security & Sovereignty Boundaries**: Tenant as security blast-radius, regional sovereignty 3. **Clean DevOps Velocity**: Content & DevOps separate from billing/tenancy 4. **Clear Monetization and Cost Attribution**: Client-level billing with subscription aggregation 5. **Regulator-Friendly Audit Trails**: Complete governance and compliance logging 6. **Future Readiness**: Sovereign, regulated, and AI workloads supported 7. **Multi-Region Native**: Designed for international/multi-national sovereign governments 8. **Decentralized Architecture**: Supports distributed governance and sovereignty ### Alignment with Industry Standards - **Azure / AWS / GCP Mental Models**: Familiar concepts with Phoenix enhancements - **Regulated Banking & Public-Sector Frameworks**: Compliance-ready architecture - **Multi-Tenant SaaS at Scale**: Enterprise-grade multi-tenancy - **Sovereign Cloud Requirements**: Regional sovereignty and data residency ### Competitive Advantages - **Superior Multi-Tenancy**: Better than Azure with finer-grained control - **Superior Billing**: Per-second granularity vs Azure's hourly - **Sovereign Identity**: Keycloak-based, no Azure dependencies - **Multi-Region Native**: Built for international/multi-national deployments - **Decentralized Architecture**: Supports distributed sovereignty - **Landing Zone Patterns**: Sovereign cloud deployments per region --- ## References ### Related Phoenix Documentation - **[Architecture Diagrams](./OPERATING_MODEL_DIAGRAMS.md)** - Visual diagrams of the operating model - **[Cloud Provider Mapping](./CLOUD_PROVIDER_MAPPING.md)** - Azure/AWS mapping and competitive analysis - **[MVP Control Plane](./MVP_CONTROL_PLANE.md)** - Minimum viable product specification - **[Multi-Region Landing Zones](./MULTI_REGION_LANDING_ZONES.md)** - Landing zone patterns and deployment - **[Migration Guide](./MIGRATION_GUIDE.md)** - Migration from existing systems and cloud providers - **[Product Specification](./PRODUCT_SPEC.md)** - Client-facing product specification ### Existing Documentation (Tenant-Based Model) > **Note**: The following documents describe the current tenant-based model. See [Migration Guide](./MIGRATION_GUIDE.md) for migration to the new operating model. - **[Tenant Management](../tenants/TENANT_MANAGEMENT.md)** - Current multi-tenant operations guide - **[Billing Guide](../tenants/BILLING_GUIDE.md)** - Current billing and cost management - **[Identity Setup](../tenants/IDENTITY_SETUP.md)** - Current Keycloak configuration ### Architecture Documentation - **[Data Model](../architecture/data-model.md)** - GraphQL schema and data model - **[GitOps Infrastructure](../../gitops/README.md)** - GitOps infrastructure - **[Architecture Diagrams](../architecture/README.md)** - System architecture diagrams --- **Last Updated**: 2025-01-09 **Version**: 1.0 **Status**: Comprehensive Operating Model Documentation