Sankofa/docs/phoenix/OPERATING_MODEL.md

# Phoenix Operating Model

**Sankofa Phoenix Cloud Services — Enterprise-Grade Operating Model for Sovereign Governments**

---

## Executive Summary

**Phoenix (Sankofa Cloud Services)** is a competing cloud services offering purpose-built to service **international and multi-national Sovereign Governments** and their contractors. Phoenix competes directly with Azure, AWS, and other cloud service providers while offering superior capabilities for sovereign deployments.

This operating model separates **commercial governance**, **technical tenancy**, and **content/devops control** while enabling clean interoperability across distributed, multi-region deployments. The model is designed to support:

- **International and multi-national sovereign governments** requiring multi-region landing zones
- **Decentralized architecture** supporting distributed sovereignty
- **Clouds for sovereignty** with regional data residency and compliance
- **Enterprise-scale multi-tenancy** superior to Azure and AWS

This document is suitable for:
- Architecture decks
- Product specifications
- Client-facing enterprise offering memos
- Implementation guides

---

## Core Management Layers (Separation of Concerns)

Phoenix is structured around **five orthogonal but linked control planes**, each with its own hierarchy and access model. Planes reference each other through IDs—not shared control.

### The Five Control Planes

1. **Commercial Plane** – *Who pays*
2. **Tenancy Plane** – *Who owns domains & identity*
3. **Subscription Plane** – *What is provisioned*
4. **Environment Plane** – *Where workloads run*
5. **Content & DevOps Plane** – *What is built, governed, and deployed*

### Design Principles

- **Orthogonal Design**: Each plane operates independently
- **ID-Based References**: Planes reference each other through IDs, not shared control
- **Separation of Concerns**: Commercial, technical, and content concerns are separated
- **Clean Interoperability**: Planes interoperate cleanly without tight coupling

---

## I. Commercial Plane — Clients (Billing Profiles)

**Purpose:** Financial ownership, invoicing, entitlements, and contractual scope.

### Entity: Client (Billing Profile)

A **Client** represents a legal entity that contracts with Phoenix for cloud services. It is the financial and contractual boundary for billing and invoicing.

#### Attributes

- **Legal Entity**: Legal name, registration number, jurisdiction
- **Contract & MSA**: Master Service Agreement, contract terms, SLAs
- **Invoicing Configuration**: Invoice format, frequency, currency, payment terms
- **Payment Instruments**: Credit cards, bank accounts, purchase orders
- **Cost Centers / Departments**: Internal cost allocation structure
- **Usage Aggregation & Chargeback**: Aggregation of usage across all tenants, chargeback to internal departments

#### Entity Schema

```graphql
type Client {
  id: ID!
  name: String!
  legalEntity: LegalEntity!
  contract: Contract
  msa: MSA
  invoicingConfig: InvoicingConfig!
  paymentInstruments: [PaymentInstrument!]!
  costCenters: [CostCenter!]!
  tenants: [Tenant!]!
  usageAggregation: UsageAggregation
  chargebackRules: [ChargebackRule!]!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type LegalEntity {
  name: String!
  registrationNumber: String
  jurisdiction: String!
  taxId: String
  address: Address!
}

type Contract {
  id: ID!
  client: Client!
  startDate: DateTime!
  endDate: DateTime
  terms: String!
  sla: SLA!
  status: ContractStatus!
}

type InvoicingConfig {
  format: InvoiceFormat!
  frequency: InvoiceFrequency!
  currency: String!
  paymentTerms: String!
  billingAddress: Address!
  emailRecipients: [String!]!
}

enum InvoiceFormat {
  PDF
  XML
  JSON
}

enum InvoiceFrequency {
  MONTHLY
  QUARTERLY
  ANNUAL
}
```

### Key Rules

1. **A Client can own multiple Tenants**
   - Rationale: A single legal entity (e.g., a government agency) may operate multiple domains/identities
   - Example: A defense contractor may have separate tenants for classified and unclassified work
   - Enforcement: Database foreign key constraint (Tenant.clientId → Client.id)

2. **A Tenant cannot span multiple Clients**
   - Rationale: Billing and contractual boundaries must be clear
   - Example: A tenant cannot be shared between two different government agencies
   - Enforcement: Database constraint (Tenant.clientId is NOT NULL and UNIQUE per tenant)

3. **Billing is never tied directly to environments or repos**
   - Rationale: Billing operates at Client/Subscription level, not at operational level
   - Example: Costs are aggregated at Subscription level, not per environment or Git repository
   - Enforcement: Billing APIs only accept Client or Subscription IDs, not Environment or Content IDs

### Relationship to Existing Billing System

The existing Phoenix billing system (documented in `docs/tenants/BILLING_GUIDE.md`) currently tracks billing at the Tenant level. The new operating model introduces Client as the billing boundary:

- **Migration Path**: Existing tenants will be assigned to a default Client (or Clients can be created and tenants assigned)
- **Billing Aggregation**: Client-level billing aggregates costs from all associated Tenants and Subscriptions
- **Backward Compatibility**: Tenant-level billing queries remain available but aggregate to Client level

### Multi-National Client Structures

For international sovereign governments:

- **Single Client, Multiple Tenants**: One Client (the government) with multiple Tenants per nation/region
- **Multiple Clients, Coordinated Tenants**: Separate Clients per nation with coordinated Tenant structures
- **Federated Billing**: Cross-border billing aggregation while maintaining sovereignty

---

## II. Tenancy Plane — Tenants (Domains)

**Purpose:** Identity, domain ownership, trust boundaries, and security isolation.

### Entity: Tenant

A **Tenant** represents an identity and domain boundary. It is the security blast-radius boundary and owns all identity, domain, and security configuration.

#### Attributes

- **Primary Domain(s)**: One or more domains owned by the tenant (e.g., `agency.gov`, `agency.sankofa.nexus`)
- **Identity Provider**: SSO configuration (Entra, Okta, Keycloak, etc.)
- **Global RBAC Namespace**: Root namespace for all RBAC within the tenant
- **Data Residency / Sovereignty Flags**: Regional data residency requirements, sovereignty flags
- **Compliance Profile**: Compliance requirements (ISO, SOC, HIPAA, government-specific standards)
- **Multi-Region Support**: Whether tenant spans multiple regions
- **Regional Data Residency Requirements**: Per-region data residency rules
- **Cross-Border Governance Settings**: Governance rules for cross-border operations

#### Entity Schema

```graphql
type Tenant {
  id: ID!
  name: String!
  primaryDomains: [String!]!
  identityProvider: IdentityProvider!
  rbacNamespace: String!
  dataResidencyFlags: [DataResidencyFlag!]!
  complianceProfile: ComplianceProfile!
  client: Client!
  subscriptions: [Subscription!]!
  environments: [Environment!]!
  regions: [Region!]!
  keycloakRealmId: String
  multiRegionEnabled: Boolean!
  regionalDataResidency: [RegionalDataResidency!]!
  crossBorderGovernance: CrossBorderGovernance
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type IdentityProvider {
  type: IdentityProviderType!
  config: JSON!
  ssoEnabled: Boolean!
  mfaRequired: Boolean!
}

enum IdentityProviderType {
  KEYCLOAK
  AZURE_AD
  OKTA
  GOOGLE_WORKSPACE
  CUSTOM_SAML
  CUSTOM_OIDC
}

type ComplianceProfile {
  standards: [ComplianceStandard!]!
  certifications: [Certification!]!
  auditRequirements: [AuditRequirement!]!
}

enum ComplianceStandard {
  ISO_27001
  ISO_27017
  ISO_27018
  SOC_2
  SOC_3
  HIPAA
  PCI_DSS
  GDPR
  CCPA
  FEDRAMP
  ITAR
  CUSTOM
}

type DataResidencyFlag {
  region: Region!
  requirement: DataResidencyRequirement!
  enforcement: DataResidencyEnforcement!
}

enum DataResidencyRequirement {
  REQUIRED
  PREFERRED
  PROHIBITED
}

enum DataResidencyEnforcement {
  HARD
  SOFT
  ADVISORY
}
```

### Key Rules

1. **One Tenant → many Subscriptions**
   - Rationale: A tenant can have multiple service subscriptions
   - Example: A tenant may have separate subscriptions for compute, data, and AI services
   - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id)

2. **One Tenant → many Environments**
   - Rationale: Environments are scoped to tenants for security isolation
   - Example: A tenant may have DEV, STAGING, and PROD environments
   - Enforcement: Environment belongs to Subscription, which belongs to Tenant

3. **Tenant is the security blast-radius boundary**
   - Rationale: Security incidents are contained within tenant boundaries
   - Example: A compromised tenant cannot access another tenant's resources
   - Enforcement: Network isolation, RBAC namespace isolation, data isolation

### Relationship to Existing Tenant Management

The existing Phoenix tenant management (documented in `docs/tenants/TENANT_MANAGEMENT.md`) aligns with this model:

- **Tenant Tiers**: FREE, STANDARD, ENTERPRISE, SOVEREIGN map to Subscription types
- **Keycloak Integration**: Each tenant gets a Keycloak realm (when `KEYCLOAK_MULTI_REALM=true`)
- **Custom Domains**: Tenant primary domains support custom domain configuration
- **Quotas**: Tenant quotas map to Subscription quotas and limits

### Keycloak Realm Mapping

- **One Tenant = One Keycloak Realm**: Each tenant has its own Keycloak realm for complete identity isolation
- **Realm Name**: Typically matches Tenant ID or primary domain
- **Federated Identity**: Tenants can federate with external identity providers (Entra, Okta, etc.)

### Multi-National Tenant Structures

For international sovereign governments:

- **Per-Nation Tenants**: Separate tenant per nation with coordinated governance
- **Federated Tenants**: Tenants that share identity federation but maintain isolation
- **Cross-Border Tenants**: Tenants that span multiple nations with regional data residency

---

## III. Subscription Plane — Subscriptions

**Purpose:** Logical containers for services, quotas, and spend.

### Entity: Subscription

A **Subscription** represents a service bundle provisioned to a tenant. It defines what services are available, quotas, limits, and cost tracking.

#### Attributes

- **Service Bundles**: Compute, data, AI, storage, networking, etc.
- **Quotas & Limits**: Resource quotas, rate limits, capacity limits
- **Cost Tracking**: Cost aggregation, budget tracking, spending alerts
- **Policy Packs**: Security policies, networking policies, data access policies
- **Feature Entitlements**: Enabled features, beta features, premium features
- **Multi-Region Subscriptions**: Whether subscription spans multiple regions

#### Entity Schema

```graphql
type Subscription {
  id: ID!
  name: String!
  tenant: Tenant!
  client: Client!
  type: SubscriptionType!
  serviceBundles: [ServiceBundle!]!
  quotas: Quotas
  limits: Limits
  costTracking: CostTracking!
  policyPacks: [PolicyPack!]!
  featureEntitlements: [FeatureEntitlement!]!
  environments: [Environment!]!
  regions: [Region!]!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

enum SubscriptionType {
  SHARED_PLATFORM
  PRODUCT
  SANDBOX
  INNOVATION
}

type ServiceBundle {
  service: ServiceType!
  enabled: Boolean!
  quotas: ServiceQuotas
  limits: ServiceLimits
}

enum ServiceType {
  COMPUTE
  STORAGE
  NETWORKING
  DATABASE
  AI_ML
  ANALYTICS
  SECURITY
  MONITORING
  BACKUP
}

type Quotas {
  compute: ComputeQuotas
  storage: StorageQuotas
  network: NetworkQuotas
  custom: JSON
}

type ComputeQuotas {
  vcpu: Int
  memory: Int # GB
  instances: Int
  gpu: Int
}

type PolicyPack {
  name: String!
  type: PolicyPackType!
  policies: [Policy!]!
  enforcement: PolicyEnforcement!
}

enum PolicyPackType {
  SECURITY
  NETWORKING
  DATA_ACCESS
  COMPLIANCE
  CUSTOM
}

enum PolicyEnforcement {
  HARD
  SOFT
  ADVISORY
}
```

### Subscription Types

1. **Shared Platform Subscription**
   - Purpose: Shared infrastructure and platform services
   - Use Cases: Common platform services, shared networking, centralized monitoring
   - Characteristics: Shared resources, cost-efficient, managed by platform team

2. **Product Subscriptions**
   - Purpose: Dedicated resources for specific products or applications
   - Use Cases: Production workloads, customer-facing applications
   - Characteristics: Dedicated resources, higher quotas, production SLAs

3. **Sandbox / Innovation Subscriptions**
   - Purpose: Experimental and development environments
   - Use Cases: Proof of concepts, experimentation, learning
   - Characteristics: Lower quotas, relaxed policies, cost-optimized

### Key Rules

1. **Subscriptions live inside a Tenant**
   - Rationale: Subscriptions inherit tenant identity and security boundaries
   - Example: All subscriptions for a tenant share the same identity provider
   - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id, NOT NULL)

2. **Subscriptions are mapped to one Client billing profile**
   - Rationale: Billing aggregation happens at Client level
   - Example: All subscription costs for a tenant roll up to the tenant's Client
   - Enforcement: Subscription.clientId → Client.id (via Tenant.clientId)

### Multi-Region Subscription Patterns

- **Regional Subscriptions**: Separate subscription per region for regional data residency
- **Global Subscriptions**: Single subscription spanning multiple regions
- **Hybrid Subscriptions**: Mix of regional and global services

---

## IV. Environment Plane — Environments

**Purpose:** Operational isolation for lifecycle stages.

### Entity: Environment

An **Environment** represents a lifecycle stage where workloads run. It provides network isolation, data isolation, and deployment policies.

#### Attributes

- **Network Isolation**: Network boundaries, firewall rules, network policies
- **Data Isolation**: Data boundaries, encryption, access controls
- **Deployment Policies**: Deployment rules, approval workflows, promotion policies
- **Runtime Secrets**: Secrets management, key rotation, access controls
- **Compliance Overlays**: Compliance requirements specific to environment
- **Regional Scope**: Region where environment is deployed

#### Entity Schema

```graphql
type Environment {
  id: ID!
  name: String!
  type: EnvironmentType!
  subscription: Subscription!
  networkIsolation: NetworkIsolation!
  dataIsolation: DataIsolation!
  deploymentPolicies: [DeploymentPolicy!]!
  runtimeSecrets: [Secret!]!
  complianceOverlays: [ComplianceOverlay!]!
  region: Region
  promotionFlow: PromotionFlow
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

enum EnvironmentType {
  DEV
  INT
  UAT
  STAGING
  PROD
  REGULATED
  SOVEREIGN
  AIR_GAPPED
}

type NetworkIsolation {
  vpcId: String
  subnetIds: [String!]!
  firewallRules: [FirewallRule!]!
  networkPolicies: [NetworkPolicy!]!
  allowedConnections: [NetworkConnection!]!
}

type DataIsolation {
  encryptionAtRest: Boolean!
  encryptionInTransit: Boolean!
  accessControls: [AccessControl!]!
  dataBoundaries: [DataBoundary!]!
}

type DeploymentPolicy {
  name: String!
  type: DeploymentPolicyType!
  rules: [DeploymentRule!]!
  approvalRequired: Boolean!
  approvers: [String!]!
}

enum DeploymentPolicyType {
  AUTOMATED
  MANUAL_APPROVAL
  POLICY_DRIVEN
  SCHEDULED
}

type PromotionFlow {
  fromEnvironment: Environment
  toEnvironment: Environment!
  policies: [PromotionPolicy!]!
  approvalRequired: Boolean!
  automated: Boolean!
}
```

### Environment Types

#### Standard Environments

1. **DEV** (Development)
   - Purpose: Developer workstations and development workloads
   - Characteristics: Relaxed policies, high developer access, cost-optimized
   - Access: Developers, DevOps engineers

2. **INT** (Integration)
   - Purpose: Integration testing and component testing
   - Characteristics: Moderate policies, limited access, test data
   - Access: QA engineers, integration testers

3. **UAT** (User Acceptance Testing)
   - Purpose: User acceptance testing and validation
   - Characteristics: Production-like policies, business user access, production-like data
   - Access: Business users, QA engineers

4. **STAGING**
   - Purpose: Pre-production validation and final testing
   - Characteristics: Production-equivalent policies, limited access, production data copies
   - Access: Release managers, senior engineers

5. **PROD** (Production)
   - Purpose: Production workloads serving end users
   - Characteristics: Strictest policies, minimal access, production data
   - Access: Operators, on-call engineers (read-only for most)

#### Specialized Environments

6. **REGULATED**
   - Purpose: Regulated workloads requiring specific compliance
   - Characteristics: Enhanced compliance overlays, audit logging, restricted access
   - Use Cases: Healthcare (HIPAA), Finance (PCI-DSS), Government (FedRAMP)

7. **SOVEREIGN**
   - Purpose: Sovereign workloads requiring data residency
   - Characteristics: Regional data residency, sovereignty flags, cross-border restrictions
   - Use Cases: Government data, national security, sovereign cloud deployments

8. **AIR-GAPPED**
   - Purpose: Classified workloads with no external connectivity
   - Characteristics: Complete network isolation, no internet access, physical security
   - Use Cases: Classified government systems, critical infrastructure

### Key Rules

1. **Environments belong to Subscriptions**
   - Rationale: Environments inherit subscription quotas and policies
   - Example: PROD environment uses Product Subscription quotas
   - Enforcement: Database foreign key (Environment.subscriptionId → Subscription.id, NOT NULL)

2. **Promotion flows are policy-driven, not manual**
   - Rationale: Automated, auditable promotion reduces human error
   - Example: Code promotion from DEV → STAGING → PROD follows defined policies
   - Enforcement: PromotionFlow policies are enforced by CI/CD pipelines

3. **PROD access is always the most restricted**
   - Rationale: Production environments require highest security
   - Example: PROD requires MFA, approval workflows, and audit logging
   - Enforcement: RBAC policies enforce stricter access controls for PROD environments

### Multi-Region Environment Patterns

- **Regional Environments**: Separate environment per region for data residency
- **Global Environments**: Single environment spanning multiple regions
- **Hybrid Environments**: Mix of regional and global components

---

## V. Content & DevOps Plane (Separate but Integrated)

**Purpose:** What is built, governed, and deployed—separate from billing and tenancy.

This plane is intentionally **not** embedded into billing or tenancy. It operates independently but integrates with the other planes through IDs.

### A. Enterprise Content Management Hierarchy

#### Entity Model

```
Enterprise
 └── Portfolio
     └── Product / Program
         └── Application / Service
             └── Component / Module
```

#### Entity Schema

```graphql
type Enterprise {
  id: ID!
  name: String!
  portfolios: [Portfolio!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Portfolio {
  id: ID!
  name: String!
  enterprise: Enterprise!
  products: [Product!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Product {
  id: ID!
  name: String!
  portfolio: Portfolio!
  applications: [Application!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Application {
  id: ID!
  name: String!
  product: Product!
  components: [Component!]!
  gitRepos: [GitRepo!]!
  ownership: Ownership!
  governance: Governance!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

type Component {
  id: ID!
  name: String!
  application: Application!
  contentType: ContentType!
  content: Content!
  ownership: Ownership!
  governance: Governance!
  version: String!
  lineage: [LineageEntry!]!
  createdAt: DateTime!
  updatedAt: DateTime!
  metadata: JSON
}

enum ContentType {
  SOURCE_CODE
  IAC
  PIPELINE
  CONFIG_TEMPLATE
  DOCUMENTATION
  DATA_SCHEMA
  AI_MODEL
  PROMPT
}

type Ownership {
  owner: String!
  team: String
  department: String
  contact: String
}

type Governance {
  approvalWorkflows: [ApprovalWorkflow!]!
  complianceTags: [ComplianceTag!]!
  accessControls: [AccessControl!]!
  retentionPolicies: [RetentionPolicy!]!
}

type ApprovalWorkflow {
  name: String!
  steps: [ApprovalStep!]!
  required: Boolean!
}

type ApprovalStep {
  approver: String!
  role: String!
  timeout: Int
}
```

#### Content Types

1. **Source Code**: Application source code, libraries, dependencies
2. **IaC (Infrastructure as Code)**: Terraform, Pulumi, Bicep, CloudFormation
3. **Pipelines**: CI/CD pipeline definitions, workflow configurations
4. **Configuration Templates**: Environment configs, deployment templates
5. **Documentation**: Technical docs, runbooks, architecture diagrams
6. **Data Schemas**: Database schemas, data models, API schemas
7. **AI Models / Prompts**: ML models, AI prompts, training data

#### Governance

- **Ownership at Each Level**: Clear ownership from Enterprise to Component
- **Approval Workflows**: Required approvals for changes at each level
- **Compliance Tagging**: Compliance tags for regulatory requirements
- **Versioning & Lineage**: Complete version history and lineage tracking

### B. Git & DevOps Integration Model

#### Git Structure

**Enterprise Git Organization**:
- Repositories mapped to **Product / Service** level
- Branch strategy enforced by policy
- Protected branches for regulated environments
- Multi-region Git repository patterns

**Repository Mapping**:
```
Enterprise Git Org
 ├── portfolio-1/
 │   ├── product-a/
 │   │   ├── application-1/ (repo)
 │   │   └── application-2/ (repo)
 │   └── product-b/
 │       └── application-3/ (repo)
 └── portfolio-2/
     └── product-c/
         └── application-4/ (repo)
```

**Branch Strategy**:
- **main/master**: Production-ready code
- **develop**: Integration branch
- **feature/***: Feature branches
- **release/***: Release branches
- **hotfix/***: Production hotfixes

**Protected Branches**:
- **PROD environments**: Require approval, no direct pushes
- **REGULATED environments**: Additional compliance checks
- **SOVEREIGN environments**: Regional approval requirements

#### CI/CD Integration

**Environment-Aware Pipelines**:
- Pipelines detect target environment from branch or configuration
- Environment-specific policies are applied automatically
- Deployment requires explicit environment selection

**Deployment Requirements**:
1. **Subscription Authorization**: Pipeline verifies subscription has required services
2. **Environment Approval**: Automated or manual approval based on environment type
3. **Policy Validation**: Security, compliance, and governance policies are validated

**GitOps for Infrastructure**:
- Infrastructure changes via Git commits
- ArgoCD syncs Git state to infrastructure
- Crossplane provisions infrastructure resources
- Integration with existing ArgoCD infrastructure (see `gitops/README.md`)

#### Promotion Flow

**Standard Promotion Flow**:
```
Code Commit
  ↓
CI (Test, Scan, Build)
  ↓
Artifact Registry
  ↓
Environment Promotion (Policy-Driven)
  ↓
Subscription Deployment
```

**Policy-Driven Promotion**:
- **Automated**: DEV → INT → UAT (automated if tests pass)
- **Approval Required**: UAT → STAGING (requires approval)
- **Strict Approval**: STAGING → PROD (requires multiple approvals, compliance checks)

**Critical Principle**:
> **Git never directly deploys to PROD without environment + subscription authorization.**

**Enforcement**:
- CI/CD pipelines check environment type
- PROD deployments require:
  - Subscription authorization
  - Environment approval workflow
  - Policy validation (security, compliance)
  - Audit logging

**Multi-Region Promotion**:
- Regional promotion flows for data residency
- Cross-region promotion with governance approval
- Sovereign promotion with regional compliance checks

---

## VI. Hierarchical Access Model (RBAC)

### Access Roles by Plane

#### 1. Commercial Access

- **Finance Admin**: Full access to Client billing, invoicing, payment instruments
- **Billing Viewer**: Read-only access to billing and cost data
- **Cost Center Owner**: Access to specific cost center data and chargeback

#### 2. Tenant Access

- **Tenant Owner**: Full control over tenant, identity, domains, subscriptions
- **Security Admin**: Security configuration, compliance, audit access
- **Identity Admin**: Identity provider configuration, user management
- **Compliance Officer**: Compliance configuration, audit, reporting

#### 3. Subscription Access

- **Subscription Owner**: Full control over subscription, quotas, policies
- **Platform Admin**: Platform services administration
- **Service Operator**: Service-specific operations
- **Read-only Auditor**: Read-only access for auditing

#### 4. Environment Access

- **Environment Owner**: Full control over environment configuration
- **Release Manager**: Promotion approval, release management
- **Operator**: Runtime operations, monitoring, troubleshooting
- **Observer**: Read-only access for monitoring

#### 5. Content & DevOps Access

- **Enterprise Architect**: Enterprise-level architecture decisions
- **Portfolio Lead**: Portfolio-level governance and decisions
- **Product Owner**: Product-level decisions and priorities
- **Dev Lead**: Development team leadership
- **Contributor**: Code contribution, development
- **Reviewer**: Code review, approval
- **Release Approver**: Release approval for production

### Cross-Plane Access

**Default Rule**: **No role crosses planes by default.**

**Explicit Delegation Required**:
- Cross-plane access must be explicitly granted
- Delegation is audited and logged
- Delegation can be time-limited or permanent

**Delegation Mechanisms**:
- **Role Delegation**: Grant role from one plane to user in another plane
- **Temporary Access**: Time-limited cross-plane access
- **Escalation**: Emergency escalation procedures

### Multi-Region RBAC

- **Regional Roles**: Roles scoped to specific regions
- **Cross-Region Roles**: Roles that span multiple regions (requires approval)
- **Federated RBAC**: RBAC across federated tenants

### Integration with Keycloak

- **Keycloak Roles**: Phoenix roles map to Keycloak roles
- **Realm-Level Roles**: Tenant-specific roles in Keycloak realm
- **Federated Roles**: Roles from federated identity providers

---

## VII. Key Rules and Constraints

### Commercial Plane Rules

1. **A Client can own multiple Tenants**
   - Rationale: Single legal entity may operate multiple domains/identities
   - Enforcement: Database foreign key (Tenant.clientId → Client.id)

2. **A Tenant cannot span multiple Clients**
   - Rationale: Billing and contractual boundaries must be clear
   - Enforcement: Database constraint (Tenant.clientId is NOT NULL and UNIQUE per tenant)

3. **Billing is never tied directly to environments or repos**
   - Rationale: Billing operates at Client/Subscription level
   - Enforcement: Billing APIs only accept Client or Subscription IDs

### Tenancy Plane Rules

4. **One Tenant → many Subscriptions**
   - Rationale: Tenant can have multiple service subscriptions
   - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id)

5. **One Tenant → many Environments**
   - Rationale: Environments are scoped to tenants for security
   - Enforcement: Environment belongs to Subscription, which belongs to Tenant

6. **Tenant is the security blast-radius boundary**
   - Rationale: Security incidents contained within tenant
   - Enforcement: Network isolation, RBAC namespace isolation, data isolation

### Subscription Plane Rules

7. **Subscriptions live inside a Tenant**
   - Rationale: Subscriptions inherit tenant identity and security
   - Enforcement: Database foreign key (Subscription.tenantId → Tenant.id, NOT NULL)

8. **Subscriptions are mapped to one Client billing profile**
   - Rationale: Billing aggregation at Client level
   - Enforcement: Subscription.clientId → Client.id (via Tenant.clientId)

### Environment Plane Rules

9. **Environments belong to Subscriptions**
   - Rationale: Environments inherit subscription quotas and policies
   - Enforcement: Database foreign key (Environment.subscriptionId → Subscription.id, NOT NULL)

10. **Promotion flows are policy-driven, not manual**
    - Rationale: Automated, auditable promotion reduces error
    - Enforcement: PromotionFlow policies enforced by CI/CD pipelines

11. **PROD access is always the most restricted**
    - Rationale: Production requires highest security
    - Enforcement: RBAC policies enforce stricter access for PROD

### Content & DevOps Plane Rules

12. **Git never directly deploys to PROD without environment + subscription authorization**
    - Rationale: Production deployments require explicit authorization
    - Enforcement: CI/CD pipelines check environment type and require approvals

13. **Content hierarchy ownership is required at each level**
    - Rationale: Clear ownership enables governance
    - Enforcement: Ownership fields are required in entity schemas

### Cross-Plane Rules

14. **No role crosses planes by default**
    - Rationale: Separation of concerns requires explicit cross-plane access
    - Enforcement: RBAC system enforces plane boundaries

15. **Cross-plane access requires explicit delegation**
    - Rationale: Auditability and security require explicit delegation
    - Enforcement: Delegation must be recorded and audited

### Violation Handling

- **Prevention**: Database constraints prevent invalid relationships
- **Detection**: Audit logs detect policy violations
- **Response**: Automated alerts and manual review for violations
- **Remediation**: Automated remediation where possible, manual intervention required for critical violations

---

## VIII. Multi-Region and Multi-National Capabilities

### Sovereign Cloud Deployments

**Per-Region/Nation Sovereign Clouds**:
- Each region/nation can have its own sovereign cloud deployment
- Complete data residency and sovereignty per region
- Regional compliance and governance

### Cross-Region Governance

**Federated Governance**:
- Governance policies can span multiple regions
- Cross-region coordination for multi-national operations
- Regional autonomy with coordinated governance

### Multi-National Tenant Structures

**Per-Nation Tenants**:
- Separate tenant per nation with coordinated governance
- Federated identity across nations
- Cross-border data sharing with governance

### Regional Data Residency

**Data Residency Enforcement**:
- Hard enforcement: Data cannot leave region
- Soft enforcement: Data preferred in region, warnings if outside
- Advisory: Recommendations for data placement

### Landing Zone Patterns

**Regional Landing Zones**:
- Landing zone per region for sovereign deployments
- Cross-region connectivity for coordination
- Regional compliance per landing zone

---

## IX. Decentralized Architecture

### How Decentralization Enables Sovereignty

**Distributed Control**:
- Control planes can be deployed per region
- Regional autonomy with coordinated governance
- No single point of control

**Sovereignty Benefits**:
- Complete control over regional infrastructure
- Data sovereignty per region
- Regulatory compliance per region

### Distributed Control Planes

**Regional Control Planes**:
- Each region can have its own control plane deployment
- Coordinated but not centralized
- Eventual consistency across regions

### Cross-Region Coordination

**Coordination Mechanisms**:
- Event-driven coordination
- API-based coordination
- Governance-based coordination

**Conflict Resolution**:
- Regional autonomy with escalation
- Governance policies for conflict resolution
- Audit trails for coordination decisions

### Federated Identity and Governance

**Federated Identity**:
- Identity federation across regions
- SSO across regions with regional control
- Multi-national identity coordination

**Federated Governance**:
- Governance policies can be federated
- Regional governance with coordination
- Cross-border governance patterns

---

## X. Integration with Existing Infrastructure

### Entity Mapping to Existing Systems

#### Proxmox Infrastructure

**Mapping**:
- **Region** → Proxmox Site
- **Cluster** → Proxmox Cluster
- **Node** → Proxmox Node
- **VM** → Proxmox VM

**Integration**:
- Environments map to Proxmox resource pools
- Subscriptions map to Proxmox quotas
- Tenants map to Proxmox access controls

#### Kubernetes Clusters

**Mapping**:
- **Environment** → Kubernetes Namespace
- **Subscription** → Kubernetes ResourceQuota
- **Tenant** → Kubernetes RBAC namespace

**Integration**:
- Environments deploy to Kubernetes namespaces
- Subscriptions enforce Kubernetes resource quotas
- Tenants enforce Kubernetes RBAC boundaries

#### Cloudflare Tunnels and Zero Trust

**Mapping**:
- **Tenant** → Cloudflare Access Policy
- **Environment** → Cloudflare Tunnel Configuration
- **Region** → Cloudflare Tunnel Endpoint

**Integration**:
- Tenant identity maps to Cloudflare Access policies
- Environments use Cloudflare tunnels for connectivity
- Regions map to Cloudflare tunnel endpoints

#### Keycloak Realms

**Mapping**:
- **Tenant** → Keycloak Realm (1:1)
- **Identity Provider** → Keycloak Identity Provider
- **RBAC Roles** → Keycloak Roles

**Integration**:
- Each tenant gets a Keycloak realm
- Tenant identity provider maps to Keycloak identity provider
- Phoenix RBAC roles map to Keycloak roles

#### ArgoCD Applications

**Mapping**:
- **Application** → ArgoCD Application
- **Environment** → ArgoCD Target Environment
- **Subscription** → ArgoCD Resource Quota

**Integration**:
- Applications deploy via ArgoCD
- Environments map to ArgoCD target environments
- Subscriptions enforce ArgoCD resource quotas

#### Crossplane Resources

**Mapping**:
- **Subscription** → Crossplane Composite Resource
- **Environment** → Crossplane Claim
- **Infrastructure** → Crossplane Managed Resources

**Integration**:
- Subscriptions provision infrastructure via Crossplane
- Environments create Crossplane claims
- Infrastructure resources managed by Crossplane

#### Monitoring and Observability

**Mapping**:
- **Tenant** → Monitoring Namespace
- **Environment** → Monitoring Labels
- **Subscription** → Cost Metrics

**Integration**:
- Monitoring scoped by tenant and environment
- Cost metrics aggregated by subscription
- Alerts configured per environment

### Resource Model Mapping

**Existing Model** (from `docs/architecture/data-model.md`):
```
Region → Site → Cluster → Node → VM/Pod/Service
```

**Operating Model Mapping**:
- **Region**: Maps to Phoenix Region (with Tenant/Subscription context)
- **Site**: Maps to Landing Zone or Environment
- **Cluster**: Maps to Subscription service bundle
- **Node**: Maps to Environment resources
- **VM/Pod/Service**: Maps to Environment workloads

### API Integration Points

**Control Plane APIs**:
- Commercial Plane API: Client and billing operations
- Tenancy Plane API: Tenant and identity operations
- Subscription Plane API: Subscription and quota operations
- Environment Plane API: Environment and deployment operations
- Content & DevOps Plane API: Content and Git operations

**Integration APIs**:
- Proxmox API integration
- Kubernetes API integration
- Cloudflare API integration
- Keycloak API integration
- ArgoCD API integration
- Crossplane API integration

---

## XI. Use Cases for Sovereign Governments

### Use Case 1: Multi-National Defense Contractor

**Scenario**: Defense contractor with classified and unclassified workloads across multiple nations.

**Entity Mapping**:
- **Client**: Defense contractor (single Client)
- **Tenants**: 
  - Tenant 1: Classified workloads (US)
  - Tenant 2: Unclassified workloads (US)
  - Tenant 3: Classified workloads (EU)
  - Tenant 4: Unclassified workloads (EU)
- **Subscriptions**: 
  - Classified Subscription (AIR-GAPPED environments)
  - Unclassified Subscription (REGULATED environments)
- **Environments**: 
  - AIR-GAPPED PROD (classified)
  - REGULATED PROD (unclassified)
- **Landing Zones**: Separate landing zones per nation

**Compliance**: ITAR, FedRAMP, regional data residency

### Use Case 2: International Healthcare Agency

**Scenario**: Healthcare agency operating across multiple countries with HIPAA requirements.

**Entity Mapping**:
- **Client**: Healthcare agency (single Client)
- **Tenants**: 
  - Tenant per country (for data residency)
- **Subscriptions**: 
  - Healthcare Subscription (HIPAA-compliant)
- **Environments**: 
  - REGULATED PROD (HIPAA)
  - REGULATED STAGING (HIPAA)
- **Landing Zones**: Per-country landing zones

**Compliance**: HIPAA, GDPR, regional healthcare regulations

### Use Case 3: Cross-Border Financial Regulator

**Scenario**: Financial regulator coordinating across multiple nations.

**Entity Mapping**:
- **Clients**: One Client per nation (coordinated)
- **Tenants**: 
  - Tenant per nation (federated)
- **Subscriptions**: 
  - Regulatory Subscription (cross-border)
- **Environments**: 
  - REGULATED PROD (financial regulations)
- **Landing Zones**: Per-nation landing zones with cross-border connectivity

**Compliance**: Financial regulations per nation, cross-border coordination

### Use Case 4: Multi-Region Public Sector Agency

**Scenario**: Public sector agency with operations across multiple regions.

**Entity Mapping**:
- **Client**: Public sector agency (single Client)
- **Tenants**: 
  - Tenant per region (for regional autonomy)
- **Subscriptions**: 
  - Public Sector Subscription
- **Environments**: 
  - PROD (public services)
  - STAGING (pre-production)
- **Landing Zones**: Per-region landing zones

**Compliance**: Government regulations, regional data residency

### Use Case 5: Air-Gapped Deployment Per Nation

**Scenario**: Classified government system with complete isolation per nation.

**Entity Mapping**:
- **Client**: Government (one per nation)
- **Tenants**: 
  - Single tenant per nation (complete isolation)
- **Subscriptions**: 
  - Classified Subscription
- **Environments**: 
  - AIR-GAPPED PROD (no external connectivity)
- **Landing Zones**: Air-gapped landing zone per nation

**Compliance**: Classified systems, national security regulations

---

## XII. Glossary

### Core Entities

- **Client (Billing Profile)**: Legal entity that contracts with Phoenix for cloud services. Financial and contractual boundary.
- **Tenant**: Identity and domain boundary. Security blast-radius. Owns identity, domain, and security configuration.
- **Subscription**: Logical container for services, quotas, and spend. Defines what services are available.
- **Environment**: Lifecycle stage where workloads run. Provides network isolation, data isolation, and deployment policies.
- **Landing Zone**: Regional deployment pattern for sovereign cloud deployments.

### Control Planes

- **Commercial Plane**: Financial ownership, invoicing, entitlements, contractual scope.
- **Tenancy Plane**: Identity, domain ownership, trust boundaries, security isolation.
- **Subscription Plane**: Logical containers for services, quotas, and spend.
- **Environment Plane**: Operational isolation for lifecycle stages.
- **Content & DevOps Plane**: What is built, governed, and deployed.

### Environment Types

- **DEV**: Development environment
- **INT**: Integration testing environment
- **UAT**: User acceptance testing environment
- **STAGING**: Pre-production validation environment
- **PROD**: Production environment
- **REGULATED**: Regulated workloads requiring specific compliance
- **SOVEREIGN**: Sovereign workloads requiring data residency
- **AIR-GAPPED**: Classified workloads with no external connectivity

### Subscription Types

- **Shared Platform Subscription**: Shared infrastructure and platform services
- **Product Subscription**: Dedicated resources for specific products
- **Sandbox Subscription**: Experimental and development environments
- **Innovation Subscription**: Innovation and learning environments

### Multi-Region Terminology

- **Sovereign Cloud**: Cloud deployment with complete regional control and data residency
- **Landing Zone**: Regional deployment pattern for sovereign deployments
- **Cross-Region Governance**: Governance policies spanning multiple regions
- **Regional Data Residency**: Requirement that data remains in specific region
- **Federated Identity**: Identity federation across regions or tenants

### Decentralized Architecture Terminology

- **Distributed Control Plane**: Control plane deployed per region
- **Federated Governance**: Governance policies federated across regions
- **Eventual Consistency**: Data consistency achieved over time across regions
- **Regional Autonomy**: Regional control with coordinated governance

### Comparison to Azure/AWS

- **Azure AD Tenant** → **Phoenix Tenant**
- **Azure Subscription** → **Phoenix Subscription**
- **Azure Resource Group** → **Phoenix Environment**
- **AWS Organization** → **Phoenix Client/Tenant**
- **AWS Account** → **Phoenix Subscription**
- **AWS Region** → **Phoenix Region/Landing Zone**

---

## XIII. Why This Works for Phoenix / Sankofa

This operating model provides:

1. **Enterprise-Scale Client Separation**: Support for large multi-tenant deployments
2. **Strong Security & Sovereignty Boundaries**: Tenant as security blast-radius, regional sovereignty
3. **Clean DevOps Velocity**: Content & DevOps separate from billing/tenancy
4. **Clear Monetization and Cost Attribution**: Client-level billing with subscription aggregation
5. **Regulator-Friendly Audit Trails**: Complete governance and compliance logging
6. **Future Readiness**: Sovereign, regulated, and AI workloads supported
7. **Multi-Region Native**: Designed for international/multi-national sovereign governments
8. **Decentralized Architecture**: Supports distributed governance and sovereignty

### Alignment with Industry Standards

- **Azure / AWS / GCP Mental Models**: Familiar concepts with Phoenix enhancements
- **Regulated Banking & Public-Sector Frameworks**: Compliance-ready architecture
- **Multi-Tenant SaaS at Scale**: Enterprise-grade multi-tenancy
- **Sovereign Cloud Requirements**: Regional sovereignty and data residency

### Competitive Advantages

- **Superior Multi-Tenancy**: Better than Azure with finer-grained control
- **Superior Billing**: Per-second granularity vs Azure's hourly
- **Sovereign Identity**: Keycloak-based, no Azure dependencies
- **Multi-Region Native**: Built for international/multi-national deployments
- **Decentralized Architecture**: Supports distributed sovereignty
- **Landing Zone Patterns**: Sovereign cloud deployments per region

---

## References

### Related Phoenix Documentation

- **[Architecture Diagrams](./OPERATING_MODEL_DIAGRAMS.md)** - Visual diagrams of the operating model
- **[Cloud Provider Mapping](./CLOUD_PROVIDER_MAPPING.md)** - Azure/AWS mapping and competitive analysis
- **[MVP Control Plane](./MVP_CONTROL_PLANE.md)** - Minimum viable product specification
- **[Multi-Region Landing Zones](./MULTI_REGION_LANDING_ZONES.md)** - Landing zone patterns and deployment
- **[Migration Guide](./MIGRATION_GUIDE.md)** - Migration from existing systems and cloud providers
- **[Product Specification](./PRODUCT_SPEC.md)** - Client-facing product specification

### Existing Documentation (Tenant-Based Model)

> **Note**: The following documents describe the current tenant-based model. See [Migration Guide](./MIGRATION_GUIDE.md) for migration to the new operating model.

- **[Tenant Management](../tenants/TENANT_MANAGEMENT.md)** - Current multi-tenant operations guide
- **[Billing Guide](../tenants/BILLING_GUIDE.md)** - Current billing and cost management
- **[Identity Setup](../tenants/IDENTITY_SETUP.md)** - Current Keycloak configuration

### Architecture Documentation

- **[Data Model](../architecture/data-model.md)** - GraphQL schema and data model
- **[GitOps Infrastructure](../../gitops/README.md)** - GitOps infrastructure
- **[Architecture Diagrams](../architecture/README.md)** - System architecture diagrams

---

**Last Updated**: 2025-01-09  
**Version**: 1.0  
**Status**: Comprehensive Operating Model Documentation