Sankofa/docs/phoenix/PLAN_REVIEW.md

Phoenix Operating Model Documentation Plan - Review

Review Date

2025-01-09

Purpose

This document reviews the plan for creating Phoenix Operating Model documentation, identifying inconsistencies and gaps that need to be addressed before implementation.

---

1. INCONSISTENCIES

1.1 Entity Model Terminology

Issue: Existing documentation uses "Tenant" as the primary entity, but the operating model introduces "Client (Billing Profile)" as a separate entity above Tenant.

Existing Docs:

• docs/tenants/TENANT_MANAGEMENT.md - Uses "Tenant" as primary entity
• docs/tenants/BILLING_GUIDE.md - Billing tied to Tenant
• docs/tenants/IDENTITY_SETUP.md - Identity tied to Tenant

Operating Model:

• Client (Billing Profile) → Tenant → Subscription → Environment
• Client owns billing, Tenant owns identity/domain

Resolution Needed:

• Clarify relationship: Client (billing) vs Tenant (identity)
• Update existing tenant docs to align with new model OR
• Document migration path from current model to new model
• Specify how existing tenant-based billing maps to Client-based billing

1.2 Billing Model Alignment

Issue: Existing billing documentation shows per-second billing tied to tenants, but operating model separates billing (Client) from tenancy.

Existing:

• Billing tracked per tenant
• Tenant quotas and limits

Operating Model:

• Billing at Client level
• Subscriptions mapped to Client billing profile
• Billing never tied directly to environments or repos

Resolution Needed:

• Document how Client billing aggregates across multiple Tenants
• Explain subscription-to-client billing mapping
• Clarify cost attribution across planes

1.3 Identity Model Alignment

Issue: Existing identity docs show Keycloak realms per tenant, but operating model has Tenant as identity boundary.

Existing:

• KEYCLOAK_MULTI_REALM=true creates realm per tenant
• Identity tied to tenant

Operating Model:

• Tenant = identity provider, domain ownership, trust boundaries
• One Tenant → many Subscriptions
• Tenant is security blast-radius boundary

Resolution Needed:

• Align Keycloak realm model with Tenant entity
• Document how multi-national governments map to tenants
• Clarify federated identity across regions

---

2. GAPS IN COVERAGE

2.1 Content & DevOps Plane - Insufficient Detail

Gap: The plan mentions Content & DevOps plane but lacks specific implementation details.

Missing Elements:

• Enterprise Content Hierarchy:

  • Enterprise → Portfolio → Product/Program → Application/Service → Component/Module
  • Ownership model at each level
  • Approval workflows per level
  • Compliance tagging per level
  • Versioning & lineage tracking

• Git Structure:

  • Enterprise Git Org structure
  • Repo mapping to Product/Service
  • Branch strategy enforcement
  • Protected branches for regulated environments
  • Multi-region Git repository patterns

• CI/CD Integration:

  • How pipelines are environment-aware
  • Subscription authorization in pipelines
  • Environment approval workflows
  • Policy validation in CI/CD
  • GitOps for infra & platform services
  • Integration with existing ArgoCD infrastructure

• Promotion Flow Details:

  • Code Commit → CI (Test, Scan) → Artifact Registry → Environment Promotion → Subscription Deployment
  • Policy-driven promotion (not manual)
  • Critical principle: Git never directly deploys to PROD without environment + subscription authorization
  • Multi-region promotion patterns

Resolution Needed: Expand Content & DevOps section with:

• Detailed entity model for content hierarchy
• Git repository structure and governance
• CI/CD pipeline architecture
• Promotion flow state machine
• Integration with existing GitOps (ArgoCD) infrastructure

2.2 Multi-Region Landing Zones - Implementation Details

Gap: Plan mentions multi-region landing zones but lacks implementation specifics.

Missing Elements:

• Landing zone architecture patterns
• How sovereign clouds are deployed per region/nation
• Cross-region connectivity patterns
• Regional data residency enforcement
• Multi-national tenant structure implementation
• Landing zone templates and automation
• Integration with existing Proxmox/Kubernetes infrastructure

Resolution Needed: Add to MULTI_REGION_LANDING_ZONES.md:

• Landing zone reference architecture
• Deployment automation patterns
• Cross-region governance mechanisms
• Data residency enforcement
• Network connectivity patterns
• Integration with existing infrastructure

2.3 Decentralized Architecture - Practical Implementation

Gap: Plan mentions decentralized nature but lacks practical implementation details.

Missing Elements:

• How control planes are distributed across regions
• Federated governance mechanisms
• Cross-region coordination protocols
• Conflict resolution in decentralized model
• Eventual consistency patterns
• Disaster recovery in decentralized model
• How decentralization differs from Azure/AWS centralized model

Resolution Needed: Add detailed section on:

• Distributed control plane architecture
• Federated governance patterns
• Cross-region coordination
• Decentralized vs centralized comparison
• Implementation patterns and best practices

2.4 Integration with Existing Infrastructure

Gap: Plan mentions integration but lacks specific details on how operating model integrates with existing systems.

Missing Elements:

• How Client/Tenant/Subscription/Environment map to:

  • Existing Proxmox infrastructure
  • Kubernetes clusters
  • Cloudflare tunnels and Zero Trust
  • Keycloak realms
  • ArgoCD applications
  • Crossplane resources
  • Monitoring and observability

• How existing resource model (Region → Site → Cluster → Node) maps to:

  • Tenant boundaries
  • Subscription boundaries
  • Environment boundaries

Resolution Needed: Add integration mapping section:

• Entity mapping to existing infrastructure
• Migration path for existing resources
• Integration patterns for each control plane
• API integration points

2.5 Multi-National Sovereign Government Scenarios

Gap: Plan mentions international/multi-national governments but lacks specific scenarios.

Missing Elements:

• Example scenarios:

  • Multi-national defense contractor with classified workloads
  • International healthcare agency with HIPAA requirements
  • Cross-border financial regulator
  • Multi-region public sector agency
  • Air-gapped deployment per nation

• How each scenario maps to:

  • Client structure (one per nation? one per agency?)
  • Tenant structure (per nation? per agency? federated?)
  • Subscription structure
  • Environment structure
  • Landing zone structure

Resolution Needed: Add use case section with:

• Detailed scenario descriptions
• Entity mapping for each scenario
• Architecture patterns per scenario
• Compliance requirements per scenario

2.6 RBAC Model - Cross-Plane Access

Gap: Plan mentions RBAC but lacks details on cross-plane access delegation.

Missing Elements:

• How roles are scoped (per plane? cross-plane?)
• Cross-plane access delegation mechanisms
• Explicit delegation requirements
• Role hierarchy across planes
• Permission inheritance patterns
• Multi-region RBAC patterns

Resolution Needed: Expand RBAC section with:

• Role definitions per plane
• Cross-plane delegation model
• Permission inheritance rules
• Multi-region RBAC patterns
• Integration with Keycloak roles

2.7 Key Rules and Constraints

Gap: Original operating model specifies key rules that need explicit documentation.

Missing Rules:

• A Client can own multiple Tenants
• A Tenant cannot span multiple Clients
• Billing is never tied directly to environments or repos
• One Tenant → many Subscriptions
• One Tenant → many Environments
• Subscriptions live inside a Tenant
• Subscriptions are mapped to one Client billing profile
• Environments belong to Subscriptions
• Promotion flows are policy-driven, not manual
• PROD access is always the most restricted
• Git never directly deploys to PROD without environment + subscription authorization
• No role crosses planes by default
• Cross-plane access requires explicit delegation

Resolution Needed: Add "Key Rules and Constraints" section to OPERATING_MODEL.md with:

• All rules explicitly stated
• Rationale for each rule
• Enforcement mechanisms
• Violation handling

---

3. MISSING ELEMENTS FROM ORIGINAL MODEL

3.1 Entity Attributes

Missing: Specific attributes for each entity type.

Client (Billing Profile):

• Legal Entity
• Contract & MSA
• Invoicing configuration
• Payment instruments
• Cost centers / departments
• Usage aggregation & chargeback

Tenant:

• Primary domain(s)
• Identity provider (SSO, Entra, Okta, etc.)
• Global RBAC namespace
• Data residency / sovereignty flags
• Compliance profile (ISO, SOC, HIPAA, etc.)

Subscription:

• Service bundles (compute, data, AI, storage, etc.)
• Quotas & limits
• Cost tracking
• Policy packs (security, networking, data access)
• Feature entitlements

Environment:

• Network isolation
• Data isolation
• Deployment policies
• Runtime secrets
• Compliance overlays

Resolution Needed: Add detailed entity schemas with all attributes.

3.2 Subscription Types

Missing: Specific subscription types mentioned in original model.

Original Model Specifies:

• Shared Platform Subscription
• Product Subscriptions
• Sandbox / Innovation Subscriptions

Resolution Needed: Document each subscription type with:

• Purpose and use cases
• Quota and limit differences
• Policy pack differences
• Cost model differences

3.3 Environment Types

Missing: Complete list of environment types.

Original Model Specifies:

• DEV
• INT
• UAT
• STAGING
• PROD
• REGULATED (optional)
• SOVEREIGN (optional)
• AIR-GAPPED (optional)

Resolution Needed: Document each environment type with:

• Purpose and characteristics
• Isolation requirements
• Access restrictions
• Promotion flow rules
• Compliance requirements

3.4 Content Types

Missing: Specific content types in Content & DevOps plane.

Original Model Specifies:

• Source code
• IaC (Terraform, Pulumi, Bicep)
• Pipelines
• Configuration templates
• Documentation
• Data schemas
• AI models / prompts

Resolution Needed: Document each content type with:

• Storage location
• Versioning strategy
• Access controls
• Governance rules

---

4. DOCUMENTATION STRUCTURE GAPS

4.1 Missing Cross-References

Gap: Plan doesn't specify how new docs relate to existing docs.

Needed:

• Cross-references between:
  • OPERATING_MODEL.md and existing tenant/billing docs
  • OPERATING_MODEL.md and architecture docs
  • OPERATING_MODEL.md and GitOps docs
  • CLOUD_PROVIDER_MAPPING.md and existing infrastructure docs

Resolution Needed: Add cross-reference section to each document.

4.2 Missing Glossary

Gap: No glossary of terms, especially for entities that differ from Azure/AWS terminology.

Needed:

• Definitions for: Client, Tenant, Subscription, Environment
• Comparison to Azure/AWS equivalents
• Multi-region terminology
• Decentralized architecture terminology

Resolution Needed: Add glossary section to OPERATING_MODEL.md.

4.3 Missing Migration Guide

Gap: No guide for migrating from existing model to new operating model.

Needed:

• Migration from tenant-based to Client/Tenant/Subscription model
• Migration from existing infrastructure to new control planes
• Migration from Azure/AWS to Phoenix

Resolution Needed: Add MIGRATION_GUIDE.md.

---

5. DIAGRAM GAPS

5.1 Missing Diagrams

Gap: Plan specifies diagrams but some are missing.

Missing:

• Entity relationship diagram showing all relationships
• Data flow diagram for cross-plane operations
• Sequence diagram for promotion flow
• Architecture diagram showing decentralized control planes
• Comparison diagram (Phoenix vs Azure vs AWS)

Resolution Needed: Add to OPERATING_MODEL_DIAGRAMS.md.

5.2 Diagram Detail Level

Gap: Diagrams may be too high-level for implementation.

Needed:

• More detailed entity relationship diagrams
• Component interaction diagrams
• API interaction diagrams
• Multi-region topology diagrams

Resolution Needed: Specify detail level for each diagram.

---

6. COMPETITIVE ANALYSIS GAPS

6.1 Feature Comparison Matrix

Gap: Plan mentions feature comparison but lacks structure.

Needed:

• Structured comparison table:
  • Multi-tenancy capabilities
  • Billing granularity
  • Identity management
  • Multi-region support
  • Decentralized architecture
  • Sovereign capabilities
  • Compliance features

Resolution Needed: Add detailed comparison matrix to CLOUD_PROVIDER_MAPPING.md.

6.2 Migration Considerations

Gap: Plan mentions migration but lacks specific considerations.

Needed:

• Migration complexity assessment
• Data migration strategies
• Identity migration strategies
• Application migration strategies
• Cost migration analysis
• Timeline estimates

Resolution Needed: Expand migration section in CLOUD_PROVIDER_MAPPING.md.

---

7. MVP GAPS

7.1 MVP Scope Definition

Gap: MVP_CONTROL_PLANE.md needs more specific scope.

Needed:

• Which features are MVP vs future
• Which planes are MVP vs future
• Which integrations are MVP vs future
• Timeline for MVP
• Success criteria for MVP

Resolution Needed: Add detailed MVP scope section.

7.2 MVP Implementation Priorities

Gap: Plan mentions priorities but lacks specific ordering.

Needed:

• Prioritized list of MVP features
• Dependencies between features
• Critical path analysis
• Risk assessment per feature

Resolution Needed: Add prioritized implementation roadmap.

---

8. RECOMMENDATIONS

8.1 Immediate Actions

1. Resolve Entity Model Inconsistencies:

   • Create mapping document showing Client vs Tenant relationship
   • Update existing tenant docs or create migration guide
   • Clarify billing model alignment

2. Expand Content & DevOps Plane:

   • Add detailed entity model
   • Document Git structure and governance
   • Detail CI/CD integration patterns
   • Specify promotion flow implementation

3. Add Key Rules Section:

   • Document all rules from original model
   • Add enforcement mechanisms
   • Specify violation handling

4. Create Integration Mapping:

   • Map new entities to existing infrastructure
   • Document integration points
   • Create migration path

8.2 Documentation Enhancements

1. Add Glossary: Define all terms, especially those differing from Azure/AWS
2. Add Cross-References: Link new docs to existing docs
3. Add Use Cases: Detailed scenarios for multi-national governments
4. Expand Diagrams: More detailed diagrams for implementation
5. Add Migration Guide: Guide for migrating to new model

8.3 Plan Updates Needed

1. Add MIGRATION_GUIDE.md to deliverables
2. Expand Content & DevOps section in all documents
3. Add Integration Mapping section to OPERATING_MODEL.md
4. Add Glossary section to OPERATING_MODEL.md
5. Add Key Rules section to OPERATING_MODEL.md
6. Expand MVP scope definition
7. Add detailed entity schemas to OPERATING_MODEL.md

---

9. PRIORITY ORDER FOR ADDRESSING GAPS

High Priority (Block Implementation)

1. Resolve entity model inconsistencies (Client vs Tenant)
2. Add key rules and constraints section
3. Expand Content & DevOps plane details
4. Create integration mapping with existing infrastructure

Medium Priority (Important for Completeness)

5. Add multi-national government use cases
6. Expand decentralized architecture details
7. Add detailed entity schemas
8. Create migration guide

Low Priority (Enhancement)

9. Add glossary
10. Expand diagrams
11. Enhance competitive analysis
12. Refine MVP scope

---

10. CONCLUSION

The plan provides a solid foundation but requires significant expansion in:

• Content & DevOps plane details
• Integration with existing infrastructure
• Entity model consistency resolution
• Key rules and constraints documentation
• Multi-national government scenarios
• Decentralized architecture implementation

Addressing these gaps will ensure the documentation is comprehensive, consistent, and actionable for implementation.