# VM Specifications - Complete List

## Overview

This document lists all VMs that need to be created for the Sankofa infrastructure, including DevOps services, application services, and infrastructure components.

**Total VMs**: 18 (16 application VMs + 2 infrastructure VMs)  
**Total Resources**: 72 CPU cores, 140 GiB RAM, 278 GiB disk

---

## Infrastructure VMs (2 VMs)

### 1. Nginx Proxy VM
- **Purpose**: DNS/SSL termination and routing between Cloudflare and publicly accessible VMs
- **Key Functions**:
  - SSL/TLS termination
  - Reverse proxy for backend services
  - Load balancing
  - DNS resolution
  - Request routing
- **VM Specs**:
  - **CPU**: 2 cores
  - **RAM**: 4 GiB
  - **Disk**: 20 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - nginx
  - certbot
  - python3-certbot-nginx
  - ufw
  - qemu-guest-agent
  - curl, wget, net-tools
- **File**: `examples/production/nginx-proxy-vm.yaml`

### 2. Cloudflare Tunnel VM
- **Purpose**: Secure tunnel connection to Cloudflare for public access
- **Key Functions**:
  - Cloudflare Tunnel daemon (cloudflared)
  - Secure outbound connections to Cloudflare
  - Tunnel configuration management
  - Health monitoring
- **VM Specs**:
  - **CPU**: 2 cores
  - **RAM**: 4 GiB
  - **Disk**: 10 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - cloudflared (installed via script)
  - ufw
  - qemu-guest-agent
  - curl, wget, net-tools
- **File**: `examples/production/cloudflare-tunnel-vm.yaml`

---

## SMOM-DBIS-138 Application VMs (16 VMs)

### Blockchain Infrastructure (12 VMs)

#### Besu Validators (4 VMs)
- **Purpose**: Hyperledger Besu blockchain validator nodes
- **VM Specs** (per VM):
  - **CPU**: 6 cores
  - **RAM**: 12 GiB
  - **Disk**: 20 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: smom-dbis-138
- **Instances**:
  - `smom-validator-01` (validator-01.yaml)
  - `smom-validator-02` (validator-02.yaml)
  - `smom-validator-03` (validator-03.yaml)
  - `smom-validator-04` (validator-04.yaml)
- **Total Resources**: 24 CPU cores, 48 GiB RAM, 80 GiB disk

#### Besu Sentries (4 VMs)
- **Purpose**: Hyperledger Besu sentry nodes (protect validators from direct internet exposure)
- **VM Specs** (per VM):
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 15 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: smom-dbis-138
- **Instances**:
  - `smom-sentry-01` (sentry-01.yaml)
  - `smom-sentry-02` (sentry-02.yaml)
  - `smom-sentry-03` (sentry-03.yaml)
  - `smom-sentry-04` (sentry-04.yaml)
- **Total Resources**: 16 CPU cores, 32 GiB RAM, 60 GiB disk

#### Besu RPC Nodes (4 VMs)
- **Purpose**: Hyperledger Besu RPC nodes (provide JSON-RPC API access)
- **VM Specs** (per VM):
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 10 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: smom-dbis-138
- **Instances**:
  - `smom-rpc-node-01` (rpc-node-01.yaml)
  - `smom-rpc-node-02` (rpc-node-02.yaml)
  - `smom-rpc-node-03` (rpc-node-03.yaml)
  - `smom-rpc-node-04` (rpc-node-04.yaml)
- **Total Resources**: 16 CPU cores, 32 GiB RAM, 40 GiB disk

### Application Services (4 VMs)

#### Services VM (1 VM)
- **Purpose**: Firefly and Cacti services
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 35 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: smom-dbis-138
- **Instance**: `smom-services` (services.yaml)
- **Services**:
  - Firefly (blockchain application framework)
  - Cacti (network monitoring)

#### Blockscout VM (1 VM)
- **Purpose**: Blockchain explorer for viewing transactions and blocks
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 12 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: smom-dbis-138
- **Instance**: `smom-blockscout` (blockscout.yaml)

#### Monitoring VM (1 VM)
- **Purpose**: Monitoring and observability stack
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 9 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: smom-dbis-138
- **Instance**: `smom-monitoring` (monitoring.yaml)

#### Management VM (1 VM) - Optional
- **Purpose**: Management and administrative tasks
- **VM Specs**:
  - **CPU**: 2 cores
  - **RAM**: 4 GiB
  - **Disk**: 2 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: smom-dbis-138
- **Instance**: `smom-management` (management.yaml)
- **Note**: Marked as optional in deployment documentation

---

## Resource Summary by Category

### Infrastructure VMs
| Component | Count | CPU | RAM | Disk |
|-----------|-------|-----|-----|------|
| Nginx Proxy | 1 | 2 | 4 GiB | 20 GiB |
| Cloudflare Tunnel | 1 | 2 | 4 GiB | 10 GiB |
| **Subtotal** | **2** | **4** | **8 GiB** | **30 GiB** |

### SMOM-DBIS-138 Application VMs
| Component | Count | CPU | RAM | Disk |
|-----------|-------|-----|-----|------|
| Validators | 4 | 24 | 48 GiB | 80 GiB |
| Sentries | 4 | 16 | 32 GiB | 60 GiB |
| RPC Nodes | 4 | 16 | 32 GiB | 40 GiB |
| Services (Firefly/Cacti) | 1 | 4 | 8 GiB | 35 GiB |
| Blockscout | 1 | 4 | 8 GiB | 12 GiB |
| Monitoring | 1 | 4 | 8 GiB | 9 GiB |
| Management (Optional) | 1 | 2 | 4 GiB | 2 GiB |
| **Subtotal** | **16** | **68** | **132 GiB** | **238 GiB** |

### Grand Total
| Category | Count | CPU | RAM | Disk |
|----------|-------|-----|-----|------|
| Infrastructure | 2 | 4 | 8 GiB | 30 GiB |
| Application | 16 | 68 | 132 GiB | 238 GiB |
| **TOTAL** | **18** | **72** | **140 GiB** | **278 GiB** |

---

## Common Configuration

All VMs share the following common configuration:

### Base Image
- **Image**: `ubuntu-22.04-cloud`
- **OS**: Ubuntu 22.04 LTS
- **Image Size**: 691MB
- **Available on**: Both sites (ml110-01 and r630-01)

### Standard Packages
All VMs include:
- `qemu-guest-agent` - For Proxmox integration
- `curl` - HTTP client
- `wget` - File download utility
- `net-tools` - Network utilities
- `apt-transport-https` - HTTPS support for apt
- `ca-certificates` - SSL certificates
- `gnupg` - GPG for package verification
- `lsb-release` - OS release information

### User Configuration
- **User**: `admin`
- **Groups**: `sudo`
- **Shell**: `/bin/bash`
- **Sudo**: NOPASSWD access
- **SSH Key**: Pre-configured with authorized key

### Guest Agent
- QEMU Guest Agent enabled and started on boot
- 30-second verification loop with status output
- Provider sets `agent: 1` in VM config

### Network
- **Bridge**: vmbr0
- **Network**: 192.168.11.0/24
- **Sites**:
  - Site 1: ml110-01 (192.168.11.10)
  - Site 2: r630-01 (192.168.11.11)

### Storage
- **Storage Pool**: local-lvm (default)
- **Alternative Pools**: local, ceph-fs, ceph-rbd

---

## Deployment Order

### Phase 1: Infrastructure (Deploy First)
1. Nginx Proxy VM
2. Cloudflare Tunnel VM

### Phase 2: Blockchain Core
3. Besu Validators (4 VMs)
4. Besu Sentries (4 VMs)
5. Besu RPC Nodes (4 VMs)

### Phase 3: Application Services
6. Services VM (Firefly/Cacti)
7. Blockscout VM
8. Monitoring VM
9. Management VM (Optional)

---

## File Locations

All VM YAML files are located in:
- **Infrastructure VMs**: `examples/production/`
  - `nginx-proxy-vm.yaml`
  - `cloudflare-tunnel-vm.yaml`
- **SMOM-DBIS-138 VMs**: `examples/production/smom-dbis-138/`
  - `validator-01.yaml` through `validator-04.yaml`
  - `sentry-01.yaml` through `sentry-04.yaml`
  - `rpc-node-01.yaml` through `rpc-node-04.yaml`
  - `services.yaml`
  - `blockscout.yaml`
  - `monitoring.yaml`
  - `management.yaml`

---

---

## Additional Infrastructure VMs (Recommended)

### Sankofa Phoenix Core Infrastructure VMs

#### 3. DNS Server VM (Primary)
- **Purpose**: Internal DNS resolution for sankofa.nexus and internal services
- **Key Functions**:
  - Authoritative DNS for sankofa.nexus domains
  - Internal service discovery
  - Split DNS for internal/external resolution
  - DNS caching and forwarding
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 50 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - bind9 (DNS server)
  - bind9utils
  - dnsutils
  - ufw
  - qemu-guest-agent
  - curl, wget, net-tools
- **DNS Zones**:
  - sankofa.nexus (authoritative)
  - *.sankofa.nexus (wildcard)
  - Internal service discovery
- **File**: `examples/production/phoenix/dns-primary.yaml`

#### 4. DNS Server VM (Secondary)
- **Purpose**: Secondary DNS server for redundancy and high availability
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 50 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**: Same as DNS Primary
- **File**: `examples/production/phoenix/dns-secondary.yaml`

#### 5. Email Server VM (Sankofa Mail)
- **Purpose**: Sankofa-branded email server for organizational email
- **Key Functions**:
  - SMTP/IMAP/POP3 services
  - Email authentication (SPF, DKIM, DMARC)
  - Webmail interface
  - Email filtering and antivirus
  - Calendar and contacts (CalDAV/CardDAV)
  - Business email routing
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 16 GiB
  - **Disk**: 200 GiB (for mail storage)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - postfix (SMTP server)
  - dovecot-core dovecot-imapd dovecot-pop3d (IMAP/POP3)
  - opendkim (DKIM signing)
  - opendmarc (DMARC validation)
  - spamassassin (spam filtering)
  - clamav (antivirus)
  - roundcube or rainloop (webmail)
  - ufw
  - qemu-guest-agent
- **Email Domains**:
  - @sankofa.nexus
  - @phoenix.sankofa.nexus
- **File**: `examples/production/phoenix/email-server.yaml`

#### 5a. AS4 Gateway VM (Business Document Exchange)
- **Purpose**: AS4 (Application Server 4) gateway for secure B2B document exchange
- **Key Functions**:
  - AS4 protocol implementation (ebMS 3.0)
  - Secure message exchange (SOAP/WS-Security)
  - Digital signatures and encryption
  - Message reliability (receipts, acknowledgments)
  - Trading partner management
  - Message routing and transformation
  - Compliance with EU eDelivery AS4 profile
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 16 GiB
  - **Disk**: 500 GiB (for message storage and archives)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - java-11-openjdk (for AS4 implementations)
  - openssl
  - xmlsec1 (XML security)
  - ufw
  - qemu-guest-agent
- **Recommended Software**:
  - **Option 1**: Holodeck B2B (open source AS4 implementation)
  - **Option 2**: AS4 Gateway (commercial)
  - **Option 3**: Hermes4AS4 (Java-based)
- **Standards Support**:
  - AS4 (OASIS ebMS 3.0)
  - WS-Security
  - X.509 certificates
  - S/MIME
  - EU eDelivery AS4 profile
- **File**: `examples/production/phoenix/as4-gateway.yaml`

#### 5b. Business Integration Gateway VM (Phoenix Logic Apps)
- **Purpose**: Workflow automation and integration platform (Azure Logic Apps equivalent)
- **Key Functions**:
  - Visual workflow designer
  - API integration and orchestration
  - Business process automation
  - Data transformation (JSON, XML, EDI)
  - Event-driven workflows
  - Scheduled tasks and triggers
  - Connector library (REST, SOAP, databases, etc.)
  - Message queuing and routing
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 16 GiB
  - **Disk**: 200 GiB (for workflow definitions and logs)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - nodejs npm
  - python3 python3-pip
  - postgresql (workflow state)
  - redis-server (message queuing)
  - nginx (reverse proxy)
  - ufw
  - qemu-guest-agent
- **Recommended Software**:
  - **Option 1**: n8n (open source workflow automation)
  - **Option 2**: Apache Airflow (workflow orchestration)
  - **Option 3**: Camunda (BPMN workflow engine)
  - **Option 4**: Temporal (workflow orchestration)
- **Integration Capabilities**:
  - REST APIs
  - SOAP services
  - Database connectors
  - File system operations
  - Email/SMS integration
  - Blockchain integration
  - AS4 gateway integration
  - Financial messaging integration
- **File**: `examples/production/phoenix/business-integration-gateway.yaml`

#### 5c. Financial Messaging Gateway VM
- **Purpose**: Financial message handling and envelope processing
- **Key Functions**:
  - SWIFT message processing
  - ISO 20022 message format support
  - Financial envelope handling (MT/MX messages)
  - Payment message processing
  - Securities message processing
  - Trade finance messages
  - Message validation and routing
  - Compliance and audit logging
  - Integration with banking systems
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 16 GiB
  - **Disk**: 500 GiB (for message archives and audit logs)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - java-11-openjdk (for financial message processing)
  - python3 python3-pip
  - postgresql (message database)
  - redis-server (message queuing)
  - openssl (encryption)
  - xmlsec1 (XML security)
  - ufw
  - qemu-guest-agent
- **Standards Support**:
  - ISO 20022 (MX messages)
  - SWIFT MT messages
  - FIX protocol
  - EDI X12 (financial transactions)
  - EDIFACT (international trade)
  - SEPA (Single Euro Payments Area)
- **Security**:
  - Message encryption
  - Digital signatures
  - PKI integration
  - Audit trails
  - Compliance reporting
- **File**: `examples/production/phoenix/financial-messaging-gateway.yaml`

#### 6. Git Server VM (Sankofa Git)
- **Purpose**: Self-hosted Git repository server (GitLab/Gitea/Forgejo)
- **Key Functions**:
  - Git repository hosting
  - Issue tracking
  - CI/CD integration
  - Code review and pull requests
  - Wiki and documentation
  - Container registry (optional)
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 16 GiB
  - **Disk**: 500 GiB (for repositories and artifacts)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - git
  - docker.io (for GitLab/Gitea containers)
  - docker-compose
  - nginx (reverse proxy)
  - postgresql (database for GitLab)
  - redis-server (caching)
  - ufw
  - qemu-guest-agent
- **Recommended Software**:
  - **Option 1**: GitLab CE (full-featured, resource-intensive)
  - **Option 2**: Gitea (lightweight, Go-based)
  - **Option 3**: Forgejo (Gitea fork, community-driven)
- **File**: `examples/production/phoenix/git-server.yaml`

#### 6a. Phoenix Codespaces IDE VM
- **Purpose**: Branded cloud-based IDE with Copilot-like AI and Agents
- **Key Functions**:
  - VS Code in browser (code-server)
  - AI-powered code completion (Copilot-like)
  - AI agents for automation and assistance
  - Git integration with Phoenix Git server
  - Multi-language support
  - Terminal access
  - Extension marketplace
  - Phoenix branding and customization
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 32 GiB (higher RAM for AI processing)
  - **Disk**: 200 GiB (for workspace storage and AI models)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - code-server (VS Code in browser)
  - docker.io (for containerized workspaces)
  - docker-compose
  - nginx (reverse proxy with SSL)
  - certbot (SSL certificates)
  - python3 python3-pip (for AI tools)
  - nodejs npm (for extensions)
  - git (Git integration)
  - build-essential (compilation tools)
  - ufw (firewall)
  - qemu-guest-agent
- **AI Integration**:
  - **Code Completion**: GitHub Copilot API or alternative (Tabby, Codeium, Cursor)
  - **AI Agents**: LangChain, AutoGPT, or custom Phoenix AI agents
  - **LLM Support**: Integration with OpenAI-compatible APIs or local models
  - **Code Analysis**: AI-powered code review and suggestions
- **Features**:
  - Phoenix-branded interface
  - Integration with Phoenix Git server
  - Workspace templates for common stacks
  - Pre-configured development environments
  - AI-powered code generation
  - Automated testing and debugging assistance
  - Multi-user support with isolation
- **File**: `examples/production/phoenix/codespaces-ide.yaml`

#### 7. Phoenix DevOps VM (CI/CD Runner)
- **Purpose**: Continuous Integration and Continuous Deployment infrastructure
- **Key Functions**:
  - CI/CD pipeline execution
  - Build artifact storage
  - Docker image building
  - Automated testing
  - Deployment automation
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 16 GiB
  - **Disk**: 200 GiB (for build artifacts and cache)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - git
  - build-essential
  - nodejs npm (for Node.js builds)
  - python3 python3-pip (for Python builds)
  - golang-go (for Go builds)
  - jq (JSON processing)
  - kubectl (Kubernetes CLI)
  - helm (Kubernetes package manager)
  - ufw
  - qemu-guest-agent
- **CI/CD Tools**:
  - **Option 1**: GitLab Runner (if using GitLab)
  - **Option 2**: Jenkins
  - **Option 3**: GitHub Actions Runner (self-hosted)
  - **Option 4**: Tekton (Kubernetes-native)
- **File**: `examples/production/phoenix/devops-runner.yaml`

#### 8. Phoenix DevOps Controller VM
- **Purpose**: CI/CD orchestration and coordination
- **Key Functions**:
  - Pipeline scheduling
  - Job queue management
  - Artifact repository
  - Secret management integration
  - Notification services
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 100 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - kubectl
  - helm
  - vault (for secret management)
  - ufw
  - qemu-guest-agent
- **File**: `examples/production/phoenix/devops-controller.yaml`

### Sankofa Phoenix Platform VMs

#### 9. Phoenix Control Plane VM (Primary)
- **Purpose**: Primary control plane for Phoenix cloud platform
- **Key Functions**:
  - Kubernetes control plane (if not using managed K8s)
  - Crossplane provider management
  - Resource orchestration
  - API gateway
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 16 GiB
  - **Disk**: 100 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: phoenix
- **Pre-installed Packages**:
  - kubernetes (kubeadm/kubelet/kubectl)
  - docker.io
  - containerd
  - ufw
  - qemu-guest-agent
- **File**: `examples/production/phoenix/control-plane-primary.yaml`

#### 10. Phoenix Control Plane VM (Secondary)
- **Purpose**: Secondary control plane for high availability
- **VM Specs**: Same as Primary
- **Site**: site-2
- **Node**: r630-01
- **File**: `examples/production/phoenix/control-plane-secondary.yaml`

#### 11. Phoenix Database VM (Primary)
- **Purpose**: Primary database for Phoenix platform services
- **VM Specs**:
  - **CPU**: 8 cores
  - **RAM**: 32 GiB
  - **Disk**: 500 GiB (for database storage)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: phoenix
- **Pre-installed Packages**:
  - postgresql-14 (or latest)
  - postgresql-contrib
  - pgbackrest (backup tool)
  - ufw
  - qemu-guest-agent
- **File**: `examples/production/phoenix/database-primary.yaml`

#### 12. Phoenix Database VM (Replica)
- **Purpose**: Database replica for high availability and read scaling
- **VM Specs**: Same as Primary
- **Site**: site-2
- **Node**: r630-01
- **File**: `examples/production/phoenix/database-replica.yaml`

### Additional Infrastructure Recommendations

#### 13. Backup Server VM
- **Purpose**: Centralized backup storage and management
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 2 TiB (large storage for backups)
  - **Storage**: local-lvm or dedicated storage pool
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - borgbackup (deduplicating backup tool)
  - restic (backup tool)
  - rsync
  - samba (SMB shares for Windows backups)
  - ufw
  - qemu-guest-agent
- **File**: `examples/production/phoenix/backup-server.yaml`

#### 14. Log Aggregation VM
- **Purpose**: Centralized log collection and analysis
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 16 GiB
  - **Disk**: 500 GiB (for log storage)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - ufw
  - qemu-guest-agent
- **Software Stack**:
  - **Option 1**: ELK Stack (Elasticsearch, Logstash, Kibana)
  - **Option 2**: Loki + Grafana (lightweight)
  - **Option 3**: Graylog
- **File**: `examples/production/phoenix/log-aggregation.yaml`

#### 15. Certificate Authority VM
- **Purpose**: Internal Certificate Authority for SSL/TLS certificates
- **VM Specs**:
  - **CPU**: 2 cores
  - **RAM**: 4 GiB
  - **Disk**: 20 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - easy-rsa (PKI management)
  - openssl
  - cfssl (Cloudflare's PKI toolkit)
  - ufw
  - qemu-guest-agent
- **File**: `examples/production/phoenix/certificate-authority.yaml`

#### 16. Monitoring VM (Phoenix)
- **Purpose**: Dedicated monitoring for Phoenix infrastructure
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 200 GiB (for metrics storage)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-2
  - **Node**: r630-01
  - **Tenant**: phoenix
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - ufw
  - qemu-guest-agent
- **Software Stack**:
  - Prometheus (metrics collection)
  - Grafana (visualization)
  - Alertmanager (alerting)
  - Node Exporter (system metrics)
- **File**: `examples/production/phoenix/monitoring.yaml`

#### 17. VPN Gateway VM
- **Purpose**: VPN server for secure remote access
- **VM Specs**:
  - **CPU**: 2 cores
  - **RAM**: 4 GiB
  - **Disk**: 20 GiB
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - wireguard (modern VPN)
  - openvpn (alternative)
  - ufw
  - qemu-guest-agent
- **File**: `examples/production/phoenix/vpn-gateway.yaml`

#### 18. Container Registry VM
- **Purpose**: Private Docker/OCI container registry
- **VM Specs**:
  - **CPU**: 4 cores
  - **RAM**: 8 GiB
  - **Disk**: 500 GiB (for container images)
  - **Storage**: local-lvm
  - **Network**: vmbr0
  - **Image**: ubuntu-22.04-cloud
  - **Site**: site-1
  - **Node**: ml110-01
  - **Tenant**: infrastructure
- **Pre-installed Packages**:
  - docker.io
  - docker-compose
  - nginx (reverse proxy)
  - ufw
  - qemu-guest-agent
- **Software**:
  - **Option 1**: Harbor (enterprise registry)
  - **Option 2**: Docker Registry (simple)
  - **Option 3**: GitLab Container Registry (if using GitLab)
- **File**: `examples/production/phoenix/container-registry.yaml`

---

## Updated Resource Summary

### Additional Infrastructure VMs
| Component | Count | CPU | RAM | Disk |
|-----------|-------|-----|-----|------|
| DNS Servers (Primary/Secondary) | 2 | 8 | 16 GiB | 100 GiB |
| Email Server | 1 | 8 | 16 GiB | 200 GiB |
| AS4 Gateway | 1 | 8 | 16 GiB | 500 GiB |
| Business Integration Gateway | 1 | 8 | 16 GiB | 200 GiB |
| Financial Messaging Gateway | 1 | 8 | 16 GiB | 500 GiB |
| Git Server | 1 | 8 | 16 GiB | 500 GiB |
| Phoenix Codespaces IDE | 1 | 8 | 32 GiB | 200 GiB |
| DevOps Runner | 1 | 8 | 16 GiB | 200 GiB |
| DevOps Controller | 1 | 4 | 8 GiB | 100 GiB |
| Phoenix Control Plane (Primary/Secondary) | 2 | 16 | 32 GiB | 200 GiB |
| Phoenix Database (Primary/Replica) | 2 | 16 | 64 GiB | 1000 GiB |
| Backup Server | 1 | 4 | 8 GiB | 2 TiB |
| Log Aggregation | 1 | 4 | 16 GiB | 500 GiB |
| Certificate Authority | 1 | 2 | 4 GiB | 20 GiB |
| Monitoring (Phoenix) | 1 | 4 | 8 GiB | 200 GiB |
| VPN Gateway | 1 | 2 | 4 GiB | 20 GiB |
| Container Registry | 1 | 4 | 8 GiB | 500 GiB |
| **Subtotal** | **20** | **122** | **300 GiB** | **7.24 TiB** |

### Complete Infrastructure Total
| Category | Count | CPU | RAM | Disk |
|----------|-------|-----|-----|------|
| Original Infrastructure | 2 | 4 | 8 GiB | 30 GiB |
| SMOM-DBIS-138 Application | 16 | 68 | 132 GiB | 238 GiB |
| Additional Infrastructure | 20 | 122 | 300 GiB | 7.24 TiB |
| **GRAND TOTAL** | **38** | **194** | **440 GiB** | **7.51 TiB** |

---

## Deployment Priority

### Phase 1: Critical Infrastructure (Deploy First)
1. DNS Servers (Primary/Secondary) - Required for all services
2. Nginx Proxy VM
3. Cloudflare Tunnel VM
4. Certificate Authority VM

### Phase 2: Core Services
5. Email Server
6. AS4 Gateway (Business Document Exchange)
7. Business Integration Gateway (Phoenix Logic Apps)
8. Financial Messaging Gateway
9. Git Server
10. Phoenix Codespaces IDE
11. Container Registry
12. VPN Gateway

### Phase 3: DevOps Infrastructure
9. DevOps Controller
10. DevOps Runner
11. Log Aggregation

### Phase 4: Phoenix Platform
12. Phoenix Control Plane (Primary/Secondary)
13. Phoenix Database (Primary/Replica)
14. Monitoring (Phoenix)

### Phase 5: Supporting Services
15. Backup Server
16. SMOM-DBIS-138 Blockchain Infrastructure
17. SMOM-DBIS-138 Application Services

---

## Deployment Optimization

### Quota Checking

**Automatic**: The Crossplane controller automatically checks quota for all VMs with tenant labels before deployment.

**Manual**: Run pre-deployment quota check:
```bash
./scripts/pre-deployment-quota-check.sh
```

**Validation**: Validate VM configurations:
```bash
./scripts/validate-and-optimize-vms.sh
```

### Command Optimization

All VM configurations use non-compounded commands for better error handling:
- Commands are separated into individual list items
- Critical operations have explicit error checking
- Non-critical operations may use `|| true` for graceful degradation

See `docs/VM_DEPLOYMENT_OPTIMIZATION.md` for detailed guidelines.

### Image Standardization

- **Standard Image**: `ubuntu-22.04-cloud` (691MB)
- **Format**: QCOW2
- **Availability**: Both sites (ml110-01 and r630-01)
- **Handling**: Controller automatically searches and imports if needed

## Notes

1. **Management VM**: Marked as optional in deployment documentation
2. **Cacti**: Combined with Firefly in the services.yaml VM
3. **Sankofa Phoenix VMs**: Now included in this comprehensive list
4. **Image Handling**: Provider automatically searches and imports images
5. **Multi-tenancy**: VMs are labeled with tenant IDs for resource isolation
6. **High Availability**: Critical services should be distributed across both sites
7. **Storage Considerations**: Large storage VMs (Git, Database, Backup) may need dedicated storage pools
8. **DNS**: Primary and secondary DNS servers provide redundancy
9. **Email**: Consider email deliverability and SPF/DKIM/DMARC configuration
10. **Git Server**: Choose GitLab for full features or Gitea/Forgejo for lightweight deployment
11. **Backup Strategy**: Implement automated backups for all critical VMs
12. **Monitoring**: Deploy monitoring before other services to track deployment health
13. **Quota Enforcement**: All tenant VMs automatically check quota before deployment
14. **Command Optimization**: All commands are non-compounded for better error handling
15. **Validation**: Use validation scripts before deployment

---

**Last Updated**: 2025-12-08  
**Status**: Production Ready - Comprehensive Infrastructure Plan