proxmox/docs/03-deployment/PHOENIX_MIGRATION_PLAN_DBIS_CHAIN138.md

# Phoenix Migration Plan: DBIS & ChainID 138 Services

**System:** Phoenix Core Migration Plan  
**Target:** Migrate DBIS Core and ChainID 138 services to Phoenix architecture  
**Version:** 1.0.0  
**Last Updated:** 2026-01-09
**Status:** Active Documentation

---

## Executive Summary

This document provides a comprehensive migration plan for migrating DBIS Core services and ChainID 138 (Besu/Blockchain) services from the legacy flat LAN architecture (192.168.11.x) to the Phoenix Core architecture on VLAN 160 (10.160.0.x).

**Migration Strategy:** Phased migration with zero-downtime DNS-based cutover. Legacy services remain operational during migration.

---

## Table of Contents

1. [Current State Analysis](#current-state-analysis)
2. [Target State Architecture](#target-state-architecture)
3. [Migration Strategy](#migration-strategy)
4. [Detailed Migration Phases](#detailed-migration-phases)
5. [VMID Allocation in Phoenix](#vmid-allocation-in-phoenix)
6. [Network Migration Plan](#network-migration-plan)
7. [Data Migration Procedures](#data-migration-procedures)
8. [Service Dependencies Mapping](#service-dependencies-mapping)
9. [Cutover Procedures](#cutover-procedures)
10. [Risk Mitigation](#risk-mitigation)
11. [Rollback Procedures](#rollback-procedures)
12. [Timeline Estimates](#timeline-estimates)

---

## Current State Analysis

### 1.1 DBIS Core Services (Current)

**Location:** ml110 (192.168.11.10)  
**Network:** Flat LAN (192.168.11.0/24)  
**Status:** Fully deployed and operational

| VMID | Service | IP Address | Status | Purpose |
|------|---------|------------|--------|---------|
| 10100 | dbis-postgres-primary | 192.168.11.100 | ✅ Running | PostgreSQL Primary |
| 10101 | dbis-postgres-replica-1 | 192.168.11.101 | ✅ Running | PostgreSQL Replica |
| 10120 | dbis-redis | 192.168.11.125 | ✅ Running | Redis Cache |
| 10130 | dbis-frontend | 192.168.11.130 | ✅ Running | Frontend Admin Console |
| 10150 | dbis-api-primary | 192.168.11.150 | ✅ Running | API Primary |
| 10151 | dbis-api-secondary | 192.168.11.151 | ✅ Running | API Secondary |

**Service Dependencies:**
- API services depend on PostgreSQL
- API services depend on Redis
- Frontend depends on API services
- All services use authentication (Keycloak - separate deployment)

**Data Volume:**
- PostgreSQL: ~10GB (estimated)
- Redis: ~1GB (estimated)
- Application code: ~5GB (estimated)

### 1.2 ChainID 138 Services (Current)

**Location:** ml110 (192.168.11.10)  
**Network:** Flat LAN (192.168.11.0/24)  
**Status:** Fully deployed and operational

#### 1.2.1 Besu Validators

| VMID | Service | IP Address | Status | Purpose |
|------|---------|------------|--------|---------|
| 1000-1004 | besu-validator-1-5 | 192.168.11.100-104 | ✅ Running | Validator nodes (ChainID 138) |

#### 1.2.2 Besu Sentries

| VMID | Service | IP Address | Status | Purpose |
|------|---------|------------|--------|---------|
| 1500-1503 | besu-sentry-1-4 | 192.168.11.150-153 | ✅ Running | Sentry nodes |

#### 1.2.3 Besu RPC Nodes

| VMID | Service | IP Address | Status | Purpose |
|------|---------|------------|--------|---------|
| 2500-2502 | besu-rpc-1-3 | 192.168.11.250-252 | ✅ Running | RPC nodes (ChainID 138) |

**Service Dependencies:**
- Validators require P2P connectivity (port 30303)
- RPC nodes depend on validators for blockchain state
- RPC nodes exposed via Cloudflare Tunnel

**Data Volume:**
- Blockchain data: ~50GB per validator (estimated)
- RPC nodes: ~50GB per node (estimated)

### 1.3 RPC Translator Services (Current)

**Location:** r630-01 (192.168.11.11)  
**Network:** Flat LAN (192.168.11.0/24)  
**Status:** Fully deployed and operational

| VMID | Service | IP Address | Status | Purpose |
|------|---------|------------|--------|---------|
| 106 | redis-rpc-translator | 192.168.11.110 | ✅ Running | Redis for nonce locking |
| 107 | web3signer-rpc-translator | 192.168.11.111 | ✅ Running | Web3Signer (v25.12.0) |
| 108 | vault-rpc-translator | 192.168.11.112 | ✅ Running | Vault secrets management |
| 2400-2402 | thirdweb-rpc-1-3 | 192.168.11.240-242 | ✅ Running | RPC Translator instances |

**Service Dependencies:**
- RPC Translators depend on Besu RPC nodes
- RPC Translators depend on Redis, Web3Signer, Vault

**Data Volume:**
- Minimal (stateless services)

### 1.4 Current Architecture Limitations

**Issues with Current Architecture:**

1. **Flat LAN:** All services on single network (192.168.11.0/24)
2. **No Network Segmentation:** Limited isolation between services
3. **VMID Conflicts:** Legacy services (7800-series) conflict with planned Phoenix deployment
4. **Single Host Dependency:** All DBIS services on ml110
5. **No VLAN-based Routing:** Cannot leverage role-based NAT pools
6. **Limited Scalability:** Hard to add new services without conflicts

---

## Target State Architecture

### 2.1 Phoenix Core Architecture

**Network:** VLAN 160 (10.160.0.0/22)  
**Gateway:** 10.160.0.1  
**VMID Range:** 8600-8699 (Phoenix Core)

**Phoenix Core Components (Phase 0 - Already Deployed):**
- VMID 8600: Phoenix API (10.160.0.10)
- VMID 8601: Phoenix Portal (10.160.0.11)
- VMID 8602: Phoenix Keycloak (10.160.0.12)
- VMID 8603: Phoenix PostgreSQL (10.160.0.13)

### 2.2 Target Architecture Benefits

**Benefits of Phoenix Architecture:**

1. **Network Segmentation:** VLAN-based isolation
2. **Scalability:** VMID range 8600-8699 supports 100 containers
3. **Role-Based NAT:** Egress NAT via Block #5 (when assigned)
4. **Zero-Downtime Migration:** DNS-based cutover
5. **Clean Separation:** No conflicts with legacy services
6. **Future-Proof:** Foundation for additional services

---

## Migration Strategy

### 3.1 Migration Principles

1. **Zero-Downtime:** DNS-based cutover, no service interruption
2. **Phased Approach:** Migrate services incrementally
3. **Parallel Operation:** Legacy and Phoenix services run simultaneously
4. **Reversible:** Rollback via DNS changes
5. **Validated:** Each phase must pass validation gates before proceeding

### 3.2 Migration Phases Overview

| Phase | Services | VMID Range | Target Network | Status |
|-------|----------|-----------|----------------|--------|
| Phase 0 | Phoenix Core | 8600-8603 | VLAN 160 | ✅ Complete |
| Phase 1 | DBIS Core Services | 8610-8615 | VLAN 160 | ⏳ Planned |
| Phase 2 | RPC Translator Services | 8620-8625 | VLAN 160 | ⏳ Planned |
| Phase 3 | Besu Services (Validators) | 8630-8639 | VLAN 110 (planned) | ⏳ Planned |
| Phase 4 | Besu Services (RPC) | 8640-8649 | VLAN 112 (planned) | ⏳ Planned |
| Phase 5 | Legacy Cleanup | N/A | N/A | ⏳ Planned |

**Note:** Besu services migration may require separate VLANs (110, 112) per network architecture. This migration plan focuses on DBIS Core and RPC Translator services initially.

---

## Detailed Migration Phases

### Phase 1: DBIS Core Services Migration

**Target:** Migrate DBIS Core services (VMIDs 10100-10151) to Phoenix architecture

#### Phase 1.1: Pre-Migration Preparation

**Prerequisites:**
- [ ] Phoenix Core (Phase 0) deployed and validated
- [ ] Network connectivity verified (VLAN 160)
- [ ] Storage capacity verified (thin1 on r630-01)
- [ ] Backup procedures tested
- [ ] Migration scripts prepared

**Pre-Migration Tasks:**

1. **Verify Current State:**
```bash
# Verify all DBIS services are running
for vmid in 10100 10101 10120 10130 10150 10151; do
    ssh root@192.168.11.10 "pct status $vmid"
done

# Verify database backups
ssh root@192.168.11.10 "pct exec 10100 -- pg_dumpall > /tmp/dbis_backup_pre_migration.sql"
```

2. **Allocate VMIDs in Phoenix:**
- VMID 8610: DBIS PostgreSQL Primary (10.160.0.20)
- VMID 8611: DBIS PostgreSQL Replica (10.160.0.21)
- VMID 8612: DBIS Redis (10.160.0.22)
- VMID 8613: DBIS Frontend (10.160.0.23)
- VMID 8614: DBIS API Primary (10.160.0.24)
- VMID 8615: DBIS API Secondary (10.160.0.25)

#### Phase 1.2: Database Migration (PostgreSQL)

**Migration Steps:**

1. **Deploy PostgreSQL Containers:**
```bash
# Create PostgreSQL Primary container (8610)
ssh root@192.168.11.11 "pct create 8610 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname dbis-postgres-primary-phoenix \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:100 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.20/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --onboot 1"

# Create PostgreSQL Replica container (8611)
ssh root@192.168.11.11 "pct create 8611 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname dbis-postgres-replica-phoenix \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:100 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.21/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --onboot 1"
```

2. **Install and Configure PostgreSQL:**
```bash
# Install PostgreSQL on primary (8610)
# (Follow PostgreSQL setup procedure from Phoenix deployment runbook)

# Restore database backup
ssh root@192.168.11.11 "pct push 8610 /tmp/dbis_backup_pre_migration.sql /tmp/"
ssh root@192.168.11.11 "pct exec 8610 -- psql -U postgres < /tmp/dbis_backup_pre_migration.sql"

# Configure replication (if replica is required)
```

3. **Verify Database:**
```bash
# Test connection
ssh root@192.168.11.11 "pct exec 8610 -- psql -U dbis -d dbis_core -c 'SELECT COUNT(*) FROM information_schema.tables;'"
```

#### Phase 1.3: Redis Migration

**Migration Steps:**

1. **Deploy Redis Container:**
```bash
# Create Redis container (8612)
ssh root@192.168.11.11 "pct create 8612 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname dbis-redis-phoenix \
    --memory 2048 \
    --cores 2 \
    --rootfs thin1:20 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.22/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --onboot 1"
```

2. **Install and Configure Redis:**
```bash
# Install Redis
ssh root@192.168.11.11 "pct exec 8612 -- bash -c 'apt-get update && apt-get install -y redis-server'"

# Configure Redis
ssh root@192.168.11.11 "pct exec 8612 -- systemctl enable redis-server"
ssh root@192.168.11.11 "pct exec 8612 -- systemctl start redis-server"
```

3. **Migrate Redis Data (if persistent data exists):**
```bash
# Export data from old Redis
ssh root@192.168.11.10 "pct exec 10120 -- redis-cli --rdb /tmp/redis_backup.rdb"

# Import data to new Redis
ssh root@192.168.11.11 "pct push 8612 /tmp/redis_backup.rdb /tmp/"
ssh root@192.168.11.11 "pct exec 8612 -- redis-cli --rdb /tmp/redis_backup.rdb"
```

#### Phase 1.4: API Services Migration

**Migration Steps:**

1. **Deploy API Containers:**
```bash
# Create API Primary container (8614)
ssh root@192.168.11.11 "pct create 8614 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname dbis-api-primary-phoenix \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.24/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --onboot 1"

# Create API Secondary container (8615)
ssh root@192.168.11.11 "pct create 8615 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname dbis-api-secondary-phoenix \
    --memory 4096 \
    --cores 4 \
    --rootfs thin1:50 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.25/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --onboot 1"
```

2. **Install Node.js and Deploy Application:**
```bash
# Install Node.js 18 (similar to Phoenix API deployment)
# Copy application code
# Install dependencies
# Configure environment variables (point to new database and Redis)
# Run migrations
# Build application
# Create systemd service
```

3. **Update Environment Variables:**
- Database: 10.160.0.20 (new PostgreSQL)
- Redis: 10.160.0.22 (new Redis)
- Keycloak: 10.160.0.12 (Phoenix Keycloak) or existing Keycloak

#### Phase 1.5: Frontend Migration

**Migration Steps:**

1. **Deploy Frontend Container:**
```bash
# Create Frontend container (8613)
ssh root@192.168.11.11 "pct create 8613 \
    local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
    --storage thin1 \
    --hostname dbis-frontend-phoenix \
    --memory 2048 \
    --cores 2 \
    --rootfs thin1:30 \
    --net0 bridge=vmbr0,name=eth0,ip=10.160.0.23/22,gw=10.160.0.1,type=veth \
    --unprivileged 1 \
    --onboot 1"
```

2. **Install Nginx and Deploy Frontend:**
```bash
# Install Nginx
# Copy frontend build files
# Configure Nginx
# Point to new API endpoints (10.160.0.24, 10.160.0.25)
```

#### Phase 1.6: Validation and Cutover

**Validation Gates:**

1. All containers running
2. Database accessible and data verified
3. Redis accessible
4. API services responding to health checks
5. Frontend accessible
6. End-to-end functionality tested

**Cutover Procedure:**

1. **Update DNS Records:**
   - `dbis-admin.d-bis.org` → 10.160.0.23 (new frontend)
   - `dbis-api.d-bis.org` → 10.160.0.24 (new API primary)
   - `dbis-api-2.d-bis.org` → 10.160.0.25 (new API secondary)

2. **Monitor Services:**
   - Check logs for errors
   - Monitor health endpoints
   - Verify user access

3. **Legacy Services:**
   - Keep legacy services running for 48 hours
   - Monitor for issues
   - Decommission after validation period

---

### Phase 2: RPC Translator Services Migration

**Target:** Migrate RPC Translator services (VMIDs 106-108, 2400-2402) to Phoenix architecture

#### Phase 2.1: Pre-Migration Preparation

**Prerequisites:**
- [ ] DBIS Core migration (Phase 1) complete (if dependencies exist)
- [ ] Network connectivity verified
- [ ] Storage capacity verified

**VMID Allocation:**
- VMID 8620: Redis (10.160.0.30)
- VMID 8621: Web3Signer (10.160.0.31)
- VMID 8622: Vault (10.160.0.32)
- VMID 8623-8625: RPC Translator instances (10.160.0.33-35)

#### Phase 2.2: Supporting Services Migration

**Migration Steps:**

1. **Migrate Redis, Web3Signer, Vault** (similar to Phase 1 patterns)
2. **Deploy RPC Translator instances**
3. **Update configuration** to point to new Besu RPC endpoints
4. **Validate services**

#### Phase 2.3: Cutover

**Cutover Procedure:**
- Update DNS records
- Monitor services
- Decommission legacy services after validation

---

### Phase 3: Besu Services Migration (Future)

**Target:** Migrate Besu services to dedicated VLANs (110, 112)

**Note:** Besu services migration may require:
- Separate VLANs (110 for validators, 112 for RPC)
- Network architecture changes
- Careful coordination with blockchain operations

**Status:** Future phase - requires additional planning

---

## VMID Allocation in Phoenix

### 4.1 VMID Allocation Table

| VMID Range | Service Category | Subnet/VLAN | Purpose |
|------------|-----------------|-------------|---------|
| 8600-8603 | Phoenix Core | VLAN 160 (10.160.0.0/22) | Core Phoenix services |
| 8610-8619 | DBIS Core | VLAN 160 (10.160.0.0/22) | DBIS services |
| 8620-8629 | RPC Translator | VLAN 160 (10.160.0.0/22) | RPC translation services |
| 8630-8639 | Reserved | TBD | Future use |
| 8640-8649 | Reserved | TBD | Future use |
| 8650-8699 | Reserved | VLAN 160 (10.160.0.0/22) | Future Phoenix services |

### 4.2 IP Address Allocation

**VLAN 160 Subnet:** 10.160.0.0/22 (1024 addresses)

| IP Range | Service | Purpose |
|----------|---------|---------|
| 10.160.0.10-13 | Phoenix Core | Core services |
| 10.160.0.20-29 | DBIS Core | DBIS services |
| 10.160.0.30-39 | RPC Translator | RPC translation services |
| 10.160.0.40-99 | Reserved | Future services |
| 10.160.0.100-255 | Reserved | Future expansion |

---

## Network Migration Plan

### 5.1 Current Network (Flat LAN)

**Current:** All services on 192.168.11.0/24  
**Gateway:** 192.168.11.1  
**Characteristics:**
- No network segmentation
- Single broadcast domain
- Limited security isolation

### 5.2 Target Network (VLAN-based)

**Target:** Services on VLAN 160 (10.160.0.0/22)  
**Gateway:** 10.160.0.1  
**Characteristics:**
- Network segmentation
- Isolated broadcast domains
- Enhanced security isolation
- Role-based NAT support

### 5.3 Migration Network Steps

1. **Verify VLAN 160 Configuration:**
   - VLAN exists on switch/router
   - Proxmox bridge supports VLAN tagging
   - ER605 routing configured

2. **Deploy Services on VLAN 160:**
   - Use static IPs in 10.160.0.0/22 range
   - Configure gateway as 10.160.0.1
   - Test connectivity

3. **DNS Cutover:**
   - Update DNS records to point to new IPs
   - Monitor services
   - Verify functionality

4. **Legacy Cleanup:**
   - Decommission legacy services after validation period
   - Reclaim IP addresses and VMIDs

---

## Data Migration Procedures

### 6.1 Database Migration

**Source:** PostgreSQL on 192.168.11.100 (VMID 10100)  
**Target:** PostgreSQL on 10.160.0.20 (VMID 8610)

**Procedure:**

1. **Pre-Migration Backup:**
```bash
# Full database backup
ssh root@192.168.11.10 "pct exec 10100 -- pg_dump -U dbis dbis_core > /tmp/dbis_backup_$(date +%Y%m%d).sql"

# Verify backup
ssh root@192.168.11.10 "ls -lh /tmp/dbis_backup_*.sql"
```

2. **Transfer Backup:**
```bash
# Copy backup to new host
scp root@192.168.11.10:/tmp/dbis_backup_*.sql root@192.168.11.11:/tmp/
```

3. **Restore Database:**
```bash
# Restore on new database
ssh root@192.168.11.11 "pct push 8610 /tmp/dbis_backup_*.sql /tmp/"
ssh root@192.168.11.11 "pct exec 8610 -- psql -U phoenix -d phoenix < /tmp/dbis_backup_*.sql"
```

4. **Verify Data:**
```bash
# Compare record counts
ssh root@192.168.11.11 "pct exec 8610 -- psql -U phoenix -d phoenix -c 'SELECT COUNT(*) FROM (SELECT table_name FROM information_schema.tables WHERE table_schema='\\''public'\\'') AS tables;'"
```

### 6.2 Redis Migration

**Source:** Redis on 192.168.11.125 (VMID 10120)  
**Target:** Redis on 10.160.0.22 (VMID 8612)

**Procedure:**

1. **Export Redis Data (if persistent):**
```bash
# Save Redis data
ssh root@192.168.11.10 "pct exec 10120 -- redis-cli SAVE"
ssh root@192.168.11.10 "pct exec 10120 -- redis-cli --rdb /tmp/redis_backup.rdb"
```

2. **Import Redis Data:**
```bash
# Copy to new Redis
scp root@192.168.11.10:/tmp/redis_backup.rdb root@192.168.11.11:/tmp/
ssh root@192.168.11.11 "pct push 8612 /tmp/redis_backup.rdb /tmp/"
ssh root@192.168.11.11 "pct exec 8612 -- cp /tmp/redis_backup.rdb /var/lib/redis/dump.rdb"
ssh root@192.168.11.11 "pct exec 8612 -- systemctl restart redis-server"
```

**Note:** Redis is often stateless. Migration may not require data transfer if Redis is used only for caching.

### 6.3 Application Code Migration

**Procedure:**

1. **Copy Application Code:**
```bash
# Copy API code
rsync -avz --exclude node_modules --exclude .git \
    root@192.168.11.10:/opt/dbis-core/ \
    root@192.168.11.11:/tmp/dbis-core-source/

# Copy to new container
ssh root@192.168.11.11 "pct push 8614 /tmp/dbis-core-source /opt/dbis-core --recursive"
```

2. **Install Dependencies and Configure:**
```bash
# Install Node.js, dependencies, configure environment
# (Follow application deployment procedures)
```

---

## Service Dependencies Mapping

### 7.1 DBIS Core Dependencies

**Internal Dependencies:**
- API → PostgreSQL (database queries)
- API → Redis (caching)
- Frontend → API (API calls)
- API → Keycloak (authentication)

**External Dependencies:**
- All services → Internet (for updates, external APIs)
- Frontend → DNS (for domain resolution)

### 7.2 RPC Translator Dependencies

**Internal Dependencies:**
- RPC Translator → Besu RPC (blockchain queries)
- RPC Translator → Redis (nonce locking)
- RPC Translator → Web3Signer (transaction signing)
- RPC Translator → Vault (secrets management)

**External Dependencies:**
- All services → Internet (for updates)

### 7.3 Dependency Migration Order

**Recommended Migration Order:**

1. **Foundation Services First:**
   - PostgreSQL
   - Redis
   - Keycloak (if shared)

2. **Application Services Second:**
   - API services
   - Frontend

3. **Supporting Services Last:**
   - Monitoring (if any)
   - Logging (if any)

---

## Cutover Procedures

### 8.1 DNS Cutover

**DNS Records to Update:**

| Record | Current IP | New IP | Type |
|--------|-----------|--------|------|
| `dbis-admin.d-bis.org` | 192.168.11.130 | 10.160.0.23 | A |
| `dbis-api.d-bis.org` | 192.168.11.150 | 10.160.0.24 | A |
| `dbis-api-2.d-bis.org` | 192.168.11.151 | 10.160.0.25 | A |

**Cutover Steps:**

1. **Pre-Cutover Validation:**
   - All new services running and validated
   - All validation gates passed
   - End-to-end testing completed

2. **Update DNS Records:**
```bash
# Using Cloudflare API or dashboard
# Update each DNS record to point to new IPs
```

3. **Monitor Services:**
   - Check logs for errors
   - Monitor health endpoints
   - Verify user access
   - Monitor for 48 hours

4. **Rollback (if needed):**
   - Revert DNS records to old IPs
   - Legacy services become active again

### 8.2 Service Cutover

**Cutover Strategy:**

1. **Parallel Operation:**
   - New services running alongside legacy services
   - DNS cutover routes traffic to new services
   - Legacy services remain running as backup

2. **Traffic Migration:**
   - DNS cutover routes all traffic to new services
   - Legacy services receive no traffic
   - Monitor for issues

3. **Validation Period:**
   - Monitor new services for 48 hours
   - Verify functionality
   - Check for errors

4. **Legacy Decommission:**
   - After validation period, decommission legacy services
   - Reclaim resources

---

## Risk Mitigation

### 9.1 Identified Risks

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Data loss during migration | Low | High | Multiple backups, verification procedures |
| Service downtime | Low | High | DNS cutover, parallel operation |
| Network connectivity issues | Medium | Medium | Pre-migration network validation |
| Application compatibility issues | Low | Medium | Pre-migration testing, validation gates |
| Configuration errors | Medium | Medium | Automated configuration, validation procedures |

### 9.2 Mitigation Strategies

**Data Loss Prevention:**
- Multiple backups before migration
- Verify backups before migration
- Test restore procedures
- Keep legacy data for 30 days

**Downtime Prevention:**
- DNS-based cutover (instant rollback)
- Parallel operation during migration
- Validation gates before cutover
- Rollback procedures documented

**Network Issues Prevention:**
- Pre-migration network validation
- Test connectivity before migration
- Verify firewall rules
- Document network configuration

**Compatibility Issues Prevention:**
- Pre-migration testing
- Validation gates
- Gradual migration (one service at a time)
- Rollback procedures

---

## Rollback Procedures

### 10.1 DNS Rollback

**If issues are discovered after DNS cutover:**

1. **Revert DNS Records:**
```bash
# Revert DNS records to old IPs
# Legacy services become active again
```

2. **Investigate Issues:**
   - Check logs
   - Identify root cause
   - Fix issues

3. **Re-attempt Migration:**
   - After fixing issues, re-attempt migration
   - Follow migration procedures again

### 10.2 Service Rollback

**If service-level rollback is needed:**

1. **Stop New Services:**
```bash
# Stop new services (keep containers for debugging)
for vmid in 8610 8611 8612 8613 8614 8615; do
    ssh root@192.168.11.11 "pct stop $vmid"
done
```

2. **Revert DNS:**
   - Revert DNS records to legacy IPs
   - Legacy services become active

3. **Keep New Services:**
   - Do not delete new containers (may contain debugging info)
   - Investigate issues
   - Fix and re-attempt

---

## Timeline Estimates

### 11.1 Phase 1: DBIS Core Migration

**Estimated Duration:** 2-4 weeks

| Task | Duration | Dependencies |
|------|----------|--------------|
| Pre-migration preparation | 2-3 days | Phoenix Core deployed |
| Database migration | 1-2 days | Preparation complete |
| Redis migration | 1 day | Database migration complete |
| API services migration | 2-3 days | Database and Redis migrated |
| Frontend migration | 1-2 days | API services migrated |
| Validation and testing | 3-5 days | All services migrated |
| DNS cutover | 1 day | Validation complete |
| Legacy decommission | 1-2 days | Cutover successful |

**Total:** 2-4 weeks (including validation period)

### 11.2 Phase 2: RPC Translator Migration

**Estimated Duration:** 1-2 weeks

| Task | Duration | Dependencies |
|------|----------|--------------|
| Pre-migration preparation | 1-2 days | Phase 1 complete (if dependencies) |
| Supporting services migration | 1-2 days | Preparation complete |
| RPC Translator migration | 2-3 days | Supporting services migrated |
| Validation and testing | 2-3 days | All services migrated |
| DNS cutover | 1 day | Validation complete |
| Legacy decommission | 1 day | Cutover successful |

**Total:** 1-2 weeks

### 11.3 Phase 3: Besu Services Migration (Future)

**Estimated Duration:** TBD (requires additional planning)

---

## Success Criteria

### 12.1 Migration Success Criteria

**Phase 1 (DBIS Core) is successful when:**
- [ ] All services running on VLAN 160
- [ ] All validation gates passed
- [ ] DNS cutover completed
- [ ] Services accessible via new IPs
- [ ] No user-reported issues for 48 hours
- [ ] Legacy services decommissioned
- [ ] Documentation updated

**Phase 2 (RPC Translator) is successful when:**
- [ ] All services running on VLAN 160
- [ ] All validation gates passed
- [ ] DNS cutover completed
- [ ] Services accessible via new IPs
- [ ] No user-reported issues for 48 hours
- [ ] Legacy services decommissioned

---

## Post-Migration Tasks

### 13.1 Immediate Post-Migration

1. **Monitor Services:**
   - Check logs daily
   - Monitor health endpoints
   - Verify user access

2. **Update Documentation:**
   - Update service documentation
   - Update network diagrams
   - Update runbooks

3. **Cleanup:**
   - Remove legacy containers (after validation period)
   - Reclaim IP addresses
   - Reclaim VMIDs

### 13.2 Long-Term Post-Migration

1. **Optimization:**
   - Optimize performance
   - Optimize resource allocation
   - Review and optimize configuration

2. **Enhancement:**
   - Implement additional security controls
   - Implement monitoring and alerting
   - Implement backup and recovery procedures

3. **Future Migration:**
   - Plan Phase 3 (Besu services) migration
   - Plan additional service migrations
   - Plan network architecture enhancements

---

## Related Documentation

- **Phoenix Deployment Runbook:** `docs/03-deployment/PHOENIX_DEPLOYMENT_RUNBOOK.md`
- **Phoenix Firewall Rules:** `docs/04-configuration/PHOENIX_VLAN160_FIREWALL_RULES.md`
- **Phoenix DNS Template:** `docs/04-configuration/PHOENIX_DNS_ZONE_TEMPLATE.md`
- **Phoenix System Boundary:** `docs/02-architecture/PHOENIX_SYSTEM_BOUNDARY_STATEMENT.md`
- **Network Architecture:** `docs/02-architecture/NETWORK_ARCHITECTURE.md`
- **VMID Allocation:** `docs/02-architecture/VMID_ALLOCATION_FINAL.md`

---

**Last Updated:** 2026-01-09  
**Status:** Draft - Ready for Review  
**Next Review:** Before Phase 1 execution