# RPC VMID Migration - Complete Summary **Last Updated:** 2026-01-31 **Document Version:** 1.0 **Status:** Active Documentation --- **Date**: 2026-01-17 **Status**: ✅ **MIGRATION COMPLETE** (12/12 migrations) --- ## Migration Overview Successfully migrated all 12 RPC nodes from old VMIDs to new VMIDs with updated IP addresses and consistent naming. --- ## Completed Migrations (12/12) ### Core/Public/Private RPC Nodes | Old VMID | Old IP | Old Name | New VMID | New IP | New Name | Status | |----------|--------|---------|----------|--------|----------|--------| | 2500 | 192.168.11.250 | besu-rpc-1 | **2101** | 192.168.11.211 | besu-rpc-core-1 | ✅ Complete | | 2501 | 192.168.11.251 | besu-rpc-2 | **2201** | 192.168.11.221 | besu-rpc-public-1 | ✅ Complete | | 2502 | 192.168.11.252 | besu-rpc-3 | **2301** | 192.168.11.232 | besu-rpc-private-1 | ✅ Complete | ### Tenant RPC Nodes | Old VMID | Old IP | Old Name | New VMID | New IP | New Name | Status | |----------|--------|---------|----------|--------|----------|--------| | 2503 | 192.168.11.253 | besu-rpc-ali-0x8a | **2303** | 192.168.11.233 | besu-rpc-ali-0x8a | ✅ Complete | | 2504 | 192.168.11.254 | besu-rpc-ali-0x1 | **2304** | 192.168.11.234 | besu-rpc-ali-0x1 | ✅ Complete | | 2505 | 192.168.11.201 | besu-rpc-luis-0x8a | **2305** | 192.168.11.235 | besu-rpc-luis-0x8a | ✅ Complete | | 2506 | 192.168.11.202 | besu-rpc-luis-0x1 | **2306** | 192.168.11.236 | besu-rpc-luis-0x1 | ✅ Complete | | 2507 | 192.168.11.203 | besu-rpc-putu-0x8a | **2307** | 192.168.11.237 | besu-rpc-putu-0x8a | ✅ Complete | | 2508 | 192.168.11.204 | besu-rpc-putu-0x1 | **2308** | 192.168.11.238 | besu-rpc-putu-0x1 | ✅ Complete | ### Thirdweb RPC Nodes | Old VMID | Old IP | Old Name | New VMID | New IP | New Name | Status | |----------|--------|---------|----------|--------|----------|--------| | 2400 | 192.168.11.240 | thirdweb-rpc-1 | **2401** | 192.168.11.241 | besu-rpc-thirdweb-0x8a-1 | ✅ Complete | | 2401 | 192.168.11.241 | thirdweb-rpc-2 | **2402** | 192.168.11.242 | besu-rpc-thirdweb-0x8a-2 | ✅ Complete | | 2402 | 192.168.11.242 | thirdweb-rpc-3 | **2403** | 192.168.11.243 | besu-rpc-thirdweb-0x8a-3 | ✅ Complete | --- ## Configuration Files Updated ### static-nodes.json - **Status**: ✅ Updated with 15 unique enodes - **Contents**: 5 validators + 10 RPC nodes (with correct new IPs) - **Missing**: 2402 and 2403 enodes (will be added once Besu services start) - **Uniqueness**: ✅ All enode IDs verified unique ### permissions-nodes.toml - **Status**: ✅ Updated with all enodes from static-nodes.json - **Contents**: 15 enodes (same as static-nodes.json) - **Format**: TOML format with proper syntax ### Deployment Status - ✅ Deployed to all running RPC nodes (11/12 - 2301 is stopped) - ✅ Files verified on all deployed nodes - ✅ Permissions set correctly (besu:besu) --- ## Current Node Status ### Running and Responding (9/12) - ✅ 2101 (192.168.11.211) - besu-rpc-core-1 - ✅ 2201 (192.168.11.221) - besu-rpc-public-1 - ✅ 2303-2308 (all 6 tenant RPCs) - All responding - ✅ 2401 (192.168.11.241) - besu-rpc-thirdweb-0x8a-1 ### Running but RPC Not Ready (2/12) - ⏳ 2402 (192.168.11.242) - besu-rpc-thirdweb-0x8a-2 (Besu starting) - ⏳ 2403 (192.168.11.243) - besu-rpc-thirdweb-0x8a-3 (Besu starting) ### Stopped (1/12) - ⏸️ 2301 (192.168.11.232) - besu-rpc-private-1 (needs start) --- ## Enode Uniqueness Verification ✅ **All enode IDs are unique!** - Verified: 15 unique enode IDs in static-nodes.json - No duplicates detected - All enodes have correct new IP addresses **Note**: 2402 and 2403 will generate new unique nodekeys when Besu starts (they don't have nodekeys yet). Their enodes will be added to static-nodes.json once available. --- ## Old VMIDs Status The following old VMIDs are ready for decommissioning after verification: | Old VMID | Status | Can Decommission | |----------|--------|------------------| | 2400 | running | ⏳ After verification | | 2500-2508 | stopped | ⏳ After verification | **Decommission Steps** (after verification): 1. Verify all new nodes are working correctly 2. Stop old VMIDs: `pct stop ` 3. Optionally destroy: `pct destroy ` (after backup) --- ## Files Updated ### Scripts - ✅ `scripts/migrate-rpc-vmids.sh` - Migration script with ordered array - ✅ `scripts/monitor-rpc-migration.sh` - Auto-monitoring and fixing script - ✅ `scripts/besu-deploy-allowlist.sh` - Updated with all new VMIDs - ✅ `scripts/verify-unique-enodes.sh` - New verification script ### Configuration Files - ✅ `smom-dbis-138/config/static-nodes.json` - Updated with new IPs - ✅ `smom-dbis-138/config/permissions-nodes.toml` - Updated with all enodes --- ## Next Steps 1. **Wait for 2402 and 2403 Besu services to start** - They will generate new unique nodekeys - Get their enodes via `admin_nodeInfo` RPC call - Verify enodes are unique - Add to static-nodes.json and permissions-nodes.toml - Redeploy updated files 2. **Start container 2301** - `pct start 2301` - Verify Besu service starts - Verify RPC connectivity 3. **Final Verification** - Run `scripts/verify-unique-enodes.sh` on all 12 nodes - Verify all RPC endpoints are responding - Check peer connections on all nodes 4. **Decommission Old VMIDs** (after verification) - Stop old VMIDs: 2400, 2500-2508 - Create backups if needed - Optionally destroy after confirmation --- ## Issues Fixed During Migration 1. ✅ **VMID Conflicts**: Resolved by ordered migration array (migrate to open VMIDs first) 2. ✅ **Network Configuration**: Fixed `pct set --net0` format (removed `net0=` prefix) 3. ✅ **Missing Rootfs**: Automatically added rootfs configuration after clone 4. ✅ **Container Locks**: Automatically unlocked containers stuck in "create" status 5. ✅ **LVM Duplicate Config**: Fixed duplicate `thin_pool_autoextend_percent` warnings 6. ✅ **Enode Uniqueness**: Verified all enode IDs are unique --- ## Summary ✅ **Migration**: 12/12 complete ✅ **Configuration**: Files deployed to 11/12 nodes ✅ **Uniqueness**: All enode IDs verified unique ✅ **Connectivity**: 9/12 nodes responding to RPC calls **Remaining**: 3 nodes need Besu services to start (2301, 2402, 2403) --- **Last Updated**: 2026-01-17