Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- Config, docs, scripts, and backup manifests - Submodule refs unchanged (m = modified content in submodules) Made-with: Cursor
141 lines
5.7 KiB
Markdown
141 lines
5.7 KiB
Markdown
# Migration Plan: r630-01 Data Pool (72% → target <70%)
|
||
|
||
**Last updated:** 2026-02-28
|
||
|
||
**Phase 1 completed (2026-02-28):** Migrated 8 CTs from data → thin1 on r630-01. **data** now **65.8%** (was 72%); **thin1** 50.6% (was 43.5%). Migrated VMIDs: 10233, 10120, 10100, 10101, 10235, 10236, 7804, 8640.
|
||
|
||
Goal: Reduce r630-01 **data** pool usage (currently ~65.8%, 280G pool) so it stays below 85% and has headroom. Options: move CTs to **r630-01 thin1** (same host) or to **r630-02 thin5** (cross-host).
|
||
|
||
---
|
||
|
||
## 1. Current layout (r630-01)
|
||
|
||
| Pool | Size | Used% | Free (approx) | Notes |
|
||
|-------|-------|-------|----------------|-------|
|
||
| **data** | 280G | **65.8%** ✓ | ~100G | After Phase 1 migrations; target <70% met. |
|
||
| **thin1** | 208G | 50.6% | ~107G | Now holds migrated CTs from Phase 1. |
|
||
|
||
**CTs on data (by size, VMID = container):**
|
||
|
||
| VMID | Allocated | Role / name (typical) | Migration priority |
|
||
|-------|-----------|------------------------|--------------------|
|
||
| 2500–2505 | 200G each | RPC (alltra etc.) | Low (critical RPC); move only if needed. |
|
||
| 2101 | 200G | Core RPC | **Do not move** (deploy/admin). |
|
||
| 1000, 1001, 1002 | 100G each | Validators | **Do not move** (consensus). |
|
||
| 1500, 1501, 1502 | 100G each | Sentries | Prefer stay; move only if necessary. |
|
||
| 10130, 10150, 10151 | 50–100G | Services | Good candidates (Phase 2). |
|
||
| 8640, 8642 | 50G each | Services | Good candidates. |
|
||
| 7804 | 20G | Gov/portal | Good candidate. |
|
||
| 10235, 10236 | 10–20G | Small services | Good candidates. |
|
||
| 10232, 10233 | 1–3G | Small | Easy wins. |
|
||
| 10100, 10101 | 8G each | Small | Easy wins. |
|
||
| 10120 | 4G | Small | Easy win. |
|
||
|
||
---
|
||
|
||
## 2. Migration options
|
||
|
||
### Option A: Same host — data → thin1 (r630-01)
|
||
|
||
- **Pros:** No node change; no IP/network change; quick (stop → vzdump → destroy → restore to thin1 → start).
|
||
- **Cons:** thin1 has ~118G free; cannot move all 200G CTs without exceeding thin1.
|
||
- **Best for:** Smaller CTs (total ~50–80G) to free space on data without touching validators/core RPC.
|
||
|
||
### Option B: Cross host — r630-01 → r630-02 thin5
|
||
|
||
- **Pros:** thin5 is empty (237G); can move one or two large CTs (e.g. 200G each).
|
||
- **Cons:** Node migration (IP may change unless static); service dependency and DNS/load balancer updates.
|
||
- **Best for:** One or two 200G RPC CTs (e.g. 2503, 2504, 2505) if you need to free a lot of data in one go.
|
||
|
||
---
|
||
|
||
## 3. Recommended phases
|
||
|
||
### Phase 1 — Same host (data → thin1)
|
||
|
||
**Goal:** Free ~50–80G on data (get data from 72% to ~55–60%) by moving smaller CTs to thin1.
|
||
|
||
**Candidates (order by size, smaller first for low risk):**
|
||
|
||
1. 10232 (1G), 10233 (3G), 10120 (4G)
|
||
2. 10100 (8G), 10101 (8G), 10235 (10G)
|
||
3. 10236 (20G), 7804 (20G)
|
||
4. Optionally one of: 8640 (50G), 8642 (50G), or 10130 (50G)
|
||
|
||
**Steps per CT (run on r630-01 or via SSH):**
|
||
|
||
```bash
|
||
# On r630-01 (192.168.11.11). Replace VMID with actual (e.g. 10232).
|
||
VMID=10232
|
||
ssh root@192.168.11.11 bash -s << 'EOF'
|
||
set -e
|
||
pct stop $VMID
|
||
vzdump $VMID --storage local --compress gzip --mode stop --remove 0
|
||
BACKUP=$(ls -t /var/lib/vz/dump/vzdump-lxc-$VMID-*.tar.gz | head -1)
|
||
pct destroy $VMID --force
|
||
pct restore $VMID $BACKUP --storage thin1
|
||
rm -f $BACKUP
|
||
pct start $VMID
|
||
EOF
|
||
```
|
||
|
||
**Automation:** Run for each VMID:
|
||
```bash
|
||
bash scripts/maintenance/migrate-ct-r630-01-data-to-thin1.sh <VMID> # e.g. 10232
|
||
bash scripts/maintenance/migrate-ct-r630-01-data-to-thin1.sh 10232 --dry-run # preview
|
||
```
|
||
Suggested order: 10232, 10233, 10120, 10100, 10101, 10235, 10236, 7804, then 8640 or 8642.
|
||
|
||
---
|
||
|
||
### Phase 2 — If data still >70%: move one large CT to r630-02 thin5
|
||
|
||
**Goal:** Free ~200G on r630-01 data by migrating one 200G CT to r630-02 and placing it on thin5.
|
||
|
||
**Candidates:** 2503, 2504, or 2505 (non–core RPC; confirm which can be moved without breaking dependencies).
|
||
|
||
**Steps:**
|
||
|
||
1. **Confirm** the CT can run on r630-02 (network, DNS, NPM proxy, etc.).
|
||
2. **Migrate** (cross-node):
|
||
`pct migrate <VMID> r630-02 --storage thin5`
|
||
Or: backup on r630-01, copy backup to r630-02, restore on r630-02 with `--storage thin5`, then destroy on r630-01.
|
||
3. **Update** any static IPs, DNS, or proxy config pointing to the old host.
|
||
4. **Verify** service and re-run `audit-proxmox-rpc-storage.sh`.
|
||
|
||
---
|
||
|
||
## 4. Scripts to use / create
|
||
|
||
| Action | Script / command |
|
||
|--------|-------------------|
|
||
| List CTs on data | `ssh root@192.168.11.11 'lvs -o lv_name,lv_size,pool_lv --noheadings pve \| grep " data "'` |
|
||
| Same-host migration (data → thin1) | New script or inline: `vzdump` → `pct destroy` → `pct restore --storage thin1` → `pct start` |
|
||
| Cross-host migration | `pct migrate <VMID> r630-02 --storage thin5` (or backup/restore to r630-02). |
|
||
| Verify after | `bash scripts/audit-proxmox-rpc-storage.sh` and `pvesm status` on both hosts. |
|
||
|
||
---
|
||
|
||
## 5. Checklist before migration
|
||
|
||
- [ ] Backup or snapshot critical CTs before Phase 1.
|
||
- [ ] Run prune on r630-01 data CTs again if not done recently: `fix-storage-r630-01-and-thin5.sh`.
|
||
- [ ] Ensure no critical operations (deploy, consensus) during the migration window.
|
||
- [ ] For Phase 2: document which VMID is moved and update DNS/NPM/load balancer.
|
||
|
||
---
|
||
|
||
## 6. Success criteria
|
||
|
||
- **Phase 1:** r630-01 data pool <70% (ideally 55–65%); thin1 still <75%.
|
||
- **Phase 2 (if run):** r630-01 data <65%; r630-02 thin5 used by migrated CT(s); all services verified.
|
||
|
||
---
|
||
|
||
## 7. Reference
|
||
|
||
- **r630-01 data:** 280G, 72% used; LVs: 1000, 1001, 1002, 1500–1502, 2101, 2500–2505, 10100, 10101, 10120, 10130, 10150, 10151, 10232, 10233, 10235, 10236, 7804, 8640, 8642.
|
||
- **r630-01 thin1:** 208G, 43.5% used; ~118G free.
|
||
- **r630-02 thin5:** 237G, 0% used (empty).
|
||
- Storage docs: `STORAGE_RECOMMENDATIONS_BY_FILL_RATE.md`, `PHYSICAL_DRIVES_AND_CONFIG.md`.
|