fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
342 lines
9.6 KiB
Markdown
342 lines
9.6 KiB
Markdown
# r630-02 Storage Review and Recommendations
|
|
|
|
**Last Updated:** 2026-01-31
|
|
**Document Version:** 1.0
|
|
**Status:** Active Documentation
|
|
|
|
---
|
|
|
|
**Date:** 2026-01-15
|
|
**Host:** r630-02 (192.168.11.12)
|
|
**Status:** ✅ **REVIEW COMPLETE**
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
**Total Storage:** ~1.85TB (8 x 232GB disks)
|
|
**Critical Issues:** 2 pools at 88%+ capacity, 1 container at 94.8% full
|
|
**Immediate Actions Required:** Container 7811 disk expansion, thin2 cleanup
|
|
**Potential Space Recovery:** ~300GB from duplicate volumes and old backups
|
|
|
|
---
|
|
|
|
## Storage Architecture
|
|
|
|
### Physical Disks
|
|
|
|
| Disk | Size | Type | Usage | Status |
|
|
|------|------|------|-------|--------|
|
|
| sda | 232.9GB | ZFS | System/Boot | Active |
|
|
| sdb | 232.9GB | ZFS | System/Boot | Active |
|
|
| sdc | 232.9GB | LVM | thin2 pool | **88.33% full** |
|
|
| sdd | 232.9GB | LVM | thin3 pool | 0% (empty) |
|
|
| sde | 232.9GB | LVM | thin4 pool | 12.69% used |
|
|
| sdf | 232.9GB | LVM | thin5 pool | 0% (empty) |
|
|
| sdg | 232.9GB | LVM | thin6 pool | 0% (empty) |
|
|
| sdh | 232.9GB | LVM | thin1 pool | **88.51% full** |
|
|
|
|
**Total Raw Capacity:** ~1.86TB
|
|
**Total Usable (LVM Thin Pools):** ~1.36TB (6 x 226GB pools)
|
|
|
|
---
|
|
|
|
## Storage Pools Status
|
|
|
|
### Active Pools
|
|
|
|
| Pool | Type | Status | Total | Used | Available | Usage % | Priority |
|
|
|------|------|--------|-------|------|-----------|---------|----------|
|
|
| **thin1-r630-02** | lvmthin | Active | 226GB | 200GB | 26GB | **88.51%** | 🔴 Critical |
|
|
| **thin2** | lvmthin | Active | 226GB | 200GB | 27GB | **88.33%** | 🔴 Critical |
|
|
| **thin4** | lvmthin | Active | 226GB | 29GB | 197GB | 12.69% | 🟢 Healthy |
|
|
| thin3 | lvmthin | Active | 226GB | 0GB | 226GB | 0.00% | 🟢 Available |
|
|
| thin5 | lvmthin | Active | 226GB | 0GB | 226GB | 0.00% | 🟢 Available |
|
|
| thin6 | lvmthin | Active | 226GB | 0GB | 226GB | 0.00% | 🟢 Available |
|
|
| local | dir | Active | 220GB | 7.3GB | 213GB | 3.31% | 🟢 Healthy |
|
|
|
|
### Inactive/Problematic Pools
|
|
|
|
| Pool | Type | Status | Issue |
|
|
|------|------|--------|-------|
|
|
| data | lvmthin | Inactive | No logical volume found |
|
|
| thin1 | lvmthin | Inactive | No logical volume found |
|
|
| local-lvm | lvmthin | Disabled | Not configured |
|
|
|
|
---
|
|
|
|
## Container Disk Usage
|
|
|
|
### Current Allocation
|
|
|
|
| Container | Name | Pool | Allocated | Used | Available | Usage % | Status |
|
|
|-----------|------|------|-----------|------|-----------|---------|--------|
|
|
| **5000** | blockscout-1 | thin2 | 200GB | 125.2GB | 60.6GB | 63.9% | 🟡 Moderate |
|
|
| **6200** | firefly-1 | thin2 | 50GB | 3.3GB | 43.1GB | 6.7% | 🟢 Healthy |
|
|
| **6201** | firefly-ali-1 | thin2 | 50GB | 2.8GB | 43.6GB | 5.7% | 🟢 Healthy |
|
|
| **7811** | mim-api-1 | thin4 | 30GB | 27.8GB | **0GB** | **94.8%** | 🔴 **CRITICAL** |
|
|
|
|
### Critical Issues
|
|
|
|
1. **Container 7811 (mim-api-1):** 94.8% full, 0GB available - **IMMEDIATE ACTION REQUIRED**
|
|
2. **thin2 Pool:** 88.33% full with 3 containers - **NEEDS CLEANUP**
|
|
3. **thin1-r630-02 Pool:** 88.51% full - **NEEDS INVESTIGATION**
|
|
|
|
---
|
|
|
|
## Duplicate Volumes Issue
|
|
|
|
### Problem Identified
|
|
|
|
**Duplicate volumes exist on both thin1 and thin2:**
|
|
|
|
| Container | thin1 Volume | thin2 Volume | Status |
|
|
|-----------|--------------|--------------|--------|
|
|
| 5000 | vm-5000-disk-0 (200GB) | vm-5000-disk-0 (200GB) | ⚠️ Duplicate |
|
|
| 6200 | vm-6200-disk-0 (50GB) | vm-6200-disk-0 (50GB) | ⚠️ Duplicate |
|
|
| 6201 | vm-6201-disk-0 (50GB) | vm-6201-disk-0 (50GB) | ⚠️ Duplicate |
|
|
|
|
**Total Duplicate Space:** ~300GB (200GB + 50GB + 50GB)
|
|
|
|
**Analysis:**
|
|
- Containers are currently using thin2 volumes (confirmed by `pct config`)
|
|
- thin1 volumes appear to be orphaned/unused
|
|
- This explains why thin1-r630-02 shows 88.51% usage despite no active containers
|
|
|
|
---
|
|
|
|
## Snapshots and Backups
|
|
|
|
### Snapshots
|
|
|
|
**Current Snapshots:** None (all containers show only "current" snapshot)
|
|
|
|
**Recommendation:** Consider creating snapshots before major changes.
|
|
|
|
### Backup Files
|
|
|
|
**Location:** `/var/lib/vz/dump`
|
|
**Total Size:** 7.2GB
|
|
**Count:** 13 backup files
|
|
**Date Range:** January 3-7, 2026
|
|
|
|
**Largest Backup Files:**
|
|
- vzdump-lxc-103-2026_01_07-12_38_26.tar.gz (1.4GB)
|
|
- vzdump-lxc-7811-2026_01_03-14_57_45.tar.gz (1.3GB)
|
|
- vzdump-lxc-130-2026_01_07-12_46_28.tar.gz (1.1GB)
|
|
- vzdump-lxc-7810-2026_01_03-14_55_23.tar.gz (781MB)
|
|
- vzdump-lxc-7810-2026_01_03-14_03_16.tar.gz (781MB)
|
|
|
|
**Note:** These backups are from containers that may no longer exist (7800, 7801, 7802, 7810, 103, 104, 105, 130).
|
|
|
|
---
|
|
|
|
## Detailed Recommendations
|
|
|
|
### 🔴 **IMMEDIATE ACTIONS (Critical)**
|
|
|
|
#### 1. Expand Container 7811 Disk (URGENT)
|
|
|
|
**Issue:** Container 7811 is 94.8% full with 0GB available.
|
|
|
|
**Action:**
|
|
```bash
|
|
# Expand container 7811 disk from 30GB to 50GB
|
|
ssh root@192.168.11.12 "pct resize 7811 rootfs +20G"
|
|
```
|
|
|
|
**Impact:** Prevents container from running out of space and crashing.
|
|
|
|
**Priority:** **CRITICAL** - Do immediately
|
|
|
|
---
|
|
|
|
#### 2. Investigate and Clean Up Duplicate Volumes on thin1
|
|
|
|
**Issue:** ~300GB of duplicate volumes on thin1-r630-02 pool.
|
|
|
|
**Steps:**
|
|
|
|
1. **Verify thin1 volumes are not in use:**
|
|
```bash
|
|
ssh root@192.168.11.12 "lvs -o lv_name,vg_name,attr | grep -E 'vm-5000|vm-6200|vm-6201' | grep thin1"
|
|
```
|
|
|
|
2. **Check if any containers reference thin1:**
|
|
```bash
|
|
ssh root@192.168.11.12 "for CTID in 5000 6200 6201; do echo \"Container \$CTID:\"; pct config \$CTID | grep rootfs; done"
|
|
```
|
|
|
|
3. **If confirmed unused, remove duplicate volumes:**
|
|
```bash
|
|
# WARNING: Only if confirmed unused!
|
|
ssh root@192.168.11.12 "lvremove /dev/thin1/vm-5000-disk-0"
|
|
ssh root@192.168.11.12 "lvremove /dev/thin1/vm-6200-disk-0"
|
|
ssh root@192.168.11.12 "lvremove /dev/thin1/vm-6201-disk-0"
|
|
```
|
|
|
|
**Potential Space Recovery:** ~300GB on thin1-r630-02
|
|
|
|
**Priority:** **HIGH** - Frees significant space
|
|
|
|
---
|
|
|
|
### 🟡 **SHORT-TERM ACTIONS (Within 1 Week)**
|
|
|
|
#### 3. Review and Clean Up Old Backup Files
|
|
|
|
**Issue:** 7.2GB of backup files from January 3-7, 2026, possibly from deleted containers.
|
|
|
|
**Action:**
|
|
|
|
1. **Identify backups for non-existent containers:**
|
|
```bash
|
|
ssh root@192.168.11.12 "pct list | awk '{print \$1}' > /tmp/active_containers.txt"
|
|
ssh root@192.168.11.12 "for file in /var/lib/vz/dump/*.tar.gz; do CTID=\$(echo \$file | grep -oP 'lxc-\K[0-9]+'); if ! grep -q \"^\$CTID\$\" /tmp/active_containers.txt; then echo \"Orphaned: \$file\"; fi; done"
|
|
```
|
|
|
|
2. **Move old backups to archive or delete:**
|
|
```bash
|
|
# Archive backups older than 30 days
|
|
ssh root@192.168.11.12 "find /var/lib/vz/dump -name '*.tar.gz' -mtime +30 -exec mv {} /var/lib/vz/archive/ \;"
|
|
```
|
|
|
|
**Potential Space Recovery:** ~7GB on local storage
|
|
|
|
**Priority:** **MEDIUM** - Good housekeeping
|
|
|
|
---
|
|
|
|
#### 4. Optimize thin2 Pool Usage
|
|
|
|
**Issue:** thin2 is 88.33% full with 3 containers.
|
|
|
|
**Options:**
|
|
|
|
**Option A: Move containers to empty pools**
|
|
- Move container 6200 or 6201 to thin3, thin5, or thin6
|
|
- Reduces thin2 usage to ~70%
|
|
|
|
**Option B: Expand thin2 pool** (if possible)
|
|
- Check if physical disk can be expanded
|
|
- Requires additional storage hardware
|
|
|
|
**Option C: Clean up container 5000**
|
|
- Container 5000 uses 125.2GB of 200GB
|
|
- Review if data can be cleaned up or archived
|
|
|
|
**Recommended:** **Option A** - Move container 6201 (stopped) to thin3
|
|
|
|
**Priority:** **MEDIUM** - Prevents future issues
|
|
|
|
---
|
|
|
|
### 🟢 **LONG-TERM ACTIONS (Within 1 Month)**
|
|
|
|
#### 5. Implement Backup Retention Policy
|
|
|
|
**Action:**
|
|
- Configure automated backup cleanup
|
|
- Keep only last 7 days of daily backups
|
|
- Keep weekly backups for 4 weeks
|
|
- Keep monthly backups for 6 months
|
|
|
|
**Implementation:**
|
|
```bash
|
|
# Add to cron or Proxmox backup job
|
|
find /var/lib/vz/dump -name '*.tar.gz' -mtime +30 -delete
|
|
```
|
|
|
|
**Priority:** **LOW** - Prevents future accumulation
|
|
|
|
---
|
|
|
|
#### 6. Monitor Storage Usage
|
|
|
|
**Action:**
|
|
- Set up alerts for pools >80% full
|
|
- Monitor container disk usage weekly
|
|
- Review storage allocation quarterly
|
|
|
|
**Priority:** **LOW** - Preventive maintenance
|
|
|
|
---
|
|
|
|
#### 7. Plan for Storage Expansion
|
|
|
|
**Current State:**
|
|
- 2 pools at 88%+ capacity
|
|
- 1 container at 94.8% full
|
|
- 3 empty pools available (thin3, thin5, thin6)
|
|
|
|
**Recommendation:**
|
|
- Utilize empty pools for new containers
|
|
- Consider storage expansion if growth continues
|
|
- Plan for 6-12 month capacity
|
|
|
|
**Priority:** **LOW** - Strategic planning
|
|
|
|
---
|
|
|
|
## Space Recovery Summary
|
|
|
|
| Action | Potential Recovery | Pool | Priority |
|
|
|--------|-------------------|------|----------|
|
|
| Remove duplicate thin1 volumes | ~300GB | thin1-r630-02 | 🔴 High |
|
|
| Clean up old backups | ~7GB | local | 🟡 Medium |
|
|
| Move containers to empty pools | N/A (redistribution) | thin2 → thin3/5/6 | 🟡 Medium |
|
|
| **Total Potential Recovery** | **~307GB** | | |
|
|
|
|
---
|
|
|
|
## Action Plan Summary
|
|
|
|
### Immediate (Today)
|
|
1. ✅ Expand container 7811 disk to 50GB
|
|
2. ✅ Verify and remove duplicate thin1 volumes
|
|
|
|
### Short-term (This Week)
|
|
3. Review and archive old backup files
|
|
4. Move container 6201 to thin3 pool
|
|
|
|
### Long-term (This Month)
|
|
5. Implement backup retention policy
|
|
6. Set up storage monitoring
|
|
7. Plan for future expansion
|
|
|
|
---
|
|
|
|
## Risk Assessment
|
|
|
|
### Low Risk Actions
|
|
- ✅ Expanding container 7811 disk (non-destructive)
|
|
- ✅ Moving stopped container 6201 to thin3
|
|
- ✅ Cleaning up old backup files
|
|
|
|
### Medium Risk Actions
|
|
- ⚠️ Removing duplicate thin1 volumes (verify first!)
|
|
- ⚠️ Moving running containers (requires downtime)
|
|
|
|
### High Risk Actions
|
|
- ❌ None identified
|
|
|
|
---
|
|
|
|
## Storage Utilization Summary
|
|
|
|
**Current Utilization:**
|
|
- **Used:** ~430GB (23% of total)
|
|
- **Available:** ~1.43TB (77% of total)
|
|
- **Critical Pools:** 2 (thin1-r630-02, thin2)
|
|
- **Empty Pools:** 3 (thin3, thin5, thin6)
|
|
|
|
**After Recommended Actions:**
|
|
- **Used:** ~130GB (7% of total)
|
|
- **Available:** ~1.73TB (93% of total)
|
|
- **Critical Pools:** 0
|
|
- **Empty Pools:** 3 (thin3, thin5, thin6)
|
|
|
|
---
|
|
|
|
**Last Updated:** 2026-01-15
|