fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
144 lines
3.9 KiB
Markdown
144 lines
3.9 KiB
Markdown
# R630-02 Container Startup Failures - Final Analysis
|
|
|
|
**Date:** January 19, 2026
|
|
**Status:** ⚠️ **STORAGE VOLUMES MISSING ON BOTH NODES**
|
|
|
|
---
|
|
|
|
## Complete Picture
|
|
|
|
### Initial Problem
|
|
- 33 containers failed to start on r630-02
|
|
- Error messages indicated missing logical volumes and startup failures
|
|
|
|
### Root Cause Discovery
|
|
|
|
1. **Containers don't exist on r630-02**
|
|
- All 33 containers have missing configuration files
|
|
- No logical volumes exist for these containers
|
|
- Containers were migrated or never created on r630-02
|
|
|
|
2. **Containers exist on pve2 but can't start**
|
|
- All 33 containers are present on pve2 (192.168.11.11)
|
|
- Containers are in "stopped" state
|
|
- **Storage volumes are missing on pve2 as well**
|
|
- Error: `no such logical volume pve/vm-XXXX-disk-X`
|
|
|
|
### Conclusion
|
|
|
|
**The containers exist on pve2, but their storage volumes are missing.** This means:
|
|
- Container configurations exist
|
|
- Logical volumes (storage) were deleted or never created
|
|
- Containers cannot start without their storage volumes
|
|
|
|
---
|
|
|
|
## Storage Status
|
|
|
|
### On r630-02:
|
|
- Storage pools: `thin1-r630-02` (active), `thin1` (inactive), `data` (inactive)
|
|
- No logical volumes for failed containers
|
|
- Containers don't exist
|
|
|
|
### On pve2:
|
|
- Containers exist but storage volumes missing
|
|
- Need to check available storage pools
|
|
- Need to recreate or migrate storage volumes
|
|
|
|
---
|
|
|
|
## Resolution Options
|
|
|
|
### Option 1: Recreate Storage Volumes on pve2 (If Data Not Critical)
|
|
|
|
If the data in these containers is not critical or can be restored:
|
|
|
|
1. **Delete and recreate containers:**
|
|
```bash
|
|
# For each container
|
|
ssh root@192.168.11.11 "pct destroy <VMID>"
|
|
# Then recreate with proper storage
|
|
```
|
|
|
|
2. **Or recreate just the storage volumes:**
|
|
```bash
|
|
# Check container storage config
|
|
ssh root@192.168.11.11 "pct config <VMID> | grep rootfs"
|
|
|
|
# Recreate volume on correct storage pool
|
|
# (requires knowing correct storage pool and size)
|
|
```
|
|
|
|
### Option 2: Restore from Backup
|
|
|
|
If backups exist:
|
|
```bash
|
|
# Restore containers from backup
|
|
vzdump --restore <backup_file> <VMID>
|
|
```
|
|
|
|
### Option 3: Migrate to Different Storage
|
|
|
|
If storage pools are misconfigured:
|
|
```bash
|
|
# Migrate to working storage pool
|
|
pct migrate <VMID> <target_node> --storage <working_storage> --restart
|
|
```
|
|
|
|
### Option 4: Check for Volumes on Other Storage Pools
|
|
|
|
Volumes might exist but on different storage pools:
|
|
```bash
|
|
# Check all storage pools
|
|
ssh root@192.168.11.11 "lvs | grep vm-3000"
|
|
ssh root@192.168.11.11 "pvesm status"
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. **Check storage configuration on pve2:**
|
|
- Identify available storage pools
|
|
- Check if volumes exist on different pools
|
|
- Verify container storage requirements
|
|
|
|
2. **Determine data criticality:**
|
|
- Are these containers critical?
|
|
- Can data be restored from backup?
|
|
- Can containers be recreated?
|
|
|
|
3. **Execute resolution:**
|
|
- Recreate volumes if data not critical
|
|
- Restore from backup if available
|
|
- Migrate to working storage if misconfigured
|
|
|
|
---
|
|
|
|
## Container List (All on pve2, Storage Missing)
|
|
|
|
| VMID | Hostname | Status | Storage Issue |
|
|
|------|----------|--------|----------------|
|
|
| 3000-3003 | ml110 (x4) | stopped | Missing LV |
|
|
| 3500-3501 | oracle/ccip | stopped | Missing LV |
|
|
| 5200 | cacti-1 | stopped | Missing LV |
|
|
| 6000 | fabric-1 | stopped | Missing LV |
|
|
| 6400 | indy-1 | stopped | Missing LV |
|
|
| 10000-10092 | order-* (12) | stopped | Missing LV |
|
|
| 10100-10151 | dbis-* (6) | stopped | Missing LV |
|
|
| 10200-10230 | order-* (5) | stopped | Missing LV |
|
|
| 10232 | CT10232 | stopped | Missing LV + Lock |
|
|
|
|
**Total: 33 containers with missing storage volumes**
|
|
|
|
---
|
|
|
|
## Recommendation
|
|
|
|
**Immediate Action:** Investigate storage configuration on pve2 to determine:
|
|
1. Which storage pools are available
|
|
2. Whether volumes exist on different pools
|
|
3. Whether containers can be recreated or need restoration
|
|
|
|
**Long-term:** Implement storage monitoring and backup procedures to prevent this issue.
|