Sankofa/docs/ceph/CRITICAL_CEPH_ISSUES.md

# Critical Ceph Storage Issues

**Date**: 2025-12-13  
**Status**: 🔴 **CRITICAL - BLOCKING VM CREATION**

---

## Executive Summary

The Ceph storage cluster has **5 critical health warnings** that are likely **blocking VM creation** for 21 VMs configured to use `ceph-fs` storage.

### Impact
- **21 VMs** are configured to use `ceph-fs` storage
- **0 VMs created** - All VMs are stuck in pending state
- **Ceph cluster unhealthy** - Multiple critical warnings

---

## Critical Issues Identified

### 1. 🔴 TOO_FEW_OSDS (CRITICAL)
- **Issue**: OSD count 2 < osd_pool_default_size 3
- **Impact**: Cannot maintain 3-way replication
- **Risk**: Data loss if an OSD fails
- **Status**: 🔴 **CRITICAL**

### 2. 🔴 UNDERSIZED PLACEMENT GROUPS (CRITICAL)
- **Issue**: PG 3.7f stuck undersized for 32 hours
- **Impact**: Data not fully replicated
- **Risk**: Data loss if storage fails
- **Status**: 🔴 **CRITICAL**

### 3. 🔴 TOO_MANY_PGS (HIGH)
- **Issue**: 288 PGs per OSD (max is 250)
- **Impact**: Performance degradation, slow operations
- **Risk**: Cluster instability
- **Status**: 🟠 **HIGH**

### 4. 🟠 POOL_TOO_MANY_PGS (MEDIUM)
- **Issue**: RBD pool has 64 PGs, should have 32
- **Impact**: Suboptimal performance
- **Risk**: Slower I/O operations
- **Status**: 🟡 **MEDIUM**

### 5. 🟠 SLOW_OPS (MEDIUM)
- **Issue**: 98 slow operations, oldest blocked for 100 seconds
- **Impact**: Cluster operations delayed
- **Risk**: Timeouts, failed operations
- **Status**: 🟡 **MEDIUM**

---

## Affected VMs

### VMs Using ceph-fs Storage (21 total)

#### Phoenix Infrastructure (7 VMs)
- `phoenix-git-server`
- `phoenix-email-server`
- `phoenix-devops-runner`
- `phoenix-codespaces-ide`
- `phoenix-financial-messaging-gateway`
- `phoenix-business-integration-gateway`
- `phoenix-as4-gateway`

#### SMOM/DBIS-138 Infrastructure (14 VMs)
- `management`
- `monitoring`
- `blockscout`
- `services`
- `rpc-node-01` through `rpc-node-04` (4 VMs)
- `validator-01` through `validator-04` (4 VMs)
- `sentry-03`, `sentry-04` (2 VMs)

### VMs Using local-lvm Storage (9 VMs)
- `basic-vm-001`
- `cloudflare-tunnel-vm`
- `large-vm-001`
- `medium-vm-001`
- `nginx-proxy-vm`
- `phoenix-dns-primary`
- (3 more test VMs)

**Note**: VMs using `local-lvm` should not be affected by Ceph issues.

---

## Root Cause Analysis

### Primary Issue: Insufficient OSDs
- **Current**: 2 OSDs
- **Required**: 3 OSDs (for 3-way replication)
- **Impact**: Cannot maintain data redundancy

### Secondary Issues
1. **PG Over-allocation**: Too many placement groups for the number of OSDs
2. **Performance Degradation**: Slow operations due to resource constraints
3. **Data Risk**: Undersized PGs mean incomplete replication

---

## Immediate Actions Required

### 🔴 URGENT (Blocking VM Creation)

1. **Add Third OSD** (if hardware available)
   ```bash
   # On Proxmox node, add new OSD
   # This requires available disk/SSD
   ```

2. **Reduce Replication Factor** (temporary workaround)
   ```bash
   # Reduce from 3 to 2 replicas (increases risk)
   ceph osd pool set <pool-name> size 2
   ceph osd pool set <pool-name> min_size 1
   ```

3. **Reduce PG Count** (performance fix)
   ```bash
   # Reduce RBD pool PGs from 64 to 32
   ceph osd pool set rbd pg_num 32
   ceph osd pool set rbd pgp_num 32
   ```

### 🟠 HIGH PRIORITY (Performance)

4. **Fix Undersized PG**
   ```bash
   # Check PG status
   ceph pg ls-by-pool rbd
   # Force recovery if needed
   ceph pg force-recovery <pg-id>
   ```

5. **Monitor Slow Operations**
   ```bash
   # Check slow ops
   ceph health detail
   # Identify blocking operations
   ```

---

## Recommended Solutions

### Option 1: Add Third OSD (BEST)
- **Action**: Add a third OSD node or disk
- **Impact**: Resolves all replication issues
- **Risk**: Low (if hardware available)
- **Timeline**: Immediate if hardware ready

### Option 2: Reduce Replication (TEMPORARY)
- **Action**: Change replication from 3 to 2
- **Impact**: Allows VM creation but increases risk
- **Risk**: Medium (data loss if 1 OSD fails)
- **Timeline**: Immediate

### Option 3: Use local-lvm for Critical VMs (WORKAROUND)
- **Action**: Change critical VMs to use local-lvm
- **Impact**: Allows deployment but loses shared storage benefits
- **Risk**: Low (no data redundancy issues)
- **Timeline**: Requires VM config changes

---

## Verification Steps

### Check Ceph Health
```bash
# SSH to Proxmox node
ssh root@ml110-01

# Check cluster health
ceph health
ceph health detail

# Check OSD status
ceph osd tree
ceph osd df

# Check pool status
ceph osd pool ls detail

# Check PG status
ceph pg ls-by-pool rbd
```

### Check VM Creation Status
```bash
# Check VMs using ceph-fs
kubectl get proxmoxvm -A -o json | \
  jq -r '.items[] | select(.spec.forProvider.storage == "ceph-fs") | .metadata.name'

# Check for storage-related errors
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox | \
  grep -i "ceph\|storage\|rbd"
```

---

## Impact Assessment

### Current Impact
- ✅ **VMs using local-lvm**: Can be created (9 VMs)
- ❌ **VMs using ceph-fs**: Blocked (21 VMs)
- ❌ **Total Progress**: 0/30 VMs (0%)

### Expected Impact After Fix
- ✅ **All VMs**: Can be created
- ✅ **Storage**: Shared storage available
- ✅ **Data Safety**: Proper replication

---

## Next Steps

### Immediate (Today)
1. 🔴 **URGENT**: Fix OSD count or reduce replication
2. 🔴 **URGENT**: Reduce PG count in RBD pool
3. 🟠 **HIGH**: Fix undersized PG
4. 🟡 **MEDIUM**: Monitor slow operations

### Short-term (This Week)
1. Add third OSD if hardware available
2. Optimize PG distribution
3. Monitor cluster health
4. Verify VM creation resumes

### Long-term (This Month)
1. Plan for proper 3-node Ceph cluster
2. Optimize storage configuration
3. Set up monitoring and alerts
4. Document storage architecture

---

## References

### Ceph Documentation
- [Ceph Placement Groups](https://docs.ceph.com/en/latest/rados/operations/placement-groups/)
- [Ceph Pool Configuration](https://docs.ceph.com/en/latest/rados/operations/pools/)
- [Ceph OSD Management](https://docs.ceph.com/en/latest/rados/operations/operating/)

### Related Issues
- VM creation blocked (0/30 VMs created)
- Ceph cluster health warnings
- Storage configuration issues

---

## Summary

### Current State
- 🔴 **Ceph Cluster**: Unhealthy (5 critical warnings)
- 🔴 **VM Creation**: Blocked (0/30 VMs)
- 🔴 **Storage**: Insufficient OSDs (2 < 3 required)
- 🟠 **Performance**: Degraded (slow operations)

### Status
🔴 **CRITICAL - IMMEDIATE ACTION REQUIRED**

**Last Updated**: 2025-12-13  
**Status**: 🔴 **BLOCKING VM CREATION**