Sankofa/docs/ceph/CRITICAL_CEPH_ISSUES.md

Critical Ceph Storage Issues

Date: 2025-12-13
Status: 🔴 CRITICAL - BLOCKING VM CREATION

---

Executive Summary

The Ceph storage cluster has 5 critical health warnings that are likely blocking VM creation for 21 VMs configured to use ceph-fs storage.

Impact

• 21 VMs are configured to use ceph-fs storage
• 0 VMs created - All VMs are stuck in pending state
• Ceph cluster unhealthy - Multiple critical warnings

---

Critical Issues Identified

1. 🔴 TOO_FEW_OSDS (CRITICAL)

• Issue: OSD count 2 < osd_pool_default_size 3
• Impact: Cannot maintain 3-way replication
• Risk: Data loss if an OSD fails
• Status: 🔴 CRITICAL

2. 🔴 UNDERSIZED PLACEMENT GROUPS (CRITICAL)

• Issue: PG 3.7f stuck undersized for 32 hours
• Impact: Data not fully replicated
• Risk: Data loss if storage fails
• Status: 🔴 CRITICAL

3. 🔴 TOO_MANY_PGS (HIGH)

• Issue: 288 PGs per OSD (max is 250)
• Impact: Performance degradation, slow operations
• Risk: Cluster instability
• Status: 🟠 HIGH

4. 🟠 POOL_TOO_MANY_PGS (MEDIUM)

• Issue: RBD pool has 64 PGs, should have 32
• Impact: Suboptimal performance
• Risk: Slower I/O operations
• Status: 🟡 MEDIUM

5. 🟠 SLOW_OPS (MEDIUM)

• Issue: 98 slow operations, oldest blocked for 100 seconds
• Impact: Cluster operations delayed
• Risk: Timeouts, failed operations
• Status: 🟡 MEDIUM

---

Affected VMs

VMs Using ceph-fs Storage (21 total)

Phoenix Infrastructure (7 VMs)

• phoenix-git-server
• phoenix-email-server
• phoenix-devops-runner
• phoenix-codespaces-ide
• phoenix-financial-messaging-gateway
• phoenix-business-integration-gateway
• phoenix-as4-gateway

SMOM/DBIS-138 Infrastructure (14 VMs)

• management
• monitoring
• blockscout
• services
• rpc-node-01 through rpc-node-04 (4 VMs)
• validator-01 through validator-04 (4 VMs)
• sentry-03, sentry-04 (2 VMs)

VMs Using local-lvm Storage (9 VMs)

• basic-vm-001
• cloudflare-tunnel-vm
• large-vm-001
• medium-vm-001
• nginx-proxy-vm
• phoenix-dns-primary
• (3 more test VMs)

Note: VMs using local-lvm should not be affected by Ceph issues.

---

Root Cause Analysis

Primary Issue: Insufficient OSDs

• Current: 2 OSDs
• Required: 3 OSDs (for 3-way replication)
• Impact: Cannot maintain data redundancy

Secondary Issues

1. PG Over-allocation: Too many placement groups for the number of OSDs
2. Performance Degradation: Slow operations due to resource constraints
3. Data Risk: Undersized PGs mean incomplete replication

---

Immediate Actions Required

🔴 URGENT (Blocking VM Creation)

1. Add Third OSD (if hardware available)

   # On Proxmox node, add new OSD
   # This requires available disk/SSD
   

2. Reduce Replication Factor (temporary workaround)

   # Reduce from 3 to 2 replicas (increases risk)
   ceph osd pool set <pool-name> size 2
   ceph osd pool set <pool-name> min_size 1
   

3. Reduce PG Count (performance fix)

   # Reduce RBD pool PGs from 64 to 32
   ceph osd pool set rbd pg_num 32
   ceph osd pool set rbd pgp_num 32
   

🟠 HIGH PRIORITY (Performance)

4. Fix Undersized PG

   # Check PG status
   ceph pg ls-by-pool rbd
   # Force recovery if needed
   ceph pg force-recovery <pg-id>
   

5. Monitor Slow Operations

   # Check slow ops
   ceph health detail
   # Identify blocking operations
   

---

Recommended Solutions

Option 1: Add Third OSD (BEST)

• Action: Add a third OSD node or disk
• Impact: Resolves all replication issues
• Risk: Low (if hardware available)
• Timeline: Immediate if hardware ready

Option 2: Reduce Replication (TEMPORARY)

• Action: Change replication from 3 to 2
• Impact: Allows VM creation but increases risk
• Risk: Medium (data loss if 1 OSD fails)
• Timeline: Immediate

Option 3: Use local-lvm for Critical VMs (WORKAROUND)

• Action: Change critical VMs to use local-lvm
• Impact: Allows deployment but loses shared storage benefits
• Risk: Low (no data redundancy issues)
• Timeline: Requires VM config changes

---

Verification Steps

Check Ceph Health

# SSH to Proxmox node
ssh root@ml110-01

# Check cluster health
ceph health
ceph health detail

# Check OSD status
ceph osd tree
ceph osd df

# Check pool status
ceph osd pool ls detail

# Check PG status
ceph pg ls-by-pool rbd

Check VM Creation Status

# Check VMs using ceph-fs
kubectl get proxmoxvm -A -o json | \
  jq -r '.items[] | select(.spec.forProvider.storage == "ceph-fs") | .metadata.name'

# Check for storage-related errors
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox | \
  grep -i "ceph\|storage\|rbd"

---

Impact Assessment

Current Impact

• ✅ VMs using local-lvm: Can be created (9 VMs)
• ❌ VMs using ceph-fs: Blocked (21 VMs)
• ❌ Total Progress: 0/30 VMs (0%)

Expected Impact After Fix

• ✅ All VMs: Can be created
• ✅ Storage: Shared storage available
• ✅ Data Safety: Proper replication

---

Next Steps

Immediate (Today)

1. 🔴 URGENT: Fix OSD count or reduce replication
2. 🔴 URGENT: Reduce PG count in RBD pool
3. 🟠 HIGH: Fix undersized PG
4. 🟡 MEDIUM: Monitor slow operations

Short-term (This Week)

1. Add third OSD if hardware available
2. Optimize PG distribution
3. Monitor cluster health
4. Verify VM creation resumes

Long-term (This Month)

1. Plan for proper 3-node Ceph cluster
2. Optimize storage configuration
3. Set up monitoring and alerts
4. Document storage architecture

---

References

Ceph Documentation

• Ceph Placement Groups
• Ceph Pool Configuration
• Ceph OSD Management

Related Issues

• VM creation blocked (0/30 VMs created)
• Ceph cluster health warnings
• Storage configuration issues

---

Summary

Current State

• 🔴 Ceph Cluster: Unhealthy (5 critical warnings)
• 🔴 VM Creation: Blocked (0/30 VMs)
• 🔴 Storage: Insufficient OSDs (2 < 3 required)
• 🟠 Performance: Degraded (slow operations)

Status

🔴 CRITICAL - IMMEDIATE ACTION REQUIRED

Last Updated: 2025-12-13
Status: 🔴 BLOCKING VM CREATION