Sankofa/docs/ceph/CEPH_CLUSTER_ISSUES.md

Ceph Cluster Instability - ML110-01

Date: 2025-12-13
Status: ⚠️ INVESTIGATING

---

Issue Summary

Symptoms

• Ceph Monitor: Constantly in "electing" state
• Slow Operations: 100+ slow operations reported
• Manager Instability: Ceph manager constantly restarting
• Connection Issues: Socket connections closing between mon0 and mon1
• Quorum Problems: Cluster appears to be having election issues

Log Patterns

- mon.ml110-01@0(electing) e2 get_health_metrics reporting 100+ slow ops
- ENGINE Bus STOPPING/STARTING (constant restarts)
- libceph: mon0/mon1 socket closed (con state OPEN)
- Session lost, hunting for new mon

---

Impact Analysis

VMs Affected

• Storage Type: ceph-fs
• Count: Multiple VMs configured to use ceph-fs
• Risk: VM creation may fail if Ceph is unavailable

Potential Issues

1. VM Creation Failures: VMs using ceph-fs may fail to create
2. Storage Unavailable: Ceph storage may be inaccessible
3. Performance Degradation: Slow operations indicate cluster stress
4. Data Risk: Instability could affect data integrity

---

Root Cause Analysis

Possible Causes

1. Network Issues: Connection problems between mon0 (ML110-01) and mon1 (R630-01)
2. Quorum Loss: Cluster may have lost quorum
3. Resource Exhaustion: System resources may be exhausted
4. Configuration Issues: Ceph configuration may be incorrect
5. Time Sync: Clock synchronization issues

---

Investigation Steps

1. Check Ceph Cluster Health

ssh root@192.168.11.10 "ceph -s"
ssh root@192.168.11.10 "ceph health detail"

2. Verify Quorum Status

ssh root@192.168.11.10 "ceph quorum_status"

3. Check Monitor Status

ssh root@192.168.11.10 "ceph mon stat"
ssh root@192.168.11.10 "ceph mon dump"

4. Check Network Connectivity

# From ML110-01 to R630-01
ssh root@192.168.11.10 "ping -c 3 192.168.11.11"
ssh root@192.168.11.10 "telnet 192.168.11.11 6789"

5. Check System Resources

ssh root@192.168.11.10 "df -h"
ssh root@192.168.11.10 "free -h"
ssh root@192.168.11.10 "iostat -x 1 3"

---

Immediate Actions

1. Verify Impact on VM Creation

• Check if any VMs have failed due to storage issues
• Monitor provider logs for Ceph-related errors
• Verify if VM creation is blocked

2. Stabilize Ceph Cluster

• Check quorum status
• Verify network connectivity
• Restart Ceph services if needed
• Check for configuration issues

3. Alternative Storage

• Consider using local-lvm for critical VMs temporarily
• Document which VMs require ceph-fs
• Plan migration if Ceph remains unstable

---

Recommendations

Short-term

1. Monitor Ceph Health: Continuously monitor cluster status
2. Check Network: Verify connectivity between nodes
3. Review Logs: Analyze Ceph logs for root cause
4. Resource Check: Verify system resources are adequate

Long-term

1. Ceph Optimization: Optimize Ceph configuration
2. Network Hardening: Ensure stable network between nodes
3. Monitoring: Set up Ceph health monitoring
4. Backup Strategy: Ensure data backup strategy is in place

---

Status

⚠️ INVESTIGATING - Ceph cluster instability detected, impact on VM deployment being assessed.

---

Last Updated: 2025-12-13
Status: ⚠️ INVESTIGATING