Sankofa/docs/ceph/ROOT_CAUSE_ANALYSIS.md

Ceph Root Cause Analysis - R630-01

Date: 2025-12-13
Status: 🔴 CRITICAL - Hardware Failure

What's Actually Broken

1. Ceph Monitor (ceph-mon@r630-01.service) - FAILED

• Status: failed (Result: signal) - Process aborted
• Root Cause: Corrupted RocksDB database
• Error: Input/output error reading /var/lib/ceph/mon/ceph-r630-01/store.db/001572.sst
• Stack Trace: RocksDBStore::get() → MonitorDBStore::get() → PaxosService::refresh() → Monitor::preinit() → ABORT

2. Ceph OSD (ceph-osd@1.service) - FAILED

• Status: failed (Result: exit-code)
• Root Cause: Cannot fetch monitor configuration
• Error: failed to fetch mon config (--no-mon-config to skip)
• Reason: OSD requires a working MON to start, but MON is down

3. No Ceph Listeners

• Port 6789 (classic msgr): No listeners
• Port 3300 (msgr2): No listeners
• Result: Cluster is completely unreachable

Underlying Hardware Issue

Failing Disk: /dev/sda on R630-01

dmesg Output Shows Critical Medium Errors:

critical medium error, dev sda, sector 22995583 op 0x0:(READ) flags 0x0
critical medium error, dev sda, sector 22922714 op 0x0:(READ) flags 0x0
Sense Key: Medium Error [current]
Add. Sense: Unrecovered read error

Impact:

• The Ceph monitor's RocksDB database (/var/lib/ceph/mon/ceph-r630-01/store.db/) is stored on /dev/sda
• Disk I/O errors are causing database corruption
• The corrupted SST file (001572.sst) cannot be read, causing the MON to abort

Configuration Status

Ceph Configuration Files

• /etc/ceph/ceph.conf: ✅ Valid
• /etc/pve/ceph.conf: ✅ Valid (matches)
• FSID: 5fb968ae-12ab-405f-b05f-0df29a168328
• MON hosts: 192.168.11.10 (ml110-01), 192.168.11.11 (r630-01)
• Network: 192.168.11.0/24 (public and cluster)

Monitor Data Directory

• Location: /var/lib/ceph/mon/ceph-r630-01/
• Status: Directory exists, but database is corrupted
• Disk Space: ✅ Sufficient (69G available on /dev/mapper/pve-root)
• Inodes: ✅ Sufficient (5.1M available)

Recovery Options

Option 1: Recover MON from Backup (Safest)

If you have a backup of /var/lib/ceph/mon/ceph-r630-01/:

1. Stop the MON service: systemctl stop ceph-mon@r630-01
2. Restore from backup
3. Start the MON: systemctl start ceph-mon@r630-01

Option 2: Recreate MON on R630-01 (Requires Quorum)

If ml110-01 MON is still healthy:

1. First: Fix or replace the failing disk /dev/sda
2. Remove the corrupted MON: pveceph mon destroy r630-01
3. Recreate the MON: pveceph mon create r630-01
4. The MON will sync from ml110-01

Option 3: Remove Corrupted SST File (Risky)

⚠️ WARNING: This may cause data loss. Only attempt if you have backups.

1. Stop the MON: systemctl stop ceph-mon@r630-01
2. Backup the store.db directory: mv /var/lib/ceph/mon/ceph-r630-01/store.db /var/lib/ceph/mon/ceph-r630-01/store.db.corrupted
3. Try to recover using Ceph tools (requires working MON on ml110-01)
4. Or recreate the MON entirely

Immediate Actions Required

1. URGENT: Address Hardware Failure

• Check disk health: Run smartctl -a /dev/sda on r630-01
• Check filesystem: Run fsck on the filesystem containing /var/lib/ceph/mon/
• Plan disk replacement: If disk is failing, replace it ASAP

2. Verify ML110-01 MON Status

Check if the other monitor is healthy:

ssh root@192.168.11.10 'systemctl status ceph-mon@ml110-01'
ssh root@192.168.11.10 'ceph -s'

3. Backup Current State

Before attempting recovery:

# Backup MON directory
tar -czf /root/ceph-mon-r630-01-backup-$(date +%Y%m%d).tar.gz /var/lib/ceph/mon/ceph-r630-01/

# Backup Ceph configuration
cp -a /etc/ceph/ /root/ceph-config-backup-$(date +%Y%m%d)/
cp -a /etc/pve/ceph.conf /root/ceph-config-backup-$(date +%Y%m%d)/

Next Steps

1. Verify ML110-01 MON is healthy (determines recovery path)
2. Check disk health on r630-01 /dev/sda
3. Decide recovery strategy based on:
   • Whether ml110-01 MON is working
   • Whether you have backups
   • Whether disk can be fixed or needs replacement
4. Execute recovery following one of the options above

Notes

• Password Security: The root password was exposed in command history. Please rotate it immediately.
• Cluster State: With only 2 MONs, losing one breaks quorum. If ml110-01 MON is also down, the cluster is unrecoverable without manual intervention.
• OSD Status: OSDs will remain down until MON is restored, as they require MON for configuration.