Sankofa/docs/ceph/ROOT_CAUSE_ANALYSIS.md

# Ceph Root Cause Analysis - R630-01

**Date**: 2025-12-13  
**Status**: 🔴 **CRITICAL - Hardware Failure**

## What's Actually Broken

### 1. Ceph Monitor (ceph-mon@r630-01.service) - **FAILED**
- **Status**: `failed (Result: signal)` - Process aborted
- **Root Cause**: Corrupted RocksDB database
- **Error**: `Input/output error` reading `/var/lib/ceph/mon/ceph-r630-01/store.db/001572.sst`
- **Stack Trace**: `RocksDBStore::get()` → `MonitorDBStore::get()` → `PaxosService::refresh()` → `Monitor::preinit()` → **ABORT**

### 2. Ceph OSD (ceph-osd@1.service) - **FAILED**
- **Status**: `failed (Result: exit-code)` 
- **Root Cause**: Cannot fetch monitor configuration
- **Error**: `failed to fetch mon config (--no-mon-config to skip)`
- **Reason**: OSD requires a working MON to start, but MON is down

### 3. No Ceph Listeners
- **Port 6789** (classic msgr): No listeners
- **Port 3300** (msgr2): No listeners
- **Result**: Cluster is completely unreachable

## Underlying Hardware Issue

### Failing Disk: `/dev/sda` on R630-01

**dmesg Output Shows Critical Medium Errors**:
```
critical medium error, dev sda, sector 22995583 op 0x0:(READ) flags 0x0
critical medium error, dev sda, sector 22922714 op 0x0:(READ) flags 0x0
Sense Key: Medium Error [current]
Add. Sense: Unrecovered read error
```

**Impact**:
- The Ceph monitor's RocksDB database (`/var/lib/ceph/mon/ceph-r630-01/store.db/`) is stored on `/dev/sda`
- Disk I/O errors are causing database corruption
- The corrupted SST file (`001572.sst`) cannot be read, causing the MON to abort

## Configuration Status

### Ceph Configuration Files
- `/etc/ceph/ceph.conf`: ✅ Valid
- `/etc/pve/ceph.conf`: ✅ Valid (matches)
- **FSID**: `5fb968ae-12ab-405f-b05f-0df29a168328`
- **MON hosts**: `192.168.11.10` (ml110-01), `192.168.11.11` (r630-01)
- **Network**: `192.168.11.0/24` (public and cluster)

### Monitor Data Directory
- **Location**: `/var/lib/ceph/mon/ceph-r630-01/`
- **Status**: Directory exists, but database is corrupted
- **Disk Space**: ✅ Sufficient (69G available on `/dev/mapper/pve-root`)
- **Inodes**: ✅ Sufficient (5.1M available)

## Recovery Options

### Option 1: Recover MON from Backup (Safest)
If you have a backup of `/var/lib/ceph/mon/ceph-r630-01/`:
1. Stop the MON service: `systemctl stop ceph-mon@r630-01`
2. Restore from backup
3. Start the MON: `systemctl start ceph-mon@r630-01`

### Option 2: Recreate MON on R630-01 (Requires Quorum)
If ml110-01 MON is still healthy:
1. **First**: Fix or replace the failing disk `/dev/sda`
2. Remove the corrupted MON: `pveceph mon destroy r630-01`
3. Recreate the MON: `pveceph mon create r630-01`
4. The MON will sync from ml110-01

### Option 3: Remove Corrupted SST File (Risky)
**⚠️ WARNING**: This may cause data loss. Only attempt if you have backups.
1. Stop the MON: `systemctl stop ceph-mon@r630-01`
2. Backup the store.db directory: `mv /var/lib/ceph/mon/ceph-r630-01/store.db /var/lib/ceph/mon/ceph-r630-01/store.db.corrupted`
3. Try to recover using Ceph tools (requires working MON on ml110-01)
4. Or recreate the MON entirely

## Immediate Actions Required

### 1. **URGENT**: Address Hardware Failure
- **Check disk health**: Run `smartctl -a /dev/sda` on r630-01
- **Check filesystem**: Run `fsck` on the filesystem containing `/var/lib/ceph/mon/`
- **Plan disk replacement**: If disk is failing, replace it ASAP

### 2. **Verify ML110-01 MON Status**
Check if the other monitor is healthy:
```bash
ssh root@192.168.11.10 'systemctl status ceph-mon@ml110-01'
ssh root@192.168.11.10 'ceph -s'
```

### 3. **Backup Current State**
Before attempting recovery:
```bash
# Backup MON directory
tar -czf /root/ceph-mon-r630-01-backup-$(date +%Y%m%d).tar.gz /var/lib/ceph/mon/ceph-r630-01/

# Backup Ceph configuration
cp -a /etc/ceph/ /root/ceph-config-backup-$(date +%Y%m%d)/
cp -a /etc/pve/ceph.conf /root/ceph-config-backup-$(date +%Y%m%d)/
```

## Next Steps

1. **Verify ML110-01 MON is healthy** (determines recovery path)
2. **Check disk health** on r630-01 `/dev/sda`
3. **Decide recovery strategy** based on:
   - Whether ml110-01 MON is working
   - Whether you have backups
   - Whether disk can be fixed or needs replacement
4. **Execute recovery** following one of the options above

## Notes

- **Password Security**: The root password was exposed in command history. Please rotate it immediately.
- **Cluster State**: With only 2 MONs, losing one breaks quorum. If ml110-01 MON is also down, the cluster is unrecoverable without manual intervention.
- **OSD Status**: OSDs will remain down until MON is restored, as they require MON for configuration.