Sankofa/docs/ceph/MON_RECOVERY_SUCCESS.md

# Ceph MON Recovery - Success

**Date**: 2025-12-13  
**Status**: ✅ **RECOVERED**

## Recovery Summary

Successfully recovered Ceph cluster by recreating the corrupted MON on r630-01.

## What Was Done

### 1. Identified Root Cause
- **Hardware Failure**: `/dev/sda` on r630-01 has critical medium errors
- **Symptom**: Corrupted RocksDB database in `/var/lib/ceph/mon/ceph-r630-01/store.db/`
- **Impact**: MON could not start, cluster lost quorum, OSDs could not start

### 2. Recovery Steps Executed

1. **Stopped and disabled failed MON service**
   ```bash
   systemctl stop ceph-mon@r630-01
   systemctl disable ceph-mon@r630-01
   ```

2. **Backed up corrupted MON directory**
   - Created backup: `/root/ceph-mon-r630-01-corrupted-*.tar.gz`
   - Note: Backup showed file corruption (file shrank during tar)

3. **Removed corrupted MON directory**
   ```bash
   rm -rf /var/lib/ceph/mon/ceph-r630-01
   ```

4. **Created new monmap from configuration**
   ```bash
   monmaptool --create --fsid 5fb968ae-12ab-405f-b05f-0df29a168328 \
     --add ml110-01 192.168.11.10 \
     --add r630-01 192.168.11.11 \
     /tmp/monmap
   ```

5. **Copied keyring from ml110-01**
   - Source: `/var/lib/ceph/mon/ceph-ml110-01/keyring`
   - Destination: `/tmp/mon-keyring` on r630-01

6. **Created new MON filesystem**
   ```bash
   mkdir -p /var/lib/ceph/mon/ceph-r630-01
   chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
   ```

7. **Fixed ownership and started MON**
   ```bash
   chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   systemctl enable ceph-mon@r630-01
   systemctl start ceph-mon@r630-01
   ```

8. **Restarted OSDs**
   - `ceph-osd@1` on r630-01: ✅ Running
   - `ceph-osd@0` on ml110-01: ✅ Running

## Current Status

### Services
- ✅ `ceph-mon@r630-01.service`: **active (running)**
- ✅ `ceph-mon@ml110-01.service`: **active (running)** (was already running)
- ✅ `ceph-osd@1.service`: **active (running)**
- ✅ `ceph-osd@0.service`: **active (running)**

### Cluster
- **Quorum**: Restored (2/2 MONs)
- **OSDs**: 2/2 running
- **Cluster Access**: Restored (commands no longer timeout)

## Remaining Issues

### 1. Hardware Failure (CRITICAL)
- **Disk**: `/dev/sda` on r630-01 still has critical medium errors
- **Risk**: MON database may corrupt again if disk continues to fail
- **Action Required**: 
  - Monitor disk health: `smartctl -a /dev/sda`
  - Plan disk replacement
  - Consider moving MON to different disk if available

### 2. Ceph Health Issues (Still Present)
- **OSD Count**: 2 OSDs < required 3 (osd_pool_default_size = 3)
- **Undersized PGs**: PG 3.7f stuck undersized for 32h+
- **PG Over-allocation**: 288 PGs per OSD (max 250), RBD pool has 64 PGs (should be 32)

### 3. Password Security
- Root password was exposed in command history
- **Action Required**: Rotate root password immediately

## Next Steps

1. **Monitor cluster health**: `ceph -s` and `ceph health detail`
2. **Address disk failure**: Check SMART status, plan replacement
3. **Add third OSD**: Use the 6x 250GB drives on r630-01 (once disk issue is resolved)
4. **Fix PG allocation**: Reduce RBD pool PGs from 64 to 32
5. **Rotate passwords**: Change root password on all nodes

## Lessons Learned

1. **Hardware monitoring is critical**: Disk failures can cascade to application layer
2. **MON quorum requires redundancy**: With only 2 MONs, losing one breaks the cluster
3. **Ownership matters**: MON directory must be owned by `ceph:ceph`
4. **Recovery is possible**: Even with corrupted database, MON can be recreated from peer

## Recovery Time

- **Start**: 19:10 PST
- **MON Running**: 19:24 PST
- **OSDs Running**: 19:25 PST
- **Total**: ~15 minutes