Sankofa/docs/ceph/RECOVERY_PLAN.md

# Ceph MON Recovery Plan - R630-01

## Current Status

✅ **ML110-01 MON**: Running (but in "probing" state - no quorum)  
❌ **R630-01 MON**: Failed (corrupted RocksDB due to failing disk)  
❌ **Cluster Quorum**: Lost (need 2/2 MONs for quorum)

## Root Cause Summary

1. **Hardware**: `/dev/sda` on r630-01 has critical medium errors
2. **Symptom**: Ceph MON RocksDB database corrupted (`001572.sst` I/O error)
3. **Impact**: MON cannot start, cluster has no quorum, OSDs cannot start

## Recovery Strategy

Since ml110-01 MON is running, we can recreate the r630-01 MON. The MON will sync cluster state from ml110-01.

### Prerequisites

1. **Fix or replace failing disk** `/dev/sda` on r630-01 (or move MON to different disk)
2. **Backup current state** (already documented)
3. **Verify ml110-01 MON is accessible** from r630-01

### Recovery Steps

#### Step 1: Stop Failed MON Service
```bash
ssh root@192.168.11.11 'systemctl stop ceph-mon@r630-01'
ssh root@192.168.11.11 'systemctl disable ceph-mon@r630-01'
```

#### Step 2: Backup Corrupted MON Directory
```bash
ssh root@192.168.11.11 'tar -czf /root/ceph-mon-r630-01-corrupted-$(date +%Y%m%d-%H%M%S).tar.gz /var/lib/ceph/mon/ceph-r630-01/'
```

#### Step 3: Remove Corrupted MON
```bash
ssh root@192.168.11.11 'pveceph mon destroy r630-01'
```

**Alternative if pveceph fails**:
```bash
ssh root@192.168.11.11 'rm -rf /var/lib/ceph/mon/ceph-r630-01'
```

#### Step 4: Recreate MON on R630-01
```bash
ssh root@192.168.11.11 'pveceph mon create r630-01'
```

The MON will:
- Create a new RocksDB database
- Sync cluster state from ml110-01
- Rejoin the quorum

#### Step 5: Verify Recovery
```bash
# Check MON status
ssh root@192.168.11.11 'systemctl status ceph-mon@r630-01'

# Check cluster quorum
ssh root@192.168.11.11 'ceph quorum_status'

# Check cluster health
ssh root@192.168.11.11 'ceph -s'
```

#### Step 6: Restart OSDs
Once quorum is restored:
```bash
ssh root@192.168.11.11 'systemctl restart ceph-osd@1'
ssh root@192.168.11.10 'systemctl restart ceph-osd@0'
```

## Disk Issue Resolution

### Option A: Move MON to Different Disk (If Available)
If r630-01 has another disk with space:
1. Create new directory on different disk: `/mnt/other-disk/ceph-mon/`
2. Update MON service to use new location
3. Recreate MON in new location

### Option B: Replace Failing Disk
1. Replace `/dev/sda` with new disk
2. Restore Proxmox installation
3. Recreate MON

### Option C: Fix Filesystem (If Disk is Recoverable)
1. Run `fsck` on filesystem
2. Check SMART status: `smartctl -a /dev/sda`
3. If disk is recoverable, proceed with MON recreation

## Risk Assessment

- **Low Risk**: Recreating MON from healthy peer (ml110-01) is standard procedure
- **Medium Risk**: If ml110-01 MON also fails during recovery, cluster becomes unrecoverable
- **High Risk**: If disk continues to fail, MON will corrupt again

## Post-Recovery Actions

1. **Monitor disk health**: Set up SMART monitoring for `/dev/sda`
2. **Add third MON**: Consider adding a third MON on a different node for redundancy
3. **Backup MON data**: Regularly backup `/var/lib/ceph/mon/` directories
4. **Monitor cluster**: Watch for recurring I/O errors

## Notes

- **Password Security**: Root password was exposed. Rotate immediately.
- **Quorum**: With 2 MONs, both must be healthy. Consider adding a third MON.
- **OSD Recovery**: OSDs will automatically recover once MON quorum is restored.