Sankofa/docs/ceph/ADD_THIRD_OSD_GUIDE.md

# Guide: Adding Third Ceph OSD Using 250GB Drive on R630-01

**Date**: 2025-12-13  
**Status**: 🔍 **READY TO IMPLEMENT**

---

## Overview

R630-01 has **6x 250GB drives** that are currently unused. We can use one of these drives to create a third Ceph OSD, which will:
- ✅ Fix the "TOO_FEW_OSDS" error (need 3 for replication)
- ✅ Allow 3-way replication to work properly
- ✅ Unblock 21 VMs waiting for ceph-fs storage
- ✅ Improve Ceph cluster health

---

## Prerequisites

1. **SSH Access** to R630-01 (192.168.11.11)
2. **Root privileges** on R630-01
3. **Ceph cluster** already configured and running
4. **One available 250GB drive** (out of 6 total)

---

## Step-by-Step Instructions

### Step 1: Verify Disk Status

SSH to R630-01 and run the verification script:

```bash
# Copy script to R630-01
scp scripts/verify-r630-disks.sh root@192.168.11.11:/tmp/

# SSH to R630-01
ssh root@192.168.11.11

# Run verification script
bash /tmp/verify-r630-disks.sh
```

**OR** run commands manually:

```bash
# List all block devices
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL

# Check all disks
fdisk -l | grep -E "^Disk /dev/sd"

# Check current Ceph OSDs
ceph osd tree
ceph health detail
```

### Step 2: Identify Available 250GB Drive

Look for drives that are:
- ✅ Around 232-260 GB in size (250GB drives)
- ✅ Not mounted (no MOUNTPOINT)
- ✅ Not used by Proxmox (not in pve-* volume groups)
- ✅ Not already a Ceph OSD

**Example output to look for**:
```
sdc    250G  disk        # Available!
sdd    250G  disk        # Available!
sde    250G  disk        # Available!
...
```

### Step 3: Verify Drive is Safe to Use

**CRITICAL**: Make sure the drive is not in use!

```bash
# Check if drive has partitions
fdisk -l /dev/sdX  # Replace sdX with your drive

# Check if drive is in a volume group
pvs | grep sdX

# Check if drive is mounted
mount | grep sdX

# Check if drive is used by Ceph
ceph-volume lvm list | grep sdX
```

**The drive should show**:
- No partitions (or only empty partition table)
- Not in any volume group
- Not mounted
- Not used by Ceph

### Step 4: Create Ceph OSD

**WARNING**: This will destroy all data on the drive!

```bash
# Replace sdX with your actual drive (e.g., sdc, sdd, etc.)
# Example: ceph-volume lvm create --data /dev/sdc

ceph-volume lvm create --data /dev/sdX
```

**Expected output**:
```
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid <uuid> --setuser ceph --setgroup ceph
...
--> ceph-volume lvm create successful for: /dev/sdc
```

### Step 5: Verify OSD is Created

```bash
# Check OSD tree (should show 3 OSDs now)
ceph osd tree

# Check OSD status
ceph osd df

# Check Ceph health (should improve)
ceph health
ceph health detail
```

**Expected improvements**:
- ✅ "TOO_FEW_OSDS" warning should disappear
- ✅ OSD count should be 3
- ✅ Health should improve (may still have other warnings)

### Step 6: Verify Replication Works

```bash
# Check pool replication settings
ceph osd pool get rbd size
ceph osd pool get rbd min_size

# Should show size=3 (or 2 if temporarily reduced)
```

### Step 7: Monitor Cluster Health

```bash
# Watch Ceph health
watch -n 5 'ceph health'

# Check for any errors
ceph health detail | grep -i error
```

---

## Troubleshooting

### Issue: Drive Not Detected

**Symptoms**: Drive doesn't show up in `lsblk` or `fdisk -l`

**Solutions**:
1. Check if drive is physically installed
2. Check PERC controller status (if using hardware RAID)
3. Check if drive needs to be initialized in PERC
4. Check system logs: `dmesg | grep -i sd`

### Issue: Drive Already Has Data

**Symptoms**: Drive shows partitions or filesystem

**Solutions**:
1. **If data is important**: Backup first!
2. **If data can be destroyed**: Wipe the drive:
   ```bash
   # WARNING: Destroys all data!
   wipefs -a /dev/sdX
   dd if=/dev/zero of=/dev/sdX bs=1M count=100
   ```

### Issue: Permission Denied

**Symptoms**: Cannot create OSD, permission errors

**Solutions**:
1. Make sure you're running as root
2. Check Ceph permissions: `ceph auth list`
3. Check if Ceph user has proper permissions

### Issue: OSD Creation Fails

**Symptoms**: `ceph-volume` command fails

**Solutions**:
1. Check Ceph cluster status: `ceph health`
2. Check if Ceph is running: `systemctl status ceph.target`
3. Check logs: `journalctl -u ceph-osd@* -n 50`
4. Try verbose mode: `ceph-volume lvm create --data /dev/sdX --verbose`

### Issue: OSD Created But Not Healthy

**Symptoms**: OSD shows up but health is still bad

**Solutions**:
1. Wait a few minutes for OSD to join cluster
2. Check OSD status: `ceph osd tree`
3. Check OSD details: `ceph osd df tree`
4. Check for errors: `ceph health detail`

---

## Verification Checklist

After creating the third OSD, verify:

- [ ] OSD appears in `ceph osd tree` (should show 3 OSDs)
- [ ] "TOO_FEW_OSDS" warning is gone
- [ ] Ceph health improves (may still have other warnings)
- [ ] OSD is "up" and "in" (not "down" or "out")
- [ ] Replication factor can be set to 3
- [ ] VMs can be created with ceph-fs storage

---

## Expected Results

### Before (Current State)
```
HEALTH_WARN 2 OSDs; TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3
```

### After (With Third OSD)
```
HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)
```

### OSD Tree (After)
```
ID  CLASS  WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
-1         1.5   root default
-3         0.9       host ml110-01
 0   hdd   0.9           osd.0         up  1.00000  1.00000
-5         0.6       host r630-01
 1   hdd   0.3           osd.1         up  1.00000  1.00000
 2   hdd   0.3           osd.2         up  1.00000  1.00000
```

---

## Next Steps After Adding Third OSD

1. **Monitor Ceph Health**: Watch for improvements
2. **Fix Other Ceph Issues**:
   - Reduce PG count (64 → 32)
   - Fix undersized placement groups
   - Resolve slow operations
3. **Verify VM Creation**: Try creating a VM with ceph-fs storage
4. **Consider Adding More OSDs**: Use remaining 5x 250GB drives for better performance

---

## Related Documentation

- [Disk Inventory](../infrastructure/DISK_INVENTORY.md) - Complete disk inventory
- [Critical Ceph Issues](./CRITICAL_CEPH_ISSUES.md) - All Ceph issues
- [Blocking Issues Summary](../BLOCKING_ISSUES_SUMMARY.md) - Overall blocking issues

---

## Summary

### Current State
- 🔴 **2 OSDs** (insufficient for 3-way replication)
- 🔴 **21 VMs blocked** (waiting for ceph-fs storage)

### After Adding Third OSD
- ✅ **3 OSDs** (sufficient for 3-way replication)
- ✅ **VMs can be created** (ceph-fs storage available)
- ✅ **Ceph health improves**

**Last Updated**: 2025-12-13  
**Status**: 🔍 **READY TO IMPLEMENT - VERIFY 250GB DRIVES FIRST**