Sankofa/docs/ceph/ADD_THIRD_OSD_GUIDE.md

Guide: Adding Third Ceph OSD Using 250GB Drive on R630-01

Date: 2025-12-13
Status: 🔍 READY TO IMPLEMENT

---

Overview

R630-01 has 6x 250GB drives that are currently unused. We can use one of these drives to create a third Ceph OSD, which will:

• ✅ Fix the "TOO_FEW_OSDS" error (need 3 for replication)
• ✅ Allow 3-way replication to work properly
• ✅ Unblock 21 VMs waiting for ceph-fs storage
• ✅ Improve Ceph cluster health

---

Prerequisites

1. SSH Access to R630-01 (192.168.11.11)
2. Root privileges on R630-01
3. Ceph cluster already configured and running
4. One available 250GB drive (out of 6 total)

---

Step-by-Step Instructions

Step 1: Verify Disk Status

SSH to R630-01 and run the verification script:

# Copy script to R630-01
scp scripts/verify-r630-disks.sh root@192.168.11.11:/tmp/

# SSH to R630-01
ssh root@192.168.11.11

# Run verification script
bash /tmp/verify-r630-disks.sh

OR run commands manually:

# List all block devices
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL

# Check all disks
fdisk -l | grep -E "^Disk /dev/sd"

# Check current Ceph OSDs
ceph osd tree
ceph health detail

Step 2: Identify Available 250GB Drive

Look for drives that are:

• ✅ Around 232-260 GB in size (250GB drives)
• ✅ Not mounted (no MOUNTPOINT)
• ✅ Not used by Proxmox (not in pve-* volume groups)
• ✅ Not already a Ceph OSD

Example output to look for:

sdc    250G  disk        # Available!
sdd    250G  disk        # Available!
sde    250G  disk        # Available!
...

Step 3: Verify Drive is Safe to Use

CRITICAL: Make sure the drive is not in use!

# Check if drive has partitions
fdisk -l /dev/sdX  # Replace sdX with your drive

# Check if drive is in a volume group
pvs | grep sdX

# Check if drive is mounted
mount | grep sdX

# Check if drive is used by Ceph
ceph-volume lvm list | grep sdX

The drive should show:

• No partitions (or only empty partition table)
• Not in any volume group
• Not mounted
• Not used by Ceph

Step 4: Create Ceph OSD

WARNING: This will destroy all data on the drive!

# Replace sdX with your actual drive (e.g., sdc, sdd, etc.)
# Example: ceph-volume lvm create --data /dev/sdc

ceph-volume lvm create --data /dev/sdX

Expected output:

Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid <uuid> --setuser ceph --setgroup ceph
...
--> ceph-volume lvm create successful for: /dev/sdc

Step 5: Verify OSD is Created

# Check OSD tree (should show 3 OSDs now)
ceph osd tree

# Check OSD status
ceph osd df

# Check Ceph health (should improve)
ceph health
ceph health detail

Expected improvements:

• ✅ "TOO_FEW_OSDS" warning should disappear
• ✅ OSD count should be 3
• ✅ Health should improve (may still have other warnings)

Step 6: Verify Replication Works

# Check pool replication settings
ceph osd pool get rbd size
ceph osd pool get rbd min_size

# Should show size=3 (or 2 if temporarily reduced)

Step 7: Monitor Cluster Health

# Watch Ceph health
watch -n 5 'ceph health'

# Check for any errors
ceph health detail | grep -i error

---

Troubleshooting

Issue: Drive Not Detected

Symptoms: Drive doesn't show up in lsblk or fdisk -l

Solutions:

1. Check if drive is physically installed
2. Check PERC controller status (if using hardware RAID)
3. Check if drive needs to be initialized in PERC
4. Check system logs: dmesg | grep -i sd

Issue: Drive Already Has Data

Symptoms: Drive shows partitions or filesystem

Solutions:

1. If data is important: Backup first!
2. If data can be destroyed: Wipe the drive:

   # WARNING: Destroys all data!
   wipefs -a /dev/sdX
   dd if=/dev/zero of=/dev/sdX bs=1M count=100
   

Issue: Permission Denied

Symptoms: Cannot create OSD, permission errors

Solutions:

1. Make sure you're running as root
2. Check Ceph permissions: ceph auth list
3. Check if Ceph user has proper permissions

Issue: OSD Creation Fails

Symptoms: ceph-volume command fails

Solutions:

1. Check Ceph cluster status: ceph health
2. Check if Ceph is running: systemctl status ceph.target
3. Check logs: journalctl -u ceph-osd@* -n 50
4. Try verbose mode: ceph-volume lvm create --data /dev/sdX --verbose

Issue: OSD Created But Not Healthy

Symptoms: OSD shows up but health is still bad

Solutions:

1. Wait a few minutes for OSD to join cluster
2. Check OSD status: ceph osd tree
3. Check OSD details: ceph osd df tree
4. Check for errors: ceph health detail

---

Verification Checklist

After creating the third OSD, verify:

• OSD appears in ceph osd tree (should show 3 OSDs)
• "TOO_FEW_OSDS" warning is gone
• Ceph health improves (may still have other warnings)
• OSD is "up" and "in" (not "down" or "out")
• Replication factor can be set to 3
• VMs can be created with ceph-fs storage

---

Expected Results

Before (Current State)

HEALTH_WARN 2 OSDs; TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3

After (With Third OSD)

HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)

OSD Tree (After)

ID  CLASS  WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
-1         1.5   root default
-3         0.9       host ml110-01
 0   hdd   0.9           osd.0         up  1.00000  1.00000
-5         0.6       host r630-01
 1   hdd   0.3           osd.1         up  1.00000  1.00000
 2   hdd   0.3           osd.2         up  1.00000  1.00000

---

Next Steps After Adding Third OSD

1. Monitor Ceph Health: Watch for improvements
2. Fix Other Ceph Issues:
   • Reduce PG count (64 → 32)
   • Fix undersized placement groups
   • Resolve slow operations
3. Verify VM Creation: Try creating a VM with ceph-fs storage
4. Consider Adding More OSDs: Use remaining 5x 250GB drives for better performance

---

Related Documentation

• Disk Inventory - Complete disk inventory
• Critical Ceph Issues - All Ceph issues
• Blocking Issues Summary - Overall blocking issues

---

Summary

Current State

• 🔴 2 OSDs (insufficient for 3-way replication)
• 🔴 21 VMs blocked (waiting for ceph-fs storage)

After Adding Third OSD

• ✅ 3 OSDs (sufficient for 3-way replication)
• ✅ VMs can be created (ceph-fs storage available)
• ✅ Ceph health improves

Last Updated: 2025-12-13
Status: 🔍 READY TO IMPLEMENT - VERIFY 250GB DRIVES FIRST