Sankofa/docs/ceph/COMPLETE_OSD_FIX.md

Complete OSD Creation Fix

Date: 2025-12-13
Status: ✅ READY TO EXECUTE

---

Current Situation

✅ Drives wiped: All 6 drives (sdc-sdh) are ready
❌ OSD creation failed: Bootstrap keyring issue
🎯 Solution: Fix bootstrap keyring and create OSDs

---

Quick Fix - Run This Script

On R630-01:

# Copy script
scp scripts/fix-and-create-all-osds.sh root@192.168.11.11:/tmp/

# SSH and run
ssh root@192.168.11.11
bash /tmp/fix-and-create-all-osds.sh

This script will:

1. ✅ Check Ceph cluster connectivity
2. ✅ Fix/bootstrap keyring issue
3. ✅ Create OSDs on all 6 drives
4. ✅ Verify results

---

Manual Method

If you prefer manual steps:

Step 1: Fix Bootstrap Keyring

# Create directory
mkdir -p /var/lib/ceph/bootstrap-osd

# Get bootstrap keyring from cluster
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# Verify
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring

Step 2: Create OSDs

# Method 1: Using ceph-volume
for drive in sdc sdd sde sdf sdg sdh; do
    ceph-volume lvm create --data /dev/$drive
done

# Method 2: Using pveceph (if available)
for drive in sdc sdd sde sdf sdg sdh; do
    pveceph create /dev/$drive
done

Step 3: Verify

# Check OSD tree
ceph osd tree

# Check health
ceph health
ceph health detail

# Check OSD details
ceph osd df

---

Expected Results

Before

• 2 OSDs (insufficient)
• HEALTH_WARN: TOO_FEW_OSDS

After (with all 6 OSDs)

• 8 OSDs total (2 existing + 6 new)
• Excellent redundancy
• HEALTH_OK or much improved
• Can handle many VMs

After (with just 1 OSD)

• 3 OSDs total (minimum for 3-way replication)
• Fixes TOO_FEW_OSDS error
• HEALTH_OK or improved
• VMs can be created

---

Troubleshooting

Issue: "ceph auth get" fails

Solution:

# Check if bootstrap-osd exists
ceph auth list | grep bootstrap

# If it doesn't exist, create it
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd'
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

Issue: "RADOS timed out"

Solution:

# Check Ceph services
systemctl status ceph.target
systemctl status ceph-mon@*

# Check monitor status
ceph mon stat

# Check network connectivity
ping <monitor-ip>

Issue: OSD creation still fails

Solution:

# Check logs
cat /tmp/ceph-osd-*.log

# Try with verbose output
ceph-volume lvm create --data /dev/sdc --verbose

# Or try pveceph
pveceph create /dev/sdc

---

Summary

Current State

• ✅ 6 drives wiped and ready
• ⚠️ Bootstrap keyring missing
• 🎯 Need to fix and create OSDs

Action

Run fix-and-create-all-osds.sh script

Expected Result

• 8 OSDs total (if all succeed)
• Or 3+ OSDs (minimum needed)
• Ceph health improves
• VMs can be created

---

Last Updated: 2025-12-13
Status: ✅ READY TO EXECUTE