Sankofa/docs/ceph/COMPLETE_OSD_FIX.md

# Complete OSD Creation Fix

**Date**: 2025-12-13  
**Status**: ✅ **READY TO EXECUTE**

---

## Current Situation

✅ **Drives wiped**: All 6 drives (sdc-sdh) are ready  
❌ **OSD creation failed**: Bootstrap keyring issue  
🎯 **Solution**: Fix bootstrap keyring and create OSDs

---

## Quick Fix - Run This Script

On R630-01:

```bash
# Copy script
scp scripts/fix-and-create-all-osds.sh root@192.168.11.11:/tmp/

# SSH and run
ssh root@192.168.11.11
bash /tmp/fix-and-create-all-osds.sh
```

This script will:
1. ✅ Check Ceph cluster connectivity
2. ✅ Fix/bootstrap keyring issue
3. ✅ Create OSDs on all 6 drives
4. ✅ Verify results

---

## Manual Method

If you prefer manual steps:

### Step 1: Fix Bootstrap Keyring

```bash
# Create directory
mkdir -p /var/lib/ceph/bootstrap-osd

# Get bootstrap keyring from cluster
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# Verify
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
```

### Step 2: Create OSDs

```bash
# Method 1: Using ceph-volume
for drive in sdc sdd sde sdf sdg sdh; do
    ceph-volume lvm create --data /dev/$drive
done

# Method 2: Using pveceph (if available)
for drive in sdc sdd sde sdf sdg sdh; do
    pveceph create /dev/$drive
done
```

### Step 3: Verify

```bash
# Check OSD tree
ceph osd tree

# Check health
ceph health
ceph health detail

# Check OSD details
ceph osd df
```

---

## Expected Results

### Before
- 2 OSDs (insufficient)
- HEALTH_WARN: TOO_FEW_OSDS

### After (with all 6 OSDs)
- **8 OSDs total** (2 existing + 6 new)
- Excellent redundancy
- HEALTH_OK or much improved
- Can handle many VMs

### After (with just 1 OSD)
- **3 OSDs total** (minimum for 3-way replication)
- Fixes TOO_FEW_OSDS error
- HEALTH_OK or improved
- VMs can be created

---

## Troubleshooting

### Issue: "ceph auth get" fails

**Solution**:
```bash
# Check if bootstrap-osd exists
ceph auth list | grep bootstrap

# If it doesn't exist, create it
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd'
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
```

### Issue: "RADOS timed out"

**Solution**:
```bash
# Check Ceph services
systemctl status ceph.target
systemctl status ceph-mon@*

# Check monitor status
ceph mon stat

# Check network connectivity
ping <monitor-ip>
```

### Issue: OSD creation still fails

**Solution**:
```bash
# Check logs
cat /tmp/ceph-osd-*.log

# Try with verbose output
ceph-volume lvm create --data /dev/sdc --verbose

# Or try pveceph
pveceph create /dev/sdc
```

---

## Summary

### Current State
- ✅ 6 drives wiped and ready
- ⚠️ Bootstrap keyring missing
- 🎯 Need to fix and create OSDs

### Action
Run `fix-and-create-all-osds.sh` script

### Expected Result
- 8 OSDs total (if all succeed)
- Or 3+ OSDs (minimum needed)
- Ceph health improves
- VMs can be created

---

**Last Updated**: 2025-12-13  
**Status**: ✅ **READY TO EXECUTE**