Sankofa/docs/ceph/CREATE_THIRD_OSD_NOW.md

# Create Third Ceph OSD - Ready to Execute

**Date**: 2025-12-13  
**Status**: ✅ **READY - ALL DRIVES IDENTIFIED**

---

## Current Status

✅ **6x 250GB drives found**: sdc, sdd, sde, sdf, sdg, sdh  
✅ **All have partitions** (sdc3, sdd3, sde3, sdf3, sdg3, sdh3 in ubuntu-vg)  
✅ **User confirmed**: 2 drives need formatting  
🎯 **Ready to create third OSD**

---

## Quick Method: Interactive Script

On R630-01, run the interactive script:

```bash
# Copy script to R630-01
scp scripts/create-third-ceph-osd.sh root@192.168.11.11:/tmp/

# SSH to R630-01
ssh root@192.168.11.11

# Run interactive script
bash /tmp/create-third-ceph-osd.sh
```

The script will:
1. Show available drives
2. Ask you to select one
3. Remove it from ubuntu-vg (if needed)
4. Wipe the drive
5. Create Ceph OSD
6. Verify creation

---

## Manual Method: Step-by-Step

### Step 1: Choose a Drive

Available drives: **sdc, sdd, sde, sdf, sdg, sdh**

**Recommendation**: Choose one that's safe to remove (you mentioned 2 need formatting - use one of those).

### Step 2: Check if Drive is in ubuntu-vg

```bash
# Check which drives are in ubuntu-vg
pvs | grep ubuntu-vg

# Example output will show drives like:
# /dev/sdc3  ubuntu-vg  lvm2  ...
```

### Step 3: Remove from ubuntu-vg (if needed)

**WARNING**: This destroys data on that logical volume!

```bash
# Replace DRIVE with your choice (e.g., sdc, sdd, etc.)

# 1. Unmount any mounted logical volumes
umount /dev/ubuntu-vg/* 2>/dev/null || true

# 2. Deactivate volume group
vgchange -a n ubuntu-vg

# 3. Remove physical volume from ubuntu-vg
pvremove /dev/DRIVE3 -y
```

### Step 4: Wipe the Drive

**WARNING**: This destroys all data on the drive!

```bash
# Wipe filesystem signatures
wipefs -a /dev/DRIVE

# Zero out first 100MB (for clean slate)
dd if=/dev/zero of=/dev/DRIVE bs=1M count=100
```

### Step 5: Create Ceph OSD

```bash
# Create Ceph OSD on the wiped drive
ceph-volume lvm create --data /dev/DRIVE
```

**Expected output**:
```
Running command: /usr/bin/ceph-osd --cluster ceph ...
--> ceph-volume lvm create successful for: /dev/DRIVE
```

### Step 6: Verify

```bash
# Check OSD tree (should show 3 OSDs now)
ceph osd tree

# Check Ceph health (should improve)
ceph health
ceph health detail

# Check OSD details
ceph osd df
```

---

## Example: Using sdc

Here's a complete example using `/dev/sdc`:

```bash
# 1. Check if in ubuntu-vg
pvs | grep sdc

# 2. If in ubuntu-vg, remove it
umount /dev/ubuntu-vg/* 2>/dev/null || true
vgchange -a n ubuntu-vg
pvremove /dev/sdc3 -y

# 3. Wipe drive
wipefs -a /dev/sdc
dd if=/dev/zero of=/dev/sdc bs=1M count=100

# 4. Create OSD
ceph-volume lvm create --data /dev/sdc

# 5. Verify
ceph osd tree
ceph health
```

---

## Expected Results

### Before
```
ID  CLASS  WEIGHT   TYPE NAME          STATUS
-1         1.18250  root default
-3         0.90970      host ml110-01
 0    hdd  0.90970          osd.0          up
-5         0.27280      host r630-01
 1    hdd  0.27280          osd.1          up
```

### After
```
ID  CLASS  WEIGHT   TYPE NAME          STATUS
-1         1.4xxx   root default
-3         0.90970      host ml110-01
 0    hdd  0.90970          osd.0          up
-5         0.6xxx       host r630-01
 1    hdd  0.27280          osd.1          up
 2    hdd  0.2xxx           osd.2          up  ← NEW!
```

### Ceph Health

**Before**:
```
HEALTH_WARN ... OSD count 2 < osd_pool_default_size 3
```

**After**:
```
HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)
```

---

## Troubleshooting

### Issue: pvremove fails

**Error**: "Can't open /dev/DRIVE3 exclusively. Mounted filesystem?"

**Solution**:
```bash
# Make sure nothing is mounted
umount /dev/ubuntu-vg/* 2>/dev/null || true
vgchange -a n ubuntu-vg

# Force remove
pvremove /dev/DRIVE3 -y -ff
```

### Issue: ceph-volume fails

**Error**: "Device /dev/DRIVE has partitions"

**Solution**:
```bash
# Make sure drive is completely wiped
wipefs -a /dev/DRIVE
dd if=/dev/zero of=/dev/DRIVE bs=1M count=100

# Try again
ceph-volume lvm create --data /dev/DRIVE
```

### Issue: OSD created but not healthy

**Solution**:
```bash
# Wait a few minutes for OSD to join cluster
sleep 60

# Check status
ceph osd tree
ceph health detail

# Check OSD logs
journalctl -u ceph-osd@2 -n 50
```

---

## Safety Checklist

Before proceeding:

- [ ] **Backup important data** (if ubuntu-vg contains data)
- [ ] **Verify ubuntu-vg is not critical** for system operation
- [ ] **Choose drive to remove** (sdc, sdd, sde, sdf, sdg, or sdh)
- [ ] **Confirm drive selection** (double-check!)
- [ ] **Have recovery plan** if something goes wrong

---

## Summary

### Current State
- ✅ 6x 250GB drives identified (sdc-sdh)
- ✅ All in ubuntu-vg volume group
- ✅ Ready to free one drive for Ceph OSD

### Action Required
1. Choose one drive (sdc-sdh)
2. Remove from ubuntu-vg (if needed)
3. Wipe drive
4. Create Ceph OSD
5. Verify health improves

### Expected Result
- ✅ 3 OSDs total (fixes TOO_FEW_OSDS error)
- ✅ Ceph health improves
- ✅ 21 VMs can be created with ceph-fs storage

---

**Last Updated**: 2025-12-13  
**Status**: ✅ **READY TO EXECUTE**