Sankofa/docs/ceph/CREATE_THIRD_OSD_NOW.md

Create Third Ceph OSD - Ready to Execute

Date: 2025-12-13
Status: ✅ READY - ALL DRIVES IDENTIFIED

---

Current Status

✅ 6x 250GB drives found: sdc, sdd, sde, sdf, sdg, sdh
✅ All have partitions (sdc3, sdd3, sde3, sdf3, sdg3, sdh3 in ubuntu-vg)
✅ User confirmed: 2 drives need formatting
🎯 Ready to create third OSD

---

Quick Method: Interactive Script

On R630-01, run the interactive script:

# Copy script to R630-01
scp scripts/create-third-ceph-osd.sh root@192.168.11.11:/tmp/

# SSH to R630-01
ssh root@192.168.11.11

# Run interactive script
bash /tmp/create-third-ceph-osd.sh

The script will:

1. Show available drives
2. Ask you to select one
3. Remove it from ubuntu-vg (if needed)
4. Wipe the drive
5. Create Ceph OSD
6. Verify creation

---

Manual Method: Step-by-Step

Step 1: Choose a Drive

Available drives: sdc, sdd, sde, sdf, sdg, sdh

Recommendation: Choose one that's safe to remove (you mentioned 2 need formatting - use one of those).

Step 2: Check if Drive is in ubuntu-vg

# Check which drives are in ubuntu-vg
pvs | grep ubuntu-vg

# Example output will show drives like:
# /dev/sdc3  ubuntu-vg  lvm2  ...

Step 3: Remove from ubuntu-vg (if needed)

WARNING: This destroys data on that logical volume!

# Replace DRIVE with your choice (e.g., sdc, sdd, etc.)

# 1. Unmount any mounted logical volumes
umount /dev/ubuntu-vg/* 2>/dev/null || true

# 2. Deactivate volume group
vgchange -a n ubuntu-vg

# 3. Remove physical volume from ubuntu-vg
pvremove /dev/DRIVE3 -y

Step 4: Wipe the Drive

WARNING: This destroys all data on the drive!

# Wipe filesystem signatures
wipefs -a /dev/DRIVE

# Zero out first 100MB (for clean slate)
dd if=/dev/zero of=/dev/DRIVE bs=1M count=100

Step 5: Create Ceph OSD

# Create Ceph OSD on the wiped drive
ceph-volume lvm create --data /dev/DRIVE

Expected output:

Running command: /usr/bin/ceph-osd --cluster ceph ...
--> ceph-volume lvm create successful for: /dev/DRIVE

Step 6: Verify

# Check OSD tree (should show 3 OSDs now)
ceph osd tree

# Check Ceph health (should improve)
ceph health
ceph health detail

# Check OSD details
ceph osd df

---

Example: Using sdc

Here's a complete example using /dev/sdc:

# 1. Check if in ubuntu-vg
pvs | grep sdc

# 2. If in ubuntu-vg, remove it
umount /dev/ubuntu-vg/* 2>/dev/null || true
vgchange -a n ubuntu-vg
pvremove /dev/sdc3 -y

# 3. Wipe drive
wipefs -a /dev/sdc
dd if=/dev/zero of=/dev/sdc bs=1M count=100

# 4. Create OSD
ceph-volume lvm create --data /dev/sdc

# 5. Verify
ceph osd tree
ceph health

---

Expected Results

Before

ID  CLASS  WEIGHT   TYPE NAME          STATUS
-1         1.18250  root default
-3         0.90970      host ml110-01
 0    hdd  0.90970          osd.0          up
-5         0.27280      host r630-01
 1    hdd  0.27280          osd.1          up

After

ID  CLASS  WEIGHT   TYPE NAME          STATUS
-1         1.4xxx   root default
-3         0.90970      host ml110-01
 0    hdd  0.90970          osd.0          up
-5         0.6xxx       host r630-01
 1    hdd  0.27280          osd.1          up
 2    hdd  0.2xxx           osd.2          up  ← NEW!

Ceph Health

Before:

HEALTH_WARN ... OSD count 2 < osd_pool_default_size 3

After:

HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)

---

Troubleshooting

Issue: pvremove fails

Error: "Can't open /dev/DRIVE3 exclusively. Mounted filesystem?"

Solution:

# Make sure nothing is mounted
umount /dev/ubuntu-vg/* 2>/dev/null || true
vgchange -a n ubuntu-vg

# Force remove
pvremove /dev/DRIVE3 -y -ff

Issue: ceph-volume fails

Error: "Device /dev/DRIVE has partitions"

Solution:

# Make sure drive is completely wiped
wipefs -a /dev/DRIVE
dd if=/dev/zero of=/dev/DRIVE bs=1M count=100

# Try again
ceph-volume lvm create --data /dev/DRIVE

Issue: OSD created but not healthy

Solution:

# Wait a few minutes for OSD to join cluster
sleep 60

# Check status
ceph osd tree
ceph health detail

# Check OSD logs
journalctl -u ceph-osd@2 -n 50

---

Safety Checklist

Before proceeding:

• Backup important data (if ubuntu-vg contains data)
• Verify ubuntu-vg is not critical for system operation
• Choose drive to remove (sdc, sdd, sde, sdf, sdg, or sdh)
• Confirm drive selection (double-check!)
• Have recovery plan if something goes wrong

---

Summary

Current State

• ✅ 6x 250GB drives identified (sdc-sdh)
• ✅ All in ubuntu-vg volume group
• ✅ Ready to free one drive for Ceph OSD

Action Required

1. Choose one drive (sdc-sdh)
2. Remove from ubuntu-vg (if needed)
3. Wipe drive
4. Create Ceph OSD
5. Verify health improves

Expected Result

• ✅ 3 OSDs total (fixes TOO_FEW_OSDS error)
• ✅ Ceph health improves
• ✅ 21 VMs can be created with ceph-fs storage

---

Last Updated: 2025-12-13
Status: ✅ READY TO EXECUTE