chore: consolidate local WIP (repo cleanup 20260707)

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/ceph/CREATE_THIRD_OSD_NOW.md

@@ -0,0 +1,264 @@
+# Create Third Ceph OSD - Ready to Execute
+
+**Date**: 2025-12-13  
+**Status**: ✅ **READY - ALL DRIVES IDENTIFIED**
+
+---
+
+## Current Status
+
+✅ **6x 250GB drives found**: sdc, sdd, sde, sdf, sdg, sdh  
+✅ **All have partitions** (sdc3, sdd3, sde3, sdf3, sdg3, sdh3 in ubuntu-vg)  
+✅ **User confirmed**: 2 drives need formatting  
+🎯 **Ready to create third OSD**
+
+---
+
+## Quick Method: Interactive Script
+
+On R630-01, run the interactive script:
+
+```bash
+# Copy script to R630-01
+scp scripts/create-third-ceph-osd.sh root@192.168.11.11:/tmp/
+
+# SSH to R630-01
+ssh root@192.168.11.11
+
+# Run interactive script
+bash /tmp/create-third-ceph-osd.sh
+```
+
+The script will:
+1. Show available drives
+2. Ask you to select one
+3. Remove it from ubuntu-vg (if needed)
+4. Wipe the drive
+5. Create Ceph OSD
+6. Verify creation
+
+---
+
+## Manual Method: Step-by-Step
+
+### Step 1: Choose a Drive
+
+Available drives: **sdc, sdd, sde, sdf, sdg, sdh**
+
+**Recommendation**: Choose one that's safe to remove (you mentioned 2 need formatting - use one of those).
+
+### Step 2: Check if Drive is in ubuntu-vg
+
+```bash
+# Check which drives are in ubuntu-vg
+pvs | grep ubuntu-vg
+
+# Example output will show drives like:
+# /dev/sdc3  ubuntu-vg  lvm2  ...
+```
+
+### Step 3: Remove from ubuntu-vg (if needed)
+
+**WARNING**: This destroys data on that logical volume!
+
+```bash
+# Replace DRIVE with your choice (e.g., sdc, sdd, etc.)
+
+# 1. Unmount any mounted logical volumes
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+
+# 2. Deactivate volume group
+vgchange -a n ubuntu-vg
+
+# 3. Remove physical volume from ubuntu-vg
+pvremove /dev/DRIVE3 -y
+```
+
+### Step 4: Wipe the Drive
+
+**WARNING**: This destroys all data on the drive!
+
+```bash
+# Wipe filesystem signatures
+wipefs -a /dev/DRIVE
+
+# Zero out first 100MB (for clean slate)
+dd if=/dev/zero of=/dev/DRIVE bs=1M count=100
+```
+
+### Step 5: Create Ceph OSD
+
+```bash
+# Create Ceph OSD on the wiped drive
+ceph-volume lvm create --data /dev/DRIVE
+```
+
+**Expected output**:
+```
+Running command: /usr/bin/ceph-osd --cluster ceph ...
+--> ceph-volume lvm create successful for: /dev/DRIVE
+```
+
+### Step 6: Verify
+
+```bash
+# Check OSD tree (should show 3 OSDs now)
+ceph osd tree
+
+# Check Ceph health (should improve)
+ceph health
+ceph health detail
+
+# Check OSD details
+ceph osd df
+```
+
+---
+
+## Example: Using sdc
+
+Here's a complete example using `/dev/sdc`:
+
+```bash
+# 1. Check if in ubuntu-vg
+pvs | grep sdc
+
+# 2. If in ubuntu-vg, remove it
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+vgchange -a n ubuntu-vg
+pvremove /dev/sdc3 -y
+
+# 3. Wipe drive
+wipefs -a /dev/sdc
+dd if=/dev/zero of=/dev/sdc bs=1M count=100
+
+# 4. Create OSD
+ceph-volume lvm create --data /dev/sdc
+
+# 5. Verify
+ceph osd tree
+ceph health
+```
+
+---
+
+## Expected Results
+
+### Before
+```
+ID  CLASS  WEIGHT   TYPE NAME          STATUS
+-1         1.18250  root default
+-3         0.90970      host ml110-01
+ 0    hdd  0.90970          osd.0          up
+-5         0.27280      host r630-01
+ 1    hdd  0.27280          osd.1          up
+```
+
+### After
+```
+ID  CLASS  WEIGHT   TYPE NAME          STATUS
+-1         1.4xxx   root default
+-3         0.90970      host ml110-01
+ 0    hdd  0.90970          osd.0          up
+-5         0.6xxx       host r630-01
+ 1    hdd  0.27280          osd.1          up
+ 2    hdd  0.2xxx           osd.2          up  ← NEW!
+```
+
+### Ceph Health
+
+**Before**:
+```
+HEALTH_WARN ... OSD count 2 < osd_pool_default_size 3
+```
+
+**After**:
+```
+HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)
+```
+
+---
+
+## Troubleshooting
+
+### Issue: pvremove fails
+
+**Error**: "Can't open /dev/DRIVE3 exclusively. Mounted filesystem?"
+
+**Solution**:
+```bash
+# Make sure nothing is mounted
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+vgchange -a n ubuntu-vg
+
+# Force remove
+pvremove /dev/DRIVE3 -y -ff
+```
+
+### Issue: ceph-volume fails
+
+**Error**: "Device /dev/DRIVE has partitions"
+
+**Solution**:
+```bash
+# Make sure drive is completely wiped
+wipefs -a /dev/DRIVE
+dd if=/dev/zero of=/dev/DRIVE bs=1M count=100
+
+# Try again
+ceph-volume lvm create --data /dev/DRIVE
+```
+
+### Issue: OSD created but not healthy
+
+**Solution**:
+```bash
+# Wait a few minutes for OSD to join cluster
+sleep 60
+
+# Check status
+ceph osd tree
+ceph health detail
+
+# Check OSD logs
+journalctl -u ceph-osd@2 -n 50
+```
+
+---
+
+## Safety Checklist
+
+Before proceeding:
+
+- [ ] **Backup important data** (if ubuntu-vg contains data)
+- [ ] **Verify ubuntu-vg is not critical** for system operation
+- [ ] **Choose drive to remove** (sdc, sdd, sde, sdf, sdg, or sdh)
+- [ ] **Confirm drive selection** (double-check!)
+- [ ] **Have recovery plan** if something goes wrong
+
+---
+
+## Summary
+
+### Current State
+- ✅ 6x 250GB drives identified (sdc-sdh)
+- ✅ All in ubuntu-vg volume group
+- ✅ Ready to free one drive for Ceph OSD
+
+### Action Required
+1. Choose one drive (sdc-sdh)
+2. Remove from ubuntu-vg (if needed)
+3. Wipe drive
+4. Create Ceph OSD
+5. Verify health improves
+
+### Expected Result
+- ✅ 3 OSDs total (fixes TOO_FEW_OSDS error)
+- ✅ Ceph health improves
+- ✅ 21 VMs can be created with ceph-fs storage
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: ✅ **READY TO EXECUTE**
+