chore: consolidate local WIP (repo cleanup 20260707)

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/ceph/ADD_THIRD_OSD_GUIDE.md

@@ -0,0 +1,284 @@
+# Guide: Adding Third Ceph OSD Using 250GB Drive on R630-01
+
+**Date**: 2025-12-13  
+**Status**: 🔍 **READY TO IMPLEMENT**
+
+---
+
+## Overview
+
+R630-01 has **6x 250GB drives** that are currently unused. We can use one of these drives to create a third Ceph OSD, which will:
+- ✅ Fix the "TOO_FEW_OSDS" error (need 3 for replication)
+- ✅ Allow 3-way replication to work properly
+- ✅ Unblock 21 VMs waiting for ceph-fs storage
+- ✅ Improve Ceph cluster health
+
+---
+
+## Prerequisites
+
+1. **SSH Access** to R630-01 (192.168.11.11)
+2. **Root privileges** on R630-01
+3. **Ceph cluster** already configured and running
+4. **One available 250GB drive** (out of 6 total)
+
+---
+
+## Step-by-Step Instructions
+
+### Step 1: Verify Disk Status
+
+SSH to R630-01 and run the verification script:
+
+```bash
+# Copy script to R630-01
+scp scripts/verify-r630-disks.sh root@192.168.11.11:/tmp/
+
+# SSH to R630-01
+ssh root@192.168.11.11
+
+# Run verification script
+bash /tmp/verify-r630-disks.sh
+```
+
+**OR** run commands manually:
+
+```bash
+# List all block devices
+lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL
+
+# Check all disks
+fdisk -l | grep -E "^Disk /dev/sd"
+
+# Check current Ceph OSDs
+ceph osd tree
+ceph health detail
+```
+
+### Step 2: Identify Available 250GB Drive
+
+Look for drives that are:
+- ✅ Around 232-260 GB in size (250GB drives)
+- ✅ Not mounted (no MOUNTPOINT)
+- ✅ Not used by Proxmox (not in pve-* volume groups)
+- ✅ Not already a Ceph OSD
+
+**Example output to look for**:
+```
+sdc    250G  disk        # Available!
+sdd    250G  disk        # Available!
+sde    250G  disk        # Available!
+...
+```
+
+### Step 3: Verify Drive is Safe to Use
+
+**CRITICAL**: Make sure the drive is not in use!
+
+```bash
+# Check if drive has partitions
+fdisk -l /dev/sdX  # Replace sdX with your drive
+
+# Check if drive is in a volume group
+pvs | grep sdX
+
+# Check if drive is mounted
+mount | grep sdX
+
+# Check if drive is used by Ceph
+ceph-volume lvm list | grep sdX
+```
+
+**The drive should show**:
+- No partitions (or only empty partition table)
+- Not in any volume group
+- Not mounted
+- Not used by Ceph
+
+### Step 4: Create Ceph OSD
+
+**WARNING**: This will destroy all data on the drive!
+
+```bash
+# Replace sdX with your actual drive (e.g., sdc, sdd, etc.)
+# Example: ceph-volume lvm create --data /dev/sdc
+
+ceph-volume lvm create --data /dev/sdX
+```
+
+**Expected output**:
+```
+Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid <uuid> --setuser ceph --setgroup ceph
+...
+--> ceph-volume lvm create successful for: /dev/sdc
+```
+
+### Step 5: Verify OSD is Created
+
+```bash
+# Check OSD tree (should show 3 OSDs now)
+ceph osd tree
+
+# Check OSD status
+ceph osd df
+
+# Check Ceph health (should improve)
+ceph health
+ceph health detail
+```
+
+**Expected improvements**:
+- ✅ "TOO_FEW_OSDS" warning should disappear
+- ✅ OSD count should be 3
+- ✅ Health should improve (may still have other warnings)
+
+### Step 6: Verify Replication Works
+
+```bash
+# Check pool replication settings
+ceph osd pool get rbd size
+ceph osd pool get rbd min_size
+
+# Should show size=3 (or 2 if temporarily reduced)
+```
+
+### Step 7: Monitor Cluster Health
+
+```bash
+# Watch Ceph health
+watch -n 5 'ceph health'
+
+# Check for any errors
+ceph health detail | grep -i error
+```
+
+---
+
+## Troubleshooting
+
+### Issue: Drive Not Detected
+
+**Symptoms**: Drive doesn't show up in `lsblk` or `fdisk -l`
+
+**Solutions**:
+1. Check if drive is physically installed
+2. Check PERC controller status (if using hardware RAID)
+3. Check if drive needs to be initialized in PERC
+4. Check system logs: `dmesg | grep -i sd`
+
+### Issue: Drive Already Has Data
+
+**Symptoms**: Drive shows partitions or filesystem
+
+**Solutions**:
+1. **If data is important**: Backup first!
+2. **If data can be destroyed**: Wipe the drive:
+   ```bash
+   # WARNING: Destroys all data!
+   wipefs -a /dev/sdX
+   dd if=/dev/zero of=/dev/sdX bs=1M count=100
+   ```
+
+### Issue: Permission Denied
+
+**Symptoms**: Cannot create OSD, permission errors
+
+**Solutions**:
+1. Make sure you're running as root
+2. Check Ceph permissions: `ceph auth list`
+3. Check if Ceph user has proper permissions
+
+### Issue: OSD Creation Fails
+
+**Symptoms**: `ceph-volume` command fails
+
+**Solutions**:
+1. Check Ceph cluster status: `ceph health`
+2. Check if Ceph is running: `systemctl status ceph.target`
+3. Check logs: `journalctl -u ceph-osd@* -n 50`
+4. Try verbose mode: `ceph-volume lvm create --data /dev/sdX --verbose`
+
+### Issue: OSD Created But Not Healthy
+
+**Symptoms**: OSD shows up but health is still bad
+
+**Solutions**:
+1. Wait a few minutes for OSD to join cluster
+2. Check OSD status: `ceph osd tree`
+3. Check OSD details: `ceph osd df tree`
+4. Check for errors: `ceph health detail`
+
+---
+
+## Verification Checklist
+
+After creating the third OSD, verify:
+
+- [ ] OSD appears in `ceph osd tree` (should show 3 OSDs)
+- [ ] "TOO_FEW_OSDS" warning is gone
+- [ ] Ceph health improves (may still have other warnings)
+- [ ] OSD is "up" and "in" (not "down" or "out")
+- [ ] Replication factor can be set to 3
+- [ ] VMs can be created with ceph-fs storage
+
+---
+
+## Expected Results
+
+### Before (Current State)
+```
+HEALTH_WARN 2 OSDs; TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3
+```
+
+### After (With Third OSD)
+```
+HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)
+```
+
+### OSD Tree (After)
+```
+ID  CLASS  WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
+-1         1.5   root default
+-3         0.9       host ml110-01
+ 0   hdd   0.9           osd.0         up  1.00000  1.00000
+-5         0.6       host r630-01
+ 1   hdd   0.3           osd.1         up  1.00000  1.00000
+ 2   hdd   0.3           osd.2         up  1.00000  1.00000
+```
+
+---
+
+## Next Steps After Adding Third OSD
+
+1. **Monitor Ceph Health**: Watch for improvements
+2. **Fix Other Ceph Issues**:
+   - Reduce PG count (64 → 32)
+   - Fix undersized placement groups
+   - Resolve slow operations
+3. **Verify VM Creation**: Try creating a VM with ceph-fs storage
+4. **Consider Adding More OSDs**: Use remaining 5x 250GB drives for better performance
+
+---
+
+## Related Documentation
+
+- [Disk Inventory](../infrastructure/DISK_INVENTORY.md) - Complete disk inventory
+- [Critical Ceph Issues](./CRITICAL_CEPH_ISSUES.md) - All Ceph issues
+- [Blocking Issues Summary](../BLOCKING_ISSUES_SUMMARY.md) - Overall blocking issues
+
+---
+
+## Summary
+
+### Current State
+- 🔴 **2 OSDs** (insufficient for 3-way replication)
+- 🔴 **21 VMs blocked** (waiting for ceph-fs storage)
+
+### After Adding Third OSD
+- ✅ **3 OSDs** (sufficient for 3-way replication)
+- ✅ **VMs can be created** (ceph-fs storage available)
+- ✅ **Ceph health improves**
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔍 **READY TO IMPLEMENT - VERIFY 250GB DRIVES FIRST**
+