chore: consolidate local WIP (repo cleanup 20260707)

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 09:41:34 -07:00
parent 477cf73005
commit 33d50fb91e
277 changed files with 44421 additions and 75 deletions
--- a/docs/ceph/ADD_THIRD_OSD_GUIDE.md
+++ b/docs/ceph/ADD_THIRD_OSD_GUIDE.md
@@ -0,0 +1,284 @@
+# Guide: Adding Third Ceph OSD Using 250GB Drive on R630-01
+
+**Date**: 2025-12-13  
+**Status**: 🔍 **READY TO IMPLEMENT**
+
+---
+
+## Overview
+
+R630-01 has **6x 250GB drives** that are currently unused. We can use one of these drives to create a third Ceph OSD, which will:
+- ✅ Fix the "TOO_FEW_OSDS" error (need 3 for replication)
+- ✅ Allow 3-way replication to work properly
+- ✅ Unblock 21 VMs waiting for ceph-fs storage
+- ✅ Improve Ceph cluster health
+
+---
+
+## Prerequisites
+
+1. **SSH Access** to R630-01 (192.168.11.11)
+2. **Root privileges** on R630-01
+3. **Ceph cluster** already configured and running
+4. **One available 250GB drive** (out of 6 total)
+
+---
+
+## Step-by-Step Instructions
+
+### Step 1: Verify Disk Status
+
+SSH to R630-01 and run the verification script:
+
+```bash
+# Copy script to R630-01
+scp scripts/verify-r630-disks.sh root@192.168.11.11:/tmp/
+
+# SSH to R630-01
+ssh root@192.168.11.11
+
+# Run verification script
+bash /tmp/verify-r630-disks.sh
+```
+
+**OR** run commands manually:
+
+```bash
+# List all block devices
+lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL
+
+# Check all disks
+fdisk -l | grep -E "^Disk /dev/sd"
+
+# Check current Ceph OSDs
+ceph osd tree
+ceph health detail
+```
+
+### Step 2: Identify Available 250GB Drive
+
+Look for drives that are:
+- ✅ Around 232-260 GB in size (250GB drives)
+- ✅ Not mounted (no MOUNTPOINT)
+- ✅ Not used by Proxmox (not in pve-* volume groups)
+- ✅ Not already a Ceph OSD
+
+**Example output to look for**:
+```
+sdc    250G  disk        # Available!
+sdd    250G  disk        # Available!
+sde    250G  disk        # Available!
+...
+```
+
+### Step 3: Verify Drive is Safe to Use
+
+**CRITICAL**: Make sure the drive is not in use!
+
+```bash
+# Check if drive has partitions
+fdisk -l /dev/sdX  # Replace sdX with your drive
+
+# Check if drive is in a volume group
+pvs | grep sdX
+
+# Check if drive is mounted
+mount | grep sdX
+
+# Check if drive is used by Ceph
+ceph-volume lvm list | grep sdX
+```
+
+**The drive should show**:
+- No partitions (or only empty partition table)
+- Not in any volume group
+- Not mounted
+- Not used by Ceph
+
+### Step 4: Create Ceph OSD
+
+**WARNING**: This will destroy all data on the drive!
+
+```bash
+# Replace sdX with your actual drive (e.g., sdc, sdd, etc.)
+# Example: ceph-volume lvm create --data /dev/sdc
+
+ceph-volume lvm create --data /dev/sdX
+```
+
+**Expected output**:
+```
+Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid <uuid> --setuser ceph --setgroup ceph
+...
+--> ceph-volume lvm create successful for: /dev/sdc
+```
+
+### Step 5: Verify OSD is Created
+
+```bash
+# Check OSD tree (should show 3 OSDs now)
+ceph osd tree
+
+# Check OSD status
+ceph osd df
+
+# Check Ceph health (should improve)
+ceph health
+ceph health detail
+```
+
+**Expected improvements**:
+- ✅ "TOO_FEW_OSDS" warning should disappear
+- ✅ OSD count should be 3
+- ✅ Health should improve (may still have other warnings)
+
+### Step 6: Verify Replication Works
+
+```bash
+# Check pool replication settings
+ceph osd pool get rbd size
+ceph osd pool get rbd min_size
+
+# Should show size=3 (or 2 if temporarily reduced)
+```
+
+### Step 7: Monitor Cluster Health
+
+```bash
+# Watch Ceph health
+watch -n 5 'ceph health'
+
+# Check for any errors
+ceph health detail | grep -i error
+```
+
+---
+
+## Troubleshooting
+
+### Issue: Drive Not Detected
+
+**Symptoms**: Drive doesn't show up in `lsblk` or `fdisk -l`
+
+**Solutions**:
+1. Check if drive is physically installed
+2. Check PERC controller status (if using hardware RAID)
+3. Check if drive needs to be initialized in PERC
+4. Check system logs: `dmesg | grep -i sd`
+
+### Issue: Drive Already Has Data
+
+**Symptoms**: Drive shows partitions or filesystem
+
+**Solutions**:
+1. **If data is important**: Backup first!
+2. **If data can be destroyed**: Wipe the drive:
+   ```bash
+   # WARNING: Destroys all data!
+   wipefs -a /dev/sdX
+   dd if=/dev/zero of=/dev/sdX bs=1M count=100
+   ```
+
+### Issue: Permission Denied
+
+**Symptoms**: Cannot create OSD, permission errors
+
+**Solutions**:
+1. Make sure you're running as root
+2. Check Ceph permissions: `ceph auth list`
+3. Check if Ceph user has proper permissions
+
+### Issue: OSD Creation Fails
+
+**Symptoms**: `ceph-volume` command fails
+
+**Solutions**:
+1. Check Ceph cluster status: `ceph health`
+2. Check if Ceph is running: `systemctl status ceph.target`
+3. Check logs: `journalctl -u ceph-osd@* -n 50`
+4. Try verbose mode: `ceph-volume lvm create --data /dev/sdX --verbose`
+
+### Issue: OSD Created But Not Healthy
+
+**Symptoms**: OSD shows up but health is still bad
+
+**Solutions**:
+1. Wait a few minutes for OSD to join cluster
+2. Check OSD status: `ceph osd tree`
+3. Check OSD details: `ceph osd df tree`
+4. Check for errors: `ceph health detail`
+
+---
+
+## Verification Checklist
+
+After creating the third OSD, verify:
+
+- [ ] OSD appears in `ceph osd tree` (should show 3 OSDs)
+- [ ] "TOO_FEW_OSDS" warning is gone
+- [ ] Ceph health improves (may still have other warnings)
+- [ ] OSD is "up" and "in" (not "down" or "out")
+- [ ] Replication factor can be set to 3
+- [ ] VMs can be created with ceph-fs storage
+
+---
+
+## Expected Results
+
+### Before (Current State)
+```
+HEALTH_WARN 2 OSDs; TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3
+```
+
+### After (With Third OSD)
+```
+HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)
+```
+
+### OSD Tree (After)
+```
+ID  CLASS  WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
+-1         1.5   root default
+-3         0.9       host ml110-01
+ 0   hdd   0.9           osd.0         up  1.00000  1.00000
+-5         0.6       host r630-01
+ 1   hdd   0.3           osd.1         up  1.00000  1.00000
+ 2   hdd   0.3           osd.2         up  1.00000  1.00000
+```
+
+---
+
+## Next Steps After Adding Third OSD
+
+1. **Monitor Ceph Health**: Watch for improvements
+2. **Fix Other Ceph Issues**:
+   - Reduce PG count (64 → 32)
+   - Fix undersized placement groups
+   - Resolve slow operations
+3. **Verify VM Creation**: Try creating a VM with ceph-fs storage
+4. **Consider Adding More OSDs**: Use remaining 5x 250GB drives for better performance
+
+---
+
+## Related Documentation
+
+- [Disk Inventory](../infrastructure/DISK_INVENTORY.md) - Complete disk inventory
+- [Critical Ceph Issues](./CRITICAL_CEPH_ISSUES.md) - All Ceph issues
+- [Blocking Issues Summary](../BLOCKING_ISSUES_SUMMARY.md) - Overall blocking issues
+
+---
+
+## Summary
+
+### Current State
+- 🔴 **2 OSDs** (insufficient for 3-way replication)
+- 🔴 **21 VMs blocked** (waiting for ceph-fs storage)
+
+### After Adding Third OSD
+- ✅ **3 OSDs** (sufficient for 3-way replication)
+- ✅ **VMs can be created** (ceph-fs storage available)
+- ✅ **Ceph health improves**
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔍 **READY TO IMPLEMENT - VERIFY 250GB DRIVES FIRST**
+
--- a/docs/ceph/CEPH_CLUSTER_ISSUES.md
+++ b/docs/ceph/CEPH_CLUSTER_ISSUES.md
@@ -0,0 +1,132 @@
+# Ceph Cluster Instability - ML110-01
+
+**Date**: 2025-12-13  
+**Status**: ⚠️ **INVESTIGATING**
+
+---
+
+## Issue Summary
+
+### Symptoms
+- **Ceph Monitor**: Constantly in "electing" state
+- **Slow Operations**: 100+ slow operations reported
+- **Manager Instability**: Ceph manager constantly restarting
+- **Connection Issues**: Socket connections closing between mon0 and mon1
+- **Quorum Problems**: Cluster appears to be having election issues
+
+### Log Patterns
+```
+- mon.ml110-01@0(electing) e2 get_health_metrics reporting 100+ slow ops
+- ENGINE Bus STOPPING/STARTING (constant restarts)
+- libceph: mon0/mon1 socket closed (con state OPEN)
+- Session lost, hunting for new mon
+```
+
+---
+
+## Impact Analysis
+
+### VMs Affected
+- **Storage Type**: `ceph-fs`
+- **Count**: Multiple VMs configured to use ceph-fs
+- **Risk**: VM creation may fail if Ceph is unavailable
+
+### Potential Issues
+1. **VM Creation Failures**: VMs using ceph-fs may fail to create
+2. **Storage Unavailable**: Ceph storage may be inaccessible
+3. **Performance Degradation**: Slow operations indicate cluster stress
+4. **Data Risk**: Instability could affect data integrity
+
+---
+
+## Root Cause Analysis
+
+### Possible Causes
+1. **Network Issues**: Connection problems between mon0 (ML110-01) and mon1 (R630-01)
+2. **Quorum Loss**: Cluster may have lost quorum
+3. **Resource Exhaustion**: System resources may be exhausted
+4. **Configuration Issues**: Ceph configuration may be incorrect
+5. **Time Sync**: Clock synchronization issues
+
+---
+
+## Investigation Steps
+
+### 1. Check Ceph Cluster Health
+```bash
+ssh root@192.168.11.10 "ceph -s"
+ssh root@192.168.11.10 "ceph health detail"
+```
+
+### 2. Verify Quorum Status
+```bash
+ssh root@192.168.11.10 "ceph quorum_status"
+```
+
+### 3. Check Monitor Status
+```bash
+ssh root@192.168.11.10 "ceph mon stat"
+ssh root@192.168.11.10 "ceph mon dump"
+```
+
+### 4. Check Network Connectivity
+```bash
+# From ML110-01 to R630-01
+ssh root@192.168.11.10 "ping -c 3 192.168.11.11"
+ssh root@192.168.11.10 "telnet 192.168.11.11 6789"
+```
+
+### 5. Check System Resources
+```bash
+ssh root@192.168.11.10 "df -h"
+ssh root@192.168.11.10 "free -h"
+ssh root@192.168.11.10 "iostat -x 1 3"
+```
+
+---
+
+## Immediate Actions
+
+### 1. Verify Impact on VM Creation
+- Check if any VMs have failed due to storage issues
+- Monitor provider logs for Ceph-related errors
+- Verify if VM creation is blocked
+
+### 2. Stabilize Ceph Cluster
+- Check quorum status
+- Verify network connectivity
+- Restart Ceph services if needed
+- Check for configuration issues
+
+### 3. Alternative Storage
+- Consider using `local-lvm` for critical VMs temporarily
+- Document which VMs require ceph-fs
+- Plan migration if Ceph remains unstable
+
+---
+
+## Recommendations
+
+### Short-term
+1. **Monitor Ceph Health**: Continuously monitor cluster status
+2. **Check Network**: Verify connectivity between nodes
+3. **Review Logs**: Analyze Ceph logs for root cause
+4. **Resource Check**: Verify system resources are adequate
+
+### Long-term
+1. **Ceph Optimization**: Optimize Ceph configuration
+2. **Network Hardening**: Ensure stable network between nodes
+3. **Monitoring**: Set up Ceph health monitoring
+4. **Backup Strategy**: Ensure data backup strategy is in place
+
+---
+
+## Status
+
+⚠️ **INVESTIGATING** - Ceph cluster instability detected, impact on VM deployment being assessed.
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: ⚠️ **INVESTIGATING**
+
--- a/docs/ceph/COMPLETE_OSD_FIX.md
+++ b/docs/ceph/COMPLETE_OSD_FIX.md
@@ -0,0 +1,169 @@
+# Complete OSD Creation Fix
+
+**Date**: 2025-12-13  
+**Status**: ✅ **READY TO EXECUTE**
+
+---
+
+## Current Situation
+
+✅ **Drives wiped**: All 6 drives (sdc-sdh) are ready  
+❌ **OSD creation failed**: Bootstrap keyring issue  
+🎯 **Solution**: Fix bootstrap keyring and create OSDs
+
+---
+
+## Quick Fix - Run This Script
+
+On R630-01:
+
+```bash
+# Copy script
+scp scripts/fix-and-create-all-osds.sh root@192.168.11.11:/tmp/
+
+# SSH and run
+ssh root@192.168.11.11
+bash /tmp/fix-and-create-all-osds.sh
+```
+
+This script will:
+1. ✅ Check Ceph cluster connectivity
+2. ✅ Fix/bootstrap keyring issue
+3. ✅ Create OSDs on all 6 drives
+4. ✅ Verify results
+
+---
+
+## Manual Method
+
+If you prefer manual steps:
+
+### Step 1: Fix Bootstrap Keyring
+
+```bash
+# Create directory
+mkdir -p /var/lib/ceph/bootstrap-osd
+
+# Get bootstrap keyring from cluster
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# Verify
+ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
+```
+
+### Step 2: Create OSDs
+
+```bash
+# Method 1: Using ceph-volume
+for drive in sdc sdd sde sdf sdg sdh; do
+    ceph-volume lvm create --data /dev/$drive
+done
+
+# Method 2: Using pveceph (if available)
+for drive in sdc sdd sde sdf sdg sdh; do
+    pveceph create /dev/$drive
+done
+```
+
+### Step 3: Verify
+
+```bash
+# Check OSD tree
+ceph osd tree
+
+# Check health
+ceph health
+ceph health detail
+
+# Check OSD details
+ceph osd df
+```
+
+---
+
+## Expected Results
+
+### Before
+- 2 OSDs (insufficient)
+- HEALTH_WARN: TOO_FEW_OSDS
+
+### After (with all 6 OSDs)
+- **8 OSDs total** (2 existing + 6 new)
+- Excellent redundancy
+- HEALTH_OK or much improved
+- Can handle many VMs
+
+### After (with just 1 OSD)
+- **3 OSDs total** (minimum for 3-way replication)
+- Fixes TOO_FEW_OSDS error
+- HEALTH_OK or improved
+- VMs can be created
+
+---
+
+## Troubleshooting
+
+### Issue: "ceph auth get" fails
+
+**Solution**:
+```bash
+# Check if bootstrap-osd exists
+ceph auth list | grep bootstrap
+
+# If it doesn't exist, create it
+ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd'
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+```
+
+### Issue: "RADOS timed out"
+
+**Solution**:
+```bash
+# Check Ceph services
+systemctl status ceph.target
+systemctl status ceph-mon@*
+
+# Check monitor status
+ceph mon stat
+
+# Check network connectivity
+ping <monitor-ip>
+```
+
+### Issue: OSD creation still fails
+
+**Solution**:
+```bash
+# Check logs
+cat /tmp/ceph-osd-*.log
+
+# Try with verbose output
+ceph-volume lvm create --data /dev/sdc --verbose
+
+# Or try pveceph
+pveceph create /dev/sdc
+```
+
+---
+
+## Summary
+
+### Current State
+- ✅ 6 drives wiped and ready
+- ⚠️ Bootstrap keyring missing
+- 🎯 Need to fix and create OSDs
+
+### Action
+Run `fix-and-create-all-osds.sh` script
+
+### Expected Result
+- 8 OSDs total (if all succeed)
+- Or 3+ OSDs (minimum needed)
+- Ceph health improves
+- VMs can be created
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: ✅ **READY TO EXECUTE**
+
--- a/docs/ceph/CREATE_THIRD_OSD_NOW.md
+++ b/docs/ceph/CREATE_THIRD_OSD_NOW.md
@@ -0,0 +1,264 @@
+# Create Third Ceph OSD - Ready to Execute
+
+**Date**: 2025-12-13  
+**Status**: ✅ **READY - ALL DRIVES IDENTIFIED**
+
+---
+
+## Current Status
+
+✅ **6x 250GB drives found**: sdc, sdd, sde, sdf, sdg, sdh  
+✅ **All have partitions** (sdc3, sdd3, sde3, sdf3, sdg3, sdh3 in ubuntu-vg)  
+✅ **User confirmed**: 2 drives need formatting  
+🎯 **Ready to create third OSD**
+
+---
+
+## Quick Method: Interactive Script
+
+On R630-01, run the interactive script:
+
+```bash
+# Copy script to R630-01
+scp scripts/create-third-ceph-osd.sh root@192.168.11.11:/tmp/
+
+# SSH to R630-01
+ssh root@192.168.11.11
+
+# Run interactive script
+bash /tmp/create-third-ceph-osd.sh
+```
+
+The script will:
+1. Show available drives
+2. Ask you to select one
+3. Remove it from ubuntu-vg (if needed)
+4. Wipe the drive
+5. Create Ceph OSD
+6. Verify creation
+
+---
+
+## Manual Method: Step-by-Step
+
+### Step 1: Choose a Drive
+
+Available drives: **sdc, sdd, sde, sdf, sdg, sdh**
+
+**Recommendation**: Choose one that's safe to remove (you mentioned 2 need formatting - use one of those).
+
+### Step 2: Check if Drive is in ubuntu-vg
+
+```bash
+# Check which drives are in ubuntu-vg
+pvs | grep ubuntu-vg
+
+# Example output will show drives like:
+# /dev/sdc3  ubuntu-vg  lvm2  ...
+```
+
+### Step 3: Remove from ubuntu-vg (if needed)
+
+**WARNING**: This destroys data on that logical volume!
+
+```bash
+# Replace DRIVE with your choice (e.g., sdc, sdd, etc.)
+
+# 1. Unmount any mounted logical volumes
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+
+# 2. Deactivate volume group
+vgchange -a n ubuntu-vg
+
+# 3. Remove physical volume from ubuntu-vg
+pvremove /dev/DRIVE3 -y
+```
+
+### Step 4: Wipe the Drive
+
+**WARNING**: This destroys all data on the drive!
+
+```bash
+# Wipe filesystem signatures
+wipefs -a /dev/DRIVE
+
+# Zero out first 100MB (for clean slate)
+dd if=/dev/zero of=/dev/DRIVE bs=1M count=100
+```
+
+### Step 5: Create Ceph OSD
+
+```bash
+# Create Ceph OSD on the wiped drive
+ceph-volume lvm create --data /dev/DRIVE
+```
+
+**Expected output**:
+```
+Running command: /usr/bin/ceph-osd --cluster ceph ...
+--> ceph-volume lvm create successful for: /dev/DRIVE
+```
+
+### Step 6: Verify
+
+```bash
+# Check OSD tree (should show 3 OSDs now)
+ceph osd tree
+
+# Check Ceph health (should improve)
+ceph health
+ceph health detail
+
+# Check OSD details
+ceph osd df
+```
+
+---
+
+## Example: Using sdc
+
+Here's a complete example using `/dev/sdc`:
+
+```bash
+# 1. Check if in ubuntu-vg
+pvs | grep sdc
+
+# 2. If in ubuntu-vg, remove it
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+vgchange -a n ubuntu-vg
+pvremove /dev/sdc3 -y
+
+# 3. Wipe drive
+wipefs -a /dev/sdc
+dd if=/dev/zero of=/dev/sdc bs=1M count=100
+
+# 4. Create OSD
+ceph-volume lvm create --data /dev/sdc
+
+# 5. Verify
+ceph osd tree
+ceph health
+```
+
+---
+
+## Expected Results
+
+### Before
+```
+ID  CLASS  WEIGHT   TYPE NAME          STATUS
+-1         1.18250  root default
+-3         0.90970      host ml110-01
+ 0    hdd  0.90970          osd.0          up
+-5         0.27280      host r630-01
+ 1    hdd  0.27280          osd.1          up
+```
+
+### After
+```
+ID  CLASS  WEIGHT   TYPE NAME          STATUS
+-1         1.4xxx   root default
+-3         0.90970      host ml110-01
+ 0    hdd  0.90970          osd.0          up
+-5         0.6xxx       host r630-01
+ 1    hdd  0.27280          osd.1          up
+ 2    hdd  0.2xxx           osd.2          up  ← NEW!
+```
+
+### Ceph Health
+
+**Before**:
+```
+HEALTH_WARN ... OSD count 2 < osd_pool_default_size 3
+```
+
+**After**:
+```
+HEALTH_OK (or HEALTH_WARN with other issues, but TOO_FEW_OSDS should be gone)
+```
+
+---
+
+## Troubleshooting
+
+### Issue: pvremove fails
+
+**Error**: "Can't open /dev/DRIVE3 exclusively. Mounted filesystem?"
+
+**Solution**:
+```bash
+# Make sure nothing is mounted
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+vgchange -a n ubuntu-vg
+
+# Force remove
+pvremove /dev/DRIVE3 -y -ff
+```
+
+### Issue: ceph-volume fails
+
+**Error**: "Device /dev/DRIVE has partitions"
+
+**Solution**:
+```bash
+# Make sure drive is completely wiped
+wipefs -a /dev/DRIVE
+dd if=/dev/zero of=/dev/DRIVE bs=1M count=100
+
+# Try again
+ceph-volume lvm create --data /dev/DRIVE
+```
+
+### Issue: OSD created but not healthy
+
+**Solution**:
+```bash
+# Wait a few minutes for OSD to join cluster
+sleep 60
+
+# Check status
+ceph osd tree
+ceph health detail
+
+# Check OSD logs
+journalctl -u ceph-osd@2 -n 50
+```
+
+---
+
+## Safety Checklist
+
+Before proceeding:
+
+- [ ] **Backup important data** (if ubuntu-vg contains data)
+- [ ] **Verify ubuntu-vg is not critical** for system operation
+- [ ] **Choose drive to remove** (sdc, sdd, sde, sdf, sdg, or sdh)
+- [ ] **Confirm drive selection** (double-check!)
+- [ ] **Have recovery plan** if something goes wrong
+
+---
+
+## Summary
+
+### Current State
+- ✅ 6x 250GB drives identified (sdc-sdh)
+- ✅ All in ubuntu-vg volume group
+- ✅ Ready to free one drive for Ceph OSD
+
+### Action Required
+1. Choose one drive (sdc-sdh)
+2. Remove from ubuntu-vg (if needed)
+3. Wipe drive
+4. Create Ceph OSD
+5. Verify health improves
+
+### Expected Result
+- ✅ 3 OSDs total (fixes TOO_FEW_OSDS error)
+- ✅ Ceph health improves
+- ✅ 21 VMs can be created with ceph-fs storage
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: ✅ **READY TO EXECUTE**
+
--- a/docs/ceph/CRITICAL_CEPH_ISSUES.md
+++ b/docs/ceph/CRITICAL_CEPH_ISSUES.md
@@ -0,0 +1,265 @@
+# Critical Ceph Storage Issues
+
+**Date**: 2025-12-13  
+**Status**: 🔴 **CRITICAL - BLOCKING VM CREATION**
+
+---
+
+## Executive Summary
+
+The Ceph storage cluster has **5 critical health warnings** that are likely **blocking VM creation** for 21 VMs configured to use `ceph-fs` storage.
+
+### Impact
+- **21 VMs** are configured to use `ceph-fs` storage
+- **0 VMs created** - All VMs are stuck in pending state
+- **Ceph cluster unhealthy** - Multiple critical warnings
+
+---
+
+## Critical Issues Identified
+
+### 1. 🔴 TOO_FEW_OSDS (CRITICAL)
+- **Issue**: OSD count 2 < osd_pool_default_size 3
+- **Impact**: Cannot maintain 3-way replication
+- **Risk**: Data loss if an OSD fails
+- **Status**: 🔴 **CRITICAL**
+
+### 2. 🔴 UNDERSIZED PLACEMENT GROUPS (CRITICAL)
+- **Issue**: PG 3.7f stuck undersized for 32 hours
+- **Impact**: Data not fully replicated
+- **Risk**: Data loss if storage fails
+- **Status**: 🔴 **CRITICAL**
+
+### 3. 🔴 TOO_MANY_PGS (HIGH)
+- **Issue**: 288 PGs per OSD (max is 250)
+- **Impact**: Performance degradation, slow operations
+- **Risk**: Cluster instability
+- **Status**: 🟠 **HIGH**
+
+### 4. 🟠 POOL_TOO_MANY_PGS (MEDIUM)
+- **Issue**: RBD pool has 64 PGs, should have 32
+- **Impact**: Suboptimal performance
+- **Risk**: Slower I/O operations
+- **Status**: 🟡 **MEDIUM**
+
+### 5. 🟠 SLOW_OPS (MEDIUM)
+- **Issue**: 98 slow operations, oldest blocked for 100 seconds
+- **Impact**: Cluster operations delayed
+- **Risk**: Timeouts, failed operations
+- **Status**: 🟡 **MEDIUM**
+
+---
+
+## Affected VMs
+
+### VMs Using ceph-fs Storage (21 total)
+
+#### Phoenix Infrastructure (7 VMs)
+- `phoenix-git-server`
+- `phoenix-email-server`
+- `phoenix-devops-runner`
+- `phoenix-codespaces-ide`
+- `phoenix-financial-messaging-gateway`
+- `phoenix-business-integration-gateway`
+- `phoenix-as4-gateway`
+
+#### SMOM/DBIS-138 Infrastructure (14 VMs)
+- `management`
+- `monitoring`
+- `blockscout`
+- `services`
+- `rpc-node-01` through `rpc-node-04` (4 VMs)
+- `validator-01` through `validator-04` (4 VMs)
+- `sentry-03`, `sentry-04` (2 VMs)
+
+### VMs Using local-lvm Storage (9 VMs)
+- `basic-vm-001`
+- `cloudflare-tunnel-vm`
+- `large-vm-001`
+- `medium-vm-001`
+- `nginx-proxy-vm`
+- `phoenix-dns-primary`
+- (3 more test VMs)
+
+**Note**: VMs using `local-lvm` should not be affected by Ceph issues.
+
+---
+
+## Root Cause Analysis
+
+### Primary Issue: Insufficient OSDs
+- **Current**: 2 OSDs
+- **Required**: 3 OSDs (for 3-way replication)
+- **Impact**: Cannot maintain data redundancy
+
+### Secondary Issues
+1. **PG Over-allocation**: Too many placement groups for the number of OSDs
+2. **Performance Degradation**: Slow operations due to resource constraints
+3. **Data Risk**: Undersized PGs mean incomplete replication
+
+---
+
+## Immediate Actions Required
+
+### 🔴 URGENT (Blocking VM Creation)
+
+1. **Add Third OSD** (if hardware available)
+   ```bash
+   # On Proxmox node, add new OSD
+   # This requires available disk/SSD
+   ```
+
+2. **Reduce Replication Factor** (temporary workaround)
+   ```bash
+   # Reduce from 3 to 2 replicas (increases risk)
+   ceph osd pool set <pool-name> size 2
+   ceph osd pool set <pool-name> min_size 1
+   ```
+
+3. **Reduce PG Count** (performance fix)
+   ```bash
+   # Reduce RBD pool PGs from 64 to 32
+   ceph osd pool set rbd pg_num 32
+   ceph osd pool set rbd pgp_num 32
+   ```
+
+### 🟠 HIGH PRIORITY (Performance)
+
+4. **Fix Undersized PG**
+   ```bash
+   # Check PG status
+   ceph pg ls-by-pool rbd
+   # Force recovery if needed
+   ceph pg force-recovery <pg-id>
+   ```
+
+5. **Monitor Slow Operations**
+   ```bash
+   # Check slow ops
+   ceph health detail
+   # Identify blocking operations
+   ```
+
+---
+
+## Recommended Solutions
+
+### Option 1: Add Third OSD (BEST)
+- **Action**: Add a third OSD node or disk
+- **Impact**: Resolves all replication issues
+- **Risk**: Low (if hardware available)
+- **Timeline**: Immediate if hardware ready
+
+### Option 2: Reduce Replication (TEMPORARY)
+- **Action**: Change replication from 3 to 2
+- **Impact**: Allows VM creation but increases risk
+- **Risk**: Medium (data loss if 1 OSD fails)
+- **Timeline**: Immediate
+
+### Option 3: Use local-lvm for Critical VMs (WORKAROUND)
+- **Action**: Change critical VMs to use local-lvm
+- **Impact**: Allows deployment but loses shared storage benefits
+- **Risk**: Low (no data redundancy issues)
+- **Timeline**: Requires VM config changes
+
+---
+
+## Verification Steps
+
+### Check Ceph Health
+```bash
+# SSH to Proxmox node
+ssh root@ml110-01
+
+# Check cluster health
+ceph health
+ceph health detail
+
+# Check OSD status
+ceph osd tree
+ceph osd df
+
+# Check pool status
+ceph osd pool ls detail
+
+# Check PG status
+ceph pg ls-by-pool rbd
+```
+
+### Check VM Creation Status
+```bash
+# Check VMs using ceph-fs
+kubectl get proxmoxvm -A -o json | \
+  jq -r '.items[] | select(.spec.forProvider.storage == "ceph-fs") | .metadata.name'
+
+# Check for storage-related errors
+kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox | \
+  grep -i "ceph\|storage\|rbd"
+```
+
+---
+
+## Impact Assessment
+
+### Current Impact
+- ✅ **VMs using local-lvm**: Can be created (9 VMs)
+- ❌ **VMs using ceph-fs**: Blocked (21 VMs)
+- ❌ **Total Progress**: 0/30 VMs (0%)
+
+### Expected Impact After Fix
+- ✅ **All VMs**: Can be created
+- ✅ **Storage**: Shared storage available
+- ✅ **Data Safety**: Proper replication
+
+---
+
+## Next Steps
+
+### Immediate (Today)
+1. 🔴 **URGENT**: Fix OSD count or reduce replication
+2. 🔴 **URGENT**: Reduce PG count in RBD pool
+3. 🟠 **HIGH**: Fix undersized PG
+4. 🟡 **MEDIUM**: Monitor slow operations
+
+### Short-term (This Week)
+1. Add third OSD if hardware available
+2. Optimize PG distribution
+3. Monitor cluster health
+4. Verify VM creation resumes
+
+### Long-term (This Month)
+1. Plan for proper 3-node Ceph cluster
+2. Optimize storage configuration
+3. Set up monitoring and alerts
+4. Document storage architecture
+
+---
+
+## References
+
+### Ceph Documentation
+- [Ceph Placement Groups](https://docs.ceph.com/en/latest/rados/operations/placement-groups/)
+- [Ceph Pool Configuration](https://docs.ceph.com/en/latest/rados/operations/pools/)
+- [Ceph OSD Management](https://docs.ceph.com/en/latest/rados/operations/operating/)
+
+### Related Issues
+- VM creation blocked (0/30 VMs created)
+- Ceph cluster health warnings
+- Storage configuration issues
+
+---
+
+## Summary
+
+### Current State
+- 🔴 **Ceph Cluster**: Unhealthy (5 critical warnings)
+- 🔴 **VM Creation**: Blocked (0/30 VMs)
+- 🔴 **Storage**: Insufficient OSDs (2 < 3 required)
+- 🟠 **Performance**: Degraded (slow operations)
+
+### Status
+🔴 **CRITICAL - IMMEDIATE ACTION REQUIRED**
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔴 **BLOCKING VM CREATION**
+
--- a/docs/ceph/DRIVES_VISIBLE_STATUS.md
+++ b/docs/ceph/DRIVES_VISIBLE_STATUS.md
@@ -0,0 +1,101 @@
+# 250GB SSDs Status - Visible and Ready
+
+**Date**: 2025-12-13  
+**Status**: ✅ **ALL 6 DRIVES VISIBLE AND READY**
+
+---
+
+## Current Status
+
+### ✅ All 6x 250GB SSDs Detected
+
+| Drive | Size | Model | Partition | Status |
+|-------|------|-------|-----------|--------|
+| **sdc** | 232.9G | CT250MX500SSD1 | sdc1 (232.9G) | ✅ Ready |
+| **sdd** | 232.9G | CT250MX500SSD1 | sdd1 (232.9G) | ✅ Ready |
+| **sde** | 232.9G | CT250MX500SSD1 | sde1 (232.9G) | ✅ Ready |
+| **sdf** | 232.9G | CT250MX500SSD1 | sdf1 (232.9G) | ✅ Ready |
+| **sdg** | 232.9G | CT250MX500SSD1 | sdg1 (232.9G) | ✅ Ready |
+| **sdh** | 232.9G | CT250MX500SSD1 | sdh1 (232.9G) | ✅ Ready |
+
+### Key Observations
+
+- ✅ **All drives visible** to the system
+- ✅ **Partitions created** (sdc1, sdd1, etc.)
+- ✅ **No filesystem** (FSTYPE empty) - clean and ready
+- ✅ **Not mounted** - available for use
+- ✅ **Proxmox can see them** - ready for storage configuration
+
+---
+
+## Proxmox Storage Status
+
+Current storage pools:
+- **ceph-fs**: inactive (mount error - needs fixing)
+- **ceph-rbd**: inactive
+- **local**: active (81.5 GB)
+- **local-lvm**: active (179.6 GB)
+
+---
+
+## Next Steps
+
+### Option 1: Create Ceph OSDs (Recommended)
+
+Use these drives for Ceph OSDs to fix the TOO_FEW_OSDS error:
+
+```bash
+# Remove partitions first (Ceph needs raw devices)
+for drive in sdc sdd sde sdf sdg sdh; do
+    wipefs -a /dev/$drive
+    ceph-volume lvm create --data /dev/$drive
+done
+```
+
+### Option 2: Add as Proxmox Storage
+
+Add them as Directory or LVM storage in Proxmox:
+
+1. **Via Web UI**: Datacenter > Storage > Add
+2. **Via CLI**: Use `pvesm` commands
+
+### Option 3: Use for Ceph OSD (One at a Time)
+
+Start with one drive to fix the immediate issue:
+
+```bash
+# Use sdc for third OSD
+wipefs -a /dev/sdc
+ceph-volume lvm create --data /dev/sdc
+
+# Verify
+ceph osd tree
+ceph health
+```
+
+---
+
+## Summary
+
+### Current State
+- ✅ 6x 250GB SSDs formatted and visible
+- ✅ Partitions created (ready for use)
+- ✅ No filesystem (clean slate)
+- ✅ Proxmox can see them
+
+### Ready For
+- ✅ Ceph OSD creation
+- ✅ Proxmox storage configuration
+- ✅ Any storage use case
+
+### Next Action
+Choose how to use the drives:
+1. **Ceph OSDs** (fixes TOO_FEW_OSDS error)
+2. **Proxmox storage** (for VMs)
+3. **Both** (some for Ceph, some for storage)
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: ✅ **DRIVES READY - CHOOSE USE CASE**
+
--- a/docs/ceph/DRIVE_VERIFICATION_RESULTS.md
+++ b/docs/ceph/DRIVE_VERIFICATION_RESULTS.md
@@ -0,0 +1,168 @@
+# Drive Verification Results - R630-01
+
+**Date**: 2025-12-13  
+**Status**: 🔍 **6x 250GB DRIVES NOT DETECTED BY OS**
+
+---
+
+## Verification Results
+
+### Disks Detected by OS
+- **sda**: 279.4 GB (300GB) - Proxmox installation (LVM)
+- **sdb**: 279.4 GB (300GB) - Ceph OSD
+
+### Expected Disks
+- **6x 250GB drives** - **NOT DETECTED** ❌
+
+---
+
+## Analysis
+
+The 6x 250GB drives are **not visible to the operating system**. This means they are likely:
+
+1. **In PERC Controller but Unconfigured** (Most Likely)
+   - Drives are physically installed
+   - Detected by PERC controller
+   - Not configured as virtual disks
+   - Not exposed to OS
+
+2. **In RAID Array**
+   - Drives are part of a RAID virtual disk
+   - Not exposed as individual disks
+   - Need to reconfigure to expose individually
+
+3. **Not Physically Installed** (Less Likely)
+   - User confirmed 8 total bays with drives
+   - But OS only sees 2 drives
+   - May need physical verification
+
+---
+
+## Current Ceph Status
+
+- **OSD Count**: 2 (insufficient for 3-way replication)
+- **Health**: HEALTH_WARN with multiple issues:
+  - TOO_FEW_OSDS: OSD count 2 < 3 required
+  - Too many PGs per OSD (288 > max 250)
+  - Degraded data redundancy
+  - Slow operations
+
+---
+
+## Next Steps
+
+### Immediate: Check PERC Controller
+
+The drives are likely in the PERC controller but not exposed. Check:
+
+1. **Install Dell OpenManage** (if not installed):
+   ```bash
+   apt-get install -y srvadmin-all
+   systemctl start dsm_sa_datamgrd
+   ```
+
+2. **Check Physical Disks in PERC**:
+   ```bash
+   omreport storage pdisk controller=0
+   ```
+
+3. **Check Virtual Disks**:
+   ```bash
+   omreport storage vdisk controller=0
+   ```
+
+4. **Check Controller Status**:
+   ```bash
+   omreport storage controller
+   ```
+
+### If Drives Found in PERC
+
+**Option A: Configure as Individual Virtual Disks** (Recommended for Ceph)
+- Create one virtual disk per physical disk
+- Expose each 250GB drive individually
+- Use for Ceph OSDs
+
+**Option B: Use PERC HBA Mode** (If Supported)
+- Configure PERC in HBA (Host Bus Adapter) mode
+- Passes drives directly to OS
+- Best for Ceph (no RAID overhead)
+
+### If Drives Not Found in PERC
+
+1. **Physical Verification**:
+   - Check drive bay LEDs
+   - Verify drives are seated properly
+   - Check backplane connections
+
+2. **Controller Issues**:
+   - Check PERC controller status
+   - May need firmware update
+   - May need controller replacement
+
+---
+
+## Commands to Run on R630-01
+
+```bash
+# 1. Check for PERC controller
+lspci | grep -iE "raid|perc|lsi|megaraid"
+
+# 2. Check kernel messages
+dmesg | grep -iE "sd[a-z]|disk|perc" | tail -20
+
+# 3. Check SCSI devices
+cat /proc/scsi/scsi
+
+# 4. Check if OpenManage available
+which omreport
+which perccli
+
+# 5. If OpenManage installed
+omreport storage pdisk controller=0
+omreport storage vdisk controller=0
+```
+
+---
+
+## Expected PERC Output
+
+If drives are in PERC, you should see:
+
+```
+Physical Disk Information
+-------------------------
+ID    Status    State    Name    Size
+0:0:0 Online    Ready    ...     300 GB
+0:0:1 Online    Ready    ...     300 GB
+0:0:2 Online    Ready    ...     250 GB  ← Should see 6 of these
+0:0:3 Online    Ready    ...     250 GB
+0:0:4 Online    Ready    ...     250 GB
+0:0:5 Online    Ready    ...     250 GB
+0:0:6 Online    Ready    ...     250 GB
+0:0:7 Online    Ready    ...     250 GB
+```
+
+---
+
+## Summary
+
+### Current State
+- ✅ 2x 300GB drives detected and in use
+- ❌ 6x 250GB drives **NOT detected by OS**
+- 🔍 Need to check PERC controller configuration
+
+### Action Required
+1. **Check PERC controller** for 6x 250GB drives
+2. **Configure drives** to expose to OS (if found)
+3. **Create Ceph OSD** on one of the 250GB drives
+4. **Fix Ceph health** issues
+
+### Priority
+🔴 **URGENT** - Need to expose 250GB drives to create third OSD
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔍 **CHECK PERC CONTROLLER FOR 250GB DRIVES**
+
--- a/docs/ceph/FIX_BOOTSTRAP_KEYRING.md
+++ b/docs/ceph/FIX_BOOTSTRAP_KEYRING.md
@@ -0,0 +1,187 @@
+# Fix Ceph Bootstrap Keyring Issue
+
+**Date**: 2025-12-13  
+**Issue**: OSD creation failing due to missing bootstrap keyring
+
+---
+
+## Problem
+
+When creating OSDs, getting error:
+```
+unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
+RADOS timed out (error connecting to the cluster)
+RuntimeError: Unable to create a new OSD id
+```
+
+---
+
+## Solution Options
+
+### Option 1: Get Bootstrap Keyring from Cluster
+
+```bash
+# Create directory if needed
+mkdir -p /var/lib/ceph/bootstrap-osd
+
+# Get bootstrap keyring from Ceph cluster
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+```
+
+### Option 2: Use Proxmox pveceph (If Available)
+
+Proxmox has its own Ceph management tool:
+
+```bash
+# Check if available
+which pveceph
+
+# Create OSD using pveceph
+pveceph create /dev/sdc
+pveceph create /dev/sdd
+# ... etc
+```
+
+### Option 3: Check Ceph Cluster Connectivity
+
+The timeout suggests the cluster might not be accessible:
+
+```bash
+# Check Ceph health
+ceph health
+
+# Check monitors
+ceph mon stat
+
+# Check if Ceph services are running
+systemctl status ceph.target
+systemctl status ceph-mon@*
+systemctl status ceph-osd@*
+```
+
+### Option 4: Create Bootstrap Keyring Manually
+
+If the cluster is accessible but keyring is missing:
+
+```bash
+# Check existing auth
+ceph auth list
+
+# If bootstrap-osd exists, export it
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# If it doesn't exist, create it
+ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring
+```
+
+---
+
+## Quick Fix Script
+
+Run this on R630-01:
+
+```bash
+# 1. Check Ceph connectivity
+ceph health
+ceph mon stat
+
+# 2. Get or create bootstrap keyring
+mkdir -p /var/lib/ceph/bootstrap-osd
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring 2>/dev/null || \
+ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# 3. Verify keyring exists
+ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# 4. Try creating OSD again
+ceph-volume lvm create --data /dev/sdc
+```
+
+---
+
+## Alternative: Use pveceph
+
+If you're using Proxmox's Ceph integration:
+
+```bash
+# Create OSDs using Proxmox tool
+pveceph create /dev/sdc
+pveceph create /dev/sdd
+pveceph create /dev/sde
+pveceph create /dev/sdf
+pveceph create /dev/sdg
+pveceph create /dev/sdh
+```
+
+---
+
+## Troubleshooting
+
+### Issue: "RADOS timed out"
+
+**Cause**: Cannot connect to Ceph cluster
+
+**Solution**:
+```bash
+# Check if monitors are running
+systemctl status ceph-mon@*
+
+# Check monitor addresses
+ceph mon dump
+
+# Check network connectivity
+ping <monitor-ip>
+```
+
+### Issue: "no keyring found"
+
+**Cause**: Bootstrap keyring missing
+
+**Solution**:
+```bash
+# Get from cluster
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# Or create new one
+ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd'
+```
+
+### Issue: "ceph command not found"
+
+**Cause**: Ceph tools not installed
+
+**Solution**:
+```bash
+# Install Ceph tools (if needed)
+apt-get update
+apt-get install -y ceph-common
+```
+
+---
+
+## Next Steps
+
+After fixing the bootstrap keyring:
+
+1. **Verify keyring exists**:
+   ```bash
+   ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
+   ```
+
+2. **Create OSDs**:
+   ```bash
+   for drive in sdc sdd sde sdf sdg sdh; do
+       ceph-volume lvm create --data /dev/$drive
+   done
+   ```
+
+3. **Verify**:
+   ```bash
+   ceph osd tree
+   ceph health
+   ```
+
+---
+
+**Last Updated**: 2025-12-13
+
--- a/docs/ceph/FIX_R630_QUORUM.md
+++ b/docs/ceph/FIX_R630_QUORUM.md
@@ -0,0 +1,168 @@
+# Fix R630-01 Monitor Quorum Issue
+
+## Problem
+R630-01 monitor is not joining the quorum. Only ml110-01 is in quorum.
+
+## Diagnosis
+
+From your terminal output:
+- Quorum only shows: `["ml110-01"]`
+- Monmap only has: `ml110-01` monitor
+- R630-01 monitor is not in the cluster
+
+## Solution Steps
+
+### Step 1: Check R630-01 Monitor Status
+
+On **r630-01**:
+
+```bash
+systemctl status ceph-mon@r630-01
+journalctl -u ceph-mon@r630-01 -n 50
+ls -la /var/lib/ceph/mon/ceph-r630-01/
+```
+
+### Step 2: Check if Monitor Directory Exists
+
+```bash
+# If directory is missing or empty, create it
+ls -la /var/lib/ceph/mon/ceph-r630-01/
+```
+
+### Step 3: Add R630-01 to Monmap
+
+On **ml110-01** (where quorum exists):
+
+```bash
+# Get current monmap
+ceph mon getmap -o /tmp/monmap
+
+# Check if r630-01 is in monmap
+monmaptool --print /tmp/monmap | grep r630-01
+
+# If not present, add it manually
+# First get FSID
+FSID=$(grep fsid /etc/pve/ceph.conf | awk '{print $3}')
+
+# Add r630-01 to monmap
+monmaptool --add r630-01 192.168.11.11 --fsid $FSID /tmp/monmap
+
+# Inject updated monmap
+ceph mon setmap -i /tmp/monmap
+```
+
+### Step 4: Create Monitor on R630-01
+
+On **r630-01**:
+
+```bash
+# Option 1: Use pveceph (recommended)
+pveceph mon create
+
+# Option 2: Manual creation if pveceph fails
+# Get monmap from ml110-01
+scp root@192.168.11.10:/tmp/monmap /tmp/
+
+# Create monitor directory
+mkdir -p /var/lib/ceph/mon/ceph-r630-01
+chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
+
+# Create keyring
+ceph-authtool --create-keyring /var/lib/ceph/mon/ceph-r630-01/keyring --gen-key -n mon.
+
+# Create monitor filesystem
+ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /var/lib/ceph/mon/ceph-r630-01/keyring
+
+# Set ownership
+chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
+```
+
+### Step 5: Start Monitor Service
+
+On **r630-01**:
+
+```bash
+systemctl enable ceph-mon@r630-01
+systemctl start ceph-mon@r630-01
+systemctl status ceph-mon@r630-01
+```
+
+### Step 6: Verify Quorum
+
+On **either node**:
+
+```bash
+ceph quorum_status
+ceph mon stat
+ceph -s
+```
+
+You should see both monitors in quorum:
+```json
+"quorum": [0, 1],
+"quorum_names": ["ml110-01", "r630-01"]
+```
+
+## Alternative: Recreate Monitor
+
+If the above doesn't work, recreate the monitor:
+
+On **r630-01**:
+
+```bash
+# Stop monitor
+systemctl stop ceph-mon@r630-01
+systemctl disable ceph-mon@r630-01
+
+# Remove monitor directory
+rm -rf /var/lib/ceph/mon/ceph-r630-01
+
+# Recreate
+pveceph mon create
+
+# Start
+systemctl enable ceph-mon@r630-01
+systemctl start ceph-mon@r630-01
+```
+
+## Troubleshooting
+
+### Monitor Service Fails to Start
+
+Check logs:
+```bash
+journalctl -u ceph-mon@r630-01 -n 100
+```
+
+Common issues:
+- **"monitor data directory does not exist"**: Run `pveceph mon create`
+- **"unable to bind"**: Check firewall, ports 6789 and 3300
+- **"authentication"**: Check keyring permissions
+
+### Monitor Not in Quorum
+
+1. Verify network connectivity:
+   ```bash
+   ping 192.168.11.10
+   ping 192.168.11.11
+   ```
+
+2. Check firewall:
+   ```bash
+   ufw allow 6789/tcp
+   ufw allow 3300/tcp
+   ```
+
+3. Verify monmap includes both monitors:
+   ```bash
+   ceph mon dump
+   ```
+
+### Still Not Working
+
+If monitor still won't join:
+1. Check if both nodes are in Proxmox cluster: `pvecm status`
+2. Verify `/etc/pve/ceph.conf` is identical on both nodes
+3. Check FSID matches on both nodes
+4. Ensure time is synchronized: `chrony sources`
+
--- a/docs/ceph/FORMAT_SSDS_FOR_PROXMOX.md
+++ b/docs/ceph/FORMAT_SSDS_FOR_PROXMOX.md
@@ -0,0 +1,146 @@
+# Format 250GB SSDs and Check Proxmox Recognition
+
+**Date**: 2025-12-13  
+**Purpose**: Format the 6x 250GB SSDs and verify Proxmox can see them
+
+---
+
+## Quick Start
+
+On R630-01:
+
+```bash
+# Copy script
+scp scripts/format-ssds-and-check-proxmox.sh root@192.168.11.11:/tmp/
+
+# SSH and run
+ssh root@192.168.11.11
+bash /tmp/format-ssds-and-check-proxmox.sh
+```
+
+---
+
+## What the Script Does
+
+1. **Checks current drive status** - Shows what's on each drive
+2. **Formats drives** - Creates GPT partition table on each
+3. **Creates partitions** - Single partition on each drive (unformatted)
+4. **Checks block devices** - Shows what the system sees
+5. **Checks Proxmox storage** - Shows Proxmox's view
+6. **Provides summary** - What to check next
+
+---
+
+## Manual Method
+
+If you prefer to do it manually:
+
+```bash
+# For each drive (sdc, sdd, sde, sdf, sdg, sdh)
+for drive in sdc sdd sde sdf sdg sdh; do
+    # Unmount any partitions
+    umount /dev/${drive}* 2>/dev/null || true
+    
+    # Create GPT partition table
+    parted -s /dev/$drive mklabel gpt
+    
+    # Create single partition
+    parted -s /dev/$drive mkpart primary 0% 100%
+    
+    # Re-read partition table
+    partprobe /dev/$drive
+done
+
+# Check results
+lsblk
+pvesm status
+```
+
+---
+
+## Check in Proxmox Web UI
+
+1. **Login**: https://192.168.11.11:8006
+2. **Navigate**: Datacenter > Storage
+3. **Click**: "Add" button
+4. **Check**: Available disks should be listed
+
+---
+
+## Expected Results
+
+### After Formatting
+
+Each drive should show:
+- GPT partition table
+- One partition (unformatted)
+- Visible to `lsblk`
+- Potentially visible to Proxmox storage manager
+
+### What Proxmox Should See
+
+- 6x drives around 250GB
+- Available for storage creation
+- Can be used for:
+  - Directory storage
+  - LVM storage
+  - Ceph OSD (after further setup)
+
+---
+
+## Next Steps After Formatting
+
+Once Proxmox recognizes the drives:
+
+1. **Option 1: Use for Ceph OSDs**
+   - Create OSDs on formatted drives
+   - Add to Ceph cluster
+
+2. **Option 2: Use as Proxmox Storage**
+   - Add as Directory storage
+   - Add as LVM storage
+   - Use for VM disks
+
+3. **Option 3: Leave for Later**
+   - Drives are ready
+   - Can be configured when needed
+
+---
+
+## Verification Commands
+
+```bash
+# Check block devices
+lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE
+
+# Check Proxmox storage
+pvesm status
+
+# Check partitions
+fdisk -l | grep -E "^Disk /dev/sd[c-h]"
+
+# Check if Proxmox can see them
+pvesm scan local
+```
+
+---
+
+## Summary
+
+### Current State
+- 6x 250GB SSDs wiped
+- Need to format and check Proxmox recognition
+
+### Action
+- Format drives with GPT partition tables
+- Check if Proxmox recognizes them
+
+### Expected Result
+- Drives formatted and visible
+- Proxmox can see them
+- Ready for storage configuration
+
+---
+
+**Last Updated**: 2025-12-13
+
--- a/docs/ceph/FREE_DRIVE_FROM_UBUNTU_VG.md
+++ b/docs/ceph/FREE_DRIVE_FROM_UBUNTU_VG.md
@@ -0,0 +1,199 @@
+# Free Drive from ubuntu-vg for Ceph OSD
+
+**Date**: 2025-12-13  
+**Status**: 🔍 **250GB DRIVES FOUND IN ubuntu-vg VOLUME GROUP**
+
+---
+
+## Current Situation
+
+The verification revealed:
+- ✅ **6x 250GB drives ARE detected** (sdc, sde, sdh, and others)
+- ⚠️ **They're in `ubuntu-vg` volume group** (100% allocated)
+- ✅ **User confirmed**: 2 drives need to be formatted
+- 🎯 **Goal**: Free up one drive for third Ceph OSD
+
+---
+
+## Step-by-Step: Free Drive from ubuntu-vg
+
+### Step 1: Check Current Status
+
+Run the check script on R630-01:
+
+```bash
+bash /tmp/check-ubuntu-vg-and-prepare-osd.sh
+```
+
+Or manually:
+
+```bash
+# Check ubuntu-vg status
+vgs ubuntu-vg
+lvs ubuntu-vg
+pvs | grep ubuntu-vg
+
+# Check what's mounted
+mount | grep ubuntu-vg
+df -h | grep ubuntu-vg
+```
+
+### Step 2: Identify Drive to Free
+
+From the screenshot, drives in ubuntu-vg include:
+- `/dev/sdc3` (247.90 GB)
+- `/dev/sde3` (247.90 GB)
+- `/dev/sdh3` (247.90 GB)
+- At least one more (partially visible)
+
+**Choose one drive** that's safe to remove (preferably one that's not critical).
+
+### Step 3: Unmount and Remove from ubuntu-vg
+
+**WARNING**: This will destroy data on the logical volume!
+
+```bash
+# 1. Unmount any mounted logical volumes (if any)
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+
+# 2. Deactivate the volume group
+vgchange -a n ubuntu-vg
+
+# 3. Remove the physical volume from ubuntu-vg
+# Replace sdX with your chosen drive (e.g., sdc, sde, sdh)
+pvremove /dev/sdX3
+
+# 4. If you want to remove the entire ubuntu-vg (if not needed):
+# vgremove ubuntu-vg
+```
+
+### Step 4: Wipe the Drive
+
+**WARNING**: This destroys all data on the drive!
+
+```bash
+# Wipe filesystem signatures
+wipefs -a /dev/sdX
+
+# Optionally, zero out the beginning (for clean slate)
+dd if=/dev/zero of=/dev/sdX bs=1M count=100
+```
+
+### Step 5: Create Ceph OSD
+
+```bash
+# Create Ceph OSD on the freed drive
+# Replace sdX with your drive (e.g., sdc, sde, sdh)
+ceph-volume lvm create --data /dev/sdX
+```
+
+### Step 6: Verify OSD Created
+
+```bash
+# Check OSD tree (should show 3 OSDs now)
+ceph osd tree
+
+# Check Ceph health (should improve)
+ceph health
+ceph health detail
+```
+
+---
+
+## Alternative: If ubuntu-vg is Needed
+
+If `ubuntu-vg` contains important data:
+
+### Option 1: Backup and Remove One Drive
+
+1. **Backup data** from ubuntu-vg
+2. **Reduce ubuntu-vg** to remove one drive
+3. **Use freed drive** for Ceph OSD
+
+### Option 2: Use Different Drive
+
+If 2 drives need formatting (as user mentioned):
+- Use one of those 2 drives for Ceph OSD
+- Format the other as needed
+
+### Option 3: Shrink ubuntu-vg
+
+1. **Shrink logical volume** in ubuntu-vg
+2. **Remove one physical volume** from ubuntu-vg
+3. **Use freed drive** for Ceph OSD
+
+---
+
+## Quick Commands Reference
+
+```bash
+# Check ubuntu-vg
+vgs ubuntu-vg
+lvs ubuntu-vg
+pvs | grep ubuntu-vg
+
+# Check what's using it
+mount | grep ubuntu-vg
+lsof | grep ubuntu-vg
+
+# Remove drive from ubuntu-vg (WARNING: Destroys data!)
+vgchange -a n ubuntu-vg
+pvremove /dev/sdX3
+
+# Wipe drive
+wipefs -a /dev/sdX
+
+# Create Ceph OSD
+ceph-volume lvm create --data /dev/sdX
+
+# Verify
+ceph osd tree
+ceph health
+```
+
+---
+
+## Safety Checklist
+
+Before removing a drive from ubuntu-vg:
+
+- [ ] **Backup any important data** (if ubuntu-vg contains data)
+- [ ] **Verify ubuntu-vg is not critical** for system operation
+- [ ] **Check if logical volumes are mounted** (unmount first)
+- [ ] **Confirm which drive to remove** (choose carefully)
+- [ ] **Have recovery plan** if something goes wrong
+
+---
+
+## Expected Results
+
+### Before
+- 2 OSDs (insufficient for 3-way replication)
+- HEALTH_WARN: TOO_FEW_OSDS
+
+### After
+- 3 OSDs (sufficient for 3-way replication)
+- HEALTH_OK or improved (TOO_FEW_OSDS warning gone)
+- VMs can be created with ceph-fs storage
+
+---
+
+## Summary
+
+### Current State
+- ✅ 6x 250GB drives found in ubuntu-vg
+- ⚠️ All drives 100% allocated
+- 🎯 Need to free one drive for Ceph OSD
+
+### Action Required
+1. Check ubuntu-vg status
+2. Identify safe drive to remove
+3. Remove from ubuntu-vg (if safe)
+4. Wipe and create Ceph OSD
+5. Verify Ceph health improves
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔍 **READY TO FREE DRIVE FROM ubuntu-vg**
+
--- a/docs/ceph/FRESH_INSTALL_SETUP.md
+++ b/docs/ceph/FRESH_INSTALL_SETUP.md
@@ -0,0 +1,227 @@
+# Fresh Ceph Setup on Proxmox VE
+
+## Prerequisites
+
+- Proxmox VE installed on both nodes
+- Nodes are in a Proxmox cluster (or standalone)
+- Network connectivity between nodes (192.168.11.0/24)
+- Root access to both nodes
+
+## Node Information
+
+- **ML110-01**: ml110-01.sankofa.nexus (192.168.11.10)
+- **R630-01**: r630-01.sankofa.nexus (192.168.11.11)
+
+## Step-by-Step Setup
+
+### Step 1: Install Ceph on Both Nodes
+
+Run on **both nodes**:
+
+```bash
+apt-get update
+apt-get install -y ceph ceph-common ceph-base
+```
+
+### Step 2: Initialize Ceph Cluster (ML110-01 only)
+
+Run on **ml110-01**:
+
+```bash
+pveceph init --network 192.168.11.0/24 --cluster-network 192.168.11.0/24
+```
+
+This will:
+- Create `/etc/pve/ceph.conf`
+- Generate cluster FSID
+- Set up bootstrap keyrings
+
+### Step 3: Create Monitors
+
+Run on **ml110-01**:
+
+```bash
+pveceph mon create
+```
+
+Run on **r630-01**:
+
+```bash
+pveceph mon create
+```
+
+### Step 4: Verify Quorum
+
+Run on **either node**:
+
+```bash
+ceph quorum_status
+ceph -s
+```
+
+You should see both monitors in quorum.
+
+### Step 5: Create OSDs
+
+#### On ML110-01
+
+Identify available drives:
+
+```bash
+lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE
+```
+
+Create OSD on available drive (replace `/dev/sdX` with actual drive):
+
+```bash
+ceph-volume lvm create --data /dev/sdX
+```
+
+#### On R630-01
+
+Create OSDs on the 6x 250GB drives (sdc, sdd, sde, sdf, sdg, sdh):
+
+```bash
+for disk in sdc sdd sde sdf sdg sdh; do
+    echo "Creating OSD on /dev/$disk..."
+    ceph-volume lvm create --data /dev/$disk
+    sleep 2
+done
+```
+
+### Step 6: Verify Cluster Health
+
+Run on **either node**:
+
+```bash
+ceph -s
+ceph osd tree
+ceph health detail
+```
+
+### Step 7: Create Storage Pools
+
+#### Create RBD Pool (for VM disks)
+
+```bash
+ceph osd pool create rbd 32 32
+ceph osd pool application enable rbd rbd
+rbd pool init rbd
+```
+
+#### Create CephFS (for shared storage)
+
+```bash
+ceph osd pool create cephfs_data 32 32
+ceph osd pool create cephfs_metadata 16 16
+ceph fs new ceph-fs cephfs_metadata cephfs_data
+```
+
+### Step 8: Add Storage to Proxmox
+
+#### Add Ceph RBD Storage
+
+In Proxmox Web UI:
+1. Datacenter → Storage → Add → RBD
+2. ID: `ceph-rbd`
+3. Pool: `rbd`
+4. Nodes: Select both nodes
+5. Add
+
+#### Add CephFS Storage
+
+In Proxmox Web UI:
+1. Datacenter → Storage → Add → CephFS
+2. ID: `ceph-fs`
+3. FS Name: `ceph-fs`
+4. Nodes: Select both nodes
+5. Add
+
+## Verification Checklist
+
+- [ ] Both monitors in quorum
+- [ ] At least 3 OSDs created (for replication factor 3)
+- [ ] Cluster health is HEALTH_OK or HEALTH_WARN (not HEALTH_ERR)
+- [ ] Storage pools created (rbd, cephfs_data, cephfs_metadata)
+- [ ] CephFS filesystem created
+- [ ] Storage added to Proxmox Web UI
+
+## Troubleshooting
+
+### Monitors Not Forming Quorum
+
+1. Check monitor services:
+   ```bash
+   systemctl status ceph-mon@$(hostname -s)
+   ```
+
+2. Check monitor logs:
+   ```bash
+   journalctl -u ceph-mon@$(hostname -s) -n 50
+   ```
+
+3. Verify network connectivity:
+   ```bash
+   ping 192.168.11.10
+   ping 192.168.11.11
+   ```
+
+4. Check firewall:
+   ```bash
+   # Allow Ceph ports
+   ufw allow 6789/tcp  # Classic msgr
+   ufw allow 3300/tcp  # msgr2
+   ```
+
+### OSD Creation Fails
+
+1. Check if drive is available:
+   ```bash
+   lsblk /dev/sdX
+   ```
+
+2. Wipe drive if needed:
+   ```bash
+   wipefs -a /dev/sdX
+   ```
+
+3. Check for existing LVM volumes:
+   ```bash
+   pvs | grep sdX
+   vgs | grep ceph
+   ```
+
+### Cluster Health Issues
+
+1. Check detailed health:
+   ```bash
+   ceph health detail
+   ```
+
+2. Check OSD status:
+   ```bash
+   ceph osd tree
+   ceph osd stat
+   ```
+
+3. Check placement groups:
+   ```bash
+   ceph pg stat
+   ```
+
+## Next Steps
+
+After Ceph is set up:
+
+1. **Fix PG allocation**: Ensure pools have appropriate PG counts
+2. **Create VMs**: Use Ceph storage for VM disks
+3. **Monitor cluster**: Set up monitoring and alerts
+4. **Backup configuration**: Backup `/etc/pve/ceph.conf` and keyrings
+
+## Important Notes
+
+- **Minimum OSDs**: For `osd_pool_default_size = 3`, you need at least 3 OSDs
+- **PG Count**: Use calculator: https://ceph.io/pgcalc/
+- **Network**: Ensure cluster network has sufficient bandwidth
+- **Backup**: Always backup Ceph configuration and keyrings
+
--- a/docs/ceph/ISSUES_SUMMARY.md
+++ b/docs/ceph/ISSUES_SUMMARY.md
@@ -0,0 +1,143 @@
+# Ceph OSD Creation Issues - Summary
+
+**Date**: 2025-12-13  
+**Status**: 🔍 **ISSUES IDENTIFIED - NEEDS DIAGNOSIS**
+
+---
+
+## Issues Identified
+
+### 1. OSD Creation Hanging/Timeout
+
+**Symptom**: `ceph-volume lvm create` hangs at "osd new" step
+
+**Root Causes** (likely):
+- ❌ **Bootstrap keyring missing** - Cannot authenticate with cluster
+- ❌ **Ceph cluster not accessible** - RADOS timeout errors
+- ❌ **Monitor connectivity issues** - Cannot reach Ceph monitors
+
+### 2. Bootstrap Keyring Issue
+
+**Error**: 
+```
+unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
+RADOS timed out (error connecting to the cluster)
+```
+
+**Possible Locations**:
+- `/var/lib/ceph/bootstrap-osd/ceph.keyring` (standard)
+- `/etc/pve/priv/ceph.client.bootstrap-osd.keyring` (Proxmox)
+
+**Status**: Unknown - needs verification
+
+### 3. Ceph Cluster Connectivity
+
+**Issue**: Commands timeout when trying to connect to cluster
+
+**Possible Causes**:
+- Monitors not running
+- Network connectivity issues
+- Firewall blocking ports
+- Cluster configuration issues
+
+---
+
+## Diagnostic Steps
+
+Run these commands on R630-01 to diagnose:
+
+### Quick Check
+
+```bash
+# 1. Check Ceph health (does it respond?)
+ceph health
+
+# 2. Check monitors
+ceph mon stat
+
+# 3. Check bootstrap keyring
+ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
+ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring
+
+# 4. Check if pveceph available
+which pveceph
+```
+
+### Full Diagnostic
+
+```bash
+# Run comprehensive diagnostic
+bash /tmp/diagnose-ceph-issues.sh
+```
+
+---
+
+## Likely Solutions
+
+### Solution 1: Use pveceph (If Proxmox-Managed)
+
+If `pveceph` is available:
+
+```bash
+for drive in sdc sdd sde sdf sdg sdh; do
+    pveceph create /dev/$drive
+done
+```
+
+### Solution 2: Fix Bootstrap Keyring
+
+If cluster is accessible:
+
+```bash
+# Get bootstrap keyring
+mkdir -p /var/lib/ceph/bootstrap-osd
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# Then create OSDs
+for drive in sdc sdd sde sdf sdg sdh; do
+    ceph-volume lvm create --data /dev/$drive
+done
+```
+
+### Solution 3: Fix Cluster Connectivity
+
+If cluster is not accessible:
+
+1. Check monitor services
+2. Check network connectivity
+3. Check firewall rules
+4. Fix connectivity issues
+5. Then retry OSD creation
+
+---
+
+## Current State
+
+- ✅ **6x 250GB drives**: Wiped and ready
+- ❌ **OSD creation**: Hanging due to authentication/connectivity
+- 🔍 **Root cause**: Needs diagnosis
+
+---
+
+## Next Action
+
+**Run diagnostic script** to understand the issue:
+
+```bash
+# On R630-01
+bash /tmp/diagnose-ceph-issues.sh
+```
+
+This will show:
+- Ceph cluster status
+- Bootstrap keyring locations
+- Monitor connectivity
+- Service status
+- Network connectivity
+
+Then we can apply the appropriate fix.
+
+---
+
+**Last Updated**: 2025-12-13
+
--- a/docs/ceph/ISSUE_ANALYSIS.md
+++ b/docs/ceph/ISSUE_ANALYSIS.md
@@ -0,0 +1,295 @@
+# Ceph OSD Creation Issue - Comprehensive Analysis
+
+**Date**: 2025-12-13  
+**Status**: 🔍 **ANALYZING ROOT CAUSES**
+
+---
+
+## Problem Summary
+
+OSD creation commands are **hanging/timing out** when trying to create OSDs on the 6x 250GB drives. The process stops at the "osd new" step, which requires cluster authentication.
+
+---
+
+## Root Cause Analysis
+
+### Issue 1: Bootstrap Keyring Missing/Inaccessible
+
+**Symptom**:
+```
+unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
+RADOS timed out (error connecting to the cluster)
+RuntimeError: Unable to create a new OSD id
+```
+
+**Root Cause**:
+- `ceph-volume` needs bootstrap keyring to authenticate with Ceph cluster
+- Keyring should be at `/var/lib/ceph/bootstrap-osd/ceph.keyring` OR `/etc/pve/priv/ceph.client.bootstrap-osd.keyring`
+- Neither location has the keyring, or it's not accessible
+
+**Why It Happens**:
+- Ceph cluster may not have bootstrap-osd client created
+- Keyring may have been deleted or never created
+- Proxmox Ceph integration may use different keyring location
+
+### Issue 2: Ceph Cluster Connectivity
+
+**Symptom**:
+- Commands hang at "osd new" step
+- "RADOS timed out" errors
+- Cannot connect to cluster
+
+**Possible Causes**:
+1. **Ceph monitors not accessible** from R630-01
+2. **Network connectivity issues** between nodes
+3. **Ceph services not running** properly
+4. **Firewall blocking** Ceph ports (6789 for monitors)
+
+### Issue 3: Ceph Cluster Configuration
+
+**Current State**:
+- 2 OSDs (one on ml110-01, one on r630-01)
+- Ceph cluster exists and is running
+- Health warnings present
+- Cluster is functional but degraded
+
+**Questions**:
+- Is this a Proxmox-managed Ceph cluster?
+- Or a standalone Ceph cluster?
+- Where are the monitors running?
+- What's the cluster configuration?
+
+---
+
+## Detailed Investigation Needed
+
+### Step 1: Check Ceph Cluster Status
+
+```bash
+# On R630-01
+ceph health
+ceph mon stat
+ceph mon dump
+ceph osd tree
+```
+
+**What to look for**:
+- Are monitors accessible?
+- What are monitor addresses?
+- Is cluster in quorum?
+- Can we connect to cluster?
+
+### Step 2: Check Bootstrap Keyring
+
+```bash
+# Check standard location
+ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# Check Proxmox location
+ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring
+
+# Check if bootstrap-osd client exists
+ceph auth list | grep bootstrap-osd
+```
+
+**What to look for**:
+- Does keyring exist?
+- Is it readable?
+- Does bootstrap-osd client exist in cluster?
+
+### Step 3: Check Ceph Services
+
+```bash
+# Check Ceph services
+systemctl status ceph.target
+systemctl status ceph-mon@*
+systemctl status ceph-osd@*
+
+# Check if monitors are running
+ps aux | grep ceph-mon
+
+# Check network connectivity
+netstat -tlnp | grep 6789
+```
+
+**What to look for**:
+- Are Ceph services running?
+- Are monitors listening on port 6789?
+- Can we reach monitor addresses?
+
+### Step 4: Check Network Connectivity
+
+```bash
+# Check if we can reach monitors
+# First, find monitor addresses
+ceph mon dump
+
+# Then test connectivity
+ping <monitor-ip>
+telnet <monitor-ip> 6789
+```
+
+**What to look for**:
+- Can we ping monitor IPs?
+- Can we connect to port 6789?
+- Are there firewall rules blocking?
+
+---
+
+## Likely Scenarios
+
+### Scenario 1: Proxmox-Managed Ceph
+
+**If Ceph was set up via Proxmox**:
+- Keyring should be in `/etc/pve/priv/`
+- May need to use `pveceph` commands instead
+- Proxmox manages Ceph configuration
+
+**Solution**:
+```bash
+# Use Proxmox Ceph tool
+pveceph create /dev/sdc
+```
+
+### Scenario 2: Standalone Ceph Cluster
+
+**If Ceph was set up manually**:
+- Keyring should be in `/var/lib/ceph/bootstrap-osd/`
+- Need to create/bootstrap keyring
+- Standard Ceph commands should work
+
+**Solution**:
+```bash
+# Create/bootstrap keyring
+mkdir -p /var/lib/ceph/bootstrap-osd
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+```
+
+### Scenario 3: Cluster Connectivity Issue
+
+**If monitors are not accessible**:
+- Network issue between nodes
+- Firewall blocking
+- Monitors not running
+
+**Solution**:
+- Fix network connectivity
+- Check firewall rules
+- Ensure monitors are running
+
+---
+
+## Diagnostic Commands
+
+Run these on R630-01 to diagnose:
+
+```bash
+# 1. Ceph cluster status
+ceph health
+ceph mon stat
+ceph osd tree
+
+# 2. Bootstrap keyring check
+ls -la /var/lib/ceph/bootstrap-osd/ 2>/dev/null
+ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring 2>/dev/null
+ceph auth list | grep bootstrap
+
+# 3. Ceph services
+systemctl status ceph.target
+systemctl status ceph-mon@*
+
+# 4. Network connectivity
+ceph mon dump | grep -E "rank|addr"
+# Then test connectivity to monitor IPs
+
+# 5. Check if pveceph available
+which pveceph
+pveceph status 2>/dev/null || echo "pveceph not available"
+```
+
+---
+
+## Recommended Fix Strategy
+
+### Option A: Use Proxmox Ceph Tool (If Available)
+
+```bash
+# Check if available
+which pveceph
+
+# If yes, use it
+for drive in sdc sdd sde sdf sdg sdh; do
+    pveceph create /dev/$drive
+done
+```
+
+### Option B: Fix Bootstrap Keyring
+
+```bash
+# 1. Check if cluster is accessible
+ceph health
+
+# 2. Get or create bootstrap keyring
+mkdir -p /var/lib/ceph/bootstrap-osd
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring 2>/dev/null || \
+ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# 3. Verify
+ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# 4. Create OSDs
+for drive in sdc sdd sde sdf sdg sdh; do
+    ceph-volume lvm create --data /dev/$drive
+done
+```
+
+### Option C: Check Cluster Connectivity First
+
+```bash
+# 1. Verify cluster is accessible
+ceph health
+ceph mon stat
+
+# 2. If not accessible, check:
+#    - Network connectivity
+#    - Firewall rules
+#    - Monitor services
+#    - Cluster configuration
+```
+
+---
+
+## Key Questions to Answer
+
+1. **Is Ceph cluster accessible from R630-01?**
+   - Can we run `ceph health` successfully?
+   - Are monitors reachable?
+
+2. **Where is the bootstrap keyring?**
+   - Does it exist?
+   - Is it in the right location?
+   - Is it readable?
+
+3. **Is this Proxmox-managed Ceph?**
+   - Should we use `pveceph`?
+   - Is Ceph integrated with Proxmox?
+
+4. **What's the cluster configuration?**
+   - Where are monitors running?
+   - What's the cluster name?
+   - What's the authentication method?
+
+---
+
+## Next Steps
+
+1. **Run diagnostic commands** to understand current state
+2. **Identify root cause** (keyring, connectivity, or configuration)
+3. **Apply appropriate fix** based on findings
+4. **Retry OSD creation** after fix
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔍 **NEEDS DIAGNOSTIC INFORMATION**
+
--- a/docs/ceph/MANUAL_VERIFICATION_STEPS.md
+++ b/docs/ceph/MANUAL_VERIFICATION_STEPS.md
@@ -0,0 +1,152 @@
+# Manual Verification Steps for 250GB Drives on R630-01
+
+**Date**: 2025-12-13  
+**Purpose**: Manual steps to verify 250GB drives when SSH access requires authentication
+
+---
+
+## Quick Verification Commands
+
+SSH to R630-01 (192.168.11.11) and run these commands:
+
+### Step 1: List All Block Devices
+
+```bash
+lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL
+```
+
+**Look for**: 6x drives around 250GB (232-260 GB range)
+
+### Step 2: List All Physical Disks
+
+```bash
+fdisk -l | grep -E "^Disk /dev/sd" | sort
+```
+
+**Look for**: Disks showing ~250GB or ~232-260 GB
+
+### Step 3: Check Current Ceph OSDs
+
+```bash
+ceph osd tree
+ceph osd df
+ceph health detail
+```
+
+**Expected**: Should show 2 OSDs currently
+
+### Step 4: Check Storage Pools
+
+```bash
+pvesm status
+```
+
+**Look for**: Storage pools and their usage
+
+### Step 5: Check Volume Groups
+
+```bash
+vgs
+pvs
+```
+
+**Look for**: Which disks are in volume groups
+
+---
+
+## One-Line Verification Script
+
+Copy and paste this entire block into R630-01:
+
+```bash
+echo "=== R630-01 Disk Verification ===" && \
+echo "" && \
+echo "1. All Block Devices:" && \
+lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL && \
+echo "" && \
+echo "2. All Physical Disks:" && \
+fdisk -l 2>/dev/null | grep -E "^Disk /dev/sd" | sort && \
+echo "" && \
+echo "3. 250GB Drives (232-260 GB):" && \
+for disk in /dev/sd[a-z]; do [ -b "$disk" ] && fdisk -l "$disk" 2>/dev/null | grep "^Disk $disk" | awk -v d="$disk" '{size=$3" "$4; if (size ~ /232|233|234|235|236|237|238|239|240|241|242|243|244|245|246|247|248|249|250|251|252|253|254|255|256|257|258|259|260/) print d": "size}'; done && \
+echo "" && \
+echo "4. Current Ceph OSDs:" && \
+ceph osd tree 2>/dev/null && \
+echo "" && \
+echo "5. Ceph Health:" && \
+ceph health 2>/dev/null && \
+echo "" && \
+echo "6. Storage Pools:" && \
+pvesm status 2>/dev/null && \
+echo "" && \
+echo "7. Volume Groups:" && \
+vgs 2>/dev/null && \
+echo "" && \
+echo "8. Physical Volumes:" && \
+pvs 2>/dev/null
+```
+
+---
+
+## What to Look For
+
+### Expected Output
+
+You should see something like:
+
+```
+NAME   SIZE TYPE MOUNTPOINT FSTYPE MODEL
+sda    279G disk            ST9300653SS      # 300GB - Proxmox
+sdb    279G disk            HUC106030CSS600  # 300GB - Ceph OSD
+sdc    250G disk                               # 250GB - Available?
+sdd    250G disk                               # 250GB - Available?
+sde    250G disk                               # 250GB - Available?
+sdf    250G disk                               # 250GB - Available?
+sdg    250G disk                               # 250GB - Available?
+sdh    250G disk                               # 250GB - Available?
+```
+
+### Key Indicators
+
+**Available Drive** (good for OSD):
+- ✅ No MOUNTPOINT (empty)
+- ✅ No FSTYPE (not formatted)
+- ✅ Not in volume group (check `pvs`)
+- ✅ Size around 232-260 GB
+
+**Drive in Use** (not available):
+- ❌ Has MOUNTPOINT (mounted)
+- ❌ Has FSTYPE (formatted)
+- ❌ In volume group (shows in `pvs`)
+
+---
+
+## After Verification
+
+Once you've identified available 250GB drives, share:
+1. How many 250GB drives were found
+2. Which device names (sdc, sdd, etc.)
+3. Which ones are available (no mount, no partitions, not in VG)
+4. Current Ceph OSD status
+
+Then we can proceed with creating the third OSD!
+
+---
+
+## Alternative: Copy Script to R630-01
+
+If you have access to copy files:
+
+```bash
+# From your local machine
+scp scripts/verify-r630-250gb-drives.sh root@192.168.11.11:/tmp/
+
+# Then on R630-01
+ssh root@192.168.11.11
+bash /tmp/verify-r630-250gb-drives.sh
+```
+
+---
+
+**Last Updated**: 2025-12-13
+
--- a/docs/ceph/MON_RECOVERY_SUCCESS.md
+++ b/docs/ceph/MON_RECOVERY_SUCCESS.md
@@ -0,0 +1,117 @@
+# Ceph MON Recovery - Success
+
+**Date**: 2025-12-13  
+**Status**: ✅ **RECOVERED**
+
+## Recovery Summary
+
+Successfully recovered Ceph cluster by recreating the corrupted MON on r630-01.
+
+## What Was Done
+
+### 1. Identified Root Cause
+- **Hardware Failure**: `/dev/sda` on r630-01 has critical medium errors
+- **Symptom**: Corrupted RocksDB database in `/var/lib/ceph/mon/ceph-r630-01/store.db/`
+- **Impact**: MON could not start, cluster lost quorum, OSDs could not start
+
+### 2. Recovery Steps Executed
+
+1. **Stopped and disabled failed MON service**
+   ```bash
+   systemctl stop ceph-mon@r630-01
+   systemctl disable ceph-mon@r630-01
+   ```
+
+2. **Backed up corrupted MON directory**
+   - Created backup: `/root/ceph-mon-r630-01-corrupted-*.tar.gz`
+   - Note: Backup showed file corruption (file shrank during tar)
+
+3. **Removed corrupted MON directory**
+   ```bash
+   rm -rf /var/lib/ceph/mon/ceph-r630-01
+   ```
+
+4. **Created new monmap from configuration**
+   ```bash
+   monmaptool --create --fsid 5fb968ae-12ab-405f-b05f-0df29a168328 \
+     --add ml110-01 192.168.11.10 \
+     --add r630-01 192.168.11.11 \
+     /tmp/monmap
+   ```
+
+5. **Copied keyring from ml110-01**
+   - Source: `/var/lib/ceph/mon/ceph-ml110-01/keyring`
+   - Destination: `/tmp/mon-keyring` on r630-01
+
+6. **Created new MON filesystem**
+   ```bash
+   mkdir -p /var/lib/ceph/mon/ceph-r630-01
+   chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
+   ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
+   ```
+
+7. **Fixed ownership and started MON**
+   ```bash
+   chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
+   systemctl enable ceph-mon@r630-01
+   systemctl start ceph-mon@r630-01
+   ```
+
+8. **Restarted OSDs**
+   - `ceph-osd@1` on r630-01: ✅ Running
+   - `ceph-osd@0` on ml110-01: ✅ Running
+
+## Current Status
+
+### Services
+- ✅ `ceph-mon@r630-01.service`: **active (running)**
+- ✅ `ceph-mon@ml110-01.service`: **active (running)** (was already running)
+- ✅ `ceph-osd@1.service`: **active (running)**
+- ✅ `ceph-osd@0.service`: **active (running)**
+
+### Cluster
+- **Quorum**: Restored (2/2 MONs)
+- **OSDs**: 2/2 running
+- **Cluster Access**: Restored (commands no longer timeout)
+
+## Remaining Issues
+
+### 1. Hardware Failure (CRITICAL)
+- **Disk**: `/dev/sda` on r630-01 still has critical medium errors
+- **Risk**: MON database may corrupt again if disk continues to fail
+- **Action Required**: 
+  - Monitor disk health: `smartctl -a /dev/sda`
+  - Plan disk replacement
+  - Consider moving MON to different disk if available
+
+### 2. Ceph Health Issues (Still Present)
+- **OSD Count**: 2 OSDs < required 3 (osd_pool_default_size = 3)
+- **Undersized PGs**: PG 3.7f stuck undersized for 32h+
+- **PG Over-allocation**: 288 PGs per OSD (max 250), RBD pool has 64 PGs (should be 32)
+
+### 3. Password Security
+- Root password was exposed in command history
+- **Action Required**: Rotate root password immediately
+
+## Next Steps
+
+1. **Monitor cluster health**: `ceph -s` and `ceph health detail`
+2. **Address disk failure**: Check SMART status, plan replacement
+3. **Add third OSD**: Use the 6x 250GB drives on r630-01 (once disk issue is resolved)
+4. **Fix PG allocation**: Reduce RBD pool PGs from 64 to 32
+5. **Rotate passwords**: Change root password on all nodes
+
+## Lessons Learned
+
+1. **Hardware monitoring is critical**: Disk failures can cascade to application layer
+2. **MON quorum requires redundancy**: With only 2 MONs, losing one breaks the cluster
+3. **Ownership matters**: MON directory must be owned by `ceph:ceph`
+4. **Recovery is possible**: Even with corrupted database, MON can be recreated from peer
+
+## Recovery Time
+
+- **Start**: 19:10 PST
+- **MON Running**: 19:24 PST
+- **OSDs Running**: 19:25 PST
+- **Total**: ~15 minutes
+
--- a/docs/ceph/OSD_CREATION_ISSUE.md
+++ b/docs/ceph/OSD_CREATION_ISSUE.md
@@ -0,0 +1,84 @@
+# OSD Creation Issue - Bootstrap Keyring
+
+**Date**: 2025-12-13  
+**Status**: ⚠️ **OSD CREATION FAILED - BOOTSTRAP KEYRING MISSING**
+
+---
+
+## What Happened
+
+✅ **Success**: All 6 drives (sdc-sdh) were successfully wiped  
+❌ **Failure**: OSD creation failed due to missing Ceph bootstrap keyring
+
+### Error Messages
+```
+unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
+RADOS timed out (error connecting to the cluster)
+RuntimeError: Unable to create a new OSD id
+```
+
+---
+
+## Root Cause
+
+The Ceph bootstrap keyring is missing or Ceph cluster is not accessible. This is needed to authenticate when creating new OSDs.
+
+---
+
+## Quick Fix
+
+Run these commands on R630-01:
+
+```bash
+# 1. Check Ceph cluster connectivity
+ceph health
+ceph mon stat
+
+# 2. Get bootstrap keyring from cluster
+mkdir -p /var/lib/ceph/bootstrap-osd
+ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# 3. Verify keyring exists
+ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# 4. Create OSDs (drives are already wiped)
+for drive in sdc sdd sde sdf sdg sdh; do
+    ceph-volume lvm create --data /dev/$drive
+done
+
+# 5. Verify
+ceph osd tree
+ceph health
+```
+
+---
+
+## Alternative: Use pveceph
+
+If you're using Proxmox's Ceph integration:
+
+```bash
+# Check if available
+which pveceph
+
+# Create OSDs
+pveceph create /dev/sdc
+pveceph create /dev/sdd
+pveceph create /dev/sde
+pveceph create /dev/sdf
+pveceph create /dev/sdg
+pveceph create /dev/sdh
+```
+
+---
+
+## Status
+
+- ✅ **Drives wiped**: All 6 drives ready
+- ⚠️ **OSD creation**: Blocked by bootstrap keyring issue
+- 🎯 **Next step**: Fix bootstrap keyring and retry OSD creation
+
+---
+
+**Last Updated**: 2025-12-13
+
--- a/docs/ceph/PERC_CONTROLLER_CHECK.md
+++ b/docs/ceph/PERC_CONTROLLER_CHECK.md
@@ -0,0 +1,192 @@
+# Checking PERC Controller for 250GB Drives
+
+**Date**: 2025-12-13  
+**Issue**: 6x 250GB drives not visible to OS, need to check PERC controller
+
+---
+
+## Current Status
+
+- **OS Detected**: Only 2x 300GB drives (sda, sdb)
+- **Expected**: 6x 250GB drives should be visible
+- **Likely Cause**: Drives not configured in PERC controller or in RAID array
+
+---
+
+## Check PERC Controller
+
+### Option 1: Install Dell OpenManage Tools
+
+```bash
+# Install Dell OpenManage Server Administrator
+apt-get update
+apt-get install -y srvadmin-all
+
+# Start services
+systemctl start dsm_sa_datamgrd
+systemctl start dsm_sa_snmpd
+systemctl start dsm_sa_eventmgrd
+
+# Check physical disks
+omreport storage pdisk controller=0
+
+# Check virtual disks (RAID arrays)
+omreport storage vdisk controller=0
+```
+
+### Option 2: Use PERC CLI (if available)
+
+```bash
+# Check if perccli is installed
+which perccli
+
+# If not installed, download from Dell
+# Then check physical disks
+perccli /c0 show
+
+# Check virtual disks
+perccli /c0/vall show
+```
+
+### Option 3: Check via /proc or /sys
+
+```bash
+# Check for PERC controller info
+cat /proc/scsi/scsi
+
+# Check block devices
+ls -la /dev/sd*
+
+# Check for additional controllers
+lspci | grep -i raid
+lspci | grep -i perc
+lspci | grep -i lsi
+```
+
+### Option 4: Check dmesg for disk detection
+
+```bash
+# Check kernel messages for disk detection
+dmesg | grep -i "sd[a-z]"
+dmesg | grep -i "disk"
+dmesg | grep -i "perc"
+dmesg | grep -i "raid"
+
+# Check recent disk events
+journalctl -k | grep -i disk | tail -50
+```
+
+---
+
+## What to Look For
+
+### If Drives Are in PERC Controller
+
+You should see:
+- 8 physical disks total
+- 2x 300GB drives (already configured)
+- 6x 250GB drives (may be unconfigured or in RAID)
+
+### If Drives Are Unconfigured
+
+They may show as:
+- "Unconfigured Good" or "Ready"
+- Not assigned to any virtual disk
+- Available for configuration
+
+### If Drives Are in RAID
+
+They may be:
+- Part of a RAID array (virtual disk)
+- Not exposed as individual disks to OS
+- Need to be removed from RAID to use individually
+
+---
+
+## Next Steps Based on Findings
+
+### Scenario 1: Drives Detected in PERC but Unconfigured
+
+**Action**: Configure drives as individual disks (non-RAID) or create separate virtual disks
+
+```bash
+# Using perccli (example - adjust for your setup)
+# Create individual virtual disks (one per physical disk)
+# This exposes each drive to the OS
+
+# Or configure PERC in HBA mode (if supported)
+# This passes drives directly to OS without RAID
+```
+
+### Scenario 2: Drives Not Detected in PERC
+
+**Possible causes**:
+- Drives not physically installed
+- Drive backplane issue
+- Controller issue
+
+**Action**: 
+1. Check physical installation
+2. Check drive bay LEDs
+3. Check PERC controller status
+4. May need to reseat drives
+
+### Scenario 3: Drives in RAID Array
+
+**Action**: 
+1. Check RAID configuration
+2. Decide if RAID array can be broken up
+3. May need to backup data and reconfigure
+4. Create individual virtual disks for Ceph OSDs
+
+---
+
+## Recommended Configuration for Ceph
+
+For Ceph OSDs, you want:
+- **Individual disks** (not in RAID)
+- **Direct access** to each physical disk
+- **PERC in HBA mode** (if supported) or individual virtual disks
+
+**Why**: Ceph handles redundancy, so RAID is not needed and can reduce performance.
+
+---
+
+## Quick Check Commands
+
+Run these on R630-01:
+
+```bash
+# 1. Check for PERC controller
+lspci | grep -iE "raid|perc|lsi|megaraid"
+
+# 2. Check kernel messages
+dmesg | grep -iE "sd[a-z]|disk|perc" | tail -20
+
+# 3. Check /proc for SCSI devices
+cat /proc/scsi/scsi
+
+# 4. Check for additional block devices
+lsblk
+fdisk -l
+
+# 5. Check if OpenManage is available
+which omreport
+which perccli
+```
+
+---
+
+## Summary
+
+The 6x 250GB drives are likely:
+1. **In PERC controller** but not configured/exposed
+2. **In a RAID array** that needs to be reconfigured
+3. **Not physically installed** (less likely if user confirmed they exist)
+
+**Next Step**: Check PERC controller status to see where the drives are.
+
+---
+
+**Last Updated**: 2025-12-13
+
--- a/docs/ceph/QUICK_START.md
+++ b/docs/ceph/QUICK_START.md
@@ -0,0 +1,72 @@
+# Quick Start: Ceph Setup on Fresh Proxmox Install
+
+## Run These Commands
+
+### On ML110-01 (192.168.11.10)
+
+```bash
+# 1. Install Ceph
+apt-get update
+apt-get install -y ceph ceph-common ceph-base
+
+# 2. Initialize cluster
+pveceph init --network 192.168.11.0/24 --cluster-network 192.168.11.0/24
+
+# 3. Create monitor
+pveceph mon create
+
+# 4. Verify
+ceph quorum_status
+```
+
+### On R630-01 (192.168.11.11)
+
+```bash
+# 1. Install Ceph
+apt-get update
+apt-get install -y ceph ceph-common ceph-base
+
+# 2. Create monitor
+pveceph mon create
+
+# 3. Verify quorum
+sleep 10
+ceph quorum_status
+```
+
+### Create OSDs on R630-01
+
+```bash
+# Create OSDs on 6x 250GB drives
+for disk in sdc sdd sde sdf sdg sdh; do
+    ceph-volume lvm create --data /dev/$disk
+    sleep 2
+done
+
+# Verify
+ceph osd tree
+ceph -s
+```
+
+### Create Storage Pools
+
+```bash
+# RBD pool for VM disks
+ceph osd pool create rbd 32 32
+ceph osd pool application enable rbd rbd
+rbd pool init rbd
+
+# CephFS for shared storage
+ceph osd pool create cephfs_data 32 32
+ceph osd pool create cephfs_metadata 16 16
+ceph fs new ceph-fs cephfs_metadata cephfs_data
+```
+
+## Verify Everything
+
+```bash
+ceph -s
+ceph osd tree
+ceph health detail
+```
+
--- a/docs/ceph/RECOVERY_PLAN.md
+++ b/docs/ceph/RECOVERY_PLAN.md
@@ -0,0 +1,113 @@
+# Ceph MON Recovery Plan - R630-01
+
+## Current Status
+
+✅ **ML110-01 MON**: Running (but in "probing" state - no quorum)  
+❌ **R630-01 MON**: Failed (corrupted RocksDB due to failing disk)  
+❌ **Cluster Quorum**: Lost (need 2/2 MONs for quorum)
+
+## Root Cause Summary
+
+1. **Hardware**: `/dev/sda` on r630-01 has critical medium errors
+2. **Symptom**: Ceph MON RocksDB database corrupted (`001572.sst` I/O error)
+3. **Impact**: MON cannot start, cluster has no quorum, OSDs cannot start
+
+## Recovery Strategy
+
+Since ml110-01 MON is running, we can recreate the r630-01 MON. The MON will sync cluster state from ml110-01.
+
+### Prerequisites
+
+1. **Fix or replace failing disk** `/dev/sda` on r630-01 (or move MON to different disk)
+2. **Backup current state** (already documented)
+3. **Verify ml110-01 MON is accessible** from r630-01
+
+### Recovery Steps
+
+#### Step 1: Stop Failed MON Service
+```bash
+ssh root@192.168.11.11 'systemctl stop ceph-mon@r630-01'
+ssh root@192.168.11.11 'systemctl disable ceph-mon@r630-01'
+```
+
+#### Step 2: Backup Corrupted MON Directory
+```bash
+ssh root@192.168.11.11 'tar -czf /root/ceph-mon-r630-01-corrupted-$(date +%Y%m%d-%H%M%S).tar.gz /var/lib/ceph/mon/ceph-r630-01/'
+```
+
+#### Step 3: Remove Corrupted MON
+```bash
+ssh root@192.168.11.11 'pveceph mon destroy r630-01'
+```
+
+**Alternative if pveceph fails**:
+```bash
+ssh root@192.168.11.11 'rm -rf /var/lib/ceph/mon/ceph-r630-01'
+```
+
+#### Step 4: Recreate MON on R630-01
+```bash
+ssh root@192.168.11.11 'pveceph mon create r630-01'
+```
+
+The MON will:
+- Create a new RocksDB database
+- Sync cluster state from ml110-01
+- Rejoin the quorum
+
+#### Step 5: Verify Recovery
+```bash
+# Check MON status
+ssh root@192.168.11.11 'systemctl status ceph-mon@r630-01'
+
+# Check cluster quorum
+ssh root@192.168.11.11 'ceph quorum_status'
+
+# Check cluster health
+ssh root@192.168.11.11 'ceph -s'
+```
+
+#### Step 6: Restart OSDs
+Once quorum is restored:
+```bash
+ssh root@192.168.11.11 'systemctl restart ceph-osd@1'
+ssh root@192.168.11.10 'systemctl restart ceph-osd@0'
+```
+
+## Disk Issue Resolution
+
+### Option A: Move MON to Different Disk (If Available)
+If r630-01 has another disk with space:
+1. Create new directory on different disk: `/mnt/other-disk/ceph-mon/`
+2. Update MON service to use new location
+3. Recreate MON in new location
+
+### Option B: Replace Failing Disk
+1. Replace `/dev/sda` with new disk
+2. Restore Proxmox installation
+3. Recreate MON
+
+### Option C: Fix Filesystem (If Disk is Recoverable)
+1. Run `fsck` on filesystem
+2. Check SMART status: `smartctl -a /dev/sda`
+3. If disk is recoverable, proceed with MON recreation
+
+## Risk Assessment
+
+- **Low Risk**: Recreating MON from healthy peer (ml110-01) is standard procedure
+- **Medium Risk**: If ml110-01 MON also fails during recovery, cluster becomes unrecoverable
+- **High Risk**: If disk continues to fail, MON will corrupt again
+
+## Post-Recovery Actions
+
+1. **Monitor disk health**: Set up SMART monitoring for `/dev/sda`
+2. **Add third MON**: Consider adding a third MON on a different node for redundancy
+3. **Backup MON data**: Regularly backup `/var/lib/ceph/mon/` directories
+4. **Monitor cluster**: Watch for recurring I/O errors
+
+## Notes
+
+- **Password Security**: Root password was exposed. Rotate immediately.
+- **Quorum**: With 2 MONs, both must be healthy. Consider adding a third MON.
+- **OSD Recovery**: OSDs will automatically recover once MON quorum is restored.
+
--- a/docs/ceph/ROOT_CAUSE_ANALYSIS.md
+++ b/docs/ceph/ROOT_CAUSE_ANALYSIS.md
@@ -0,0 +1,119 @@
+# Ceph Root Cause Analysis - R630-01
+
+**Date**: 2025-12-13  
+**Status**: 🔴 **CRITICAL - Hardware Failure**
+
+## What's Actually Broken
+
+### 1. Ceph Monitor (ceph-mon@r630-01.service) - **FAILED**
+- **Status**: `failed (Result: signal)` - Process aborted
+- **Root Cause**: Corrupted RocksDB database
+- **Error**: `Input/output error` reading `/var/lib/ceph/mon/ceph-r630-01/store.db/001572.sst`
+- **Stack Trace**: `RocksDBStore::get()` → `MonitorDBStore::get()` → `PaxosService::refresh()` → `Monitor::preinit()` → **ABORT**
+
+### 2. Ceph OSD (ceph-osd@1.service) - **FAILED**
+- **Status**: `failed (Result: exit-code)` 
+- **Root Cause**: Cannot fetch monitor configuration
+- **Error**: `failed to fetch mon config (--no-mon-config to skip)`
+- **Reason**: OSD requires a working MON to start, but MON is down
+
+### 3. No Ceph Listeners
+- **Port 6789** (classic msgr): No listeners
+- **Port 3300** (msgr2): No listeners
+- **Result**: Cluster is completely unreachable
+
+## Underlying Hardware Issue
+
+### Failing Disk: `/dev/sda` on R630-01
+
+**dmesg Output Shows Critical Medium Errors**:
+```
+critical medium error, dev sda, sector 22995583 op 0x0:(READ) flags 0x0
+critical medium error, dev sda, sector 22922714 op 0x0:(READ) flags 0x0
+Sense Key: Medium Error [current]
+Add. Sense: Unrecovered read error
+```
+
+**Impact**:
+- The Ceph monitor's RocksDB database (`/var/lib/ceph/mon/ceph-r630-01/store.db/`) is stored on `/dev/sda`
+- Disk I/O errors are causing database corruption
+- The corrupted SST file (`001572.sst`) cannot be read, causing the MON to abort
+
+## Configuration Status
+
+### Ceph Configuration Files
+- `/etc/ceph/ceph.conf`: ✅ Valid
+- `/etc/pve/ceph.conf`: ✅ Valid (matches)
+- **FSID**: `5fb968ae-12ab-405f-b05f-0df29a168328`
+- **MON hosts**: `192.168.11.10` (ml110-01), `192.168.11.11` (r630-01)
+- **Network**: `192.168.11.0/24` (public and cluster)
+
+### Monitor Data Directory
+- **Location**: `/var/lib/ceph/mon/ceph-r630-01/`
+- **Status**: Directory exists, but database is corrupted
+- **Disk Space**: ✅ Sufficient (69G available on `/dev/mapper/pve-root`)
+- **Inodes**: ✅ Sufficient (5.1M available)
+
+## Recovery Options
+
+### Option 1: Recover MON from Backup (Safest)
+If you have a backup of `/var/lib/ceph/mon/ceph-r630-01/`:
+1. Stop the MON service: `systemctl stop ceph-mon@r630-01`
+2. Restore from backup
+3. Start the MON: `systemctl start ceph-mon@r630-01`
+
+### Option 2: Recreate MON on R630-01 (Requires Quorum)
+If ml110-01 MON is still healthy:
+1. **First**: Fix or replace the failing disk `/dev/sda`
+2. Remove the corrupted MON: `pveceph mon destroy r630-01`
+3. Recreate the MON: `pveceph mon create r630-01`
+4. The MON will sync from ml110-01
+
+### Option 3: Remove Corrupted SST File (Risky)
+**⚠️ WARNING**: This may cause data loss. Only attempt if you have backups.
+1. Stop the MON: `systemctl stop ceph-mon@r630-01`
+2. Backup the store.db directory: `mv /var/lib/ceph/mon/ceph-r630-01/store.db /var/lib/ceph/mon/ceph-r630-01/store.db.corrupted`
+3. Try to recover using Ceph tools (requires working MON on ml110-01)
+4. Or recreate the MON entirely
+
+## Immediate Actions Required
+
+### 1. **URGENT**: Address Hardware Failure
+- **Check disk health**: Run `smartctl -a /dev/sda` on r630-01
+- **Check filesystem**: Run `fsck` on the filesystem containing `/var/lib/ceph/mon/`
+- **Plan disk replacement**: If disk is failing, replace it ASAP
+
+### 2. **Verify ML110-01 MON Status**
+Check if the other monitor is healthy:
+```bash
+ssh root@192.168.11.10 'systemctl status ceph-mon@ml110-01'
+ssh root@192.168.11.10 'ceph -s'
+```
+
+### 3. **Backup Current State**
+Before attempting recovery:
+```bash
+# Backup MON directory
+tar -czf /root/ceph-mon-r630-01-backup-$(date +%Y%m%d).tar.gz /var/lib/ceph/mon/ceph-r630-01/
+
+# Backup Ceph configuration
+cp -a /etc/ceph/ /root/ceph-config-backup-$(date +%Y%m%d)/
+cp -a /etc/pve/ceph.conf /root/ceph-config-backup-$(date +%Y%m%d)/
+```
+
+## Next Steps
+
+1. **Verify ML110-01 MON is healthy** (determines recovery path)
+2. **Check disk health** on r630-01 `/dev/sda`
+3. **Decide recovery strategy** based on:
+   - Whether ml110-01 MON is working
+   - Whether you have backups
+   - Whether disk can be fixed or needs replacement
+4. **Execute recovery** following one of the options above
+
+## Notes
+
+- **Password Security**: The root password was exposed in command history. Please rotate it immediately.
+- **Cluster State**: With only 2 MONs, losing one breaks quorum. If ml110-01 MON is also down, the cluster is unrecoverable without manual intervention.
+- **OSD Status**: OSDs will remain down until MON is restored, as they require MON for configuration.
+
--- a/docs/ceph/ROOT_CAUSE_IDENTIFIED.md
+++ b/docs/ceph/ROOT_CAUSE_IDENTIFIED.md
@@ -0,0 +1,111 @@
+# Root Cause Identified - Ceph Services Failed
+
+**Date**: 2025-12-13  
+**Status**: ✅ **ROOT CAUSE FOUND**
+
+---
+
+## Key Findings
+
+### ✅ Good News
+- **Bootstrap keyring EXISTS**: `/var/lib/ceph/bootstrap-osd/ceph.keyring` ✓
+- **pveceph available**: Proxmox Ceph tool is installed ✓
+- **Ceph config files exist**: Both `/etc/ceph/ceph.conf` and `/etc/pve/ceph.conf` ✓
+- **Ceph target active**: Systemd target is active ✓
+
+### ❌ Critical Issues
+- **Ceph monitor service FAILED**: `ceph-mon@r630-01.service` - **FAILED**
+- **Ceph OSD service FAILED**: `ceph-osd@1.service` - **FAILED**
+- **No monitor listening**: Port 6789 has no listeners
+- **Cluster not accessible**: Timeout when trying to connect
+
+---
+
+## Root Cause
+
+**Ceph services are FAILED on R630-01**. This is why:
+- OSD creation hangs (can't connect to cluster)
+- Commands timeout (no monitors running)
+- Cannot authenticate (services not running)
+
+---
+
+## Solution
+
+### Step 1: Check Service Status and Logs
+
+```bash
+# Check why services failed
+systemctl status ceph-mon@r630-01.service
+systemctl status ceph-osd@1.service
+
+# Check logs
+journalctl -u ceph-mon@r630-01.service -n 50
+journalctl -u ceph-osd@1.service -n 50
+```
+
+### Step 2: Fix and Start Services
+
+```bash
+# Try to start monitor
+systemctl start ceph-mon@r630-01.service
+
+# Try to start OSD
+systemctl start ceph-osd@1.service
+
+# Check status
+systemctl status ceph-mon@r630-01.service
+systemctl status ceph-osd@1.service
+```
+
+### Step 3: Verify Cluster Accessibility
+
+```bash
+# Test cluster connectivity
+ceph health
+ceph mon stat
+ceph osd tree
+```
+
+### Step 4: Retry OSD Creation
+
+Once services are running:
+
+```bash
+# Option A: Use pveceph (recommended for Proxmox)
+for drive in sdc sdd sde sdf sdg sdh; do
+    pveceph create /dev/$drive
+done
+
+# Option B: Use ceph-volume (bootstrap keyring exists)
+for drive in sdc sdd sde sdf sdg sdh; do
+    ceph-volume lvm create --data /dev/$drive
+done
+```
+
+---
+
+## Why Services Failed
+
+Common reasons:
+1. **Configuration issues** - Corrupted config
+2. **Disk issues** - OSD disk problems
+3. **Cluster quorum** - Lost quorum
+4. **Network issues** - Cannot reach other nodes
+5. **Permission issues** - Keyring permissions
+
+---
+
+## Next Steps
+
+1. **Check service logs** to understand why they failed
+2. **Fix the issues** preventing services from starting
+3. **Start services** and verify they stay running
+4. **Test cluster connectivity**
+5. **Create OSDs** on the 6x 250GB drives
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: ✅ **ROOT CAUSE IDENTIFIED - FIX SERVICES FIRST**
+
--- a/docs/ceph/VERIFY_250GB_DRIVES.md
+++ b/docs/ceph/VERIFY_250GB_DRIVES.md
@@ -0,0 +1,216 @@
+# Verify 250GB Drives on R630-01
+
+**Date**: 2025-12-13  
+**Purpose**: Verify status of 6x 250GB drives on R630-01 for Ceph OSD creation
+
+---
+
+## Quick Start
+
+### Option 1: Run Verification Script (Recommended)
+
+```bash
+# Copy script to R630-01
+scp scripts/verify-r630-250gb-drives.sh root@192.168.11.11:/tmp/
+
+# SSH to R630-01
+ssh root@192.168.11.11
+
+# Run verification script
+bash /tmp/verify-r630-250gb-drives.sh > /tmp/disk-verification.log 2>&1
+
+# Review results
+cat /tmp/disk-verification.log
+```
+
+### Option 2: Manual Verification Commands
+
+SSH to R630-01 and run these commands:
+
+```bash
+# 1. List all block devices
+lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL
+
+# 2. List all physical disks
+fdisk -l | grep -E "^Disk /dev/sd"
+
+# 3. Find 250GB drives (232-260 GB range)
+for disk in /dev/sd[a-z]; do
+    if [ -b "$disk" ]; then
+        size=$(fdisk -l "$disk" 2>/dev/null | grep "^Disk $disk" | awk '{print $3, $4}')
+        echo "$disk: $size"
+    fi
+done
+
+# 4. Check current Ceph OSDs
+ceph osd tree
+ceph osd df
+
+# 5. Check storage pools
+pvesm status
+
+# 6. Check volume groups
+vgs
+pvs
+```
+
+---
+
+## What to Look For
+
+### Expected Findings
+
+1. **6x 250GB drives** should be visible in `lsblk` or `fdisk -l`
+2. **Drive sizes** should be around 232-260 GB (250GB drives)
+3. **Available drives** should have:
+   - ✅ No mount point
+   - ✅ No partitions (or empty partition table)
+   - ✅ Not in any volume group
+   - ✅ Not used by Ceph
+
+### Drive Identification
+
+Look for drives like:
+```
+sdc    250G  disk        # Available!
+sdd    250G  disk        # Available!
+sde    250GB disk        # Available!
+...
+```
+
+### Current Known Disks
+
+- **sda**: 279.4 GB (300GB) - Proxmox installation
+- **sdb**: 279.4 GB (300GB) - Ceph OSD
+- **sdc-sdh**: Should be 6x 250GB drives (to be verified)
+
+---
+
+## Verification Checklist
+
+After running the script, check:
+
+- [ ] Are 6x 250GB drives detected?
+- [ ] What are their device names? (sdc, sdd, sde, etc.)
+- [ ] Are they initialized? (do they show up in fdisk?)
+- [ ] Do they have partitions?
+- [ ] Are they mounted?
+- [ ] Are they in any volume group?
+- [ ] Are they used by Ceph?
+- [ ] Which one(s) are available for OSD creation?
+
+---
+
+## Expected Output Examples
+
+### Good: Available Drive
+```
+sdc    250G  disk        # No mount, no partitions, not in VG
+```
+
+### Warning: Drive in Use
+```
+sdc    250G  disk
+└─sdc1 250G  part  /mnt/something  ext4  # Has partition and mount
+```
+
+### Warning: Drive in Volume Group
+```
+sdc    250G  disk
+└─sdc1 250G  part        LVM2_member  # In LVM volume group
+```
+
+---
+
+## Next Steps After Verification
+
+### If Drives Are Available
+
+1. **Choose one drive** (e.g., sdc)
+2. **Verify it's safe** (no data, not in use)
+3. **Create Ceph OSD**:
+   ```bash
+   ceph-volume lvm create --data /dev/sdc
+   ```
+4. **Verify OSD created**:
+   ```bash
+   ceph osd tree
+   ceph health
+   ```
+
+### If Drives Are Not Detected
+
+1. **Check PERC controller** (if using hardware RAID):
+   ```bash
+   omreport storage pdisk
+   ```
+2. **Check system logs**:
+   ```bash
+   dmesg | grep -i sd
+   journalctl -k | grep -i disk
+   ```
+3. **Physical inspection** - Verify drives are installed
+
+### If Drives Are in Use
+
+1. **If in RAID**: Check PERC controller configuration
+2. **If in LVM**: Check if they're part of a storage pool
+3. **If mounted**: Check what's using them
+4. **Decide**: Can they be safely repurposed?
+
+---
+
+## Troubleshooting
+
+### Issue: Drives Not Showing Up
+
+**Possible causes**:
+- Drives not physically installed
+- PERC controller not configured
+- Drives need to be initialized
+- System needs reboot to detect drives
+
+**Solutions**:
+1. Check physical installation
+2. Check PERC controller: `omreport storage pdisk`
+3. Initialize drives in PERC if needed
+4. Reboot system if drives were just installed
+
+### Issue: Drives Show But Can't Use
+
+**Possible causes**:
+- Drives in hardware RAID
+- Drives have partitions
+- Drives in volume group
+
+**Solutions**:
+1. Check PERC RAID configuration
+2. Wipe partitions if safe: `wipefs -a /dev/sdX`
+3. Remove from volume group if safe: `vgremove <vg-name>`
+
+---
+
+## Related Documentation
+
+- [Add Third OSD Guide](./ADD_THIRD_OSD_GUIDE.md) - How to create OSD after verification
+- [Disk Inventory](../infrastructure/DISK_INVENTORY.md) - Complete disk inventory
+- [Critical Ceph Issues](./CRITICAL_CEPH_ISSUES.md) - Ceph issues to fix
+
+---
+
+## Summary
+
+### Goal
+Verify that 6x 250GB drives exist and identify which ones are available for Ceph OSD creation.
+
+### Expected Result
+At least one 250GB drive available and ready to use for creating the third Ceph OSD.
+
+### After Verification
+- If available: Create OSD immediately
+- If not available: Investigate why and fix
+- If partially available: Use what's available
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔍 **READY TO VERIFY**
+
--- a/docs/ceph/WIPE_ALL_DRIVES.md
+++ b/docs/ceph/WIPE_ALL_DRIVES.md
@@ -0,0 +1,216 @@
+# Wipe All 6x 250GB Drives
+
+**Date**: 2025-12-13  
+**Status**: ✅ **READY TO EXECUTE**
+
+---
+
+## Overview
+
+Wipe all 6x 250GB drives (sdc, sdd, sde, sdf, sdg, sdh) to prepare them for Ceph OSDs.
+
+**WARNING**: This will destroy ALL data on these drives!
+
+---
+
+## Option 1: Wipe Only (Then Create OSDs Manually)
+
+### Script: `wipe-all-250gb-drives.sh`
+
+This script will:
+1. Remove drives from ubuntu-vg
+2. Wipe all 6 drives
+3. Leave them ready for OSD creation
+
+**Usage**:
+```bash
+# Copy to R630-01
+scp scripts/wipe-all-250gb-drives.sh root@192.168.11.11:/tmp/
+
+# SSH and run
+ssh root@192.168.11.11
+bash /tmp/wipe-all-250gb-drives.sh
+```
+
+**After wiping**, create OSDs manually:
+```bash
+# Create third OSD (minimum needed)
+ceph-volume lvm create --data /dev/sdc
+
+# Or create OSDs on all drives (for better performance)
+ceph-volume lvm create --data /dev/sdc
+ceph-volume lvm create --data /dev/sdd
+ceph-volume lvm create --data /dev/sde
+ceph-volume lvm create --data /dev/sdf
+ceph-volume lvm create --data /dev/sdg
+ceph-volume lvm create --data /dev/sdh
+```
+
+---
+
+## Option 2: Wipe AND Create OSDs (Automated)
+
+### Script: `wipe-and-create-osds.sh`
+
+This script will:
+1. Remove drives from ubuntu-vg
+2. Wipe all 6 drives
+3. Create Ceph OSDs on all 6 drives automatically
+
+**Usage**:
+```bash
+# Copy to R630-01
+scp scripts/wipe-and-create-osds.sh root@192.168.11.11:/tmp/
+
+# SSH and run
+ssh root@192.168.11.11
+bash /tmp/wipe-and-create-osds.sh
+```
+
+**This is the recommended option** - it does everything in one go!
+
+---
+
+## Manual Method
+
+If you prefer to do it manually:
+
+```bash
+# 1. Remove from ubuntu-vg
+umount /dev/ubuntu-vg/* 2>/dev/null || true
+vgchange -a n ubuntu-vg
+for drive in sdc sdd sde sdf sdg sdh; do
+    pvremove /dev/${drive}3 -y -ff 2>/dev/null || true
+done
+
+# 2. Wipe all drives
+for drive in sdc sdd sde sdf sdg sdh; do
+    umount /dev/${drive}* 2>/dev/null || true
+    wipefs -a /dev/$drive
+    dd if=/dev/zero of=/dev/$drive bs=1M count=100
+done
+
+# 3. Create OSDs
+for drive in sdc sdd sde sdf sdg sdh; do
+    ceph-volume lvm create --data /dev/$drive
+done
+
+# 4. Verify
+ceph osd tree
+ceph health
+```
+
+---
+
+## Expected Results
+
+### Before
+- 2 OSDs (insufficient)
+- HEALTH_WARN: TOO_FEW_OSDS
+
+### After (with all 6 OSDs)
+- **8 OSDs total** (2 existing + 6 new)
+- Excellent redundancy and performance
+- HEALTH_OK or much improved
+- Can handle many more VMs
+
+### After (with just 1 OSD)
+- **3 OSDs total** (minimum for 3-way replication)
+- Fixes TOO_FEW_OSDS error
+- HEALTH_OK or improved
+- VMs can be created
+
+---
+
+## Recommendations
+
+### Minimum (Fix Current Issue)
+- Create OSD on **1 drive** (e.g., sdc)
+- Fixes TOO_FEW_OSDS error
+- Allows VM creation
+
+### Recommended (Better Performance)
+- Create OSDs on **all 6 drives**
+- Much better performance
+- Better data distribution
+- More redundancy
+
+### Best Practice
+- Use all available drives for OSDs
+- Better utilization of hardware
+- Improved Ceph performance
+
+---
+
+## Safety Checklist
+
+Before running:
+
+- [ ] **Backup important data** (if ubuntu-vg contains data)
+- [ ] **Verify ubuntu-vg is not critical** for system operation
+- [ ] **Confirm all 6 drives** are the ones you want to wipe
+- [ ] **Have recovery plan** if something goes wrong
+- [ ] **Type 'yes' to confirm** when script asks
+
+---
+
+## Troubleshooting
+
+### Issue: pvremove fails
+
+**Solution**:
+```bash
+# Force remove
+vgchange -a n ubuntu-vg
+pvremove /dev/sdX3 -y -ff
+```
+
+### Issue: wipefs fails
+
+**Solution**:
+```bash
+# Unmount first
+umount /dev/sdX* 2>/dev/null || true
+# Then wipe
+wipefs -a /dev/sdX
+```
+
+### Issue: OSD creation fails
+
+**Solution**:
+```bash
+# Make sure drive is completely wiped
+wipefs -a /dev/sdX
+dd if=/dev/zero of=/dev/sdX bs=1M count=100
+
+# Check Ceph status
+ceph health
+ceph osd tree
+
+# Try again
+ceph-volume lvm create --data /dev/sdX
+```
+
+---
+
+## Summary
+
+### Current State
+- 6x 250GB drives in ubuntu-vg
+- Need to be wiped and prepared for Ceph
+
+### Action
+- Run `wipe-and-create-osds.sh` (recommended)
+- Or run `wipe-all-250gb-drives.sh` then create OSDs manually
+
+### Expected Result
+- 8 OSDs total (if all 6 created)
+- Or 3 OSDs minimum (if just 1 created)
+- Ceph health improves significantly
+- VMs can be created with ceph-fs storage
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: ✅ **READY TO EXECUTE**
+