Sankofa/docs/ceph/VERIFY_250GB_DRIVES.md

# Verify 250GB Drives on R630-01

**Date**: 2025-12-13  
**Purpose**: Verify status of 6x 250GB drives on R630-01 for Ceph OSD creation

---

## Quick Start

### Option 1: Run Verification Script (Recommended)

```bash
# Copy script to R630-01
scp scripts/verify-r630-250gb-drives.sh root@192.168.11.11:/tmp/

# SSH to R630-01
ssh root@192.168.11.11

# Run verification script
bash /tmp/verify-r630-250gb-drives.sh > /tmp/disk-verification.log 2>&1

# Review results
cat /tmp/disk-verification.log
```

### Option 2: Manual Verification Commands

SSH to R630-01 and run these commands:

```bash
# 1. List all block devices
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL

# 2. List all physical disks
fdisk -l | grep -E "^Disk /dev/sd"

# 3. Find 250GB drives (232-260 GB range)
for disk in /dev/sd[a-z]; do
    if [ -b "$disk" ]; then
        size=$(fdisk -l "$disk" 2>/dev/null | grep "^Disk $disk" | awk '{print $3, $4}')
        echo "$disk: $size"
    fi
done

# 4. Check current Ceph OSDs
ceph osd tree
ceph osd df

# 5. Check storage pools
pvesm status

# 6. Check volume groups
vgs
pvs
```

---

## What to Look For

### Expected Findings

1. **6x 250GB drives** should be visible in `lsblk` or `fdisk -l`
2. **Drive sizes** should be around 232-260 GB (250GB drives)
3. **Available drives** should have:
   - ✅ No mount point
   - ✅ No partitions (or empty partition table)
   - ✅ Not in any volume group
   - ✅ Not used by Ceph

### Drive Identification

Look for drives like:
```
sdc    250G  disk        # Available!
sdd    250G  disk        # Available!
sde    250GB disk        # Available!
...
```

### Current Known Disks

- **sda**: 279.4 GB (300GB) - Proxmox installation
- **sdb**: 279.4 GB (300GB) - Ceph OSD
- **sdc-sdh**: Should be 6x 250GB drives (to be verified)

---

## Verification Checklist

After running the script, check:

- [ ] Are 6x 250GB drives detected?
- [ ] What are their device names? (sdc, sdd, sde, etc.)
- [ ] Are they initialized? (do they show up in fdisk?)
- [ ] Do they have partitions?
- [ ] Are they mounted?
- [ ] Are they in any volume group?
- [ ] Are they used by Ceph?
- [ ] Which one(s) are available for OSD creation?

---

## Expected Output Examples

### Good: Available Drive
```
sdc    250G  disk        # No mount, no partitions, not in VG
```

### Warning: Drive in Use
```
sdc    250G  disk
└─sdc1 250G  part  /mnt/something  ext4  # Has partition and mount
```

### Warning: Drive in Volume Group
```
sdc    250G  disk
└─sdc1 250G  part        LVM2_member  # In LVM volume group
```

---

## Next Steps After Verification

### If Drives Are Available

1. **Choose one drive** (e.g., sdc)
2. **Verify it's safe** (no data, not in use)
3. **Create Ceph OSD**:
   ```bash
   ceph-volume lvm create --data /dev/sdc
   ```
4. **Verify OSD created**:
   ```bash
   ceph osd tree
   ceph health
   ```

### If Drives Are Not Detected

1. **Check PERC controller** (if using hardware RAID):
   ```bash
   omreport storage pdisk
   ```
2. **Check system logs**:
   ```bash
   dmesg | grep -i sd
   journalctl -k | grep -i disk
   ```
3. **Physical inspection** - Verify drives are installed

### If Drives Are in Use

1. **If in RAID**: Check PERC controller configuration
2. **If in LVM**: Check if they're part of a storage pool
3. **If mounted**: Check what's using them
4. **Decide**: Can they be safely repurposed?

---

## Troubleshooting

### Issue: Drives Not Showing Up

**Possible causes**:
- Drives not physically installed
- PERC controller not configured
- Drives need to be initialized
- System needs reboot to detect drives

**Solutions**:
1. Check physical installation
2. Check PERC controller: `omreport storage pdisk`
3. Initialize drives in PERC if needed
4. Reboot system if drives were just installed

### Issue: Drives Show But Can't Use

**Possible causes**:
- Drives in hardware RAID
- Drives have partitions
- Drives in volume group

**Solutions**:
1. Check PERC RAID configuration
2. Wipe partitions if safe: `wipefs -a /dev/sdX`
3. Remove from volume group if safe: `vgremove <vg-name>`

---

## Related Documentation

- [Add Third OSD Guide](./ADD_THIRD_OSD_GUIDE.md) - How to create OSD after verification
- [Disk Inventory](../infrastructure/DISK_INVENTORY.md) - Complete disk inventory
- [Critical Ceph Issues](./CRITICAL_CEPH_ISSUES.md) - Ceph issues to fix

---

## Summary

### Goal
Verify that 6x 250GB drives exist and identify which ones are available for Ceph OSD creation.

### Expected Result
At least one 250GB drive available and ready to use for creating the third Ceph OSD.

### After Verification
- If available: Create OSD immediately
- If not available: Investigate why and fix
- If partially available: Use what's available

**Last Updated**: 2025-12-13  
**Status**: 🔍 **READY TO VERIFY**