Sankofa/docs/infrastructure/DISK_INVENTORY.md

# Physical Disk Inventory - ML110-01 and R630-01

**Date**: 2025-12-13  
**Purpose**: Document physical disk configuration for Ceph OSD planning

---

## Summary

Both nodes currently have **2 physical disks each**, with one disk used for Proxmox installation and one disk used for Ceph OSD. This results in **2 OSDs total**, which is insufficient for 3-way replication.

**IMPORTANT UPDATE**: R630-01 actually has **8 total drive bays** with **6 additional 250GB drives** that may be available for use.

### Current Ceph Configuration
- **OSD Count**: 2 (one per node)
- **Required for 3-way replication**: 3 OSDs minimum
- **Status**: 🔴 **INSUFFICIENT** - Cannot maintain 3-way replication
- **Available Disks**: 6x 250GB drives on R630-01 (potential for additional OSDs)

---

## ML110-01 (HPE ML110 Gen9)

**IP Address**: 192.168.11.10  
**Hostname**: ml110-01  
**Form Factor**: Tower Server

### Physical Disks

| Disk | Model | Size | Type | Usage | Status |
|------|-------|------|------|-------|--------|
| **sda** | Seagate ST1000DM003-1ER162 | 931.5 GB | SATA HDD | Proxmox Installation (LVM) | ✅ Active |
| **sdb** | Seagate ST1000DM003-1ER162 | 931.5 GB | SATA HDD | Ceph OSD | ✅ Active |

### Storage Configuration

**Primary Disk (sda)**:
- **Partition Layout**:
  - `pve-swap`: 8 GB
  - `pve-root`: 96 GB
  - `pve-data`: 794.3 GB (for VMs using local-lvm)
- **Storage Pool**: `local-lvm` (794.3 GB available)

**Secondary Disk (sdb)**:
- **Usage**: Ceph OSD block device
- **Storage Pool**: `ceph-fs` (384 GB available)
- **Status**: Active Ceph OSD

### Storage Controller
- **Type**: Intel C610/X99 series chipset 6-Port SATA Controller (AHCI mode)
- **RAID**: Software-based (LVM, Ceph) - no hardware RAID controller

### Available Drive Bays
- **Form Factor**: Tower server
- **Drive Bays**: [To be determined - likely 2-4 internal bays]
- **Expansion**: Limited by tower form factor

### Recommendations
- ✅ **Current Configuration**: Adequate for current setup
- ⚠️ **Ceph Expansion**: Limited expansion options (tower form factor)
- 💡 **Option**: Add external storage or use network storage for third OSD

---

## R630-01 (Dell PowerEdge R630 High Memory)

**IP Address**: 192.168.11.11  
**Hostname**: r630-01  
**Form Factor**: 1U Rack Server  
**Serial Number**: HNQ3FB2

### Physical Disks (8 Total Bays)

| Bay | Disk | Model | Size | Type | Usage | Status |
|-----|------|-------|------|------|-------|--------|
| **1** | sda | Seagate ST9300653SS | 279.4 GB | SAS SSD | Proxmox Installation (LVM) | ✅ Active |
| **2** | sdb | HUC106030CSS600 | 279.4 GB | SAS SSD | Ceph OSD | ✅ Active |
| **3** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
| **4** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
| **5** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
| **6** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
| **7** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
| **8** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |

**Note**: The 6x 250GB drives need to be verified - they may be:
- Not detected by the OS
- Not initialized/formatted
- Used for something else (RAID, backup, etc.)
- Available for Ceph OSD creation

### Storage Configuration

**Primary Disk (sda - Bay 1)**:
- **Partition Layout**:
  - `pve-swap`: 8 GB
  - `pve-root`: 79.6 GB
  - `pve-data`: 171.3 GB (for VMs using local-lvm)
- **Storage Pool**: `local-lvm` (171.3 GB available)

**Secondary Disk (sdb - Bay 2)**:
- **Usage**: Ceph OSD block device
- **Storage Pool**: Ceph OSD
- **Status**: Active Ceph OSD

**6x 250GB Drives (Bays 3-8)**:
- **Status**: ⚠️ **NEEDS VERIFICATION**
- **Potential Usage**: 
  - Available for additional Ceph OSDs (ideal!)
  - Could create 3rd OSD immediately
  - Could create more OSDs for better performance

### Storage Controller
- **Type**: Dell PERC H730 Mini (LSI MegaRAID SAS-3 3108 [Invader])
- **RAID**: Hardware RAID controller available
- **Mode**: Currently using individual disks (not in RAID)

### Available Drive Bays
- **Form Factor**: 1U Rack Server
- **Total Drive Bays**: **8x 2.5" hot-swappable drive bays**
- **Currently Used**: 2 bays (sda, sdb)
- **Additional Drives**: **6x 250GB drives** (status unknown)
- **Storage Options**: SATA, SAS, NVMe (with riser card)

### Recommendations
- ✅ **Excellent Expansion Potential**: 6 additional drives available
- ✅ **Hot-Swappable**: Can add/configure drives without downtime
- 🔍 **URGENT**: Verify status of 6x 250GB drives
- 💡 **Best Option**: Use one of the 250GB drives for third OSD
- 💡 **Future Expansion**: Can use remaining 5 drives for more OSDs

---

## Ceph OSD Status

### Current OSDs
1. **OSD 0**: ML110-01 (sdb) - 931.5 GB
2. **OSD 1**: R630-01 (sdb) - 279.4 GB

### Issues
- 🔴 **Only 2 OSDs** - Cannot maintain 3-way replication
- 🔴 **Mismatched Sizes** - 931.5 GB vs 279.4 GB (suboptimal)
- 🔴 **Insufficient for Production** - Need minimum 3 OSDs

### Potential Additional OSDs
- ✅ **6x 250GB drives available** on R630-01
- 💡 **Can create 3rd OSD immediately** (if drives are usable)
- 💡 **Can create more OSDs** for better performance and redundancy

---

## Recommendations for Adding Third OSD

### Option 1: Use Existing 250GB Drive on R630-01 (BEST) ✅

**Advantages**:
- ✅ **No hardware purchase needed** - Drive already installed
- ✅ **Hot-swappable** - Can configure without downtime
- ✅ **Immediate solution** - Can create OSD right away
- ✅ **Size appropriate** - 250GB is close to existing 279.4GB OSD

**Steps**:
1. **Verify drive is detected**:
   ```bash
   ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL"
   ```

2. **Check if drive is initialized**:
   ```bash
   ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"
   ```

3. **Create Ceph OSD** (if drive is available):
   ```bash
   # On R630-01
   ceph-volume lvm create --data /dev/sdX  # Replace sdX with actual device
   ```

4. **Verify OSD joins cluster**:
   ```bash
   ceph osd tree
   ceph health
   ```

### Option 2: Add New Disk to R630-01

**Advantages**:
- ✅ **8 empty drive bays** available (if 250GB drives not usable)
- ✅ **Hot-swappable** - can add without downtime
- ✅ **Better performance** - Can choose SSD

**Disadvantages**:
- ⚠️ Requires hardware purchase
- ⚠️ Takes time to procure and install

### Option 3: Reduce Replication Factor (TEMPORARY WORKAROUND)

**Action**: Change replication from 3 to 2

**Advantages**:
- ✅ **Immediate** - No hardware needed
- ✅ **Allows VM creation** - Unblocks 21 VMs

**Disadvantages**:
- ⚠️ **Increased Risk** - Data loss if 1 OSD fails
- ⚠️ **Not Production-Ready** - Should be temporary only
- ⚠️ **Requires Monitoring** - Need to watch for failures

**Steps**:
```bash
# Reduce replication factor
ceph osd pool set rbd size 2
ceph osd pool set rbd min_size 1

# Verify
ceph osd pool get rbd size
ceph osd pool get rbd min_size
```

---

## Disk Size Considerations

### Current Mismatch
- **ML110-01 OSD**: 931.5 GB
- **R630-01 OSD (current)**: 279.4 GB
- **R630-01 OSD (potential)**: 250 GB
- **Difference**: Significant size mismatch

### Impact
- ⚠️ **Uneven Data Distribution** - More data on larger OSD
- ⚠️ **Suboptimal Performance** - Larger OSD handles more load
- ⚠️ **Wasted Space** - Cannot use full capacity of larger OSD

### Recommendations
1. **For Third OSD**: Use 250GB drive (close to existing 279.4GB)
2. **For Future**: Consider replacing smaller OSDs with larger ones
3. **For Production**: Use similar-sized disks for better balance
4. **For Optimal Setup**: Consider using multiple 250GB drives for more OSDs

---

## Next Steps

### Immediate (To Fix Ceph Issues)

1. **URGENT - Verify 250GB Drives**:
   ```bash
   # Check if drives are detected
   ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL"
   
   # Check all block devices
   ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"
   
   # Check Ceph OSD status
   ssh root@192.168.11.11 "ceph osd tree"
   ```

2. **Option A - Use Existing 250GB Drive** (BEST):
   - Verify drive is detected and usable
   - Create Ceph OSD on one of the 250GB drives
   - Verify cluster health improves

3. **Option B - Reduce Replication** (TEMPORARY):
   - Change replication from 3 to 2
   - Monitor for failures
   - Plan to add third OSD using 250GB drive

### Short-term (This Week)

1. Verify and configure 250GB drives
2. Create third OSD on R630-01
3. Verify Ceph cluster health improves
4. Monitor OSD performance
5. Consider adding more OSDs from remaining 250GB drives

### Long-term (This Month)

1. Add more OSDs from remaining 250GB drives (better performance)
2. Balance OSD sizes for optimal distribution
3. Plan for storage expansion
4. Document storage architecture

---

## Verification Commands

### Check Current Disks on R630-01
```bash
# List all block devices
ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL"

# Check all disks
ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"

# Check for uninitialized drives
ssh root@192.168.11.11 "ls -la /dev/sd*"
```

### Check Ceph OSDs
```bash
# On either node
ceph osd tree
ceph osd df
ceph osd pool ls detail
ceph health detail
```

### Check Available Drive Bays
```bash
# Check via iDRAC/IPMI (if available)
# Or physical inspection
# Or check PERC controller status
ssh root@192.168.11.11 "omreport storage vdisk"  # If Dell tools installed
```

---

## Summary

### Current State
- **ML110-01**: 2 disks (1 Proxmox, 1 Ceph OSD)
- **R630-01**: 8 total bays
  - 2x 300GB drives (1 Proxmox, 1 Ceph OSD)
  - 6x 250GB drives (status unknown - needs verification)
- **Total OSDs**: 2 (insufficient for 3-way replication)

### Best Solution
✅ **Use one of the 6x 250GB drives on R630-01** - No hardware purchase needed, immediate solution

### Action Required
🔍 **URGENT**: Verify status of 6x 250GB drives and configure one for third OSD

### Temporary Workaround
⚠️ **Reduce replication to 2** - Allows VM creation but increases risk (use only if 250GB drives not usable)

**Last Updated**: 2025-12-13  
**Status**: 🔍 **VERIFY 250GB DRIVES - POTENTIAL IMMEDIATE SOLUTION AVAILABLE**