Sankofa/docs/infrastructure/DISK_INVENTORY.md

Physical Disk Inventory - ML110-01 and R630-01

Date: 2025-12-13
Purpose: Document physical disk configuration for Ceph OSD planning

---

Summary

Both nodes currently have 2 physical disks each, with one disk used for Proxmox installation and one disk used for Ceph OSD. This results in 2 OSDs total, which is insufficient for 3-way replication.

IMPORTANT UPDATE: R630-01 actually has 8 total drive bays with 6 additional 250GB drives that may be available for use.

Current Ceph Configuration

• OSD Count: 2 (one per node)
• Required for 3-way replication: 3 OSDs minimum
• Status: 🔴 INSUFFICIENT - Cannot maintain 3-way replication
• Available Disks: 6x 250GB drives on R630-01 (potential for additional OSDs)

---

ML110-01 (HPE ML110 Gen9)

IP Address: 192.168.11.10
Hostname: ml110-01
Form Factor: Tower Server

Physical Disks

Disk	Model	Size	Type	Usage	Status
sda	Seagate ST1000DM003-1ER162	931.5 GB	SATA HDD	Proxmox Installation (LVM)	✅ Active
sdb	Seagate ST1000DM003-1ER162	931.5 GB	SATA HDD	Ceph OSD	✅ Active

Storage Configuration

Primary Disk (sda):

• Partition Layout:
  • pve-swap: 8 GB
  • pve-root: 96 GB
  • pve-data: 794.3 GB (for VMs using local-lvm)
• Storage Pool: local-lvm (794.3 GB available)

Secondary Disk (sdb):

• Usage: Ceph OSD block device
• Storage Pool: ceph-fs (384 GB available)
• Status: Active Ceph OSD

Storage Controller

• Type: Intel C610/X99 series chipset 6-Port SATA Controller (AHCI mode)
• RAID: Software-based (LVM, Ceph) - no hardware RAID controller

Available Drive Bays

• Form Factor: Tower server
• Drive Bays: [To be determined - likely 2-4 internal bays]
• Expansion: Limited by tower form factor

Recommendations

• ✅ Current Configuration: Adequate for current setup
• ⚠️ Ceph Expansion: Limited expansion options (tower form factor)
• 💡 Option: Add external storage or use network storage for third OSD

---

R630-01 (Dell PowerEdge R630 High Memory)

IP Address: 192.168.11.11
Hostname: r630-01
Form Factor: 1U Rack Server
Serial Number: HNQ3FB2

Physical Disks (8 Total Bays)

Bay	Disk	Model	Size	Type	Usage	Status
1	sda	Seagate ST9300653SS	279.4 GB	SAS SSD	Proxmox Installation (LVM)	✅ Active
2	sdb	HUC106030CSS600	279.4 GB	SAS SSD	Ceph OSD	✅ Active
3	?	[Unknown]	250 GB	[Unknown]	UNUSED	⚠️ Available?
4	?	[Unknown]	250 GB	[Unknown]	UNUSED	⚠️ Available?
5	?	[Unknown]	250 GB	[Unknown]	UNUSED	⚠️ Available?
6	?	[Unknown]	250 GB	[Unknown]	UNUSED	⚠️ Available?
7	?	[Unknown]	250 GB	[Unknown]	UNUSED	⚠️ Available?
8	?	[Unknown]	250 GB	[Unknown]	UNUSED	⚠️ Available?

Note: The 6x 250GB drives need to be verified - they may be:

• Not detected by the OS
• Not initialized/formatted
• Used for something else (RAID, backup, etc.)
• Available for Ceph OSD creation

Storage Configuration

Primary Disk (sda - Bay 1):

• Partition Layout:
  • pve-swap: 8 GB
  • pve-root: 79.6 GB
  • pve-data: 171.3 GB (for VMs using local-lvm)
• Storage Pool: local-lvm (171.3 GB available)

Secondary Disk (sdb - Bay 2):

• Usage: Ceph OSD block device
• Storage Pool: Ceph OSD
• Status: Active Ceph OSD

6x 250GB Drives (Bays 3-8):

• Status: ⚠️ NEEDS VERIFICATION
• Potential Usage:
  • Available for additional Ceph OSDs (ideal!)
  • Could create 3rd OSD immediately
  • Could create more OSDs for better performance

Storage Controller

• Type: Dell PERC H730 Mini (LSI MegaRAID SAS-3 3108 [Invader])
• RAID: Hardware RAID controller available
• Mode: Currently using individual disks (not in RAID)

Available Drive Bays

• Form Factor: 1U Rack Server
• Total Drive Bays: 8x 2.5" hot-swappable drive bays
• Currently Used: 2 bays (sda, sdb)
• Additional Drives: 6x 250GB drives (status unknown)
• Storage Options: SATA, SAS, NVMe (with riser card)

Recommendations

• ✅ Excellent Expansion Potential: 6 additional drives available
• ✅ Hot-Swappable: Can add/configure drives without downtime
• 🔍 URGENT: Verify status of 6x 250GB drives
• 💡 Best Option: Use one of the 250GB drives for third OSD
• 💡 Future Expansion: Can use remaining 5 drives for more OSDs

---

Ceph OSD Status

Current OSDs

1. OSD 0: ML110-01 (sdb) - 931.5 GB
2. OSD 1: R630-01 (sdb) - 279.4 GB

Issues

• 🔴 Only 2 OSDs - Cannot maintain 3-way replication
• 🔴 Mismatched Sizes - 931.5 GB vs 279.4 GB (suboptimal)
• 🔴 Insufficient for Production - Need minimum 3 OSDs

Potential Additional OSDs

• ✅ 6x 250GB drives available on R630-01
• 💡 Can create 3rd OSD immediately (if drives are usable)
• 💡 Can create more OSDs for better performance and redundancy

---

Recommendations for Adding Third OSD

Option 1: Use Existing 250GB Drive on R630-01 (BEST) ✅

Advantages:

• ✅ No hardware purchase needed - Drive already installed
• ✅ Hot-swappable - Can configure without downtime
• ✅ Immediate solution - Can create OSD right away
• ✅ Size appropriate - 250GB is close to existing 279.4GB OSD

Steps:

1. Verify drive is detected:

   ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL"
   

2. Check if drive is initialized:

   ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"
   

3. Create Ceph OSD (if drive is available):

   # On R630-01
   ceph-volume lvm create --data /dev/sdX  # Replace sdX with actual device
   

4. Verify OSD joins cluster:

   ceph osd tree
   ceph health
   

Option 2: Add New Disk to R630-01

Advantages:

• ✅ 8 empty drive bays available (if 250GB drives not usable)
• ✅ Hot-swappable - can add without downtime
• ✅ Better performance - Can choose SSD

Disadvantages:

• ⚠️ Requires hardware purchase
• ⚠️ Takes time to procure and install

Option 3: Reduce Replication Factor (TEMPORARY WORKAROUND)

Action: Change replication from 3 to 2

Advantages:

• ✅ Immediate - No hardware needed
• ✅ Allows VM creation - Unblocks 21 VMs

Disadvantages:

• ⚠️ Increased Risk - Data loss if 1 OSD fails
• ⚠️ Not Production-Ready - Should be temporary only
• ⚠️ Requires Monitoring - Need to watch for failures

Steps:

# Reduce replication factor
ceph osd pool set rbd size 2
ceph osd pool set rbd min_size 1

# Verify
ceph osd pool get rbd size
ceph osd pool get rbd min_size

---

Disk Size Considerations

Current Mismatch

• ML110-01 OSD: 931.5 GB
• R630-01 OSD (current): 279.4 GB
• R630-01 OSD (potential): 250 GB
• Difference: Significant size mismatch

Impact

• ⚠️ Uneven Data Distribution - More data on larger OSD
• ⚠️ Suboptimal Performance - Larger OSD handles more load
• ⚠️ Wasted Space - Cannot use full capacity of larger OSD

Recommendations

1. For Third OSD: Use 250GB drive (close to existing 279.4GB)
2. For Future: Consider replacing smaller OSDs with larger ones
3. For Production: Use similar-sized disks for better balance
4. For Optimal Setup: Consider using multiple 250GB drives for more OSDs

---

Next Steps

Immediate (To Fix Ceph Issues)

1. URGENT - Verify 250GB Drives:

   # Check if drives are detected
   ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL"
   
   # Check all block devices
   ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"
   
   # Check Ceph OSD status
   ssh root@192.168.11.11 "ceph osd tree"
   

2. Option A - Use Existing 250GB Drive (BEST):

   • Verify drive is detected and usable
   • Create Ceph OSD on one of the 250GB drives
   • Verify cluster health improves

3. Option B - Reduce Replication (TEMPORARY):

   • Change replication from 3 to 2
   • Monitor for failures
   • Plan to add third OSD using 250GB drive

Short-term (This Week)

1. Verify and configure 250GB drives
2. Create third OSD on R630-01
3. Verify Ceph cluster health improves
4. Monitor OSD performance
5. Consider adding more OSDs from remaining 250GB drives

Long-term (This Month)

1. Add more OSDs from remaining 250GB drives (better performance)
2. Balance OSD sizes for optimal distribution
3. Plan for storage expansion
4. Document storage architecture

---

Verification Commands

Check Current Disks on R630-01

# List all block devices
ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL"

# Check all disks
ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"

# Check for uninitialized drives
ssh root@192.168.11.11 "ls -la /dev/sd*"

Check Ceph OSDs

# On either node
ceph osd tree
ceph osd df
ceph osd pool ls detail
ceph health detail

Check Available Drive Bays

# Check via iDRAC/IPMI (if available)
# Or physical inspection
# Or check PERC controller status
ssh root@192.168.11.11 "omreport storage vdisk"  # If Dell tools installed

---

Summary

Current State

• ML110-01: 2 disks (1 Proxmox, 1 Ceph OSD)
• R630-01: 8 total bays
  • 2x 300GB drives (1 Proxmox, 1 Ceph OSD)
  • 6x 250GB drives (status unknown - needs verification)
• Total OSDs: 2 (insufficient for 3-way replication)

Best Solution

✅ Use one of the 6x 250GB drives on R630-01 - No hardware purchase needed, immediate solution

Action Required

🔍 URGENT: Verify status of 6x 250GB drives and configure one for third OSD

Temporary Workaround

⚠️ Reduce replication to 2 - Allows VM creation but increases risk (use only if 250GB drives not usable)

Last Updated: 2025-12-13
Status: 🔍 VERIFY 250GB DRIVES - POTENTIAL IMMEDIATE SOLUTION AVAILABLE