Sankofa/docs/infrastructure/NEXT_STEPS.md

Infrastructure Next Steps

Date: 2025-12-15
Status: Assessment Complete - Action Items Identified

Current State Summary

✅ Completed

1. SSH Key Authentication: Set up passwordless SSH access to both servers
   • ✅ r630-01 (192.168.11.11) - SSH key configured
   • ✅ ml110-01 (192.168.11.10) - SSH key configured
2. Disk Layout Documentation: Comprehensive fdisk output captured
   • ✅ r630-01 disk layout documented
   • ✅ ml110-01 disk layout documented

⚠️ Current Issues Identified

1. Ceph Installation Status (CRITICAL)

• Issue: Only ceph-common and ceph-fuse packages installed, missing ceph-mon and ceph-osd
• Status: pveceph command reports "binary not installed"
• Current State:
  • /var/lib/ceph/ directories exist but are empty (no mon/ or osd/ subdirectories)
  • Ceph client tools available but cluster services not installed
• Previous State: Documentation indicates MON recovery was successful on 2025-12-13, but cluster appears to have been removed
• Action Required: Install full Ceph packages or reinitialize cluster

2. Disk Health - R630-01 /dev/sda (WARNING)

• Model: ST9300653SS (Seagate 300GB SAS, 15K RPM)
• SMART Status: OK (but showing signs of wear)
• Age: Manufactured 2014, ~45,532 hours powered on (~5.2 years)
• Concerns:
  • 2,186 elements in grown defect list
  • 456,913 total uncorrected read errors
  • 379,382,059 total errors corrected
• Risk: Disk is aging and showing significant error correction activity
• Action Required: Monitor closely, plan for replacement

3. Ceph Cluster Accessibility

• Status: Cannot access cluster (ceph -s fails with config errors)
• Possible Causes:
  • Ceph not fully installed
  • Configuration files missing
  • Cluster not initialized
• Action Required: Verify Ceph installation and configuration

Immediate Next Steps

Priority 1: Verify Ceph Installation Status

Check if Ceph is installed:

# On both servers
ssh root@192.168.11.10 "dpkg -l | grep ceph"
ssh root@192.168.11.11 "dpkg -l | grep ceph"

# Check for Ceph packages
ssh root@192.168.11.10 "apt list --installed | grep ceph"
ssh root@192.168.11.11 "apt list --installed | grep ceph"

If Ceph is not installed:

• Follow Proxmox Ceph installation guide
• Or use existing scripts in scripts/ directory

If Ceph is installed but binaries missing:

• Reinstall Ceph packages: apt install --reinstall ceph-common ceph-mon ceph-osd
• Verify Proxmox Ceph integration: pveceph install

Priority 2: Assess Disk Health and Plan Replacement

Monitor disk health:

# Check SMART attributes
ssh root@192.168.11.11 "smartctl -a /dev/sda"

# Check for I/O errors in dmesg
ssh root@192.168.11.11 "dmesg | grep -i 'sda.*error' | tail -20"

# Check filesystem health
ssh root@192.168.11.11 "fsck -n /dev/mapper/pve-root"

Plan disk replacement:

• R630-01 has /dev/sdb (279.4GB) available as secondary disk
• Consider moving critical services (like Ceph MON) to /dev/sdb
• Or plan full disk replacement for /dev/sda

Priority 3: Verify Ceph Cluster State

If Ceph is installed, check cluster status:

# Check if Ceph services exist
ssh root@192.168.11.10 "systemctl list-units | grep ceph"
ssh root@192.168.11.11 "systemctl list-units | grep ceph"

# Check for Ceph configuration
ssh root@192.168.11.10 "ls -la /etc/ceph/ /etc/pve/ceph.conf"
ssh root@192.168.11.11 "ls -la /etc/ceph/ /etc/pve/ceph.conf"

# Check for MON data directories
ssh root@192.168.11.10 "ls -la /var/lib/ceph/mon/"
ssh root@192.168.11.11 "ls -la /var/lib/ceph/mon/"

# Check for OSD data
ssh root@192.168.11.10 "ls -la /var/lib/ceph/osd/"
ssh root@192.168.11.11 "ls -la /var/lib/ceph/osd/"

Medium-Term Actions

1. Ceph Cluster Setup/Recovery

If cluster needs to be initialized:

• Use existing scripts: scripts/fresh-ceph-install-r630.sh
• Or follow Proxmox Ceph setup guide

If cluster exists but is inaccessible:

• Review recovery documentation: docs/ceph/MON_RECOVERY_SUCCESS.md
• Check if MON services need to be recreated
• Verify network connectivity between nodes

2. Add Additional Ceph OSDs

R630-01 has 6x 250GB SSDs available:

• /dev/sdc through /dev/sdh (232.89 GiB each)
• Total capacity: ~1.4TB
• Use script: scripts/setup-ceph-osds-r630.sh
• Or manual setup following: docs/ceph/ADD_THIRD_OSD_GUIDE.md

Benefits:

• Increase cluster capacity
• Improve redundancy (currently only 2 OSDs)
• Fix "undersized PGs" warnings

3. Disk Replacement Planning

For R630-01 /dev/sda:

• Monitor SMART attributes weekly
• Plan replacement when errors increase
• Consider moving MON to /dev/sdb as interim solution
• Document replacement procedure

Long-Term Improvements

1. Infrastructure Monitoring

• Set up SMART monitoring for all disks
• Configure alerts for disk health degradation
• Monitor Ceph cluster health automatically

2. Backup Strategy

• Regular backups of Ceph MON data directories
• Backup Proxmox configurations
• Document recovery procedures

3. Redundancy Improvements

• Add third Ceph MON (if possible)
• Ensure OSD count meets replication requirements
• Plan for hardware failures

4. Documentation Updates

• Update disk layout docs as changes occur
• Document all recovery procedures
• Maintain runbook for common issues

Available Scripts and Resources

Ceph Setup Scripts

• scripts/fresh-ceph-install-r630.sh - Fresh Ceph installation
• scripts/setup-ceph-osds-r630.sh - OSD setup for R630-01
• scripts/reinitialize-ceph-cluster.sh - Cluster reinitialization

Disk Management Scripts

• scripts/analyze-r630-disk-layout.sh - Disk analysis
• scripts/verify-r630-250gb-drives.sh - SSD verification

SSH Helper Scripts

• scripts/ssh-r630-01.sh - SSH to R630-01
• scripts/ssh-ml110-01.sh - SSH to ML110-01

Documentation

• docs/infrastructure/ - Infrastructure documentation
• docs/ceph/ - Ceph-specific documentation

Recommended Action Order

1. Verify Ceph installation (15 minutes)

   • Check packages, binaries, configuration
   • Determine if reinstall needed

2. Assess cluster state (30 minutes)

   • Check services, data directories
   • Determine if recovery needed

3. Monitor disk health (ongoing)

   • Set up monitoring
   • Plan replacement timeline

4. Add OSDs (1-2 hours)

   • Once cluster is stable
   • Use 6x SSDs on R630-01

5. Plan disk replacement (planning)

   • Schedule replacement
   • Document procedure

Notes

• All SSH access is now configured with key authentication
• Disk layouts are fully documented
• Previous recovery documentation exists but cluster state needs verification
• Disk health on R630-01 /dev/sda requires attention