Sankofa/docs/infrastructure/NEXT_STEPS.md

# Infrastructure Next Steps

**Date**: 2025-12-15  
**Status**: Assessment Complete - Action Items Identified

## Current State Summary

### ✅ Completed
1. **SSH Key Authentication**: Set up passwordless SSH access to both servers
   - ✅ r630-01 (192.168.11.11) - SSH key configured
   - ✅ ml110-01 (192.168.11.10) - SSH key configured
2. **Disk Layout Documentation**: Comprehensive fdisk output captured
   - ✅ r630-01 disk layout documented
   - ✅ ml110-01 disk layout documented

### ⚠️ Current Issues Identified

#### 1. Ceph Installation Status (CRITICAL)
- **Issue**: Only `ceph-common` and `ceph-fuse` packages installed, missing `ceph-mon` and `ceph-osd`
- **Status**: `pveceph` command reports "binary not installed"
- **Current State**: 
  - `/var/lib/ceph/` directories exist but are empty (no mon/ or osd/ subdirectories)
  - Ceph client tools available but cluster services not installed
- **Previous State**: Documentation indicates MON recovery was successful on 2025-12-13, but cluster appears to have been removed
- **Action Required**: Install full Ceph packages or reinitialize cluster

#### 2. Disk Health - R630-01 `/dev/sda` (WARNING)
- **Model**: ST9300653SS (Seagate 300GB SAS, 15K RPM)
- **SMART Status**: OK (but showing signs of wear)
- **Age**: Manufactured 2014, ~45,532 hours powered on (~5.2 years)
- **Concerns**:
  - 2,186 elements in grown defect list
  - 456,913 total uncorrected read errors
  - 379,382,059 total errors corrected
- **Risk**: Disk is aging and showing significant error correction activity
- **Action Required**: Monitor closely, plan for replacement

#### 3. Ceph Cluster Accessibility
- **Status**: Cannot access cluster (`ceph -s` fails with config errors)
- **Possible Causes**:
  - Ceph not fully installed
  - Configuration files missing
  - Cluster not initialized
- **Action Required**: Verify Ceph installation and configuration

## Immediate Next Steps

### Priority 1: Verify Ceph Installation Status

**Check if Ceph is installed:**
```bash
# On both servers
ssh root@192.168.11.10 "dpkg -l | grep ceph"
ssh root@192.168.11.11 "dpkg -l | grep ceph"

# Check for Ceph packages
ssh root@192.168.11.10 "apt list --installed | grep ceph"
ssh root@192.168.11.11 "apt list --installed | grep ceph"
```

**If Ceph is not installed:**
- Follow Proxmox Ceph installation guide
- Or use existing scripts in `scripts/` directory

**If Ceph is installed but binaries missing:**
- Reinstall Ceph packages: `apt install --reinstall ceph-common ceph-mon ceph-osd`
- Verify Proxmox Ceph integration: `pveceph install`

### Priority 2: Assess Disk Health and Plan Replacement

**Monitor disk health:**
```bash
# Check SMART attributes
ssh root@192.168.11.11 "smartctl -a /dev/sda"

# Check for I/O errors in dmesg
ssh root@192.168.11.11 "dmesg | grep -i 'sda.*error' | tail -20"

# Check filesystem health
ssh root@192.168.11.11 "fsck -n /dev/mapper/pve-root"
```

**Plan disk replacement:**
- R630-01 has `/dev/sdb` (279.4GB) available as secondary disk
- Consider moving critical services (like Ceph MON) to `/dev/sdb`
- Or plan full disk replacement for `/dev/sda`

### Priority 3: Verify Ceph Cluster State

**If Ceph is installed, check cluster status:**
```bash
# Check if Ceph services exist
ssh root@192.168.11.10 "systemctl list-units | grep ceph"
ssh root@192.168.11.11 "systemctl list-units | grep ceph"

# Check for Ceph configuration
ssh root@192.168.11.10 "ls -la /etc/ceph/ /etc/pve/ceph.conf"
ssh root@192.168.11.11 "ls -la /etc/ceph/ /etc/pve/ceph.conf"

# Check for MON data directories
ssh root@192.168.11.10 "ls -la /var/lib/ceph/mon/"
ssh root@192.168.11.11 "ls -la /var/lib/ceph/mon/"

# Check for OSD data
ssh root@192.168.11.10 "ls -la /var/lib/ceph/osd/"
ssh root@192.168.11.11 "ls -la /var/lib/ceph/osd/"
```

## Medium-Term Actions

### 1. Ceph Cluster Setup/Recovery

**If cluster needs to be initialized:**
- Use existing scripts: `scripts/fresh-ceph-install-r630.sh`
- Or follow Proxmox Ceph setup guide

**If cluster exists but is inaccessible:**
- Review recovery documentation: `docs/ceph/MON_RECOVERY_SUCCESS.md`
- Check if MON services need to be recreated
- Verify network connectivity between nodes

### 2. Add Additional Ceph OSDs

**R630-01 has 6x 250GB SSDs available:**
- `/dev/sdc` through `/dev/sdh` (232.89 GiB each)
- Total capacity: ~1.4TB
- Use script: `scripts/setup-ceph-osds-r630.sh`
- Or manual setup following: `docs/ceph/ADD_THIRD_OSD_GUIDE.md`

**Benefits:**
- Increase cluster capacity
- Improve redundancy (currently only 2 OSDs)
- Fix "undersized PGs" warnings

### 3. Disk Replacement Planning

**For R630-01 `/dev/sda`:**
- Monitor SMART attributes weekly
- Plan replacement when errors increase
- Consider moving MON to `/dev/sdb` as interim solution
- Document replacement procedure

## Long-Term Improvements

### 1. Infrastructure Monitoring
- Set up SMART monitoring for all disks
- Configure alerts for disk health degradation
- Monitor Ceph cluster health automatically

### 2. Backup Strategy
- Regular backups of Ceph MON data directories
- Backup Proxmox configurations
- Document recovery procedures

### 3. Redundancy Improvements
- Add third Ceph MON (if possible)
- Ensure OSD count meets replication requirements
- Plan for hardware failures

### 4. Documentation Updates
- Update disk layout docs as changes occur
- Document all recovery procedures
- Maintain runbook for common issues

## Available Scripts and Resources

### Ceph Setup Scripts
- `scripts/fresh-ceph-install-r630.sh` - Fresh Ceph installation
- `scripts/setup-ceph-osds-r630.sh` - OSD setup for R630-01
- `scripts/reinitialize-ceph-cluster.sh` - Cluster reinitialization

### Disk Management Scripts
- `scripts/analyze-r630-disk-layout.sh` - Disk analysis
- `scripts/verify-r630-250gb-drives.sh` - SSD verification

### SSH Helper Scripts
- `scripts/ssh-r630-01.sh` - SSH to R630-01
- `scripts/ssh-ml110-01.sh` - SSH to ML110-01

### Documentation
- `docs/infrastructure/` - Infrastructure documentation
- `docs/ceph/` - Ceph-specific documentation

## Recommended Action Order

1. **Verify Ceph installation** (15 minutes)
   - Check packages, binaries, configuration
   - Determine if reinstall needed

2. **Assess cluster state** (30 minutes)
   - Check services, data directories
   - Determine if recovery needed

3. **Monitor disk health** (ongoing)
   - Set up monitoring
   - Plan replacement timeline

4. **Add OSDs** (1-2 hours)
   - Once cluster is stable
   - Use 6x SSDs on R630-01

5. **Plan disk replacement** (planning)
   - Schedule replacement
   - Document procedure

## Notes

- All SSH access is now configured with key authentication
- Disk layouts are fully documented
- Previous recovery documentation exists but cluster state needs verification
- Disk health on R630-01 `/dev/sda` requires attention