chore: consolidate local WIP (repo cleanup 20260707)

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/infrastructure/NEXT_STEPS.md

@@ -0,0 +1,212 @@
+# Infrastructure Next Steps
+
+**Date**: 2025-12-15  
+**Status**: Assessment Complete - Action Items Identified
+
+## Current State Summary
+
+### ✅ Completed
+1. **SSH Key Authentication**: Set up passwordless SSH access to both servers
+   - ✅ r630-01 (192.168.11.11) - SSH key configured
+   - ✅ ml110-01 (192.168.11.10) - SSH key configured
+2. **Disk Layout Documentation**: Comprehensive fdisk output captured
+   - ✅ r630-01 disk layout documented
+   - ✅ ml110-01 disk layout documented
+
+### ⚠️ Current Issues Identified
+
+#### 1. Ceph Installation Status (CRITICAL)
+- **Issue**: Only `ceph-common` and `ceph-fuse` packages installed, missing `ceph-mon` and `ceph-osd`
+- **Status**: `pveceph` command reports "binary not installed"
+- **Current State**: 
+  - `/var/lib/ceph/` directories exist but are empty (no mon/ or osd/ subdirectories)
+  - Ceph client tools available but cluster services not installed
+- **Previous State**: Documentation indicates MON recovery was successful on 2025-12-13, but cluster appears to have been removed
+- **Action Required**: Install full Ceph packages or reinitialize cluster
+
+#### 2. Disk Health - R630-01 `/dev/sda` (WARNING)
+- **Model**: ST9300653SS (Seagate 300GB SAS, 15K RPM)
+- **SMART Status**: OK (but showing signs of wear)
+- **Age**: Manufactured 2014, ~45,532 hours powered on (~5.2 years)
+- **Concerns**:
+  - 2,186 elements in grown defect list
+  - 456,913 total uncorrected read errors
+  - 379,382,059 total errors corrected
+- **Risk**: Disk is aging and showing significant error correction activity
+- **Action Required**: Monitor closely, plan for replacement
+
+#### 3. Ceph Cluster Accessibility
+- **Status**: Cannot access cluster (`ceph -s` fails with config errors)
+- **Possible Causes**:
+  - Ceph not fully installed
+  - Configuration files missing
+  - Cluster not initialized
+- **Action Required**: Verify Ceph installation and configuration
+
+## Immediate Next Steps
+
+### Priority 1: Verify Ceph Installation Status
+
+**Check if Ceph is installed:**
+```bash
+# On both servers
+ssh root@192.168.11.10 "dpkg -l | grep ceph"
+ssh root@192.168.11.11 "dpkg -l | grep ceph"
+
+# Check for Ceph packages
+ssh root@192.168.11.10 "apt list --installed | grep ceph"
+ssh root@192.168.11.11 "apt list --installed | grep ceph"
+```
+
+**If Ceph is not installed:**
+- Follow Proxmox Ceph installation guide
+- Or use existing scripts in `scripts/` directory
+
+**If Ceph is installed but binaries missing:**
+- Reinstall Ceph packages: `apt install --reinstall ceph-common ceph-mon ceph-osd`
+- Verify Proxmox Ceph integration: `pveceph install`
+
+### Priority 2: Assess Disk Health and Plan Replacement
+
+**Monitor disk health:**
+```bash
+# Check SMART attributes
+ssh root@192.168.11.11 "smartctl -a /dev/sda"
+
+# Check for I/O errors in dmesg
+ssh root@192.168.11.11 "dmesg | grep -i 'sda.*error' | tail -20"
+
+# Check filesystem health
+ssh root@192.168.11.11 "fsck -n /dev/mapper/pve-root"
+```
+
+**Plan disk replacement:**
+- R630-01 has `/dev/sdb` (279.4GB) available as secondary disk
+- Consider moving critical services (like Ceph MON) to `/dev/sdb`
+- Or plan full disk replacement for `/dev/sda`
+
+### Priority 3: Verify Ceph Cluster State
+
+**If Ceph is installed, check cluster status:**
+```bash
+# Check if Ceph services exist
+ssh root@192.168.11.10 "systemctl list-units | grep ceph"
+ssh root@192.168.11.11 "systemctl list-units | grep ceph"
+
+# Check for Ceph configuration
+ssh root@192.168.11.10 "ls -la /etc/ceph/ /etc/pve/ceph.conf"
+ssh root@192.168.11.11 "ls -la /etc/ceph/ /etc/pve/ceph.conf"
+
+# Check for MON data directories
+ssh root@192.168.11.10 "ls -la /var/lib/ceph/mon/"
+ssh root@192.168.11.11 "ls -la /var/lib/ceph/mon/"
+
+# Check for OSD data
+ssh root@192.168.11.10 "ls -la /var/lib/ceph/osd/"
+ssh root@192.168.11.11 "ls -la /var/lib/ceph/osd/"
+```
+
+## Medium-Term Actions
+
+### 1. Ceph Cluster Setup/Recovery
+
+**If cluster needs to be initialized:**
+- Use existing scripts: `scripts/fresh-ceph-install-r630.sh`
+- Or follow Proxmox Ceph setup guide
+
+**If cluster exists but is inaccessible:**
+- Review recovery documentation: `docs/ceph/MON_RECOVERY_SUCCESS.md`
+- Check if MON services need to be recreated
+- Verify network connectivity between nodes
+
+### 2. Add Additional Ceph OSDs
+
+**R630-01 has 6x 250GB SSDs available:**
+- `/dev/sdc` through `/dev/sdh` (232.89 GiB each)
+- Total capacity: ~1.4TB
+- Use script: `scripts/setup-ceph-osds-r630.sh`
+- Or manual setup following: `docs/ceph/ADD_THIRD_OSD_GUIDE.md`
+
+**Benefits:**
+- Increase cluster capacity
+- Improve redundancy (currently only 2 OSDs)
+- Fix "undersized PGs" warnings
+
+### 3. Disk Replacement Planning
+
+**For R630-01 `/dev/sda`:**
+- Monitor SMART attributes weekly
+- Plan replacement when errors increase
+- Consider moving MON to `/dev/sdb` as interim solution
+- Document replacement procedure
+
+## Long-Term Improvements
+
+### 1. Infrastructure Monitoring
+- Set up SMART monitoring for all disks
+- Configure alerts for disk health degradation
+- Monitor Ceph cluster health automatically
+
+### 2. Backup Strategy
+- Regular backups of Ceph MON data directories
+- Backup Proxmox configurations
+- Document recovery procedures
+
+### 3. Redundancy Improvements
+- Add third Ceph MON (if possible)
+- Ensure OSD count meets replication requirements
+- Plan for hardware failures
+
+### 4. Documentation Updates
+- Update disk layout docs as changes occur
+- Document all recovery procedures
+- Maintain runbook for common issues
+
+## Available Scripts and Resources
+
+### Ceph Setup Scripts
+- `scripts/fresh-ceph-install-r630.sh` - Fresh Ceph installation
+- `scripts/setup-ceph-osds-r630.sh` - OSD setup for R630-01
+- `scripts/reinitialize-ceph-cluster.sh` - Cluster reinitialization
+
+### Disk Management Scripts
+- `scripts/analyze-r630-disk-layout.sh` - Disk analysis
+- `scripts/verify-r630-250gb-drives.sh` - SSD verification
+
+### SSH Helper Scripts
+- `scripts/ssh-r630-01.sh` - SSH to R630-01
+- `scripts/ssh-ml110-01.sh` - SSH to ML110-01
+
+### Documentation
+- `docs/infrastructure/` - Infrastructure documentation
+- `docs/ceph/` - Ceph-specific documentation
+
+## Recommended Action Order
+
+1. **Verify Ceph installation** (15 minutes)
+   - Check packages, binaries, configuration
+   - Determine if reinstall needed
+
+2. **Assess cluster state** (30 minutes)
+   - Check services, data directories
+   - Determine if recovery needed
+
+3. **Monitor disk health** (ongoing)
+   - Set up monitoring
+   - Plan replacement timeline
+
+4. **Add OSDs** (1-2 hours)
+   - Once cluster is stable
+   - Use 6x SSDs on R630-01
+
+5. **Plan disk replacement** (planning)
+   - Schedule replacement
+   - Document procedure
+
+## Notes
+
+- All SSH access is now configured with key authentication
+- Disk layouts are fully documented
+- Previous recovery documentation exists but cluster state needs verification
+- Disk health on R630-01 `/dev/sda` requires attention
+