116 lines
3.6 KiB
Markdown
116 lines
3.6 KiB
Markdown
|
# Ceph Setup Progress Summary - sfvalley-01
|
||
|
|
|
||
|
|
**Date**: 2025-12-15 15:25 PST
|
||
|
|
**Status**: ⚠️ **PARTIAL SUCCESS - Monitor Filesystem Created, Service Issues Remain**
|
||
|
|
|
||
|
|
## Completed Steps
|
||
|
|
|
||
|
|
### ✅ Infrastructure Setup
|
||
|
|
1. **SSH Key Authentication**: Configured for both servers
|
||
|
|
2. **Disk Layout Documentation**: fdisk output documented for both servers
|
||
|
|
3. **Ceph Packages**: Installed on both ml110-01 and r630-01 (19.2.3-pve2)
|
||
|
|
4. **Proxmox Cluster**: r630-01 successfully joined cluster "sfvalley-01"
|
||
|
|
|
||
|
|
### ✅ Configuration
|
||
|
|
1. **Ceph Configuration**: Updated and synced across cluster
|
||
|
|
- Network: 192.168.11.0/24 (corrected from 192.168.11.11/24)
|
||
|
|
- FSID: 021d20d3-446f-42cb-a219-5e01213b7b2d
|
||
|
|
- Monitor hosts: 192.168.11.10, 192.168.11.11
|
||
|
|
- Both monitor sections configured
|
||
|
|
|
||
|
|
### ✅ Monitor Setup Progress
|
||
|
|
1. **Monmap Extracted**: Successfully extracted from ml110-01 monitor
|
||
|
|
2. **Monitor Keyring**: Retrieved from ml110-01
|
||
|
|
3. **Monitor Filesystem Created**: `/var/lib/ceph/mon/ceph-r630-01/` filesystem created with:
|
||
|
|
- keyring
|
||
|
|
- kv_backend
|
||
|
|
- store.db directory
|
||
|
|
|
||
|
|
## Current Issues
|
||
|
|
|
||
|
|
### ⚠️ Monitor Service
|
||
|
|
- **Status**: Monitor filesystem exists but service fails to start
|
||
|
|
- **Error**: Service exits with code 1
|
||
|
|
- **Location**: `/var/lib/ceph/mon/ceph-r630-01/` exists with proper structure
|
||
|
|
- **Next Step**: Investigate why service fails despite filesystem being created
|
||
|
|
|
||
|
|
### ⚠️ Cluster Accessibility
|
||
|
|
- **Status**: Ceph commands timeout
|
||
|
|
- **Impact**: Cannot verify quorum, cannot create OSDs
|
||
|
|
- **Root Cause**: ML110-01 monitor in "probing" state, cannot form quorum
|
||
|
|
- **Next Step**: Resolve monitor quorum issue on ml110-01
|
||
|
|
|
||
|
|
### ⚠️ OSD Creation
|
||
|
|
- **Status**: Cannot create OSDs (cluster not accessible)
|
||
|
|
- **Available Disks**: 6x 232.9G SSDs on r630-01 (/dev/sdc through /dev/sdh)
|
||
|
|
- **Next Step**: Create OSDs once cluster is operational
|
||
|
|
|
||
|
|
## Technical Details
|
||
|
|
|
||
|
|
### Monitor Filesystem Structure (r630-01)
|
||
|
|
```
|
||
|
|
/var/lib/ceph/mon/ceph-r630-01/
|
||
|
|
├── keyring (77 bytes)
|
||
|
|
├── kv_backend (8 bytes)
|
||
|
|
└── store.db/ (database directory)
|
||
|
|
```
|
||
|
|
|
||
|
|
### Configuration Files
|
||
|
|
- **Location**: `/etc/pve/ceph.conf` (synced via Proxmox cluster)
|
||
|
|
- **Status**: ✅ Correct and synced
|
||
|
|
- **Monitors Configured**: ml110-01, r630-01
|
||
|
|
|
||
|
|
## Next Steps
|
||
|
|
|
||
|
|
### Priority 1: Fix Monitor Service
|
||
|
|
1. Check detailed error logs: `journalctl -xeu ceph-mon@r630-01.service`
|
||
|
|
2. Verify filesystem permissions and ownership
|
||
|
|
3. Check if monitor can connect to ml110-01
|
||
|
|
4. Resolve service startup issue
|
||
|
|
|
||
|
|
### Priority 2: Establish Monitor Quorum
|
||
|
|
1. Investigate why ml110-01 monitor is in "probing" state
|
||
|
|
2. Check monitor logs for errors
|
||
|
|
3. Verify network connectivity between monitors
|
||
|
|
4. Ensure both monitors can form quorum
|
||
|
|
|
||
|
|
### Priority 3: Create OSDs
|
||
|
|
1. Once cluster is accessible, create OSDs on r630-01:
|
||
|
|
```bash
|
||
|
|
pveceph osd create /dev/sdc
|
||
|
|
pveceph osd create /dev/sdd
|
||
|
|
pveceph osd create /dev/sde
|
||
|
|
pveceph osd create /dev/sdf
|
||
|
|
pveceph osd create /dev/sdg
|
||
|
|
pveceph osd create /dev/sdh
|
||
|
|
```
|
||
|
|
|
||
|
|
## Commands Reference
|
||
|
|
|
||
|
|
### Check Monitor Status
|
||
|
|
```bash
|
||
|
|
systemctl status ceph-mon@r630-01.service
|
||
|
|
journalctl -u ceph-mon@r630-01.service -n 20
|
||
|
|
```
|
||
|
|
|
||
|
|
### Check Cluster Status
|
||
|
|
```bash
|
||
|
|
ceph quorum_status
|
||
|
|
ceph -s
|
||
|
|
ceph mon dump
|
||
|
|
```
|
||
|
|
|
||
|
|
### Create OSDs (once cluster is operational)
|
||
|
|
```bash
|
||
|
|
pveceph osd create /dev/sdc
|
||
|
|
# Repeat for sdd, sde, sdf, sdg, sdh
|
||
|
|
```
|
||
|
|
|
||
|
|
## Notes
|
||
|
|
|
||
|
|
- Monitor filesystem was successfully created using monmap and keyring from ml110-01
|
||
|
|
- Configuration is correct and synced across Proxmox cluster
|
||
|
|
- The main blocker is the monitor service not starting and cluster not being accessible
|
||
|
|
- May need to investigate deeper into monitor logs or use Proxmox web UI for final setup
|
||
|
|
|