# Ceph Cluster Current State - sfvalley-01

**Date**: 2025-12-15 15:18 PST  
**Status**: ⚠️ **CLUSTER NOT FULLY OPERATIONAL**

## Summary

The Ceph cluster on ml110-01 exists but is not responding to commands. The monitor is running but in "probing" state and cannot form quorum, which prevents:
- Adding r630-01 monitor
- Creating OSDs
- Querying cluster status

## Current Configuration

### Both Nodes (Synced via Proxmox cluster)
```
[global]
	auth_client_required = cephx
	auth_cluster_required = cephx
	auth_service_required = cephx
	cluster_network = 192.168.11.0/24
	fsid = 021d20d3-446f-42cb-a219-5e01213b7b2d
	mon_allow_pool_delete = true
	mon_host = 192.168.11.10 192.168.11.11
	ms_bind_ipv4 = true
	ms_bind_ipv6 = false
	osd_pool_default_min_size = 2
	osd_pool_default_size = 3
	public_network = 192.168.11.0/24

[client]
	keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
	keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.ml110-01]
	public_addr = 192.168.11.10

[mon.r630-01]
	public_addr = 192.168.11.11
```

## Service Status

### ML110-01
- ✅ ceph-mon@ml110-01.service: **ACTIVE (running)**
- ⚠️ Monitor state: "probing" (cannot form quorum)
- ❌ Ceph commands: Timeout (cluster not accessible)

### R630-01
- ❌ ceph-mon@r630-01.service: **FAILED** (filesystem not created)
- ⚠️ Cannot create monitor without monmap from ml110-01
- ⚠️ Cannot create OSDs without cluster access

## Available Disks for OSDs (R630-01)

- `/dev/sdc`: 232.9G (CT250MX500SSD1)
- `/dev/sdd`: 232.9G (CT250MX500SSD1)
- `/dev/sde`: 232.9G (CT250MX500SSD1)
- `/dev/sdf`: 232.9G (CT250MX500SSD1)
- `/dev/sdg`: 232.9G (CT250MX500SSD1)
- `/dev/sdh`: 232.9G (CT250MX500SSD1)

**Total Available**: ~1.4TB across 6 SSDs

## Blocking Issues

1. **Monitor Quorum Not Established**
   - ML110-01 monitor in "probing" state
   - Cannot form quorum with single monitor
   - R630-01 monitor cannot be added without quorum

2. **Cluster Commands Timeout**
   - `ceph -s`: Timeout
   - `ceph mon getmap`: Timeout
   - `ceph quorum_status`: Timeout
   - Cannot query or manage cluster

3. **Monitor Filesystem Not Created**
   - R630-01 monitor directory exists but is empty
   - Needs monmap and keyring from ml110-01
   - Cannot create without cluster access

## Recommended Solutions

### Option 1: Use Proxmox Web UI (Recommended)
The Proxmox web interface may handle the monitor and OSD creation more reliably:
1. Access Proxmox web UI
2. Navigate to: Datacenter > Ceph > Monitors
3. Add r630-01 monitor through UI
4. Navigate to: Datacenter > Ceph > OSDs
5. Create OSDs on r630-01 disks through UI

### Option 2: Reinitialize Ceph Cluster
If the cluster is not critical or can be reinitialized:
1. Stop Ceph on ml110-01
2. Remove existing Ceph data
3. Reinitialize cluster with proper configuration
4. Add r630-01 monitor
5. Add OSDs

### Option 3: Fix Monitor Quorum
If there's existing data that must be preserved:
1. Investigate why ml110-01 monitor is in "probing" state
2. Check monitor logs for errors
3. Fix underlying issue preventing quorum
4. Then add r630-01 monitor

## Next Steps

1. **Investigate Monitor Issue**: Check why ml110-01 monitor cannot form quorum
2. **Use Web UI**: Try adding monitor and OSDs through Proxmox web interface
3. **Or Reinitialize**: If acceptable, reinitialize cluster with proper setup

## Notes

- Configuration is correct and synced across cluster
- Network connectivity is working (ping successful)
- Monitor ports are open (6789, 3300)
- Issue appears to be with monitor quorum formation
- May require cluster reinitialization or web UI approach