124 lines
3.5 KiB
Markdown
124 lines
3.5 KiB
Markdown
|
# Ceph Cluster Current State - sfvalley-01
|
||
|
|
|
||
|
|
**Date**: 2025-12-15 15:18 PST
|
||
|
|
**Status**: ⚠️ **CLUSTER NOT FULLY OPERATIONAL**
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
|
||
|
|
The Ceph cluster on ml110-01 exists but is not responding to commands. The monitor is running but in "probing" state and cannot form quorum, which prevents:
|
||
|
|
- Adding r630-01 monitor
|
||
|
|
- Creating OSDs
|
||
|
|
- Querying cluster status
|
||
|
|
|
||
|
|
## Current Configuration
|
||
|
|
|
||
|
|
### Both Nodes (Synced via Proxmox cluster)
|
||
|
|
```
|
||
|
|
[global]
|
||
|
|
auth_client_required = cephx
|
||
|
|
auth_cluster_required = cephx
|
||
|
|
auth_service_required = cephx
|
||
|
|
cluster_network = 192.168.11.0/24
|
||
|
|
fsid = 021d20d3-446f-42cb-a219-5e01213b7b2d
|
||
|
|
mon_allow_pool_delete = true
|
||
|
|
mon_host = 192.168.11.10 192.168.11.11
|
||
|
|
ms_bind_ipv4 = true
|
||
|
|
ms_bind_ipv6 = false
|
||
|
|
osd_pool_default_min_size = 2
|
||
|
|
osd_pool_default_size = 3
|
||
|
|
public_network = 192.168.11.0/24
|
||
|
|
|
||
|
|
[client]
|
||
|
|
keyring = /etc/pve/priv/$cluster.$name.keyring
|
||
|
|
|
||
|
|
[client.crash]
|
||
|
|
keyring = /etc/pve/ceph/$cluster.$name.keyring
|
||
|
|
|
||
|
|
[mon.ml110-01]
|
||
|
|
public_addr = 192.168.11.10
|
||
|
|
|
||
|
|
[mon.r630-01]
|
||
|
|
public_addr = 192.168.11.11
|
||
|
|
```
|
||
|
|
|
||
|
|
## Service Status
|
||
|
|
|
||
|
|
### ML110-01
|
||
|
|
- ✅ ceph-mon@ml110-01.service: **ACTIVE (running)**
|
||
|
|
- ⚠️ Monitor state: "probing" (cannot form quorum)
|
||
|
|
- ❌ Ceph commands: Timeout (cluster not accessible)
|
||
|
|
|
||
|
|
### R630-01
|
||
|
|
- ❌ ceph-mon@r630-01.service: **FAILED** (filesystem not created)
|
||
|
|
- ⚠️ Cannot create monitor without monmap from ml110-01
|
||
|
|
- ⚠️ Cannot create OSDs without cluster access
|
||
|
|
|
||
|
|
## Available Disks for OSDs (R630-01)
|
||
|
|
|
||
|
|
- `/dev/sdc`: 232.9G (CT250MX500SSD1)
|
||
|
|
- `/dev/sdd`: 232.9G (CT250MX500SSD1)
|
||
|
|
- `/dev/sde`: 232.9G (CT250MX500SSD1)
|
||
|
|
- `/dev/sdf`: 232.9G (CT250MX500SSD1)
|
||
|
|
- `/dev/sdg`: 232.9G (CT250MX500SSD1)
|
||
|
|
- `/dev/sdh`: 232.9G (CT250MX500SSD1)
|
||
|
|
|
||
|
|
**Total Available**: ~1.4TB across 6 SSDs
|
||
|
|
|
||
|
|
## Blocking Issues
|
||
|
|
|
||
|
|
1. **Monitor Quorum Not Established**
|
||
|
|
- ML110-01 monitor in "probing" state
|
||
|
|
- Cannot form quorum with single monitor
|
||
|
|
- R630-01 monitor cannot be added without quorum
|
||
|
|
|
||
|
|
2. **Cluster Commands Timeout**
|
||
|
|
- `ceph -s`: Timeout
|
||
|
|
- `ceph mon getmap`: Timeout
|
||
|
|
- `ceph quorum_status`: Timeout
|
||
|
|
- Cannot query or manage cluster
|
||
|
|
|
||
|
|
3. **Monitor Filesystem Not Created**
|
||
|
|
- R630-01 monitor directory exists but is empty
|
||
|
|
- Needs monmap and keyring from ml110-01
|
||
|
|
- Cannot create without cluster access
|
||
|
|
|
||
|
|
## Recommended Solutions
|
||
|
|
|
||
|
|
### Option 1: Use Proxmox Web UI (Recommended)
|
||
|
|
The Proxmox web interface may handle the monitor and OSD creation more reliably:
|
||
|
|
1. Access Proxmox web UI
|
||
|
|
2. Navigate to: Datacenter > Ceph > Monitors
|
||
|
|
3. Add r630-01 monitor through UI
|
||
|
|
4. Navigate to: Datacenter > Ceph > OSDs
|
||
|
|
5. Create OSDs on r630-01 disks through UI
|
||
|
|
|
||
|
|
### Option 2: Reinitialize Ceph Cluster
|
||
|
|
If the cluster is not critical or can be reinitialized:
|
||
|
|
1. Stop Ceph on ml110-01
|
||
|
|
2. Remove existing Ceph data
|
||
|
|
3. Reinitialize cluster with proper configuration
|
||
|
|
4. Add r630-01 monitor
|
||
|
|
5. Add OSDs
|
||
|
|
|
||
|
|
### Option 3: Fix Monitor Quorum
|
||
|
|
If there's existing data that must be preserved:
|
||
|
|
1. Investigate why ml110-01 monitor is in "probing" state
|
||
|
|
2. Check monitor logs for errors
|
||
|
|
3. Fix underlying issue preventing quorum
|
||
|
|
4. Then add r630-01 monitor
|
||
|
|
|
||
|
|
## Next Steps
|
||
|
|
|
||
|
|
1. **Investigate Monitor Issue**: Check why ml110-01 monitor cannot form quorum
|
||
|
|
2. **Use Web UI**: Try adding monitor and OSDs through Proxmox web interface
|
||
|
|
3. **Or Reinitialize**: If acceptable, reinitialize cluster with proper setup
|
||
|
|
|
||
|
|
## Notes
|
||
|
|
|
||
|
|
- Configuration is correct and synced across cluster
|
||
|
|
- Network connectivity is working (ping successful)
|
||
|
|
- Monitor ports are open (6789, 3300)
|
||
|
|
- Issue appears to be with monitor quorum formation
|
||
|
|
- May require cluster reinitialization or web UI approach
|
||
|
|
|