33d50fb91e
Some checks failed
API CI / API Lint (push) Successful in 47s
API CI / API Type Check (push) Failing after 47s
API CI / API Test (push) Successful in 1m0s
API CI / API Build (push) Failing after 50s
API CI / Build Docker Image (push) Has been skipped
Build Crossplane Provider / build (push) Failing after 5m51s
CD Pipeline / Deploy to Staging (push) Failing after 29s
CI Pipeline / Lint and Type Check (push) Failing after 36s
CI Pipeline / Build (push) Has been skipped
CI Pipeline / Test Backend (push) Failing after 1m33s
CI Pipeline / Test Frontend (push) Failing after 30s
CI Pipeline / Security Scan (push) Failing after 1m16s
Crossplane Provider CI / Go Test (push) Failing after 3m23s
Crossplane Provider CI / Go Lint (push) Failing after 7m27s
Crossplane Provider CI / Go Build (push) Failing after 3m27s
Deploy to Staging / Deploy to Staging (push) Failing after 30s
Portal CI / Portal Lint (push) Failing after 21s
Portal CI / Portal Type Check (push) Failing after 21s
Portal CI / Portal Test (push) Failing after 21s
Portal CI / Portal Build (push) Failing after 22s
Test Suite / frontend-tests (push) Failing after 30s
Test Suite / api-tests (push) Failing after 49s
Test Suite / blockchain-tests (push) Failing after 30s
Type Check / type-check (map[directory:. name:root]) (push) Failing after 23s
Type Check / type-check (map[directory:api name:api]) (push) Failing after 21s
Type Check / type-check (map[directory:portal name:portal]) (push) Failing after 19s
Validate Configuration Files / validate (push) Failing after 1m52s
CD Pipeline / Deploy to Production (push) Has been skipped
Co-authored-by: Cursor <cursoragent@cursor.com>
124 lines
3.5 KiB
Markdown
124 lines
3.5 KiB
Markdown
# Ceph Cluster Current State - sfvalley-01
|
|
|
|
**Date**: 2025-12-15 15:18 PST
|
|
**Status**: ⚠️ **CLUSTER NOT FULLY OPERATIONAL**
|
|
|
|
## Summary
|
|
|
|
The Ceph cluster on ml110-01 exists but is not responding to commands. The monitor is running but in "probing" state and cannot form quorum, which prevents:
|
|
- Adding r630-01 monitor
|
|
- Creating OSDs
|
|
- Querying cluster status
|
|
|
|
## Current Configuration
|
|
|
|
### Both Nodes (Synced via Proxmox cluster)
|
|
```
|
|
[global]
|
|
auth_client_required = cephx
|
|
auth_cluster_required = cephx
|
|
auth_service_required = cephx
|
|
cluster_network = 192.168.11.0/24
|
|
fsid = 021d20d3-446f-42cb-a219-5e01213b7b2d
|
|
mon_allow_pool_delete = true
|
|
mon_host = 192.168.11.10 192.168.11.11
|
|
ms_bind_ipv4 = true
|
|
ms_bind_ipv6 = false
|
|
osd_pool_default_min_size = 2
|
|
osd_pool_default_size = 3
|
|
public_network = 192.168.11.0/24
|
|
|
|
[client]
|
|
keyring = /etc/pve/priv/$cluster.$name.keyring
|
|
|
|
[client.crash]
|
|
keyring = /etc/pve/ceph/$cluster.$name.keyring
|
|
|
|
[mon.ml110-01]
|
|
public_addr = 192.168.11.10
|
|
|
|
[mon.r630-01]
|
|
public_addr = 192.168.11.11
|
|
```
|
|
|
|
## Service Status
|
|
|
|
### ML110-01
|
|
- ✅ ceph-mon@ml110-01.service: **ACTIVE (running)**
|
|
- ⚠️ Monitor state: "probing" (cannot form quorum)
|
|
- ❌ Ceph commands: Timeout (cluster not accessible)
|
|
|
|
### R630-01
|
|
- ❌ ceph-mon@r630-01.service: **FAILED** (filesystem not created)
|
|
- ⚠️ Cannot create monitor without monmap from ml110-01
|
|
- ⚠️ Cannot create OSDs without cluster access
|
|
|
|
## Available Disks for OSDs (R630-01)
|
|
|
|
- `/dev/sdc`: 232.9G (CT250MX500SSD1)
|
|
- `/dev/sdd`: 232.9G (CT250MX500SSD1)
|
|
- `/dev/sde`: 232.9G (CT250MX500SSD1)
|
|
- `/dev/sdf`: 232.9G (CT250MX500SSD1)
|
|
- `/dev/sdg`: 232.9G (CT250MX500SSD1)
|
|
- `/dev/sdh`: 232.9G (CT250MX500SSD1)
|
|
|
|
**Total Available**: ~1.4TB across 6 SSDs
|
|
|
|
## Blocking Issues
|
|
|
|
1. **Monitor Quorum Not Established**
|
|
- ML110-01 monitor in "probing" state
|
|
- Cannot form quorum with single monitor
|
|
- R630-01 monitor cannot be added without quorum
|
|
|
|
2. **Cluster Commands Timeout**
|
|
- `ceph -s`: Timeout
|
|
- `ceph mon getmap`: Timeout
|
|
- `ceph quorum_status`: Timeout
|
|
- Cannot query or manage cluster
|
|
|
|
3. **Monitor Filesystem Not Created**
|
|
- R630-01 monitor directory exists but is empty
|
|
- Needs monmap and keyring from ml110-01
|
|
- Cannot create without cluster access
|
|
|
|
## Recommended Solutions
|
|
|
|
### Option 1: Use Proxmox Web UI (Recommended)
|
|
The Proxmox web interface may handle the monitor and OSD creation more reliably:
|
|
1. Access Proxmox web UI
|
|
2. Navigate to: Datacenter > Ceph > Monitors
|
|
3. Add r630-01 monitor through UI
|
|
4. Navigate to: Datacenter > Ceph > OSDs
|
|
5. Create OSDs on r630-01 disks through UI
|
|
|
|
### Option 2: Reinitialize Ceph Cluster
|
|
If the cluster is not critical or can be reinitialized:
|
|
1. Stop Ceph on ml110-01
|
|
2. Remove existing Ceph data
|
|
3. Reinitialize cluster with proper configuration
|
|
4. Add r630-01 monitor
|
|
5. Add OSDs
|
|
|
|
### Option 3: Fix Monitor Quorum
|
|
If there's existing data that must be preserved:
|
|
1. Investigate why ml110-01 monitor is in "probing" state
|
|
2. Check monitor logs for errors
|
|
3. Fix underlying issue preventing quorum
|
|
4. Then add r630-01 monitor
|
|
|
|
## Next Steps
|
|
|
|
1. **Investigate Monitor Issue**: Check why ml110-01 monitor cannot form quorum
|
|
2. **Use Web UI**: Try adding monitor and OSDs through Proxmox web interface
|
|
3. **Or Reinitialize**: If acceptable, reinitialize cluster with proper setup
|
|
|
|
## Notes
|
|
|
|
- Configuration is correct and synced across cluster
|
|
- Network connectivity is working (ping successful)
|
|
- Monitor ports are open (6789, 3300)
|
|
- Issue appears to be with monitor quorum formation
|
|
- May require cluster reinitialization or web UI approach
|
|
|