33d50fb91e
Some checks failed
API CI / API Lint (push) Successful in 47s
API CI / API Type Check (push) Failing after 47s
API CI / API Test (push) Successful in 1m0s
API CI / API Build (push) Failing after 50s
API CI / Build Docker Image (push) Has been skipped
Build Crossplane Provider / build (push) Failing after 5m51s
CD Pipeline / Deploy to Staging (push) Failing after 29s
CI Pipeline / Lint and Type Check (push) Failing after 36s
CI Pipeline / Build (push) Has been skipped
CI Pipeline / Test Backend (push) Failing after 1m33s
CI Pipeline / Test Frontend (push) Failing after 30s
CI Pipeline / Security Scan (push) Failing after 1m16s
Crossplane Provider CI / Go Test (push) Failing after 3m23s
Crossplane Provider CI / Go Lint (push) Failing after 7m27s
Crossplane Provider CI / Go Build (push) Failing after 3m27s
Deploy to Staging / Deploy to Staging (push) Failing after 30s
Portal CI / Portal Lint (push) Failing after 21s
Portal CI / Portal Type Check (push) Failing after 21s
Portal CI / Portal Test (push) Failing after 21s
Portal CI / Portal Build (push) Failing after 22s
Test Suite / frontend-tests (push) Failing after 30s
Test Suite / api-tests (push) Failing after 49s
Test Suite / blockchain-tests (push) Failing after 30s
Type Check / type-check (map[directory:. name:root]) (push) Failing after 23s
Type Check / type-check (map[directory:api name:api]) (push) Failing after 21s
Type Check / type-check (map[directory:portal name:portal]) (push) Failing after 19s
Validate Configuration Files / validate (push) Failing after 1m52s
CD Pipeline / Deploy to Production (push) Has been skipped
Co-authored-by: Cursor <cursoragent@cursor.com>
133 lines
3.3 KiB
Markdown
133 lines
3.3 KiB
Markdown
# Ceph Cluster Instability - ML110-01
|
|
|
|
**Date**: 2025-12-13
|
|
**Status**: ⚠️ **INVESTIGATING**
|
|
|
|
---
|
|
|
|
## Issue Summary
|
|
|
|
### Symptoms
|
|
- **Ceph Monitor**: Constantly in "electing" state
|
|
- **Slow Operations**: 100+ slow operations reported
|
|
- **Manager Instability**: Ceph manager constantly restarting
|
|
- **Connection Issues**: Socket connections closing between mon0 and mon1
|
|
- **Quorum Problems**: Cluster appears to be having election issues
|
|
|
|
### Log Patterns
|
|
```
|
|
- mon.ml110-01@0(electing) e2 get_health_metrics reporting 100+ slow ops
|
|
- ENGINE Bus STOPPING/STARTING (constant restarts)
|
|
- libceph: mon0/mon1 socket closed (con state OPEN)
|
|
- Session lost, hunting for new mon
|
|
```
|
|
|
|
---
|
|
|
|
## Impact Analysis
|
|
|
|
### VMs Affected
|
|
- **Storage Type**: `ceph-fs`
|
|
- **Count**: Multiple VMs configured to use ceph-fs
|
|
- **Risk**: VM creation may fail if Ceph is unavailable
|
|
|
|
### Potential Issues
|
|
1. **VM Creation Failures**: VMs using ceph-fs may fail to create
|
|
2. **Storage Unavailable**: Ceph storage may be inaccessible
|
|
3. **Performance Degradation**: Slow operations indicate cluster stress
|
|
4. **Data Risk**: Instability could affect data integrity
|
|
|
|
---
|
|
|
|
## Root Cause Analysis
|
|
|
|
### Possible Causes
|
|
1. **Network Issues**: Connection problems between mon0 (ML110-01) and mon1 (R630-01)
|
|
2. **Quorum Loss**: Cluster may have lost quorum
|
|
3. **Resource Exhaustion**: System resources may be exhausted
|
|
4. **Configuration Issues**: Ceph configuration may be incorrect
|
|
5. **Time Sync**: Clock synchronization issues
|
|
|
|
---
|
|
|
|
## Investigation Steps
|
|
|
|
### 1. Check Ceph Cluster Health
|
|
```bash
|
|
ssh root@192.168.11.10 "ceph -s"
|
|
ssh root@192.168.11.10 "ceph health detail"
|
|
```
|
|
|
|
### 2. Verify Quorum Status
|
|
```bash
|
|
ssh root@192.168.11.10 "ceph quorum_status"
|
|
```
|
|
|
|
### 3. Check Monitor Status
|
|
```bash
|
|
ssh root@192.168.11.10 "ceph mon stat"
|
|
ssh root@192.168.11.10 "ceph mon dump"
|
|
```
|
|
|
|
### 4. Check Network Connectivity
|
|
```bash
|
|
# From ML110-01 to R630-01
|
|
ssh root@192.168.11.10 "ping -c 3 192.168.11.11"
|
|
ssh root@192.168.11.10 "telnet 192.168.11.11 6789"
|
|
```
|
|
|
|
### 5. Check System Resources
|
|
```bash
|
|
ssh root@192.168.11.10 "df -h"
|
|
ssh root@192.168.11.10 "free -h"
|
|
ssh root@192.168.11.10 "iostat -x 1 3"
|
|
```
|
|
|
|
---
|
|
|
|
## Immediate Actions
|
|
|
|
### 1. Verify Impact on VM Creation
|
|
- Check if any VMs have failed due to storage issues
|
|
- Monitor provider logs for Ceph-related errors
|
|
- Verify if VM creation is blocked
|
|
|
|
### 2. Stabilize Ceph Cluster
|
|
- Check quorum status
|
|
- Verify network connectivity
|
|
- Restart Ceph services if needed
|
|
- Check for configuration issues
|
|
|
|
### 3. Alternative Storage
|
|
- Consider using `local-lvm` for critical VMs temporarily
|
|
- Document which VMs require ceph-fs
|
|
- Plan migration if Ceph remains unstable
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
### Short-term
|
|
1. **Monitor Ceph Health**: Continuously monitor cluster status
|
|
2. **Check Network**: Verify connectivity between nodes
|
|
3. **Review Logs**: Analyze Ceph logs for root cause
|
|
4. **Resource Check**: Verify system resources are adequate
|
|
|
|
### Long-term
|
|
1. **Ceph Optimization**: Optimize Ceph configuration
|
|
2. **Network Hardening**: Ensure stable network between nodes
|
|
3. **Monitoring**: Set up Ceph health monitoring
|
|
4. **Backup Strategy**: Ensure data backup strategy is in place
|
|
|
|
---
|
|
|
|
## Status
|
|
|
|
⚠️ **INVESTIGATING** - Ceph cluster instability detected, impact on VM deployment being assessed.
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-12-13
|
|
**Status**: ⚠️ **INVESTIGATING**
|
|
|