chore: consolidate local WIP (repo cleanup 20260707)
Some checks failed
API CI / API Lint (push) Successful in 47s
API CI / API Type Check (push) Failing after 47s
API CI / API Test (push) Successful in 1m0s
API CI / API Build (push) Failing after 50s
API CI / Build Docker Image (push) Has been skipped
Build Crossplane Provider / build (push) Failing after 5m51s
CD Pipeline / Deploy to Staging (push) Failing after 29s
CI Pipeline / Lint and Type Check (push) Failing after 36s
CI Pipeline / Build (push) Has been skipped
CI Pipeline / Test Backend (push) Failing after 1m33s
CI Pipeline / Test Frontend (push) Failing after 30s
CI Pipeline / Security Scan (push) Failing after 1m16s
Crossplane Provider CI / Go Test (push) Failing after 3m23s
Crossplane Provider CI / Go Lint (push) Failing after 7m27s
Crossplane Provider CI / Go Build (push) Failing after 3m27s
Deploy to Staging / Deploy to Staging (push) Failing after 30s
Portal CI / Portal Lint (push) Failing after 21s
Portal CI / Portal Type Check (push) Failing after 21s
Portal CI / Portal Test (push) Failing after 21s
Portal CI / Portal Build (push) Failing after 22s
Test Suite / frontend-tests (push) Failing after 30s
Test Suite / api-tests (push) Failing after 49s
Test Suite / blockchain-tests (push) Failing after 30s
Type Check / type-check (map[directory:. name:root]) (push) Failing after 23s
Type Check / type-check (map[directory:api name:api]) (push) Failing after 21s
Type Check / type-check (map[directory:portal name:portal]) (push) Failing after 19s
Validate Configuration Files / validate (push) Failing after 1m52s
CD Pipeline / Deploy to Production (push) Has been skipped
Some checks failed
API CI / API Lint (push) Successful in 47s
API CI / API Type Check (push) Failing after 47s
API CI / API Test (push) Successful in 1m0s
API CI / API Build (push) Failing after 50s
API CI / Build Docker Image (push) Has been skipped
Build Crossplane Provider / build (push) Failing after 5m51s
CD Pipeline / Deploy to Staging (push) Failing after 29s
CI Pipeline / Lint and Type Check (push) Failing after 36s
CI Pipeline / Build (push) Has been skipped
CI Pipeline / Test Backend (push) Failing after 1m33s
CI Pipeline / Test Frontend (push) Failing after 30s
CI Pipeline / Security Scan (push) Failing after 1m16s
Crossplane Provider CI / Go Test (push) Failing after 3m23s
Crossplane Provider CI / Go Lint (push) Failing after 7m27s
Crossplane Provider CI / Go Build (push) Failing after 3m27s
Deploy to Staging / Deploy to Staging (push) Failing after 30s
Portal CI / Portal Lint (push) Failing after 21s
Portal CI / Portal Type Check (push) Failing after 21s
Portal CI / Portal Test (push) Failing after 21s
Portal CI / Portal Build (push) Failing after 22s
Test Suite / frontend-tests (push) Failing after 30s
Test Suite / api-tests (push) Failing after 49s
Test Suite / blockchain-tests (push) Failing after 30s
Type Check / type-check (map[directory:. name:root]) (push) Failing after 23s
Type Check / type-check (map[directory:api name:api]) (push) Failing after 21s
Type Check / type-check (map[directory:portal name:portal]) (push) Failing after 19s
Validate Configuration Files / validate (push) Failing after 1m52s
CD Pipeline / Deploy to Production (push) Has been skipped
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
169
docs/ceph/COMPLETE_OSD_FIX.md
Normal file
169
docs/ceph/COMPLETE_OSD_FIX.md
Normal file
@@ -0,0 +1,169 @@
|
||||
# Complete OSD Creation Fix
|
||||
|
||||
**Date**: 2025-12-13
|
||||
**Status**: ✅ **READY TO EXECUTE**
|
||||
|
||||
---
|
||||
|
||||
## Current Situation
|
||||
|
||||
✅ **Drives wiped**: All 6 drives (sdc-sdh) are ready
|
||||
❌ **OSD creation failed**: Bootstrap keyring issue
|
||||
🎯 **Solution**: Fix bootstrap keyring and create OSDs
|
||||
|
||||
---
|
||||
|
||||
## Quick Fix - Run This Script
|
||||
|
||||
On R630-01:
|
||||
|
||||
```bash
|
||||
# Copy script
|
||||
scp scripts/fix-and-create-all-osds.sh root@192.168.11.11:/tmp/
|
||||
|
||||
# SSH and run
|
||||
ssh root@192.168.11.11
|
||||
bash /tmp/fix-and-create-all-osds.sh
|
||||
```
|
||||
|
||||
This script will:
|
||||
1. ✅ Check Ceph cluster connectivity
|
||||
2. ✅ Fix/bootstrap keyring issue
|
||||
3. ✅ Create OSDs on all 6 drives
|
||||
4. ✅ Verify results
|
||||
|
||||
---
|
||||
|
||||
## Manual Method
|
||||
|
||||
If you prefer manual steps:
|
||||
|
||||
### Step 1: Fix Bootstrap Keyring
|
||||
|
||||
```bash
|
||||
# Create directory
|
||||
mkdir -p /var/lib/ceph/bootstrap-osd
|
||||
|
||||
# Get bootstrap keyring from cluster
|
||||
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
|
||||
|
||||
# Verify
|
||||
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
|
||||
```
|
||||
|
||||
### Step 2: Create OSDs
|
||||
|
||||
```bash
|
||||
# Method 1: Using ceph-volume
|
||||
for drive in sdc sdd sde sdf sdg sdh; do
|
||||
ceph-volume lvm create --data /dev/$drive
|
||||
done
|
||||
|
||||
# Method 2: Using pveceph (if available)
|
||||
for drive in sdc sdd sde sdf sdg sdh; do
|
||||
pveceph create /dev/$drive
|
||||
done
|
||||
```
|
||||
|
||||
### Step 3: Verify
|
||||
|
||||
```bash
|
||||
# Check OSD tree
|
||||
ceph osd tree
|
||||
|
||||
# Check health
|
||||
ceph health
|
||||
ceph health detail
|
||||
|
||||
# Check OSD details
|
||||
ceph osd df
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Before
|
||||
- 2 OSDs (insufficient)
|
||||
- HEALTH_WARN: TOO_FEW_OSDS
|
||||
|
||||
### After (with all 6 OSDs)
|
||||
- **8 OSDs total** (2 existing + 6 new)
|
||||
- Excellent redundancy
|
||||
- HEALTH_OK or much improved
|
||||
- Can handle many VMs
|
||||
|
||||
### After (with just 1 OSD)
|
||||
- **3 OSDs total** (minimum for 3-way replication)
|
||||
- Fixes TOO_FEW_OSDS error
|
||||
- HEALTH_OK or improved
|
||||
- VMs can be created
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: "ceph auth get" fails
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check if bootstrap-osd exists
|
||||
ceph auth list | grep bootstrap
|
||||
|
||||
# If it doesn't exist, create it
|
||||
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd'
|
||||
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
|
||||
```
|
||||
|
||||
### Issue: "RADOS timed out"
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check Ceph services
|
||||
systemctl status ceph.target
|
||||
systemctl status ceph-mon@*
|
||||
|
||||
# Check monitor status
|
||||
ceph mon stat
|
||||
|
||||
# Check network connectivity
|
||||
ping <monitor-ip>
|
||||
```
|
||||
|
||||
### Issue: OSD creation still fails
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check logs
|
||||
cat /tmp/ceph-osd-*.log
|
||||
|
||||
# Try with verbose output
|
||||
ceph-volume lvm create --data /dev/sdc --verbose
|
||||
|
||||
# Or try pveceph
|
||||
pveceph create /dev/sdc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
### Current State
|
||||
- ✅ 6 drives wiped and ready
|
||||
- ⚠️ Bootstrap keyring missing
|
||||
- 🎯 Need to fix and create OSDs
|
||||
|
||||
### Action
|
||||
Run `fix-and-create-all-osds.sh` script
|
||||
|
||||
### Expected Result
|
||||
- 8 OSDs total (if all succeed)
|
||||
- Or 3+ OSDs (minimum needed)
|
||||
- Ceph health improves
|
||||
- VMs can be created
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-13
|
||||
**Status**: ✅ **READY TO EXECUTE**
|
||||
|
||||
Reference in New Issue
Block a user