Sankofa/docs/ceph/ISSUES_SUMMARY.md

Ceph OSD Creation Issues - Summary

Date: 2025-12-13
Status: 🔍 ISSUES IDENTIFIED - NEEDS DIAGNOSIS

---

Issues Identified

1. OSD Creation Hanging/Timeout

Symptom: ceph-volume lvm create hangs at "osd new" step

Root Causes (likely):

• ❌ Bootstrap keyring missing - Cannot authenticate with cluster
• ❌ Ceph cluster not accessible - RADOS timeout errors
• ❌ Monitor connectivity issues - Cannot reach Ceph monitors

2. Bootstrap Keyring Issue

Error:

unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
RADOS timed out (error connecting to the cluster)

Possible Locations:

• /var/lib/ceph/bootstrap-osd/ceph.keyring (standard)
• /etc/pve/priv/ceph.client.bootstrap-osd.keyring (Proxmox)

Status: Unknown - needs verification

3. Ceph Cluster Connectivity

Issue: Commands timeout when trying to connect to cluster

Possible Causes:

• Monitors not running
• Network connectivity issues
• Firewall blocking ports
• Cluster configuration issues

---

Diagnostic Steps

Run these commands on R630-01 to diagnose:

Quick Check

# 1. Check Ceph health (does it respond?)
ceph health

# 2. Check monitors
ceph mon stat

# 3. Check bootstrap keyring
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring

# 4. Check if pveceph available
which pveceph

Full Diagnostic

# Run comprehensive diagnostic
bash /tmp/diagnose-ceph-issues.sh

---

Likely Solutions

Solution 1: Use pveceph (If Proxmox-Managed)

If pveceph is available:

for drive in sdc sdd sde sdf sdg sdh; do
    pveceph create /dev/$drive
done

Solution 2: Fix Bootstrap Keyring

If cluster is accessible:

# Get bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# Then create OSDs
for drive in sdc sdd sde sdf sdg sdh; do
    ceph-volume lvm create --data /dev/$drive
done

Solution 3: Fix Cluster Connectivity

If cluster is not accessible:

1. Check monitor services
2. Check network connectivity
3. Check firewall rules
4. Fix connectivity issues
5. Then retry OSD creation

---

Current State

• ✅ 6x 250GB drives: Wiped and ready
• ❌ OSD creation: Hanging due to authentication/connectivity
• 🔍 Root cause: Needs diagnosis

---

Next Action

Run diagnostic script to understand the issue:

# On R630-01
bash /tmp/diagnose-ceph-issues.sh

This will show:

• Ceph cluster status
• Bootstrap keyring locations
• Monitor connectivity
• Service status
• Network connectivity

Then we can apply the appropriate fix.

---

Last Updated: 2025-12-13