Sankofa/docs/ceph/ISSUE_ANALYSIS.md

Ceph OSD Creation Issue - Comprehensive Analysis

Date: 2025-12-13
Status: 🔍 ANALYZING ROOT CAUSES

---

Problem Summary

OSD creation commands are hanging/timing out when trying to create OSDs on the 6x 250GB drives. The process stops at the "osd new" step, which requires cluster authentication.

---

Root Cause Analysis

Issue 1: Bootstrap Keyring Missing/Inaccessible

Symptom:

unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
RADOS timed out (error connecting to the cluster)
RuntimeError: Unable to create a new OSD id

Root Cause:

• ceph-volume needs bootstrap keyring to authenticate with Ceph cluster
• Keyring should be at /var/lib/ceph/bootstrap-osd/ceph.keyring OR /etc/pve/priv/ceph.client.bootstrap-osd.keyring
• Neither location has the keyring, or it's not accessible

Why It Happens:

• Ceph cluster may not have bootstrap-osd client created
• Keyring may have been deleted or never created
• Proxmox Ceph integration may use different keyring location

Issue 2: Ceph Cluster Connectivity

Symptom:

• Commands hang at "osd new" step
• "RADOS timed out" errors
• Cannot connect to cluster

Possible Causes:

1. Ceph monitors not accessible from R630-01
2. Network connectivity issues between nodes
3. Ceph services not running properly
4. Firewall blocking Ceph ports (6789 for monitors)

Issue 3: Ceph Cluster Configuration

Current State:

• 2 OSDs (one on ml110-01, one on r630-01)
• Ceph cluster exists and is running
• Health warnings present
• Cluster is functional but degraded

Questions:

• Is this a Proxmox-managed Ceph cluster?
• Or a standalone Ceph cluster?
• Where are the monitors running?
• What's the cluster configuration?

---

Detailed Investigation Needed

Step 1: Check Ceph Cluster Status

# On R630-01
ceph health
ceph mon stat
ceph mon dump
ceph osd tree

What to look for:

• Are monitors accessible?
• What are monitor addresses?
• Is cluster in quorum?
• Can we connect to cluster?

Step 2: Check Bootstrap Keyring

# Check standard location
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring

# Check Proxmox location
ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring

# Check if bootstrap-osd client exists
ceph auth list | grep bootstrap-osd

What to look for:

• Does keyring exist?
• Is it readable?
• Does bootstrap-osd client exist in cluster?

Step 3: Check Ceph Services

# Check Ceph services
systemctl status ceph.target
systemctl status ceph-mon@*
systemctl status ceph-osd@*

# Check if monitors are running
ps aux | grep ceph-mon

# Check network connectivity
netstat -tlnp | grep 6789

What to look for:

• Are Ceph services running?
• Are monitors listening on port 6789?
• Can we reach monitor addresses?

Step 4: Check Network Connectivity

# Check if we can reach monitors
# First, find monitor addresses
ceph mon dump

# Then test connectivity
ping <monitor-ip>
telnet <monitor-ip> 6789

What to look for:

• Can we ping monitor IPs?
• Can we connect to port 6789?
• Are there firewall rules blocking?

---

Likely Scenarios

Scenario 1: Proxmox-Managed Ceph

If Ceph was set up via Proxmox:

• Keyring should be in /etc/pve/priv/
• May need to use pveceph commands instead
• Proxmox manages Ceph configuration

Solution:

# Use Proxmox Ceph tool
pveceph create /dev/sdc

Scenario 2: Standalone Ceph Cluster

If Ceph was set up manually:

• Keyring should be in /var/lib/ceph/bootstrap-osd/
• Need to create/bootstrap keyring
• Standard Ceph commands should work

Solution:

# Create/bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

Scenario 3: Cluster Connectivity Issue

If monitors are not accessible:

• Network issue between nodes
• Firewall blocking
• Monitors not running

Solution:

• Fix network connectivity
• Check firewall rules
• Ensure monitors are running

---

Diagnostic Commands

Run these on R630-01 to diagnose:

# 1. Ceph cluster status
ceph health
ceph mon stat
ceph osd tree

# 2. Bootstrap keyring check
ls -la /var/lib/ceph/bootstrap-osd/ 2>/dev/null
ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring 2>/dev/null
ceph auth list | grep bootstrap

# 3. Ceph services
systemctl status ceph.target
systemctl status ceph-mon@*

# 4. Network connectivity
ceph mon dump | grep -E "rank|addr"
# Then test connectivity to monitor IPs

# 5. Check if pveceph available
which pveceph
pveceph status 2>/dev/null || echo "pveceph not available"

---

Recommended Fix Strategy

Option A: Use Proxmox Ceph Tool (If Available)

# Check if available
which pveceph

# If yes, use it
for drive in sdc sdd sde sdf sdg sdh; do
    pveceph create /dev/$drive
done

Option B: Fix Bootstrap Keyring

# 1. Check if cluster is accessible
ceph health

# 2. Get or create bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring 2>/dev/null || \
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring

# 3. Verify
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring

# 4. Create OSDs
for drive in sdc sdd sde sdf sdg sdh; do
    ceph-volume lvm create --data /dev/$drive
done

Option C: Check Cluster Connectivity First

# 1. Verify cluster is accessible
ceph health
ceph mon stat

# 2. If not accessible, check:
#    - Network connectivity
#    - Firewall rules
#    - Monitor services
#    - Cluster configuration

---

Key Questions to Answer

1. Is Ceph cluster accessible from R630-01?

   • Can we run ceph health successfully?
   • Are monitors reachable?

2. Where is the bootstrap keyring?

   • Does it exist?
   • Is it in the right location?
   • Is it readable?

3. Is this Proxmox-managed Ceph?

   • Should we use pveceph?
   • Is Ceph integrated with Proxmox?

4. What's the cluster configuration?

   • Where are monitors running?
   • What's the cluster name?
   • What's the authentication method?

---

Next Steps

1. Run diagnostic commands to understand current state
2. Identify root cause (keyring, connectivity, or configuration)
3. Apply appropriate fix based on findings
4. Retry OSD creation after fix

---

Last Updated: 2025-12-13
Status: 🔍 NEEDS DIAGNOSTIC INFORMATION