Sankofa/docs/ceph/ISSUE_ANALYSIS.md

# Ceph OSD Creation Issue - Comprehensive Analysis

**Date**: 2025-12-13  
**Status**: 🔍 **ANALYZING ROOT CAUSES**

---

## Problem Summary

OSD creation commands are **hanging/timing out** when trying to create OSDs on the 6x 250GB drives. The process stops at the "osd new" step, which requires cluster authentication.

---

## Root Cause Analysis

### Issue 1: Bootstrap Keyring Missing/Inaccessible

**Symptom**:
```
unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
RADOS timed out (error connecting to the cluster)
RuntimeError: Unable to create a new OSD id
```

**Root Cause**:
- `ceph-volume` needs bootstrap keyring to authenticate with Ceph cluster
- Keyring should be at `/var/lib/ceph/bootstrap-osd/ceph.keyring` OR `/etc/pve/priv/ceph.client.bootstrap-osd.keyring`
- Neither location has the keyring, or it's not accessible

**Why It Happens**:
- Ceph cluster may not have bootstrap-osd client created
- Keyring may have been deleted or never created
- Proxmox Ceph integration may use different keyring location

### Issue 2: Ceph Cluster Connectivity

**Symptom**:
- Commands hang at "osd new" step
- "RADOS timed out" errors
- Cannot connect to cluster

**Possible Causes**:
1. **Ceph monitors not accessible** from R630-01
2. **Network connectivity issues** between nodes
3. **Ceph services not running** properly
4. **Firewall blocking** Ceph ports (6789 for monitors)

### Issue 3: Ceph Cluster Configuration

**Current State**:
- 2 OSDs (one on ml110-01, one on r630-01)
- Ceph cluster exists and is running
- Health warnings present
- Cluster is functional but degraded

**Questions**:
- Is this a Proxmox-managed Ceph cluster?
- Or a standalone Ceph cluster?
- Where are the monitors running?
- What's the cluster configuration?

---

## Detailed Investigation Needed

### Step 1: Check Ceph Cluster Status

```bash
# On R630-01
ceph health
ceph mon stat
ceph mon dump
ceph osd tree
```

**What to look for**:
- Are monitors accessible?
- What are monitor addresses?
- Is cluster in quorum?
- Can we connect to cluster?

### Step 2: Check Bootstrap Keyring

```bash
# Check standard location
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring

# Check Proxmox location
ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring

# Check if bootstrap-osd client exists
ceph auth list | grep bootstrap-osd
```

**What to look for**:
- Does keyring exist?
- Is it readable?
- Does bootstrap-osd client exist in cluster?

### Step 3: Check Ceph Services

```bash
# Check Ceph services
systemctl status ceph.target
systemctl status ceph-mon@*
systemctl status ceph-osd@*

# Check if monitors are running
ps aux | grep ceph-mon

# Check network connectivity
netstat -tlnp | grep 6789
```

**What to look for**:
- Are Ceph services running?
- Are monitors listening on port 6789?
- Can we reach monitor addresses?

### Step 4: Check Network Connectivity

```bash
# Check if we can reach monitors
# First, find monitor addresses
ceph mon dump

# Then test connectivity
ping <monitor-ip>
telnet <monitor-ip> 6789
```

**What to look for**:
- Can we ping monitor IPs?
- Can we connect to port 6789?
- Are there firewall rules blocking?

---

## Likely Scenarios

### Scenario 1: Proxmox-Managed Ceph

**If Ceph was set up via Proxmox**:
- Keyring should be in `/etc/pve/priv/`
- May need to use `pveceph` commands instead
- Proxmox manages Ceph configuration

**Solution**:
```bash
# Use Proxmox Ceph tool
pveceph create /dev/sdc
```

### Scenario 2: Standalone Ceph Cluster

**If Ceph was set up manually**:
- Keyring should be in `/var/lib/ceph/bootstrap-osd/`
- Need to create/bootstrap keyring
- Standard Ceph commands should work

**Solution**:
```bash
# Create/bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
```

### Scenario 3: Cluster Connectivity Issue

**If monitors are not accessible**:
- Network issue between nodes
- Firewall blocking
- Monitors not running

**Solution**:
- Fix network connectivity
- Check firewall rules
- Ensure monitors are running

---

## Diagnostic Commands

Run these on R630-01 to diagnose:

```bash
# 1. Ceph cluster status
ceph health
ceph mon stat
ceph osd tree

# 2. Bootstrap keyring check
ls -la /var/lib/ceph/bootstrap-osd/ 2>/dev/null
ls -la /etc/pve/priv/ceph.client.bootstrap-osd.keyring 2>/dev/null
ceph auth list | grep bootstrap

# 3. Ceph services
systemctl status ceph.target
systemctl status ceph-mon@*

# 4. Network connectivity
ceph mon dump | grep -E "rank|addr"
# Then test connectivity to monitor IPs

# 5. Check if pveceph available
which pveceph
pveceph status 2>/dev/null || echo "pveceph not available"
```

---

## Recommended Fix Strategy

### Option A: Use Proxmox Ceph Tool (If Available)

```bash
# Check if available
which pveceph

# If yes, use it
for drive in sdc sdd sde sdf sdg sdh; do
    pveceph create /dev/$drive
done
```

### Option B: Fix Bootstrap Keyring

```bash
# 1. Check if cluster is accessible
ceph health

# 2. Get or create bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring 2>/dev/null || \
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring

# 3. Verify
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring

# 4. Create OSDs
for drive in sdc sdd sde sdf sdg sdh; do
    ceph-volume lvm create --data /dev/$drive
done
```

### Option C: Check Cluster Connectivity First

```bash
# 1. Verify cluster is accessible
ceph health
ceph mon stat

# 2. If not accessible, check:
#    - Network connectivity
#    - Firewall rules
#    - Monitor services
#    - Cluster configuration
```

---

## Key Questions to Answer

1. **Is Ceph cluster accessible from R630-01?**
   - Can we run `ceph health` successfully?
   - Are monitors reachable?

2. **Where is the bootstrap keyring?**
   - Does it exist?
   - Is it in the right location?
   - Is it readable?

3. **Is this Proxmox-managed Ceph?**
   - Should we use `pveceph`?
   - Is Ceph integrated with Proxmox?

4. **What's the cluster configuration?**
   - Where are monitors running?
   - What's the cluster name?
   - What's the authentication method?

---

## Next Steps

1. **Run diagnostic commands** to understand current state
2. **Identify root cause** (keyring, connectivity, or configuration)
3. **Apply appropriate fix** based on findings
4. **Retry OSD creation** after fix

---

**Last Updated**: 2025-12-13  
**Status**: 🔍 **NEEDS DIAGNOSTIC INFORMATION**