Sankofa/docs/ceph/FIX_BOOTSTRAP_KEYRING.md

# Fix Ceph Bootstrap Keyring Issue

**Date**: 2025-12-13  
**Issue**: OSD creation failing due to missing bootstrap keyring

---

## Problem

When creating OSDs, getting error:
```
unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
RADOS timed out (error connecting to the cluster)
RuntimeError: Unable to create a new OSD id
```

---

## Solution Options

### Option 1: Get Bootstrap Keyring from Cluster

```bash
# Create directory if needed
mkdir -p /var/lib/ceph/bootstrap-osd

# Get bootstrap keyring from Ceph cluster
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring
```

### Option 2: Use Proxmox pveceph (If Available)

Proxmox has its own Ceph management tool:

```bash
# Check if available
which pveceph

# Create OSD using pveceph
pveceph create /dev/sdc
pveceph create /dev/sdd
# ... etc
```

### Option 3: Check Ceph Cluster Connectivity

The timeout suggests the cluster might not be accessible:

```bash
# Check Ceph health
ceph health

# Check monitors
ceph mon stat

# Check if Ceph services are running
systemctl status ceph.target
systemctl status ceph-mon@*
systemctl status ceph-osd@*
```

### Option 4: Create Bootstrap Keyring Manually

If the cluster is accessible but keyring is missing:

```bash
# Check existing auth
ceph auth list

# If bootstrap-osd exists, export it
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# If it doesn't exist, create it
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring
```

---

## Quick Fix Script

Run this on R630-01:

```bash
# 1. Check Ceph connectivity
ceph health
ceph mon stat

# 2. Get or create bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring 2>/dev/null || \
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring

# 3. Verify keyring exists
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring

# 4. Try creating OSD again
ceph-volume lvm create --data /dev/sdc
```

---

## Alternative: Use pveceph

If you're using Proxmox's Ceph integration:

```bash
# Create OSDs using Proxmox tool
pveceph create /dev/sdc
pveceph create /dev/sdd
pveceph create /dev/sde
pveceph create /dev/sdf
pveceph create /dev/sdg
pveceph create /dev/sdh
```

---

## Troubleshooting

### Issue: "RADOS timed out"

**Cause**: Cannot connect to Ceph cluster

**Solution**:
```bash
# Check if monitors are running
systemctl status ceph-mon@*

# Check monitor addresses
ceph mon dump

# Check network connectivity
ping <monitor-ip>
```

### Issue: "no keyring found"

**Cause**: Bootstrap keyring missing

**Solution**:
```bash
# Get from cluster
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# Or create new one
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd'
```

### Issue: "ceph command not found"

**Cause**: Ceph tools not installed

**Solution**:
```bash
# Install Ceph tools (if needed)
apt-get update
apt-get install -y ceph-common
```

---

## Next Steps

After fixing the bootstrap keyring:

1. **Verify keyring exists**:
   ```bash
   ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
   ```

2. **Create OSDs**:
   ```bash
   for drive in sdc sdd sde sdf sdg sdh; do
       ceph-volume lvm create --data /dev/$drive
   done
   ```

3. **Verify**:
   ```bash
   ceph osd tree
   ceph health
   ```

---

**Last Updated**: 2025-12-13