Sankofa/docs/ceph/FIX_BOOTSTRAP_KEYRING.md

Fix Ceph Bootstrap Keyring Issue

Date: 2025-12-13
Issue: OSD creation failing due to missing bootstrap keyring

---

Problem

When creating OSDs, getting error:

unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring
RADOS timed out (error connecting to the cluster)
RuntimeError: Unable to create a new OSD id

---

Solution Options

Option 1: Get Bootstrap Keyring from Cluster

# Create directory if needed
mkdir -p /var/lib/ceph/bootstrap-osd

# Get bootstrap keyring from Ceph cluster
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

Option 2: Use Proxmox pveceph (If Available)

Proxmox has its own Ceph management tool:

# Check if available
which pveceph

# Create OSD using pveceph
pveceph create /dev/sdc
pveceph create /dev/sdd
# ... etc

Option 3: Check Ceph Cluster Connectivity

The timeout suggests the cluster might not be accessible:

# Check Ceph health
ceph health

# Check monitors
ceph mon stat

# Check if Ceph services are running
systemctl status ceph.target
systemctl status ceph-mon@*
systemctl status ceph-osd@*

Option 4: Create Bootstrap Keyring Manually

If the cluster is accessible but keyring is missing:

# Check existing auth
ceph auth list

# If bootstrap-osd exists, export it
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# If it doesn't exist, create it
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring

---

Quick Fix Script

Run this on R630-01:

# 1. Check Ceph connectivity
ceph health
ceph mon stat

# 2. Get or create bootstrap keyring
mkdir -p /var/lib/ceph/bootstrap-osd
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring 2>/dev/null || \
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i /var/lib/ceph/bootstrap-osd/ceph.keyring

# 3. Verify keyring exists
ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring

# 4. Try creating OSD again
ceph-volume lvm create --data /dev/sdc

---

Alternative: Use pveceph

If you're using Proxmox's Ceph integration:

# Create OSDs using Proxmox tool
pveceph create /dev/sdc
pveceph create /dev/sdd
pveceph create /dev/sde
pveceph create /dev/sdf
pveceph create /dev/sdg
pveceph create /dev/sdh

---

Troubleshooting

Issue: "RADOS timed out"

Cause: Cannot connect to Ceph cluster

Solution:

# Check if monitors are running
systemctl status ceph-mon@*

# Check monitor addresses
ceph mon dump

# Check network connectivity
ping <monitor-ip>

Issue: "no keyring found"

Cause: Bootstrap keyring missing

Solution:

# Get from cluster
ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

# Or create new one
ceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd'

Issue: "ceph command not found"

Cause: Ceph tools not installed

Solution:

# Install Ceph tools (if needed)
apt-get update
apt-get install -y ceph-common

---

Next Steps

After fixing the bootstrap keyring:

1. Verify keyring exists:

   ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
   

2. Create OSDs:

   for drive in sdc sdd sde sdf sdg sdh; do
       ceph-volume lvm create --data /dev/$drive
   done
   

3. Verify:

   ceph osd tree
   ceph health
   

---

Last Updated: 2025-12-13