Sankofa/docs/ceph/FIX_R630_QUORUM.md

Fix R630-01 Monitor Quorum Issue

Problem

R630-01 monitor is not joining the quorum. Only ml110-01 is in quorum.

Diagnosis

From your terminal output:

• Quorum only shows: ["ml110-01"]
• Monmap only has: ml110-01 monitor
• R630-01 monitor is not in the cluster

Solution Steps

Step 1: Check R630-01 Monitor Status

On r630-01:

systemctl status ceph-mon@r630-01
journalctl -u ceph-mon@r630-01 -n 50
ls -la /var/lib/ceph/mon/ceph-r630-01/

Step 2: Check if Monitor Directory Exists

# If directory is missing or empty, create it
ls -la /var/lib/ceph/mon/ceph-r630-01/

Step 3: Add R630-01 to Monmap

On ml110-01 (where quorum exists):

# Get current monmap
ceph mon getmap -o /tmp/monmap

# Check if r630-01 is in monmap
monmaptool --print /tmp/monmap | grep r630-01

# If not present, add it manually
# First get FSID
FSID=$(grep fsid /etc/pve/ceph.conf | awk '{print $3}')

# Add r630-01 to monmap
monmaptool --add r630-01 192.168.11.11 --fsid $FSID /tmp/monmap

# Inject updated monmap
ceph mon setmap -i /tmp/monmap

Step 4: Create Monitor on R630-01

On r630-01:

# Option 1: Use pveceph (recommended)
pveceph mon create

# Option 2: Manual creation if pveceph fails
# Get monmap from ml110-01
scp root@192.168.11.10:/tmp/monmap /tmp/

# Create monitor directory
mkdir -p /var/lib/ceph/mon/ceph-r630-01
chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01

# Create keyring
ceph-authtool --create-keyring /var/lib/ceph/mon/ceph-r630-01/keyring --gen-key -n mon.

# Create monitor filesystem
ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /var/lib/ceph/mon/ceph-r630-01/keyring

# Set ownership
chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01

Step 5: Start Monitor Service

On r630-01:

systemctl enable ceph-mon@r630-01
systemctl start ceph-mon@r630-01
systemctl status ceph-mon@r630-01

Step 6: Verify Quorum

On either node:

ceph quorum_status
ceph mon stat
ceph -s

You should see both monitors in quorum:

"quorum": [0, 1],
"quorum_names": ["ml110-01", "r630-01"]

Alternative: Recreate Monitor

If the above doesn't work, recreate the monitor:

On r630-01:

# Stop monitor
systemctl stop ceph-mon@r630-01
systemctl disable ceph-mon@r630-01

# Remove monitor directory
rm -rf /var/lib/ceph/mon/ceph-r630-01

# Recreate
pveceph mon create

# Start
systemctl enable ceph-mon@r630-01
systemctl start ceph-mon@r630-01

Troubleshooting

Monitor Service Fails to Start

Check logs:

journalctl -u ceph-mon@r630-01 -n 100

Common issues:

• "monitor data directory does not exist": Run pveceph mon create
• "unable to bind": Check firewall, ports 6789 and 3300
• "authentication": Check keyring permissions

Monitor Not in Quorum

1. Verify network connectivity:

   ping 192.168.11.10
   ping 192.168.11.11
   

2. Check firewall:

   ufw allow 6789/tcp
   ufw allow 3300/tcp
   

3. Verify monmap includes both monitors:

   ceph mon dump
   

Still Not Working

If monitor still won't join:

1. Check if both nodes are in Proxmox cluster: pvecm status
2. Verify /etc/pve/ceph.conf is identical on both nodes
3. Check FSID matches on both nodes
4. Ensure time is synchronized: chrony sources