Sankofa/docs/ceph/FIX_R630_QUORUM.md

# Fix R630-01 Monitor Quorum Issue

## Problem
R630-01 monitor is not joining the quorum. Only ml110-01 is in quorum.

## Diagnosis

From your terminal output:
- Quorum only shows: `["ml110-01"]`
- Monmap only has: `ml110-01` monitor
- R630-01 monitor is not in the cluster

## Solution Steps

### Step 1: Check R630-01 Monitor Status

On **r630-01**:

```bash
systemctl status ceph-mon@r630-01
journalctl -u ceph-mon@r630-01 -n 50
ls -la /var/lib/ceph/mon/ceph-r630-01/
```

### Step 2: Check if Monitor Directory Exists

```bash
# If directory is missing or empty, create it
ls -la /var/lib/ceph/mon/ceph-r630-01/
```

### Step 3: Add R630-01 to Monmap

On **ml110-01** (where quorum exists):

```bash
# Get current monmap
ceph mon getmap -o /tmp/monmap

# Check if r630-01 is in monmap
monmaptool --print /tmp/monmap | grep r630-01

# If not present, add it manually
# First get FSID
FSID=$(grep fsid /etc/pve/ceph.conf | awk '{print $3}')

# Add r630-01 to monmap
monmaptool --add r630-01 192.168.11.11 --fsid $FSID /tmp/monmap

# Inject updated monmap
ceph mon setmap -i /tmp/monmap
```

### Step 4: Create Monitor on R630-01

On **r630-01**:

```bash
# Option 1: Use pveceph (recommended)
pveceph mon create

# Option 2: Manual creation if pveceph fails
# Get monmap from ml110-01
scp root@192.168.11.10:/tmp/monmap /tmp/

# Create monitor directory
mkdir -p /var/lib/ceph/mon/ceph-r630-01
chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01

# Create keyring
ceph-authtool --create-keyring /var/lib/ceph/mon/ceph-r630-01/keyring --gen-key -n mon.

# Create monitor filesystem
ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /var/lib/ceph/mon/ceph-r630-01/keyring

# Set ownership
chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
```

### Step 5: Start Monitor Service

On **r630-01**:

```bash
systemctl enable ceph-mon@r630-01
systemctl start ceph-mon@r630-01
systemctl status ceph-mon@r630-01
```

### Step 6: Verify Quorum

On **either node**:

```bash
ceph quorum_status
ceph mon stat
ceph -s
```

You should see both monitors in quorum:
```json
"quorum": [0, 1],
"quorum_names": ["ml110-01", "r630-01"]
```

## Alternative: Recreate Monitor

If the above doesn't work, recreate the monitor:

On **r630-01**:

```bash
# Stop monitor
systemctl stop ceph-mon@r630-01
systemctl disable ceph-mon@r630-01

# Remove monitor directory
rm -rf /var/lib/ceph/mon/ceph-r630-01

# Recreate
pveceph mon create

# Start
systemctl enable ceph-mon@r630-01
systemctl start ceph-mon@r630-01
```

## Troubleshooting

### Monitor Service Fails to Start

Check logs:
```bash
journalctl -u ceph-mon@r630-01 -n 100
```

Common issues:
- **"monitor data directory does not exist"**: Run `pveceph mon create`
- **"unable to bind"**: Check firewall, ports 6789 and 3300
- **"authentication"**: Check keyring permissions

### Monitor Not in Quorum

1. Verify network connectivity:
   ```bash
   ping 192.168.11.10
   ping 192.168.11.11
   ```

2. Check firewall:
   ```bash
   ufw allow 6789/tcp
   ufw allow 3300/tcp
   ```

3. Verify monmap includes both monitors:
   ```bash
   ceph mon dump
   ```

### Still Not Working

If monitor still won't join:
1. Check if both nodes are in Proxmox cluster: `pvecm status`
2. Verify `/etc/pve/ceph.conf` is identical on both nodes
3. Check FSID matches on both nodes
4. Ensure time is synchronized: `chrony sources`