Sankofa/docs/infrastructure/CEPH_CLUSTER_SETUP.md

# Ceph Cluster Setup - sfvalley-01

**Date**: 2025-12-15  
**Cluster Name**: sfvalley-01 (to be configured)  
**Status**: ⚠️ **IN PROGRESS - Monitor Configuration Needed**

## Current State

### Installation Status
- ✅ **Ceph Packages**: Installed on both servers (version 19.2.3-pve2)
  - ml110-01 (192.168.11.10): ✅ Installed
  - r630-01 (192.168.11.11): ✅ Installed

### Cluster Configuration
- **FSID**: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
- **Network**: 192.168.11.0/24 (public and cluster)
- **Monitors**: 
  - ml110-01: Created (may need reconfiguration)
  - r630-01: Created separately (needs to join main cluster)

### Current Issues
1. **Cluster Name**: Currently using default "ceph" name, needs to be "sfvalley-01"
2. **Monitor Configuration**: 
   - ml110-01: ✅ Running (cluster f601f0e2-cd09-402f-9e15-4b1c9a7a7b25)
   - r630-01: ❌ Failed - Monitor filesystem not created
3. **Cluster Access**: Timeout issues when querying cluster status (may be normal during setup)

### Required Actions
1. **Add r630-01 monitor to existing cluster**:
   ```bash
   # On ml110-01: Get monmap and keyring
   ceph mon getmap -o /tmp/monmap
   scp /tmp/monmap root@192.168.11.11:/tmp/
   scp /etc/pve/priv/ceph.mon.keyring root@192.168.11.11:/tmp/mon-keyring
   
   # On r630-01: Create monitor filesystem
   mkdir -p /var/lib/ceph/mon/ceph-r630-01
   chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
   chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   systemctl start ceph-mon@r630-01.service
   ```

2. **Update cluster name to "sfvalley-01"** (if needed):
   - This may require reinitializing if cluster name change is required
   - Or use "sfvalley-01" as display name in Proxmox UI

## Configuration Files

### `/etc/pve/ceph.conf` (Current)
```
[global]
	auth_client_required = cephx
	auth_cluster_required = cephx
	auth_service_required = cephx
	cluster_network = 192.168.11.0/24
	fsid = f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
	mon_allow_pool_delete = true
	mon_host = 192.168.11.10
	ms_bind_ipv4 = true
	ms_bind_ipv6 = false
	osd_pool_default_min_size = 2
	osd_pool_default_size = 3
	public_network = 192.168.11.0/24

[client]
	keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
	keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.ml110-01]
	public_addr = 192.168.11.10
```

## Next Steps to Configure sfvalley-01 Cluster

### Option 1: Reinitialize with Cluster Name (Recommended if cluster is new/empty)
Since the cluster appears to be newly created and may not have data, we can reinitialize with the proper cluster name:

1. **Stop all Ceph services**
2. **Remove existing cluster data**
3. **Reinitialize with cluster name "sfvalley-01"**
4. **Create monitors on both nodes**

### Option 2: Update Existing Cluster (If cluster has data)
If the cluster already has important data:
1. Update configuration files to reference "sfvalley-01"
2. Update keyring paths
3. Restart services

## Cluster Name Configuration

In Ceph, the cluster name affects:
- Configuration file location: `/etc/ceph/sfvalley-01.conf`
- Data directories: `/var/lib/ceph/mon/sfvalley-01-*`
- Keyring paths: `/etc/pve/priv/sfvalley-01.*.keyring`
- Command usage: `ceph -n sfvalley-01 -s`

## Network Configuration

- **Public Network**: 192.168.11.0/24
- **Cluster Network**: 192.168.11.0/24
- **Monitor Hosts**: 
  - ml110-01: 192.168.11.10
  - r630-01: 192.168.11.11

## Related Documentation

- `docs/infrastructure/NEXT_STEPS.md` - Overall infrastructure plan
- `docs/infrastructure/CEPH_INSTALLATION_ISSUE.md` - Installation troubleshooting
- `scripts/fresh-ceph-install-r630.sh` - Fresh installation script