Sankofa/docs/infrastructure/CEPH_CLUSTER_SETUP.md

Ceph Cluster Setup - sfvalley-01

Date: 2025-12-15
Cluster Name: sfvalley-01 (to be configured)
Status: ⚠️ IN PROGRESS - Monitor Configuration Needed

Current State

Installation Status

• ✅ Ceph Packages: Installed on both servers (version 19.2.3-pve2)
  • ml110-01 (192.168.11.10): ✅ Installed
  • r630-01 (192.168.11.11): ✅ Installed

Cluster Configuration

• FSID: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
• Network: 192.168.11.0/24 (public and cluster)
• Monitors:
  • ml110-01: Created (may need reconfiguration)
  • r630-01: Created separately (needs to join main cluster)

Current Issues

1. Cluster Name: Currently using default "ceph" name, needs to be "sfvalley-01"
2. Monitor Configuration:
   • ml110-01: ✅ Running (cluster f601f0e2-cd09-402f-9e15-4b1c9a7a7b25)
   • r630-01: ❌ Failed - Monitor filesystem not created
3. Cluster Access: Timeout issues when querying cluster status (may be normal during setup)

Required Actions

1. Add r630-01 monitor to existing cluster:

   # On ml110-01: Get monmap and keyring
   ceph mon getmap -o /tmp/monmap
   scp /tmp/monmap root@192.168.11.11:/tmp/
   scp /etc/pve/priv/ceph.mon.keyring root@192.168.11.11:/tmp/mon-keyring
   
   # On r630-01: Create monitor filesystem
   mkdir -p /var/lib/ceph/mon/ceph-r630-01
   chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
   chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
   systemctl start ceph-mon@r630-01.service
   

2. Update cluster name to "sfvalley-01" (if needed):

   • This may require reinitializing if cluster name change is required
   • Or use "sfvalley-01" as display name in Proxmox UI

Configuration Files

/etc/pve/ceph.conf (Current)

[global]
	auth_client_required = cephx
	auth_cluster_required = cephx
	auth_service_required = cephx
	cluster_network = 192.168.11.0/24
	fsid = f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
	mon_allow_pool_delete = true
	mon_host = 192.168.11.10
	ms_bind_ipv4 = true
	ms_bind_ipv6 = false
	osd_pool_default_min_size = 2
	osd_pool_default_size = 3
	public_network = 192.168.11.0/24

[client]
	keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
	keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.ml110-01]
	public_addr = 192.168.11.10

Next Steps to Configure sfvalley-01 Cluster

Option 1: Reinitialize with Cluster Name (Recommended if cluster is new/empty)

Since the cluster appears to be newly created and may not have data, we can reinitialize with the proper cluster name:

1. Stop all Ceph services
2. Remove existing cluster data
3. Reinitialize with cluster name "sfvalley-01"
4. Create monitors on both nodes

Option 2: Update Existing Cluster (If cluster has data)

If the cluster already has important data:

1. Update configuration files to reference "sfvalley-01"
2. Update keyring paths
3. Restart services

Cluster Name Configuration

In Ceph, the cluster name affects:

• Configuration file location: /etc/ceph/sfvalley-01.conf
• Data directories: /var/lib/ceph/mon/sfvalley-01-*
• Keyring paths: /etc/pve/priv/sfvalley-01.*.keyring
• Command usage: ceph -n sfvalley-01 -s

Network Configuration

• Public Network: 192.168.11.0/24
• Cluster Network: 192.168.11.0/24
• Monitor Hosts:
  • ml110-01: 192.168.11.10
  • r630-01: 192.168.11.11

Related Documentation

• docs/infrastructure/NEXT_STEPS.md - Overall infrastructure plan
• docs/infrastructure/CEPH_INSTALLATION_ISSUE.md - Installation troubleshooting
• scripts/fresh-ceph-install-r630.sh - Fresh installation script