Sankofa/docs/infrastructure/COMPLETE_STATUS_REPORT.md

Complete Status Report - sfvalley-01 Cluster

Date: 2025-12-15 14:55 PST
Cluster Name: sfvalley-01
Report Type: Pre-Change Status Poll

Proxmox Cluster Status

✅ HEALTHY - Quorum Established

Cluster Information:

• Name: sfvalley-01
• Config Version: 2
• Transport: knet
• Secure Auth: Enabled

Quorum Status:

• Status: ✅ Quorate
• Expected Votes: 2
• Total Votes: 2
• Quorum: 2

Cluster Members:

• Node 1: ml110-01 (192.168.11.10) - Node ID: 0x00000001
• Node 2: r630-01 (192.168.11.11) - Node ID: 0x00000002

Status: ✅ Both nodes are active and in quorum

---

Ceph Cluster Status

⚠️ PARTIALLY OPERATIONAL - Monitor Issues

Cluster Configuration:

• FSID: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
• Network: 192.168.11.0/24 (public and cluster)
• Pool Settings:
  • Default size: 3
  • Default min_size: 2

ML110-01 (192.168.11.10) Status

Services:

• ✅ ceph-mon@ml110-01.service: ACTIVE (running)

  • Status: Running but in "probing" state
  • Started: Mon 2025-12-15 14:39:30 PST
  • Memory: 142.7M
  • Issue: Reporting slow operations (1800+ slow ops)

• ✅ ceph-mgr@ml110-01.service: ACTIVE (running)

  • Status: Running
  • Started: Mon 2025-12-15 14:39:33 PST
  • Memory: 327.3M
  • Issue: Authentication errors with monitor

Data Directories:

• ✅ /var/lib/ceph/mon/ceph-ml110-01/ - Exists
• ✅ /var/lib/ceph/mgr/ceph-ml110-01/ - Exists
• ⚠️ /var/lib/ceph/osd/ - Empty (no OSDs)

R630-01 (192.168.11.11) Status

Services:

• ❌ ceph-mon@r630-01.service: FAILED

  • Status: Failed (exit-code)
  • Error: Monitor data directory does not exist - needs 'mkfs'
  • Last attempt: Mon 2025-12-15 14:43:39 PST
  • Issue: Monitor filesystem not created

• ✅ ceph-mgr@r630-01.service: ACTIVE (running)

  • Status: Running
  • Started: Mon 2025-12-15 14:39:47 PST
  • Memory: 326M
  • Issue: Reporting status from non-daemon mon.r630-01

Data Directories:

• ⚠️ /var/lib/ceph/mon/ceph-r630-01/ - Exists but EMPTY (needs mkfs)
• ✅ /var/lib/ceph/mgr/ceph-r630-01/ - Exists
• ⚠️ /var/lib/ceph/osd/ - Empty (no OSDs)

Ceph Cluster Access

Status: ⚠️ TIMEOUT ISSUES

• ceph -s: Timeout
• ceph mon dump: Timeout
• ceph osd tree: Timeout
• pveceph status: Timeout

Root Cause: Monitor on ml110-01 is in "probing" state and cannot form quorum without r630-01 monitor. This causes all Ceph commands to timeout.

---

Configuration Files

/etc/pve/ceph.conf (Synced across cluster)

Both nodes have identical configuration:

[global]
	auth_client_required = cephx
	auth_cluster_required = cephx
	auth_service_required = cephx
	cluster_network = 192.168.11.0/24
	fsid = f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
	mon_allow_pool_delete = true
	mon_host = 192.168.11.10
	ms_bind_ipv4 = true
	ms_bind_ipv6 = false
	osd_pool_default_min_size = 2
	osd_pool_default_size = 3
	public_network = 192.168.11.0/24

[client]
	keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
	keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.ml110-01]
	public_addr = 192.168.11.10

Note: Configuration only lists ml110-01 in mon_host. r630-01 monitor needs to be added.

---

Issues Identified

Critical Issues

1. R630-01 Monitor Not Operational

   • Monitor service failed
   • Monitor filesystem not created
   • Needs: ceph-mon --mkfs to create filesystem
   • Impact: No quorum, cluster commands timeout

2. Monitor Quorum Not Established

   • Only 1 monitor running (ml110-01)
   • ml110-01 monitor in "probing" state
   • Cannot form quorum with single monitor
   • Impact: Cluster inaccessible

3. Ceph Commands Timing Out

   • All ceph commands timeout
   • Cannot query cluster status
   • Impact: Cannot manage cluster

Warning Issues

1. No OSDs Configured

   • No OSDs on either node
   • Cluster has no storage
   • Impact: Cannot store data

2. Monitor Configuration Incomplete

   • mon_host only lists ml110-01
   • r630-01 monitor not in configuration
   • Impact: r630-01 monitor cannot be discovered

3. Slow Operations on Monitor

   • ml110-01 monitor reporting 1800+ slow operations
   • May indicate performance issues
   • Impact: Degraded performance

---

Required Actions

Priority 1: Fix R630-01 Monitor (CRITICAL)

Steps:

1. Get monmap and keyring from ml110-01
2. Create monitor filesystem on r630-01
3. Start monitor service
4. Verify quorum established

Commands:

# On ml110-01
ceph mon getmap -o /tmp/monmap
scp /tmp/monmap root@192.168.11.11:/tmp/
scp /etc/pve/priv/ceph.mon.keyring root@192.168.11.11:/tmp/mon-keyring

# On r630-01
mkdir -p /var/lib/ceph/mon/ceph-r630-01
chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
systemctl start ceph-mon@r630-01.service

Priority 2: Update Monitor Configuration

Steps:

1. Add r630-01 to mon_host in ceph.conf
2. Add [mon.r630-01] section
3. Verify configuration syncs to both nodes

Priority 3: Verify Quorum

Steps:

1. Check monitor quorum: ceph quorum_status
2. Verify both monitors in quorum
3. Test cluster access: ceph -s

Priority 4: Add OSDs

Steps:

1. Identify available disks (6x 250GB SSDs on r630-01)
2. Create OSDs using pveceph osd create
3. Verify OSD status

---

Summary

✅ Working

• Proxmox cluster: Healthy, quorum established
• Ceph packages: Installed on both nodes
• Ceph manager: Running on both nodes
• Ceph monitor: Running on ml110-01
• Configuration: Synced across cluster

⚠️ Needs Attention

• Ceph monitor: r630-01 monitor needs filesystem creation
• Monitor quorum: Cannot form quorum with single monitor
• Cluster access: Commands timing out
• OSDs: No OSDs configured

❌ Blocking Issues

• R630-01 monitor not operational (blocks quorum)
• Cluster commands timing out (blocks management)

---

Next Steps

1. Fix r630-01 monitor (see Priority 1 above)
2. Update monitor configuration to include r630-01
3. Verify quorum and cluster access
4. Add OSDs once cluster is operational
5. Monitor performance for slow operations

---

Notes

• Proxmox cluster "sfvalley-01" is healthy and operational
• Ceph cluster exists but needs monitor quorum to be fully operational
• All configuration is synced via Proxmox cluster filesystem
• Both nodes have Ceph installed and partially configured
• Once monitor quorum is established, cluster should become fully accessible