114 lines
3.6 KiB
Markdown
114 lines
3.6 KiB
Markdown
|
# Ceph Cluster Setup - sfvalley-01
|
||
|
|
|
||
|
|
**Date**: 2025-12-15
|
||
|
|
**Cluster Name**: sfvalley-01 (to be configured)
|
||
|
|
**Status**: ⚠️ **IN PROGRESS - Monitor Configuration Needed**
|
||
|
|
|
||
|
|
## Current State
|
||
|
|
|
||
|
|
### Installation Status
|
||
|
|
- ✅ **Ceph Packages**: Installed on both servers (version 19.2.3-pve2)
|
||
|
|
- ml110-01 (192.168.11.10): ✅ Installed
|
||
|
|
- r630-01 (192.168.11.11): ✅ Installed
|
||
|
|
|
||
|
|
### Cluster Configuration
|
||
|
|
- **FSID**: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
|
||
|
|
- **Network**: 192.168.11.0/24 (public and cluster)
|
||
|
|
- **Monitors**:
|
||
|
|
- ml110-01: Created (may need reconfiguration)
|
||
|
|
- r630-01: Created separately (needs to join main cluster)
|
||
|
|
|
||
|
|
### Current Issues
|
||
|
|
1. **Cluster Name**: Currently using default "ceph" name, needs to be "sfvalley-01"
|
||
|
|
2. **Monitor Configuration**:
|
||
|
|
- ml110-01: ✅ Running (cluster f601f0e2-cd09-402f-9e15-4b1c9a7a7b25)
|
||
|
|
- r630-01: ❌ Failed - Monitor filesystem not created
|
||
|
|
3. **Cluster Access**: Timeout issues when querying cluster status (may be normal during setup)
|
||
|
|
|
||
|
|
### Required Actions
|
||
|
|
1. **Add r630-01 monitor to existing cluster**:
|
||
|
|
```bash
|
||
|
|
# On ml110-01: Get monmap and keyring
|
||
|
|
ceph mon getmap -o /tmp/monmap
|
||
|
|
scp /tmp/monmap root@192.168.11.11:/tmp/
|
||
|
|
scp /etc/pve/priv/ceph.mon.keyring root@192.168.11.11:/tmp/mon-keyring
|
||
|
|
|
||
|
|
# On r630-01: Create monitor filesystem
|
||
|
|
mkdir -p /var/lib/ceph/mon/ceph-r630-01
|
||
|
|
chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
|
||
|
|
ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
|
||
|
|
chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
|
||
|
|
systemctl start ceph-mon@r630-01.service
|
||
|
|
```
|
||
|
|
|
||
|
|
2. **Update cluster name to "sfvalley-01"** (if needed):
|
||
|
|
- This may require reinitializing if cluster name change is required
|
||
|
|
- Or use "sfvalley-01" as display name in Proxmox UI
|
||
|
|
|
||
|
|
## Configuration Files
|
||
|
|
|
||
|
|
### `/etc/pve/ceph.conf` (Current)
|
||
|
|
```
|
||
|
|
[global]
|
||
|
|
auth_client_required = cephx
|
||
|
|
auth_cluster_required = cephx
|
||
|
|
auth_service_required = cephx
|
||
|
|
cluster_network = 192.168.11.0/24
|
||
|
|
fsid = f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
|
||
|
|
mon_allow_pool_delete = true
|
||
|
|
mon_host = 192.168.11.10
|
||
|
|
ms_bind_ipv4 = true
|
||
|
|
ms_bind_ipv6 = false
|
||
|
|
osd_pool_default_min_size = 2
|
||
|
|
osd_pool_default_size = 3
|
||
|
|
public_network = 192.168.11.0/24
|
||
|
|
|
||
|
|
[client]
|
||
|
|
keyring = /etc/pve/priv/$cluster.$name.keyring
|
||
|
|
|
||
|
|
[client.crash]
|
||
|
|
keyring = /etc/pve/ceph/$cluster.$name.keyring
|
||
|
|
|
||
|
|
[mon.ml110-01]
|
||
|
|
public_addr = 192.168.11.10
|
||
|
|
```
|
||
|
|
|
||
|
|
## Next Steps to Configure sfvalley-01 Cluster
|
||
|
|
|
||
|
|
### Option 1: Reinitialize with Cluster Name (Recommended if cluster is new/empty)
|
||
|
|
Since the cluster appears to be newly created and may not have data, we can reinitialize with the proper cluster name:
|
||
|
|
|
||
|
|
1. **Stop all Ceph services**
|
||
|
|
2. **Remove existing cluster data**
|
||
|
|
3. **Reinitialize with cluster name "sfvalley-01"**
|
||
|
|
4. **Create monitors on both nodes**
|
||
|
|
|
||
|
|
### Option 2: Update Existing Cluster (If cluster has data)
|
||
|
|
If the cluster already has important data:
|
||
|
|
1. Update configuration files to reference "sfvalley-01"
|
||
|
|
2. Update keyring paths
|
||
|
|
3. Restart services
|
||
|
|
|
||
|
|
## Cluster Name Configuration
|
||
|
|
|
||
|
|
In Ceph, the cluster name affects:
|
||
|
|
- Configuration file location: `/etc/ceph/sfvalley-01.conf`
|
||
|
|
- Data directories: `/var/lib/ceph/mon/sfvalley-01-*`
|
||
|
|
- Keyring paths: `/etc/pve/priv/sfvalley-01.*.keyring`
|
||
|
|
- Command usage: `ceph -n sfvalley-01 -s`
|
||
|
|
|
||
|
|
## Network Configuration
|
||
|
|
|
||
|
|
- **Public Network**: 192.168.11.0/24
|
||
|
|
- **Cluster Network**: 192.168.11.0/24
|
||
|
|
- **Monitor Hosts**:
|
||
|
|
- ml110-01: 192.168.11.10
|
||
|
|
- r630-01: 192.168.11.11
|
||
|
|
|
||
|
|
## Related Documentation
|
||
|
|
|
||
|
|
- `docs/infrastructure/NEXT_STEPS.md` - Overall infrastructure plan
|
||
|
|
- `docs/infrastructure/CEPH_INSTALLATION_ISSUE.md` - Installation troubleshooting
|
||
|
|
- `scripts/fresh-ceph-install-r630.sh` - Fresh installation script
|
||
|
|
|