# Fresh Ceph Setup on Proxmox VE

## Prerequisites

- Proxmox VE installed on both nodes
- Nodes are in a Proxmox cluster (or standalone)
- Network connectivity between nodes (192.168.11.0/24)
- Root access to both nodes

## Node Information

- **ML110-01**: ml110-01.sankofa.nexus (192.168.11.10)
- **R630-01**: r630-01.sankofa.nexus (192.168.11.11)

## Step-by-Step Setup

### Step 1: Install Ceph on Both Nodes

Run on **both nodes**:

```bash
apt-get update
apt-get install -y ceph ceph-common ceph-base
```

### Step 2: Initialize Ceph Cluster (ML110-01 only)

Run on **ml110-01**:

```bash
pveceph init --network 192.168.11.0/24 --cluster-network 192.168.11.0/24
```

This will:
- Create `/etc/pve/ceph.conf`
- Generate cluster FSID
- Set up bootstrap keyrings

### Step 3: Create Monitors

Run on **ml110-01**:

```bash
pveceph mon create
```

Run on **r630-01**:

```bash
pveceph mon create
```

### Step 4: Verify Quorum

Run on **either node**:

```bash
ceph quorum_status
ceph -s
```

You should see both monitors in quorum.

### Step 5: Create OSDs

#### On ML110-01

Identify available drives:

```bash
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE
```

Create OSD on available drive (replace `/dev/sdX` with actual drive):

```bash
ceph-volume lvm create --data /dev/sdX
```

#### On R630-01

Create OSDs on the 6x 250GB drives (sdc, sdd, sde, sdf, sdg, sdh):

```bash
for disk in sdc sdd sde sdf sdg sdh; do
    echo "Creating OSD on /dev/$disk..."
    ceph-volume lvm create --data /dev/$disk
    sleep 2
done
```

### Step 6: Verify Cluster Health

Run on **either node**:

```bash
ceph -s
ceph osd tree
ceph health detail
```

### Step 7: Create Storage Pools

#### Create RBD Pool (for VM disks)

```bash
ceph osd pool create rbd 32 32
ceph osd pool application enable rbd rbd
rbd pool init rbd
```

#### Create CephFS (for shared storage)

```bash
ceph osd pool create cephfs_data 32 32
ceph osd pool create cephfs_metadata 16 16
ceph fs new ceph-fs cephfs_metadata cephfs_data
```

### Step 8: Add Storage to Proxmox

#### Add Ceph RBD Storage

In Proxmox Web UI:
1. Datacenter → Storage → Add → RBD
2. ID: `ceph-rbd`
3. Pool: `rbd`
4. Nodes: Select both nodes
5. Add

#### Add CephFS Storage

In Proxmox Web UI:
1. Datacenter → Storage → Add → CephFS
2. ID: `ceph-fs`
3. FS Name: `ceph-fs`
4. Nodes: Select both nodes
5. Add

## Verification Checklist

- [ ] Both monitors in quorum
- [ ] At least 3 OSDs created (for replication factor 3)
- [ ] Cluster health is HEALTH_OK or HEALTH_WARN (not HEALTH_ERR)
- [ ] Storage pools created (rbd, cephfs_data, cephfs_metadata)
- [ ] CephFS filesystem created
- [ ] Storage added to Proxmox Web UI

## Troubleshooting

### Monitors Not Forming Quorum

1. Check monitor services:
   ```bash
   systemctl status ceph-mon@$(hostname -s)
   ```

2. Check monitor logs:
   ```bash
   journalctl -u ceph-mon@$(hostname -s) -n 50
   ```

3. Verify network connectivity:
   ```bash
   ping 192.168.11.10
   ping 192.168.11.11
   ```

4. Check firewall:
   ```bash
   # Allow Ceph ports
   ufw allow 6789/tcp  # Classic msgr
   ufw allow 3300/tcp  # msgr2
   ```

### OSD Creation Fails

1. Check if drive is available:
   ```bash
   lsblk /dev/sdX
   ```

2. Wipe drive if needed:
   ```bash
   wipefs -a /dev/sdX
   ```

3. Check for existing LVM volumes:
   ```bash
   pvs | grep sdX
   vgs | grep ceph
   ```

### Cluster Health Issues

1. Check detailed health:
   ```bash
   ceph health detail
   ```

2. Check OSD status:
   ```bash
   ceph osd tree
   ceph osd stat
   ```

3. Check placement groups:
   ```bash
   ceph pg stat
   ```

## Next Steps

After Ceph is set up:

1. **Fix PG allocation**: Ensure pools have appropriate PG counts
2. **Create VMs**: Use Ceph storage for VM disks
3. **Monitor cluster**: Set up monitoring and alerts
4. **Backup configuration**: Backup `/etc/pve/ceph.conf` and keyrings

## Important Notes

- **Minimum OSDs**: For `osd_pool_default_size = 3`, you need at least 3 OSDs
- **PG Count**: Use calculator: https://ceph.io/pgcalc/
- **Network**: Ensure cluster network has sufficient bandwidth
- **Backup**: Always backup Ceph configuration and keyrings