# Proxmox Cluster Setup Guide

**Last Updated**: 2024-12-19

## Overview

This guide explains how to create a Proxmox cluster between ML110-01 and R630-01.

## Prerequisites

- ✅ Both instances on same network (192.168.11.0/24) - **Met**
- ✅ Network connectivity between instances - **Confirmed**
- ✅ API access to both instances - **Working**
- ⚠️ SSH access to nodes (for corosync configuration)
- ⚠️ Firewall rules for clustering ports (5404-5405)

## Cluster Configuration

- **Cluster Name**: sankofa-cluster
- **Node 1**: ML110-01 (192.168.11.10)
- **Node 2**: R630-01 (192.168.11.11)

## Method 1: Using Proxmox Web UI (Recommended)

### Step 1: Create Cluster on First Node

1. Log in to ML110-01 web UI: https://ml110-01.sankofa.nexus:8006
2. Go to: **Datacenter** → **Cluster**
3. Click **Create Cluster**
4. Enter cluster name: `sankofa-cluster`
5. Click **Create**

### Step 2: Add Second Node

1. Log in to R630-01 web UI: https://r630-01.sankofa.nexus:8006
2. Go to: **Datacenter** → **Cluster**
3. Click **Join Cluster**
4. Enter:
   - **Cluster Name**: `sankofa-cluster`
   - **Node IP**: `192.168.11.10` (ML110-01)
   - **Root Password**: (for ML110-01)
5. Click **Join**

### Step 3: Verify Cluster

On either node:
- Go to **Datacenter** → **Cluster**
- You should see both nodes listed
- Both nodes should show status "Online"

## Method 2: Using SSH and pvecm (Command Line)

### Step 1: Create Cluster on First Node

SSH into ML110-01:
```bash
ssh root@192.168.11.10

# Create cluster
pvecm create sankofa-cluster

# Verify
pvecm status
```

### Step 2: Add Second Node

SSH into R630-01:
```bash
ssh root@192.168.11.11

# Join cluster
pvecm add 192.168.11.10

# Verify
pvecm status
pvecm nodes
```

### Step 3: Configure Quorum (2-Node Cluster)

For a 2-node cluster, you need to configure quorum:
```bash
# On either node
pvecm expected 2
pvecm status
```

## Method 3: Using API (Limited)

The Proxmox API has limited cluster management capabilities. For full cluster creation, use Web UI or SSH.

### Check Cluster Status via API

```bash
source .env

# Check nodes in cluster
curl -k -H "Authorization: PVEAPIToken ${PROXMOX_TOKEN_ML110_01}" \
  https://192.168.11.10:8006/api2/json/cluster/config/nodes

# Check cluster status
curl -k -H "Authorization: PVEAPIToken ${PROXMOX_TOKEN_ML110_01}" \
  https://192.168.11.10:8006/api2/json/cluster/status
```

## Firewall Configuration

Ensure these ports are open between nodes:

- **8006**: Proxmox API (HTTPS)
- **5404-5405**: Corosync (cluster communication)
- **22**: SSH (for cluster operations)
- **3128**: Spice proxy (optional)

### Configure Firewall on Proxmox

```bash
# On each node, allow cluster traffic
pve-firewall localnet add 192.168.11.0/24
pve-firewall refresh
```

## Verification

### Check Cluster Status

```bash
# Via API
curl -k -H "Authorization: PVEAPIToken ${PROXMOX_TOKEN_ML110_01}" \
  https://192.168.11.10:8006/api2/json/cluster/status

# Via SSH (on node)
pvecm status
pvecm nodes
```

### Test Cluster Operations

1. Create a VM on ML110-01
2. Verify it appears in cluster view
3. Try migrating VM between nodes
4. Verify storage is accessible from both nodes

## Troubleshooting

### Nodes Can't Join Cluster

1. **Check network connectivity**:
   ```bash
   ping <other-node-ip>
   ```

2. **Check firewall**:
   ```bash
   iptables -L -n | grep <other-node-ip>
   ```

3. **Check corosync**:
   ```bash
   systemctl status corosync
   corosync-cmapctl | grep members
   ```

### Quorum Issues

For 2-node cluster:
```bash
# Set expected votes
pvecm expected 2

# Check quorum
pvecm status
```

### Cluster Split-Brain

If cluster splits:
```bash
# On majority node
pvecm expected 2

# On minority node (if needed)
pvecm expected 1
```

## Post-Cluster Setup

After cluster is created:

1. **Verify both nodes visible**:
   - Check Datacenter → Cluster in web UI
   - Both nodes should be listed

2. **Configure shared storage** (if needed):
   - Set up NFS, Ceph, or other shared storage
   - Add storage to cluster

3. **Test VM operations**:
   - Create VM on one node
   - Verify it's visible on both nodes
   - Test migration

4. **Update Crossplane ProviderConfig**:
   - Cluster name can be used in provider config
   - VMs can be created on cluster level

## Related Documentation

- [Inter-Instance Connectivity](./INTER_INSTANCE_CONNECTIVITY.md)
- [Deployment Guide](./DEPLOYMENT_GUIDE.md)
- [Task List](./TASK_LIST.md)