Sankofa/docs/proxmox/INTER_INSTANCE_CONNECTIVITY.md

# Proxmox Inter-Instance Connectivity

**Last Updated**: 2024-12-19

## Overview

This document describes the connectivity status between the two Proxmox instances and how to verify they can see each other.

## Instance Configuration

- **Instance 1**: ML110-01
  - IP: 192.168.11.10
  - FQDN: ml110-01.sankofa.nexus
  - Site: us-sfvalley

- **Instance 2**: R630-01
  - IP: 192.168.11.11
  - FQDN: r630-01.sankofa.nexus
  - Site: us-sfvalley-2

## Network Configuration

Both instances are on the same network:
- **Subnet**: 192.168.11.0/24
- **Network Type**: Private LAN
- **Gateway**: (to be determined)

## Connectivity Test Results

### From External Machine (This Machine)

✅ **Both instances are reachable:**
- ML110-01 → R630-01: ✅ Reachable (HTTP 401 - auth required)
- R630-01 → ML110-01: ✅ Reachable (HTTP 401 - auth required)
- Both respond to API calls with authentication ✅

### API-Based Tests

Using Proxmox API tokens:

1. **ML110-01 can reach R630-01 API**: ✅
   - Version: 9.1.1
   - Authentication: Successful

2. **R630-01 can reach ML110-01 API**: ✅
   - Version: 9.1.1
   - Authentication: Successful

### Cluster Status

Check cluster membership:
```bash
# From ML110-01
curl -k -H "Authorization: PVEAPIToken <token>" \
  https://192.168.11.10:8006/api2/json/cluster/status

# From R630-01
curl -k -H "Authorization: PVEAPIToken <token>" \
  https://192.168.11.11:8006/api2/json/cluster/status
```

## Verification Methods

### Method 1: API-Based (From External Machine)

```bash
source .env

# Test ML110-01 → R630-01
curl -k -H "Authorization: PVEAPIToken ${PROXMOX_TOKEN_R630_01}" \
  https://192.168.11.11:8006/api2/json/version

# Test R630-01 → ML110-01
curl -k -H "Authorization: PVEAPIToken ${PROXMOX_TOKEN_ML110_01}" \
  https://192.168.11.10:8006/api2/json/version
```

### Method 2: SSH-Based (From Within Instances)

Requires SSH access to instances:

```bash
# SSH into ML110-01
ssh root@192.168.11.10

# Test connectivity to R630-01
curl -k https://192.168.11.11:8006/api2/json/version
ping -c 3 192.168.11.11

# SSH into R630-01
ssh root@192.168.11.11

# Test connectivity to ML110-01
curl -k https://192.168.11.10:8006/api2/json/version
ping -c 3 192.168.11.10
```

### Method 3: Using Proxmox Web UI

1. Log in to ML110-01 web UI
2. Go to: Datacenter → Cluster
3. Check if R630-01 appears in cluster members
4. Repeat from R630-01

## Cluster Requirements

If you want to cluster these instances:

### Network Requirements
- ✅ Same subnet (192.168.11.0/24) - **Met**
- ✅ Network connectivity - **Confirmed**
- ⚠️ Firewall rules for clustering ports

### Firewall Ports for Clustering

Required ports:
- **8006**: Proxmox API (HTTPS)
- **5404-5405**: Corosync (cluster communication)
- **22**: SSH (for cluster operations)
- **3128**: Spice proxy (optional)

### Cluster Setup Steps

1. **Prepare first node (ML110-01)**:
   ```bash
   # On ML110-01
   pvecm create <cluster-name>
   ```

2. **Add second node (R630-01)**:
   ```bash
   # On R630-01
   pvecm add <first-node-ip> -link0 <first-node-ip>
   ```

3. **Verify cluster**:
   ```bash
   # On either node
   pvecm status
   pvecm nodes
   ```

## Current Status

### ✅ Confirmed
- Both instances are on the same network
- Both instances are reachable via IP and FQDN
- API connectivity works from external machine
- Both instances respond to authenticated API calls

### ⚠️ To Verify
- Direct connectivity from within instances (requires SSH)
- Cluster membership status
- Firewall configuration for clustering
- Corosync port accessibility

## Troubleshooting

### Instances Can't See Each Other

1. **Check network connectivity**:
   ```bash
   ping <other-instance-ip>
   ```

2. **Check firewall rules**:
   ```bash
   # On Proxmox
   iptables -L -n | grep <other-instance-ip>
   pve-firewall status
   ```

3. **Check routing**:
   ```bash
   ip route show
   route -n
   ```

### Cluster Issues

1. **Check corosync**:
   ```bash
   systemctl status corosync
   corosync-cmapctl | grep members
   ```

2. **Check quorum**:
   ```bash
   pvecm status
   pvecm expected 2  # For 2-node cluster
   ```

## Related Documentation

- [Deployment Readiness](./DEPLOYMENT_READINESS.md)
- [Network Configuration](./NETWORK_CONFIGURATION.md)
- [Cluster Setup Guide](./CLUSTER_SETUP.md)