Sankofa/docs/proxmox/acme-commands.md

# Proxmox ACME Setup - Complete Command Reference

This document provides all commands needed for the Proxmox ACME certificate setup process.

## Prerequisites

### Verify Network Connectivity

```bash
# Test Cloudflare API connectivity from Proxmox nodes
curl -I https://api.cloudflare.com

# Test DNS resolution for nodes
dig ml110-01.sankofa.nexus
dig r630-01.sankofa.nexus

# Test HTTPS connectivity to Proxmox nodes
curl -k -I https://ml110-01.sankofa.nexus:8006
curl -k -I https://r630-01.sankofa.nexus:8006
```

## Step 1: Create Cloudflare API Token

**Note**: This step must be done via Cloudflare Web UI (no CLI commands).

1. Go to: https://dash.cloudflare.com/profile/api-tokens
2. Click "Create Token"
3. Use "Edit zone DNS" template
4. Select zone: `sankofa.nexus`
5. Copy token and save securely

### Verify Token (Optional - Test from Local Machine)

```bash
# Replace YOUR_TOKEN with actual token
TOKEN="YOUR_CLOUDFLARE_API_TOKEN"
ZONE_ID="YOUR_ZONE_ID"  # Get from Cloudflare dashboard

# Test token can read DNS records
curl -X GET "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" | jq '.result[] | select(.name | contains("sankofa.nexus"))'

# Test token can create DNS records (creates test record - delete after)
curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"type":"TXT","name":"_acme-test.sankofa.nexus","content":"test-validation","ttl":120}' | jq

# Delete test record
RECORD_ID="RECORD_ID_FROM_ABOVE"
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${RECORD_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json"
```

## Step 2: Configure ACME via Proxmox CLI (Alternative to Web UI)

### SSH to ml110-01 (Admin Node)

```bash
# SSH to admin node
ssh root@ml110-01.sankofa.nexus
# or
ssh root@192.168.11.10
```

### Register ACME Account via CLI

```bash
# Register Let's Encrypt account
pvesh create /cluster/acme/account \
  --name letsencrypt-prod \
  --directory https://acme-v02.api.letsencrypt.org/directory \
  --contact email@example.com \
  --account

# Verify account registration
pvesh get /cluster/acme/account/letsencrypt-prod
```

### Add Cloudflare DNS Plugin via CLI

```bash
# Replace YOUR_TOKEN with actual Cloudflare API token
CLOUDFLARE_TOKEN="YOUR_CLOUDFLARE_API_TOKEN"

# Add Cloudflare DNS plugin
pvesh create /cluster/acme/plugins \
  --id cloudflare-dns \
  --type cloudflare \
  --data "api={${CLOUDFLARE_TOKEN}} domain=sankofa.nexus"

# Verify plugin configuration
pvesh get /cluster/acme/plugins/cloudflare-dns
```

### Order Certificates via CLI

```bash
# Order wildcard certificate
pvesh create /cluster/acme/certificate \
  --name wildcard-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "*.sankofa.nexus"

# Order single-host certificate for ml110-01
pvesh create /cluster/acme/certificate \
  --name ml110-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "ml110-01.sankofa.nexus"

# Order single-host certificate for r630-01
pvesh create /cluster/acme/certificate \
  --name r630-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "r630-01.sankofa.nexus"

# Check certificate order status
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus
pvesh get /cluster/acme/certificate/ml110-01-sankofa-nexus
pvesh get /cluster/acme/certificate/r630-01-sankofa-nexus

# List all ACME certificates
pvesh get /cluster/acme/certificate
```

### Apply Certificate to Nodes via CLI

```bash
# Apply wildcard certificate to ml110-01 node
pvesh create /nodes/ml110-01/certificates/custom \
  --filename wildcard-sankofa-nexus.pem \
  --certificates wildcard-sankofa-nexus

# Apply wildcard certificate to r630-01 node
pvesh create /nodes/r630-01/certificates/custom \
  --filename wildcard-sankofa-nexus.pem \
  --certificates wildcard-sankofa-nexus

# Verify certificate assignment
pvesh get /nodes/ml110-01/certificates/custom
pvesh get /nodes/r630-01/certificates/custom
```

**Note**: Certificate application to nodes is typically done automatically when certificates are ordered. The above commands may need adjustment based on Proxmox version.

### Alternative: Apply via Web UI Certificate Manager

If CLI certificate application doesn't work:

1. Go to: `https://ml110-01.sankofa.nexus:8006`
2. Navigate: **Datacenter** → **Certificates**
3. Select certificate: `wildcard-sankofa-nexus`
4. Click **"Edit"**
5. Check boxes for: `ml110-01` and `r630-01`
6. Click **"OK"**

## Step 3: Verify Certificate Installation

### Check Certificate Status on Nodes

```bash
# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Check ACME certificates
pvesh get /cluster/acme/certificate

# Check node certificates
pvesh get /nodes/ml110-01/certificates/custom

# View certificate file
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem | openssl x509 -noout -text

# Check certificate expiration
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem | openssl x509 -noout -dates

# Verify issuer (should show Let's Encrypt)
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem | openssl x509 -noout -issuer
```

### Test SSL Connection from Local Machine

```bash
# Test ml110-01 certificate
echo | openssl s_client -connect ml110-01.sankofa.nexus:8006 -servername ml110-01.sankofa.nexus 2>/dev/null | \
  openssl x509 -noout -dates -subject -issuer

# Test r630-01 certificate
echo | openssl s_client -connect r630-01.sankofa.nexus:8006 -servername r630-01.sankofa.nexus 2>/dev/null | \
  openssl x509 -noout -dates -subject -issuer

# Check certificate chain
echo | openssl s_client -connect ml110-01.sankofa.nexus:8006 -servername ml110-01.sankofa.nexus -showcerts 2>/dev/null | \
  openssl x509 -noout -text | grep -A 5 "Certificate chain"
```

### Run Verification Script

```bash
# From project root
cd /home/intlc/projects/Sankofa

# Run verification script
./scripts/verify-proxmox-certs.sh

# Or with explicit path
bash scripts/verify-proxmox-certs.sh
```

### Check Proxmox Logs for ACME

```bash
# On ml110-01
ssh root@ml110-01.sankofa.nexus

# View ACME-related logs
journalctl -u pvedaemon -f | grep -i acme

# Check recent ACME activity
journalctl -u pvedaemon --since "1 hour ago" | grep -i acme

# View ACME certificate order logs
journalctl -u pvedaemon | grep -i "wildcard-sankofa-nexus"
```

## Step 4: Deploy Updated Tunnel Configurations

### Copy Updated Configs to Proxmox Nodes

```bash
# From project root
cd /home/intlc/projects/Sankofa

# Copy config to ml110-01 (Site 1)
scp cloudflare/tunnel-configs/proxmox-site-1.yaml root@ml110-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/proxmox-site-1.yaml

# Copy config to r630-01 (Site 2)
scp cloudflare/tunnel-configs/proxmox-site-2.yaml root@r630-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/proxmox-site-2.yaml

# If Site 3 exists, copy config
# scp cloudflare/tunnel-configs/proxmox-site-3.yaml root@r630-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/proxmox-site-3.yaml
```

### Verify Config Files on Nodes

```bash
# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Verify config file exists and is correct
cat /etc/cloudflared/tunnel-configs/proxmox-site-1.yaml

# Verify skipVerify is removed (should NOT appear in file)
grep -i "skipverify" /etc/cloudflared/tunnel-configs/proxmox-site-1.yaml
# Should return no results

# Check YAML syntax
cloudflared tunnel ingress validate --config /etc/cloudflared/tunnel-configs/proxmox-site-1.yaml
```

### Restart Cloudflare Tunnel Services

```bash
# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Check tunnel service status
systemctl status cloudflared-tunnel

# Restart tunnel service
systemctl restart cloudflared-tunnel

# Verify service started successfully
systemctl status cloudflared-tunnel

# View tunnel logs
journalctl -u cloudflared-tunnel -f

# Check for errors (especially certificate verification)
journalctl -u cloudflared-tunnel --since "5 minutes ago" | grep -i error
```

```bash
# On r630-01
ssh root@r630-01.sankofa.nexus

# Same commands as above
systemctl status cloudflared-tunnel
systemctl restart cloudflared-tunnel
systemctl status cloudflared-tunnel
journalctl -u cloudflared-tunnel -f
```

### Verify Tunnel Connectivity

```bash
# Test tunnel connectivity via Cloudflare
# From local machine or any device with internet

# Test ml110-01 tunnel
curl -I https://ml110-01.sankofa.nexus

# Test ml110-01 API tunnel
curl -I https://ml110-01-api.sankofa.nexus

# Test r630-01 tunnel
curl -I https://r630-01.sankofa.nexus

# Test r630-01 API tunnel
curl -I https://r630-01-api.sankofa.nexus

# Verify certificate is valid (should show Let's Encrypt)
echo | openssl s_client -connect ml110-01.sankofa.nexus:443 -servername ml110-01.sankofa.nexus 2>/dev/null | \
  openssl x509 -noout -issuer
```

## Step 5: Monitor Certificate Renewal

### Check Certificate Expiration Dates

```bash
# Run verification script to check expiration
./scripts/verify-proxmox-certs.sh

# Or check manually
for node in ml110-01.sankofa.nexus r630-01.sankofa.nexus; do
  echo "Checking ${node}..."
  echo | openssl s_client -connect ${node}:8006 -servername ${node} 2>/dev/null | \
    openssl x509 -noout -dates
done
```

### Monitor ACME Renewal Process

```bash
# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Watch ACME logs in real-time
journalctl -u pvedaemon -f | grep -i acme

# Check renewal schedule
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus | jq '.data.renewal'

# Manually trigger renewal (if needed)
pvesh post /cluster/acme/certificate/wildcard-sankofa-nexus/renew
```

### Set Up Certificate Expiration Monitoring

```bash
# Create monitoring script
cat > /usr/local/bin/check-proxmox-certs.sh <<'EOF'
#!/bin/bash
# Check certificate expiration and alert if expiring soon

NODES=("ml110-01.sankofa.nexus:8006" "r630-01.sankofa.nexus:8006")
WARN_DAYS=30

for node_port in "${NODES[@]}"; do
  node=$(echo $node_port | cut -d: -f1)
  port=$(echo $node_port | cut -d: -f2)
  
  expiry=$(echo | openssl s_client -connect ${node}:${port} -servername ${node} 2>/dev/null | \
           openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  
  if [ -n "$expiry" ]; then
    expiry_epoch=$(date -d "$expiry" +%s 2>/dev/null || date -j -f "%b %d %H:%M:%S %Y %Z" "$expiry" +%s 2>/dev/null)
    now_epoch=$(date +%s)
    days=$(( (expiry_epoch - now_epoch) / 86400 ))
    
    if [ $days -lt $WARN_DAYS ]; then
      echo "WARNING: Certificate for ${node} expires in ${days} days (${expiry})"
    else
      echo "OK: Certificate for ${node} expires in ${days} days"
    fi
  fi
done
EOF

chmod +x /usr/local/bin/check-proxmox-certs.sh

# Add to cron for daily checks
(crontab -l 2>/dev/null; echo "0 9 * * * /usr/local/bin/check-proxmox-certs.sh | mail -s 'Proxmox Certificate Status' admin@example.com") | crontab -
```

## Step 6: Troubleshooting Commands

### If Certificate Order Fails

```bash
# Check ACME logs
journalctl -u pvedaemon | grep -i acme | tail -50

# Verify DNS plugin configuration
pvesh get /cluster/acme/plugins/cloudflare-dns

# Test Cloudflare API connectivity
curl -I https://api.cloudflare.com

# Check DNS TXT record creation
dig _acme-challenge.sankofa.nexus TXT +short

# Verify token still works
curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "Authorization: Bearer YOUR_TOKEN"
```

### If Certificate Not Applied to Node

```bash
# Check certificate exists
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus

# Check node certificate directory
ls -la /etc/pve/nodes/ml110-01/

# Restart pveproxy to reload certificates
systemctl restart pveproxy

# Check pveproxy logs
journalctl -u pveproxy -f
```

### If Tunnel Fails After Removing skipVerify

```bash
# Check tunnel logs for certificate errors
journalctl -u cloudflared-tunnel --since "10 minutes ago" | grep -i "certificate\|tls\|ssl"

# Temporarily re-enable skipVerify for debugging (NOT recommended)
# Edit config file and add back:
#   tls:
#     skipVerify: true
# Then restart tunnel

# Verify certificate is accessible from tunnel service
curl -k https://localhost:8006
```

### Reset ACME Configuration (If Needed)

```bash
# Delete certificate
pvesh delete /cluster/acme/certificate/wildcard-sankofa-nexus

# Delete DNS plugin
pvesh delete /cluster/acme/plugins/cloudflare-dns

# Delete ACME account
pvesh delete /cluster/acme/account/letsencrypt-prod

# Start fresh and reconfigure
```

## Quick Reference: All-in-One Setup Commands

### Complete Setup on ml110-01 (via SSH)

```bash
# SSH to node
ssh root@ml110-01.sankofa.nexus

# Set variables
CLOUDFLARE_TOKEN="YOUR_TOKEN_HERE"
EMAIL="admin@example.com"

# Register ACME account
pvesh create /cluster/acme/account \
  --name letsencrypt-prod \
  --directory https://acme-v02.api.letsencrypt.org/directory \
  --contact ${EMAIL}

# Add DNS plugin
pvesh create /cluster/acme/plugins \
  --id cloudflare-dns \
  --type cloudflare \
  --data "api={${CLOUDFLARE_TOKEN}} domain=sankofa.nexus"

# Order wildcard certificate
pvesh create /cluster/acme/certificate \
  --name wildcard-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "*.sankofa.nexus"

# Order single-host certificates
pvesh create /cluster/acme/certificate \
  --name ml110-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "ml110-01.sankofa.nexus"

pvesh create /cluster/acme/certificate \
  --name r630-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "r630-01.sankofa.nexus"

# Verify certificates
pvesh get /cluster/acme/certificate

# Check certificate details
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus | jq
```

### Deploy Tunnel Configs (from local machine)

```bash
# From project root
cd /home/intlc/projects/Sankofa

# Copy configs
scp cloudflare/tunnel-configs/proxmox-site-1.yaml root@ml110-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/
scp cloudflare/tunnel-configs/proxmox-site-2.yaml root@r630-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/

# Restart services
ssh root@ml110-01.sankofa.nexus "systemctl restart cloudflared-tunnel && systemctl status cloudflared-tunnel"
ssh root@r630-01.sankofa.nexus "systemctl restart cloudflared-tunnel && systemctl status cloudflared-tunnel"
```

### Full Verification (from local machine)

```bash
# Run verification script
./scripts/verify-proxmox-certs.sh

# Manual verification
for node in ml110-01.sankofa.nexus r630-01.sankofa.nexus; do
  echo "=== ${node} ==="
  echo | openssl s_client -connect ${node}:8006 -servername ${node} 2>/dev/null | \
    openssl x509 -noout -dates -subject -issuer | head -5
  echo
done
```

## Related Documentation

- [ACME Setup Guide](./acme-setup.md) - Complete step-by-step guide
- [TLS Configuration](./TLS_CONFIGURATION.md) - General TLS configuration
- [Cloudflare Tunnel Configs](../cloudflare/tunnel-configs/) - Tunnel configuration files