Sankofa/docs/proxmox/acme-commands.md

Proxmox ACME Setup - Complete Command Reference

This document provides all commands needed for the Proxmox ACME certificate setup process.

Prerequisites

Verify Network Connectivity

# Test Cloudflare API connectivity from Proxmox nodes
curl -I https://api.cloudflare.com

# Test DNS resolution for nodes
dig ml110-01.sankofa.nexus
dig r630-01.sankofa.nexus

# Test HTTPS connectivity to Proxmox nodes
curl -k -I https://ml110-01.sankofa.nexus:8006
curl -k -I https://r630-01.sankofa.nexus:8006

Step 1: Create Cloudflare API Token

Note: This step must be done via Cloudflare Web UI (no CLI commands).

1. Go to: https://dash.cloudflare.com/profile/api-tokens
2. Click "Create Token"
3. Use "Edit zone DNS" template
4. Select zone: sankofa.nexus
5. Copy token and save securely

Verify Token (Optional - Test from Local Machine)

# Replace YOUR_TOKEN with actual token
TOKEN="YOUR_CLOUDFLARE_API_TOKEN"
ZONE_ID="YOUR_ZONE_ID"  # Get from Cloudflare dashboard

# Test token can read DNS records
curl -X GET "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" | jq '.result[] | select(.name | contains("sankofa.nexus"))'

# Test token can create DNS records (creates test record - delete after)
curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"type":"TXT","name":"_acme-test.sankofa.nexus","content":"test-validation","ttl":120}' | jq

# Delete test record
RECORD_ID="RECORD_ID_FROM_ABOVE"
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${RECORD_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json"

Step 2: Configure ACME via Proxmox CLI (Alternative to Web UI)

SSH to ml110-01 (Admin Node)

# SSH to admin node
ssh root@ml110-01.sankofa.nexus
# or
ssh root@192.168.11.10

Register ACME Account via CLI

# Register Let's Encrypt account
pvesh create /cluster/acme/account \
  --name letsencrypt-prod \
  --directory https://acme-v02.api.letsencrypt.org/directory \
  --contact email@example.com \
  --account

# Verify account registration
pvesh get /cluster/acme/account/letsencrypt-prod

Add Cloudflare DNS Plugin via CLI

# Replace YOUR_TOKEN with actual Cloudflare API token
CLOUDFLARE_TOKEN="YOUR_CLOUDFLARE_API_TOKEN"

# Add Cloudflare DNS plugin
pvesh create /cluster/acme/plugins \
  --id cloudflare-dns \
  --type cloudflare \
  --data "api={${CLOUDFLARE_TOKEN}} domain=sankofa.nexus"

# Verify plugin configuration
pvesh get /cluster/acme/plugins/cloudflare-dns

Order Certificates via CLI

# Order wildcard certificate
pvesh create /cluster/acme/certificate \
  --name wildcard-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "*.sankofa.nexus"

# Order single-host certificate for ml110-01
pvesh create /cluster/acme/certificate \
  --name ml110-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "ml110-01.sankofa.nexus"

# Order single-host certificate for r630-01
pvesh create /cluster/acme/certificate \
  --name r630-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "r630-01.sankofa.nexus"

# Check certificate order status
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus
pvesh get /cluster/acme/certificate/ml110-01-sankofa-nexus
pvesh get /cluster/acme/certificate/r630-01-sankofa-nexus

# List all ACME certificates
pvesh get /cluster/acme/certificate

Apply Certificate to Nodes via CLI

# Apply wildcard certificate to ml110-01 node
pvesh create /nodes/ml110-01/certificates/custom \
  --filename wildcard-sankofa-nexus.pem \
  --certificates wildcard-sankofa-nexus

# Apply wildcard certificate to r630-01 node
pvesh create /nodes/r630-01/certificates/custom \
  --filename wildcard-sankofa-nexus.pem \
  --certificates wildcard-sankofa-nexus

# Verify certificate assignment
pvesh get /nodes/ml110-01/certificates/custom
pvesh get /nodes/r630-01/certificates/custom

Note: Certificate application to nodes is typically done automatically when certificates are ordered. The above commands may need adjustment based on Proxmox version.

Alternative: Apply via Web UI Certificate Manager

If CLI certificate application doesn't work:

1. Go to: https://ml110-01.sankofa.nexus:8006
2. Navigate: Datacenter → Certificates
3. Select certificate: wildcard-sankofa-nexus
4. Click "Edit"
5. Check boxes for: ml110-01 and r630-01
6. Click "OK"

Step 3: Verify Certificate Installation

Check Certificate Status on Nodes

# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Check ACME certificates
pvesh get /cluster/acme/certificate

# Check node certificates
pvesh get /nodes/ml110-01/certificates/custom

# View certificate file
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem | openssl x509 -noout -text

# Check certificate expiration
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem | openssl x509 -noout -dates

# Verify issuer (should show Let's Encrypt)
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem | openssl x509 -noout -issuer

Test SSL Connection from Local Machine

# Test ml110-01 certificate
echo | openssl s_client -connect ml110-01.sankofa.nexus:8006 -servername ml110-01.sankofa.nexus 2>/dev/null | \
  openssl x509 -noout -dates -subject -issuer

# Test r630-01 certificate
echo | openssl s_client -connect r630-01.sankofa.nexus:8006 -servername r630-01.sankofa.nexus 2>/dev/null | \
  openssl x509 -noout -dates -subject -issuer

# Check certificate chain
echo | openssl s_client -connect ml110-01.sankofa.nexus:8006 -servername ml110-01.sankofa.nexus -showcerts 2>/dev/null | \
  openssl x509 -noout -text | grep -A 5 "Certificate chain"

Run Verification Script

# From project root
cd /home/intlc/projects/Sankofa

# Run verification script
./scripts/verify-proxmox-certs.sh

# Or with explicit path
bash scripts/verify-proxmox-certs.sh

Check Proxmox Logs for ACME

# On ml110-01
ssh root@ml110-01.sankofa.nexus

# View ACME-related logs
journalctl -u pvedaemon -f | grep -i acme

# Check recent ACME activity
journalctl -u pvedaemon --since "1 hour ago" | grep -i acme

# View ACME certificate order logs
journalctl -u pvedaemon | grep -i "wildcard-sankofa-nexus"

Step 4: Deploy Updated Tunnel Configurations

Copy Updated Configs to Proxmox Nodes

# From project root
cd /home/intlc/projects/Sankofa

# Copy config to ml110-01 (Site 1)
scp cloudflare/tunnel-configs/proxmox-site-1.yaml root@ml110-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/proxmox-site-1.yaml

# Copy config to r630-01 (Site 2)
scp cloudflare/tunnel-configs/proxmox-site-2.yaml root@r630-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/proxmox-site-2.yaml

# If Site 3 exists, copy config
# scp cloudflare/tunnel-configs/proxmox-site-3.yaml root@r630-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/proxmox-site-3.yaml

Verify Config Files on Nodes

# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Verify config file exists and is correct
cat /etc/cloudflared/tunnel-configs/proxmox-site-1.yaml

# Verify skipVerify is removed (should NOT appear in file)
grep -i "skipverify" /etc/cloudflared/tunnel-configs/proxmox-site-1.yaml
# Should return no results

# Check YAML syntax
cloudflared tunnel ingress validate --config /etc/cloudflared/tunnel-configs/proxmox-site-1.yaml

Restart Cloudflare Tunnel Services

# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Check tunnel service status
systemctl status cloudflared-tunnel

# Restart tunnel service
systemctl restart cloudflared-tunnel

# Verify service started successfully
systemctl status cloudflared-tunnel

# View tunnel logs
journalctl -u cloudflared-tunnel -f

# Check for errors (especially certificate verification)
journalctl -u cloudflared-tunnel --since "5 minutes ago" | grep -i error

# On r630-01
ssh root@r630-01.sankofa.nexus

# Same commands as above
systemctl status cloudflared-tunnel
systemctl restart cloudflared-tunnel
systemctl status cloudflared-tunnel
journalctl -u cloudflared-tunnel -f

Verify Tunnel Connectivity

# Test tunnel connectivity via Cloudflare
# From local machine or any device with internet

# Test ml110-01 tunnel
curl -I https://ml110-01.sankofa.nexus

# Test ml110-01 API tunnel
curl -I https://ml110-01-api.sankofa.nexus

# Test r630-01 tunnel
curl -I https://r630-01.sankofa.nexus

# Test r630-01 API tunnel
curl -I https://r630-01-api.sankofa.nexus

# Verify certificate is valid (should show Let's Encrypt)
echo | openssl s_client -connect ml110-01.sankofa.nexus:443 -servername ml110-01.sankofa.nexus 2>/dev/null | \
  openssl x509 -noout -issuer

Step 5: Monitor Certificate Renewal

Check Certificate Expiration Dates

# Run verification script to check expiration
./scripts/verify-proxmox-certs.sh

# Or check manually
for node in ml110-01.sankofa.nexus r630-01.sankofa.nexus; do
  echo "Checking ${node}..."
  echo | openssl s_client -connect ${node}:8006 -servername ${node} 2>/dev/null | \
    openssl x509 -noout -dates
done

Monitor ACME Renewal Process

# On ml110-01
ssh root@ml110-01.sankofa.nexus

# Watch ACME logs in real-time
journalctl -u pvedaemon -f | grep -i acme

# Check renewal schedule
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus | jq '.data.renewal'

# Manually trigger renewal (if needed)
pvesh post /cluster/acme/certificate/wildcard-sankofa-nexus/renew

Set Up Certificate Expiration Monitoring

# Create monitoring script
cat > /usr/local/bin/check-proxmox-certs.sh <<'EOF'
#!/bin/bash
# Check certificate expiration and alert if expiring soon

NODES=("ml110-01.sankofa.nexus:8006" "r630-01.sankofa.nexus:8006")
WARN_DAYS=30

for node_port in "${NODES[@]}"; do
  node=$(echo $node_port | cut -d: -f1)
  port=$(echo $node_port | cut -d: -f2)
  
  expiry=$(echo | openssl s_client -connect ${node}:${port} -servername ${node} 2>/dev/null | \
           openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  
  if [ -n "$expiry" ]; then
    expiry_epoch=$(date -d "$expiry" +%s 2>/dev/null || date -j -f "%b %d %H:%M:%S %Y %Z" "$expiry" +%s 2>/dev/null)
    now_epoch=$(date +%s)
    days=$(( (expiry_epoch - now_epoch) / 86400 ))
    
    if [ $days -lt $WARN_DAYS ]; then
      echo "WARNING: Certificate for ${node} expires in ${days} days (${expiry})"
    else
      echo "OK: Certificate for ${node} expires in ${days} days"
    fi
  fi
done
EOF

chmod +x /usr/local/bin/check-proxmox-certs.sh

# Add to cron for daily checks
(crontab -l 2>/dev/null; echo "0 9 * * * /usr/local/bin/check-proxmox-certs.sh | mail -s 'Proxmox Certificate Status' admin@example.com") | crontab -

Step 6: Troubleshooting Commands

If Certificate Order Fails

# Check ACME logs
journalctl -u pvedaemon | grep -i acme | tail -50

# Verify DNS plugin configuration
pvesh get /cluster/acme/plugins/cloudflare-dns

# Test Cloudflare API connectivity
curl -I https://api.cloudflare.com

# Check DNS TXT record creation
dig _acme-challenge.sankofa.nexus TXT +short

# Verify token still works
curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "Authorization: Bearer YOUR_TOKEN"

If Certificate Not Applied to Node

# Check certificate exists
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus

# Check node certificate directory
ls -la /etc/pve/nodes/ml110-01/

# Restart pveproxy to reload certificates
systemctl restart pveproxy

# Check pveproxy logs
journalctl -u pveproxy -f

If Tunnel Fails After Removing skipVerify

# Check tunnel logs for certificate errors
journalctl -u cloudflared-tunnel --since "10 minutes ago" | grep -i "certificate\|tls\|ssl"

# Temporarily re-enable skipVerify for debugging (NOT recommended)
# Edit config file and add back:
#   tls:
#     skipVerify: true
# Then restart tunnel

# Verify certificate is accessible from tunnel service
curl -k https://localhost:8006

Reset ACME Configuration (If Needed)

# Delete certificate
pvesh delete /cluster/acme/certificate/wildcard-sankofa-nexus

# Delete DNS plugin
pvesh delete /cluster/acme/plugins/cloudflare-dns

# Delete ACME account
pvesh delete /cluster/acme/account/letsencrypt-prod

# Start fresh and reconfigure

Quick Reference: All-in-One Setup Commands

Complete Setup on ml110-01 (via SSH)

# SSH to node
ssh root@ml110-01.sankofa.nexus

# Set variables
CLOUDFLARE_TOKEN="YOUR_TOKEN_HERE"
EMAIL="admin@example.com"

# Register ACME account
pvesh create /cluster/acme/account \
  --name letsencrypt-prod \
  --directory https://acme-v02.api.letsencrypt.org/directory \
  --contact ${EMAIL}

# Add DNS plugin
pvesh create /cluster/acme/plugins \
  --id cloudflare-dns \
  --type cloudflare \
  --data "api={${CLOUDFLARE_TOKEN}} domain=sankofa.nexus"

# Order wildcard certificate
pvesh create /cluster/acme/certificate \
  --name wildcard-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "*.sankofa.nexus"

# Order single-host certificates
pvesh create /cluster/acme/certificate \
  --name ml110-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "ml110-01.sankofa.nexus"

pvesh create /cluster/acme/certificate \
  --name r630-01-sankofa-nexus \
  --acme letsencrypt-prod \
  --plugin cloudflare-dns \
  --domain "r630-01.sankofa.nexus"

# Verify certificates
pvesh get /cluster/acme/certificate

# Check certificate details
pvesh get /cluster/acme/certificate/wildcard-sankofa-nexus | jq

Deploy Tunnel Configs (from local machine)

# From project root
cd /home/intlc/projects/Sankofa

# Copy configs
scp cloudflare/tunnel-configs/proxmox-site-1.yaml root@ml110-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/
scp cloudflare/tunnel-configs/proxmox-site-2.yaml root@r630-01.sankofa.nexus:/etc/cloudflared/tunnel-configs/

# Restart services
ssh root@ml110-01.sankofa.nexus "systemctl restart cloudflared-tunnel && systemctl status cloudflared-tunnel"
ssh root@r630-01.sankofa.nexus "systemctl restart cloudflared-tunnel && systemctl status cloudflared-tunnel"

Full Verification (from local machine)

# Run verification script
./scripts/verify-proxmox-certs.sh

# Manual verification
for node in ml110-01.sankofa.nexus r630-01.sankofa.nexus; do
  echo "=== ${node} ==="
  echo | openssl s_client -connect ${node}:8006 -servername ${node} 2>/dev/null | \
    openssl x509 -noout -dates -subject -issuer | head -5
  echo
done

Related Documentation

• ACME Setup Guide - Complete step-by-step guide
• TLS Configuration - General TLS configuration
• Cloudflare Tunnel Configs - Tunnel configuration files