Sankofa/docs/proxmox/acme-setup.md

Proxmox ACME Certificate Setup with Cloudflare DNS-01

Overview

This guide provides step-by-step instructions for configuring Let's Encrypt certificates on Proxmox nodes using Cloudflare DNS-01 validation. This setup enables automatic certificate issuance and renewal without exposing Proxmox to the internet.

Prerequisites

Domain Requirements

• Domain must be managed by Cloudflare DNS
• Domain must be active in Cloudflare account
• DNS records for Proxmox nodes must exist (or will be created automatically)

Current Setup:

• Domain: sankofa.nexus
• Proxmox Nodes:
  • ml110-01.sankofa.nexus (192.168.11.10) - Admin functions
  • r630-01.sankofa.nexus (192.168.11.11) - VMs and heavy lifting

Proxmox Requirements

• Proxmox VE 6.0+ (ACME plugin included)
• Administrative access to Proxmox web UI
• Network connectivity from Proxmox to Cloudflare API (outbound HTTPS)

Cloudflare Requirements

• Cloudflare account with DNS management access
• Ability to create API tokens

Step 1: Create Cloudflare API Token

Purpose

The API token allows Proxmox ACME plugin to create and delete DNS TXT records for domain validation.

Instructions

1. Log in to Cloudflare Dashboard

   • Go to https://dash.cloudflare.com
   • Select your account

2. Navigate to API Tokens

   • Click your profile icon (top right)
   • Select "My Profile"
   • Click "API Tokens" in the left sidebar
   • Click "Create Token"

3. Configure Token Permissions

   Option A: Use Template (Recommended)

   • Scroll to "Edit zone DNS" template
   • Click "Use template"
   • Continue to Zone Resources

   Option B: Custom Token

   • Click "Create Custom Token"
   • Token Name: proxmox-acme-dns01 (or descriptive name)
   • Permissions:
     • Resource: Zone
     • Permission: DNS
     • Permission Level: Edit
   • Click "Continue to summary"

4. Configure Zone Resources

   • Include: Select "Specific zone"
   • Zone: Select sankofa.nexus
   • DNS: Edit (should be pre-selected)
   • Review permissions summary

5. Create Token

   • Click "Continue to summary"
   • Review configuration
   • Click "Create Token"
   • CRITICAL: Copy the token immediately (shown only once)
   • Store securely (password manager recommended)

Security Best Practices

• Scope: Token scoped to sankofa.nexus zone only (not "All zones")
• Permissions: DNS Edit only (minimal required permissions)
• Naming: Use descriptive token name for easy identification
• Rotation: Rotate token periodically (recommended: every 90 days)
• Storage: Never commit token to version control
• Access: Limit who has access to the token

Token Format

The token will look like:

abcdef1234567890abcdef1234567890abcdef12

Step 2: Configure ACME Account in Proxmox

On ml110-01 (Admin Node)

1. Access Proxmox Web UI

   • Navigate to https://ml110-01.sankofa.nexus:8006
   • Log in with administrative credentials

2. Navigate to ACME Accounts

   • Click "Datacenter" in the left sidebar
   • Expand "ACME" section
   • Click "Accounts"

3. Add ACME Account

   • Click "Add" button
   • Directory: Select "Let's Encrypt" (production)
   • Email: Enter your administrative email address
     • Used for certificate expiration notifications
     • Example: admin@sankofa.nexus
   • Name: letsencrypt-prod
   • Click "Register"

4. Verify Account Creation

   • Account should appear in the list
   • Status should show as active/registered

Step 3: Configure Cloudflare DNS Plugin

1. Navigate to ACME DNS Plugins

   • In "Datacenter" → "ACME" section
   • Click "DNS Plugin"

2. Add DNS Plugin

   • Click "Add" button
   • ID: cloudflare-dns (or descriptive identifier)
   • Plugin: Select "cloudflare" from dropdown
   • API Token: Paste the Cloudflare API token from Step 1
   • Domain: sankofa.nexus
   • Click "Create"

3. Verify Plugin Configuration

   • Plugin should appear in the list
   • Status should be valid

Troubleshooting Plugin Configuration

Error: "Invalid API token"

• Verify token was copied correctly (no extra spaces)
• Ensure token has DNS Edit permissions for sankofa.nexus zone
• Check token hasn't been deleted or rotated in Cloudflare

Error: "Zone not found"

• Verify domain spelling matches exactly
• Ensure domain is in your Cloudflare account
• Check you have DNS management access to the zone

Step 4: Order Certificates

Wildcard Certificate (Recommended)

1. Navigate to ACME Certificates

   • In "Datacenter" → "ACME" section
   • Click "Certificates"

2. Order Wildcard Certificate

   • Click "Order Certificate"
   • ACME Account: Select letsencrypt-prod
   • DNS Plugin: Select cloudflare-dns
   • Domain: *.sankofa.nexus
   • Name: wildcard-sankofa-nexus (optional, for identification)
   • Click "Order"

3. Monitor Certificate Order

   • Certificate status will show as "Ordering..."
   • Proxmox will automatically:
     • Create DNS TXT record via Cloudflare API
     • Wait for DNS propagation
     • Validate domain ownership
     • Download certificate
   • Process typically takes 1-2 minutes

4. Verify Certificate

   • Status should change to "Valid" or "Active"
   • Certificate should show expiration date (~90 days from now)
   • Click certificate to view details (SANs, issuer, etc.)

Single-Host Certificates (Optional)

While the wildcard certificate covers all subdomains, you may want specific certificates for clarity or compliance.

Certificate for ml110-01

1. Order Certificate

   • Click "Order Certificate"
   • ACME Account: letsencrypt-prod
   • DNS Plugin: cloudflare-dns
   • Domain: ml110-01.sankofa.nexus
   • Name: ml110-01-sankofa-nexus
   • Click "Order"

2. Wait for Validation

   • Same process as wildcard certificate
   • Status should become "Valid"

Certificate for r630-01

1. Order Certificate

   • Click "Order Certificate"
   • ACME Account: letsencrypt-prod
   • DNS Plugin: cloudflare-dns
   • Domain: r630-01.sankofa.nexus
   • Name: r630-01-sankofa-nexus
   • Click "Order"

2. Wait for Validation

   • Status should become "Valid"

Step 5: Apply Certificates to Nodes

Apply Wildcard Certificate to All Nodes

1. Navigate to Certificates

   • Click "Datacenter" in left sidebar
   • Click "Certificates" (not under ACME section)

2. Select Certificate

   • Find wildcard-sankofa-nexus in the list
   • Click on the certificate name

3. Edit Certificate Assignment

   • Click "Edit" button
   • Check boxes for both nodes:
     • ml110-01 (Admin node)
     • r630-01 (VM node)
   • Click "OK"

4. Verify Application

   • Certificate should show as assigned to both nodes
   • Nodes will automatically use the certificate for HTTPS

Apply Single-Host Certificates (if created)

Single-host certificates are automatically used by the matching node when present.

To verify:

• Check certificate details show correct hostname
• Certificate should automatically appear on the matching node

Step 6: Verify Certificate Installation

Method 1: Proxmox Web UI

1. Check Certificate Status

   • Datacenter → Certificates
   • Verify certificates show as "Valid"
   • Check expiration dates (should be ~90 days)

2. Test HTTPS Connection

   • Access Proxmox web UI via HTTPS
   • Browser should show valid certificate (green lock icon)
   • Certificate should show as issued by "Let's Encrypt"
   • Click lock icon to view certificate details

Method 2: Command Line (from Proxmox node)

# Check certificate on ml110-01
openssl s_client -connect ml110-01.sankofa.nexus:8006 -servername ml110-01.sankofa.nexus < /dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer

# Check certificate on r630-01
openssl s_client -connect r630-01.sankofa.nexus:8006 -servername r630-01.sankofa.nexus < /dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer

Expected output:

• Issuer: Should show "Let's Encrypt"
• Subject: Should include node hostname
• Valid From/To: Should show current date and ~90 days future

Method 3: Using Verification Script

Run the provided verification script:

./scripts/verify-proxmox-certs.sh

This script will:

• Check certificate validity on both nodes
• Verify expiration dates
• Test SSL connections
• Report certificate status

Step 7: Update Cloudflare Tunnel Configurations

After certificates are installed, update tunnel configurations to remove skipVerify: true.

Files to Update

• cloudflare/tunnel-configs/proxmox-site-1.yaml (ml110-01)
• cloudflare/tunnel-configs/proxmox-site-2.yaml (r630-01)
• cloudflare/tunnel-configs/proxmox-site-3.yaml (if applicable)

Changes Required

Before:

originRequest:
  tls:
    skipVerify: true

After:

originRequest:
  tls:
    skipVerify: false

Or simply remove the tls section entirely (defaults to skipVerify: false).

Apply Changes

1. Update configuration files
2. Copy updated configs to Proxmox nodes
3. Restart cloudflared tunnel service:

   systemctl restart cloudflared-tunnel
   

4. Verify tunnel connectivity:

   systemctl status cloudflared-tunnel
   

Auto-Renewal

How It Works

Proxmox ACME plugin automatically renews certificates before expiration:

• Renewal typically occurs 30 days before expiration
• Process is automatic (no manual intervention)
• Renewal uses same DNS-01 validation process
• No downtime required

Verification

1. Check Auto-Renewal Status

   • Datacenter → ACME → Certificates
   • Certificates should show auto-renewal enabled
   • Check renewal date (should be ~30 days before expiration)

2. Monitor Renewal

   • Proxmox logs renewal attempts
   • Check logs if renewal fails:

     journalctl -u pvedaemon -f | grep -i acme
     

3. Email Notifications

   • Let's Encrypt sends expiration warnings to account email
   • Proxmox may also send notifications on renewal failure

Troubleshooting

Certificate Order Fails

Symptom: Certificate status shows "Failed" or "Error"

Possible Causes:

1. Invalid API Token

   • Verify token in DNS Plugin configuration
   • Test token in Cloudflare API directly
   • Create new token if necessary

2. DNS Propagation Delay

   • Wait 5-10 minutes and retry
   • Check DNS TXT records manually:

     dig _acme-challenge.sankofa.nexus TXT
     

3. Rate Limiting

   • Let's Encrypt has rate limits (50 certs/week/domain)
   • Wait and retry later
   • Use staging environment for testing

4. Network Connectivity

   • Verify Proxmox can reach Cloudflare API (outbound HTTPS)
   • Check firewall rules
   • Test connectivity:

     curl -I https://api.cloudflare.com
     

Solution: Check Proxmox logs:

journalctl -u pvedaemon | grep -i acme | tail -50

Certificate Not Applied to Node

Symptom: Node still uses self-signed certificate

Possible Causes:

1. Certificate not assigned to node
2. Node needs restart (rare)
3. Certificate format issue

Solution:

1. Check certificate assignment in Datacenter → Certificates
2. Verify certificate is valid and not expired
3. Restart pveproxy service if needed:

   systemctl restart pveproxy
   

DNS TXT Records Not Created

Symptom: Certificate validation fails, TXT records missing

Possible Causes:

1. API token lacks permissions
2. Token expired or deleted
3. Cloudflare API error

Solution:

1. Verify API token in DNS Plugin configuration
2. Test token manually:

   curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
     -H "Authorization: Bearer YOUR_TOKEN"
   

3. Check Cloudflare API status page

Auto-Renewal Fails

Symptom: Certificate expires, not renewed automatically

Possible Causes:

1. API token expired or rotated
2. Network connectivity issues
3. Let's Encrypt rate limiting
4. DNS plugin configuration changed

Solution:

1. Check renewal logs:

   journalctl -u pvedaemon | grep -i "acme\|renewal" | tail -100
   

2. Verify API token is still valid
3. Manually trigger renewal:
   • Go to Datacenter → ACME → Certificates
   • Select certificate
   • Click "Renew" (if available)

Certificate Shows as Invalid in Browser

Symptom: Browser shows certificate error despite valid certificate

Possible Causes:

1. Browser cache issue
2. Certificate chain incomplete
3. Wrong certificate applied

Solution:

1. Clear browser cache and cookies
2. Try incognito/private browsing mode
3. Verify certificate via command line (see Step 6)
4. Check certificate chain is complete

Certificate Management

View Certificate Details

In Proxmox Web UI:

• Datacenter → Certificates
• Click certificate name to view details:
  • Subject Alternative Names (SANs)
  • Issuer
  • Valid from/to dates
  • Serial number
  • Public key fingerprint

Via Command Line:

# View certificate file
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem

# Parse certificate details
openssl x509 -in /etc/pve/nodes/ml110-01/pveproxy-ssl.pem -text -noout

Revoke Certificate

If a certificate is compromised or no longer needed:

1. Via Proxmox Web UI:

   • Datacenter → ACME → Certificates
   • Select certificate
   • Click "Revoke"
   • Confirm revocation

2. Note: Revoked certificates cannot be reused

   • Order new certificate if needed
   • Let's Encrypt tracks revoked certificates

Delete Certificate

To remove a certificate from Proxmox:

1. Remove from Nodes:

   • Datacenter → Certificates
   • Select certificate
   • Click "Edit"
   • Uncheck all nodes
   • Click "OK"

2. Delete Certificate:

   • Datacenter → ACME → Certificates
   • Select certificate
   • Click "Delete"
   • Confirm deletion

Token Rotation

When to Rotate

• Every 90 days (recommended)
• After security incident
• If token is compromised
• As part of security policy

Rotation Procedure

1. Create New Token

   • Follow Step 1 to create new Cloudflare API token
   • Use same permissions and zone scope

2. Update DNS Plugin

   • Datacenter → ACME → DNS Plugin
   • Select cloudflare-dns plugin
   • Click "Edit"
   • Replace API Token with new token
   • Click "OK"

3. Verify Certificates Still Work

   • Check certificate status
   • Test HTTPS connections
   • Monitor for 24-48 hours

4. Revoke Old Token

   • Go to Cloudflare Dashboard
   • My Profile → API Tokens
   • Find old token
   • Click "Revoke"
   • Confirm revocation

Testing New Token

Before rotating, test new token works:

# Test token can read DNS records
curl -X GET "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records" \
  -H "Authorization: Bearer NEW_TOKEN" \
  -H "Content-Type: application/json"

# Test token can create DNS records (careful!)
# This creates a test record - delete it after testing
curl -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records" \
  -H "Authorization: Bearer NEW_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"type":"TXT","name":"_acme-test.sankofa.nexus","content":"test","ttl":120}'

Security Considerations

API Token Security

• Never commit tokens to version control
• Store tokens in password manager
• Use separate tokens for different purposes
• Rotate tokens regularly
• Monitor token usage in Cloudflare

Certificate Security

• Let's Encrypt certificates are public (anyone can view them)
• Private keys are stored securely on Proxmox
• Automatic renewal prevents expiration
• Monitor certificate expiration dates

Network Security

• Proxmox nodes should be behind firewall
• Only allow necessary outbound connections
• Monitor Cloudflare API access
• Use VPN or private network when possible

Access Control

• Limit who can modify ACME configuration
• Audit certificate changes
• Review certificate assignments regularly
• Monitor for unauthorized certificate issuance

Compliance

Certificate Requirements

• Valid certificates required for compliance (SOC 2, ISO 27001)
• Automatic renewal ensures continuous validity
• Certificate audit trail in Proxmox logs
• Let's Encrypt certificates meet industry standards

Monitoring and Auditing

• Log all certificate operations
• Monitor certificate expiration
• Audit API token usage
• Review certificate assignments quarterly

Related Documentation

• TLS Configuration Guide - General TLS/SSL configuration
• DNS Configuration - DNS setup for Proxmox
• Cloudflare Domain Setup - Cloudflare integration
• Cloudflare Tunnel Configs - Tunnel configuration files

Verification Checklist

After completing setup, verify:

• Cloudflare API token created and stored securely
• ACME account registered in Proxmox
• Cloudflare DNS plugin configured
• Wildcard certificate ordered and valid
• Single-host certificates ordered (if applicable)
• Certificates applied to both nodes
• HTTPS connections show valid certificates
• Auto-renewal configured and verified
• Tunnel configurations updated (skipVerify removed)
• Verification script passes
• Email notifications configured

Support

Proxmox Resources

• Proxmox ACME Documentation
• Proxmox Forum

Let's Encrypt Resources

• Let's Encrypt Documentation
• Let's Encrypt Community

Cloudflare Resources

• Cloudflare API Documentation
• Cloudflare DNS API

Last Updated

• Date: 2024-12-19
• Status: Complete
• Next Review: 2025-03-19