Sankofa/docs/proxmox/acme-setup.md

# Proxmox ACME Certificate Setup with Cloudflare DNS-01

## Overview

This guide provides step-by-step instructions for configuring Let's Encrypt certificates on Proxmox nodes using Cloudflare DNS-01 validation. This setup enables automatic certificate issuance and renewal without exposing Proxmox to the internet.

## Prerequisites

### Domain Requirements

- Domain must be managed by Cloudflare DNS
- Domain must be active in Cloudflare account
- DNS records for Proxmox nodes must exist (or will be created automatically)

**Current Setup:**
- **Domain**: `sankofa.nexus`
- **Proxmox Nodes**: 
  - `ml110-01.sankofa.nexus` (192.168.11.10) - Admin functions
  - `r630-01.sankofa.nexus` (192.168.11.11) - VMs and heavy lifting

### Proxmox Requirements

- Proxmox VE 6.0+ (ACME plugin included)
- Administrative access to Proxmox web UI
- Network connectivity from Proxmox to Cloudflare API (outbound HTTPS)

### Cloudflare Requirements

- Cloudflare account with DNS management access
- Ability to create API tokens

## Step 1: Create Cloudflare API Token

### Purpose

The API token allows Proxmox ACME plugin to create and delete DNS TXT records for domain validation.

### Instructions

1. **Log in to Cloudflare Dashboard**
   - Go to [https://dash.cloudflare.com](https://dash.cloudflare.com)
   - Select your account

2. **Navigate to API Tokens**
   - Click your profile icon (top right)
   - Select **"My Profile"**
   - Click **"API Tokens"** in the left sidebar
   - Click **"Create Token"**

3. **Configure Token Permissions**

   **Option A: Use Template (Recommended)**
   - Scroll to **"Edit zone DNS"** template
   - Click **"Use template"**
   - Continue to Zone Resources

   **Option B: Custom Token**
   - Click **"Create Custom Token"**
   - **Token Name**: `proxmox-acme-dns01` (or descriptive name)
   - **Permissions**: 
     - Resource: `Zone`
     - Permission: `DNS`
     - Permission Level: `Edit`
   - Click **"Continue to summary"**

4. **Configure Zone Resources**
   - **Include**: Select **"Specific zone"**
   - **Zone**: Select `sankofa.nexus`
   - **DNS**: `Edit` (should be pre-selected)
   - Review permissions summary

5. **Create Token**
   - Click **"Continue to summary"**
   - Review configuration
   - Click **"Create Token"**
   - **CRITICAL**: Copy the token immediately (shown only once)
   - Store securely (password manager recommended)

### Security Best Practices

- **Scope**: Token scoped to `sankofa.nexus` zone only (not "All zones")
- **Permissions**: DNS Edit only (minimal required permissions)
- **Naming**: Use descriptive token name for easy identification
- **Rotation**: Rotate token periodically (recommended: every 90 days)
- **Storage**: Never commit token to version control
- **Access**: Limit who has access to the token

### Token Format

The token will look like:
```
abcdef1234567890abcdef1234567890abcdef12
```

## Step 2: Configure ACME Account in Proxmox

### On ml110-01 (Admin Node)

1. **Access Proxmox Web UI**
   - Navigate to `https://ml110-01.sankofa.nexus:8006`
   - Log in with administrative credentials

2. **Navigate to ACME Accounts**
   - Click **"Datacenter"** in the left sidebar
   - Expand **"ACME"** section
   - Click **"Accounts"**

3. **Add ACME Account**
   - Click **"Add"** button
   - **Directory**: Select **"Let's Encrypt"** (production)
   - **Email**: Enter your administrative email address
     - Used for certificate expiration notifications
     - Example: `admin@sankofa.nexus`
   - **Name**: `letsencrypt-prod`
   - Click **"Register"**

4. **Verify Account Creation**
   - Account should appear in the list
   - Status should show as active/registered

## Step 3: Configure Cloudflare DNS Plugin

1. **Navigate to ACME DNS Plugins**
   - In **"Datacenter"** → **"ACME"** section
   - Click **"DNS Plugin"**

2. **Add DNS Plugin**
   - Click **"Add"** button
   - **ID**: `cloudflare-dns` (or descriptive identifier)
   - **Plugin**: Select **"cloudflare"** from dropdown
   - **API Token**: Paste the Cloudflare API token from Step 1
   - **Domain**: `sankofa.nexus`
   - Click **"Create"**

3. **Verify Plugin Configuration**
   - Plugin should appear in the list
   - Status should be valid

### Troubleshooting Plugin Configuration

**Error: "Invalid API token"**
- Verify token was copied correctly (no extra spaces)
- Ensure token has DNS Edit permissions for `sankofa.nexus` zone
- Check token hasn't been deleted or rotated in Cloudflare

**Error: "Zone not found"**
- Verify domain spelling matches exactly
- Ensure domain is in your Cloudflare account
- Check you have DNS management access to the zone

## Step 4: Order Certificates

### Wildcard Certificate (Recommended)

1. **Navigate to ACME Certificates**
   - In **"Datacenter"** → **"ACME"** section
   - Click **"Certificates"**

2. **Order Wildcard Certificate**
   - Click **"Order Certificate"**
   - **ACME Account**: Select `letsencrypt-prod`
   - **DNS Plugin**: Select `cloudflare-dns`
   - **Domain**: `*.sankofa.nexus`
   - **Name**: `wildcard-sankofa-nexus` (optional, for identification)
   - Click **"Order"**

3. **Monitor Certificate Order**
   - Certificate status will show as "Ordering..."
   - Proxmox will automatically:
     - Create DNS TXT record via Cloudflare API
     - Wait for DNS propagation
     - Validate domain ownership
     - Download certificate
   - Process typically takes 1-2 minutes

4. **Verify Certificate**
   - Status should change to "Valid" or "Active"
   - Certificate should show expiration date (~90 days from now)
   - Click certificate to view details (SANs, issuer, etc.)

### Single-Host Certificates (Optional)

While the wildcard certificate covers all subdomains, you may want specific certificates for clarity or compliance.

#### Certificate for ml110-01

1. **Order Certificate**
   - Click **"Order Certificate"**
   - **ACME Account**: `letsencrypt-prod`
   - **DNS Plugin**: `cloudflare-dns`
   - **Domain**: `ml110-01.sankofa.nexus`
   - **Name**: `ml110-01-sankofa-nexus`
   - Click **"Order"**

2. **Wait for Validation**
   - Same process as wildcard certificate
   - Status should become "Valid"

#### Certificate for r630-01

1. **Order Certificate**
   - Click **"Order Certificate"**
   - **ACME Account**: `letsencrypt-prod`
   - **DNS Plugin**: `cloudflare-dns`
   - **Domain**: `r630-01.sankofa.nexus`
   - **Name**: `r630-01-sankofa-nexus`
   - Click **"Order"**

2. **Wait for Validation**
   - Status should become "Valid"

## Step 5: Apply Certificates to Nodes

### Apply Wildcard Certificate to All Nodes

1. **Navigate to Certificates**
   - Click **"Datacenter"** in left sidebar
   - Click **"Certificates"** (not under ACME section)

2. **Select Certificate**
   - Find `wildcard-sankofa-nexus` in the list
   - Click on the certificate name

3. **Edit Certificate Assignment**
   - Click **"Edit"** button
   - Check boxes for both nodes:
     - `ml110-01` (Admin node)
     - `r630-01` (VM node)
   - Click **"OK"**

4. **Verify Application**
   - Certificate should show as assigned to both nodes
   - Nodes will automatically use the certificate for HTTPS

### Apply Single-Host Certificates (if created)

Single-host certificates are automatically used by the matching node when present.

To verify:
- Check certificate details show correct hostname
- Certificate should automatically appear on the matching node

## Step 6: Verify Certificate Installation

### Method 1: Proxmox Web UI

1. **Check Certificate Status**
   - **Datacenter** → **Certificates**
   - Verify certificates show as "Valid"
   - Check expiration dates (should be ~90 days)

2. **Test HTTPS Connection**
   - Access Proxmox web UI via HTTPS
   - Browser should show valid certificate (green lock icon)
   - Certificate should show as issued by "Let's Encrypt"
   - Click lock icon to view certificate details

### Method 2: Command Line (from Proxmox node)

```bash
# Check certificate on ml110-01
openssl s_client -connect ml110-01.sankofa.nexus:8006 -servername ml110-01.sankofa.nexus < /dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer

# Check certificate on r630-01
openssl s_client -connect r630-01.sankofa.nexus:8006 -servername r630-01.sankofa.nexus < /dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer
```

Expected output:
- **Issuer**: Should show "Let's Encrypt"
- **Subject**: Should include node hostname
- **Valid From/To**: Should show current date and ~90 days future

### Method 3: Using Verification Script

Run the provided verification script:

```bash
./scripts/verify-proxmox-certs.sh
```

This script will:
- Check certificate validity on both nodes
- Verify expiration dates
- Test SSL connections
- Report certificate status

## Step 7: Update Cloudflare Tunnel Configurations

After certificates are installed, update tunnel configurations to remove `skipVerify: true`.

### Files to Update

- `cloudflare/tunnel-configs/proxmox-site-1.yaml` (ml110-01)
- `cloudflare/tunnel-configs/proxmox-site-2.yaml` (r630-01)
- `cloudflare/tunnel-configs/proxmox-site-3.yaml` (if applicable)

### Changes Required

**Before:**
```yaml
originRequest:
  tls:
    skipVerify: true
```

**After:**
```yaml
originRequest:
  tls:
    skipVerify: false
```

Or simply remove the `tls` section entirely (defaults to `skipVerify: false`).

### Apply Changes

1. Update configuration files
2. Copy updated configs to Proxmox nodes
3. Restart cloudflared tunnel service:
   ```bash
   systemctl restart cloudflared-tunnel
   ```
4. Verify tunnel connectivity:
   ```bash
   systemctl status cloudflared-tunnel
   ```

## Auto-Renewal

### How It Works

Proxmox ACME plugin automatically renews certificates before expiration:
- Renewal typically occurs 30 days before expiration
- Process is automatic (no manual intervention)
- Renewal uses same DNS-01 validation process
- No downtime required

### Verification

1. **Check Auto-Renewal Status**
   - **Datacenter** → **ACME** → **Certificates**
   - Certificates should show auto-renewal enabled
   - Check renewal date (should be ~30 days before expiration)

2. **Monitor Renewal**
   - Proxmox logs renewal attempts
   - Check logs if renewal fails:
     ```bash
     journalctl -u pvedaemon -f | grep -i acme
     ```

3. **Email Notifications**
   - Let's Encrypt sends expiration warnings to account email
   - Proxmox may also send notifications on renewal failure

## Troubleshooting

### Certificate Order Fails

**Symptom**: Certificate status shows "Failed" or "Error"

**Possible Causes**:
1. **Invalid API Token**
   - Verify token in DNS Plugin configuration
   - Test token in Cloudflare API directly
   - Create new token if necessary

2. **DNS Propagation Delay**
   - Wait 5-10 minutes and retry
   - Check DNS TXT records manually:
     ```bash
     dig _acme-challenge.sankofa.nexus TXT
     ```

3. **Rate Limiting**
   - Let's Encrypt has rate limits (50 certs/week/domain)
   - Wait and retry later
   - Use staging environment for testing

4. **Network Connectivity**
   - Verify Proxmox can reach Cloudflare API (outbound HTTPS)
   - Check firewall rules
   - Test connectivity:
     ```bash
     curl -I https://api.cloudflare.com
     ```

**Solution**: Check Proxmox logs:
```bash
journalctl -u pvedaemon | grep -i acme | tail -50
```

### Certificate Not Applied to Node

**Symptom**: Node still uses self-signed certificate

**Possible Causes**:
1. Certificate not assigned to node
2. Node needs restart (rare)
3. Certificate format issue

**Solution**:
1. Check certificate assignment in **Datacenter** → **Certificates**
2. Verify certificate is valid and not expired
3. Restart pveproxy service if needed:
   ```bash
   systemctl restart pveproxy
   ```

### DNS TXT Records Not Created

**Symptom**: Certificate validation fails, TXT records missing

**Possible Causes**:
1. API token lacks permissions
2. Token expired or deleted
3. Cloudflare API error

**Solution**:
1. Verify API token in DNS Plugin configuration
2. Test token manually:
   ```bash
   curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
     -H "Authorization: Bearer YOUR_TOKEN"
   ```
3. Check Cloudflare API status page

### Auto-Renewal Fails

**Symptom**: Certificate expires, not renewed automatically

**Possible Causes**:
1. API token expired or rotated
2. Network connectivity issues
3. Let's Encrypt rate limiting
4. DNS plugin configuration changed

**Solution**:
1. Check renewal logs:
   ```bash
   journalctl -u pvedaemon | grep -i "acme\|renewal" | tail -100
   ```
2. Verify API token is still valid
3. Manually trigger renewal:
   - Go to **Datacenter** → **ACME** → **Certificates**
   - Select certificate
   - Click **"Renew"** (if available)

### Certificate Shows as Invalid in Browser

**Symptom**: Browser shows certificate error despite valid certificate

**Possible Causes**:
1. Browser cache issue
2. Certificate chain incomplete
3. Wrong certificate applied

**Solution**:
1. Clear browser cache and cookies
2. Try incognito/private browsing mode
3. Verify certificate via command line (see Step 6)
4. Check certificate chain is complete

## Certificate Management

### View Certificate Details

**In Proxmox Web UI**:
- **Datacenter** → **Certificates**
- Click certificate name to view details:
  - Subject Alternative Names (SANs)
  - Issuer
  - Valid from/to dates
  - Serial number
  - Public key fingerprint

**Via Command Line**:
```bash
# View certificate file
cat /etc/pve/nodes/ml110-01/pveproxy-ssl.pem

# Parse certificate details
openssl x509 -in /etc/pve/nodes/ml110-01/pveproxy-ssl.pem -text -noout
```

### Revoke Certificate

If a certificate is compromised or no longer needed:

1. **Via Proxmox Web UI**:
   - **Datacenter** → **ACME** → **Certificates**
   - Select certificate
   - Click **"Revoke"**
   - Confirm revocation

2. **Note**: Revoked certificates cannot be reused
   - Order new certificate if needed
   - Let's Encrypt tracks revoked certificates

### Delete Certificate

To remove a certificate from Proxmox:

1. **Remove from Nodes**:
   - **Datacenter** → **Certificates**
   - Select certificate
   - Click **"Edit"**
   - Uncheck all nodes
   - Click **"OK"**

2. **Delete Certificate**:
   - **Datacenter** → **ACME** → **Certificates**
   - Select certificate
   - Click **"Delete"**
   - Confirm deletion

## Token Rotation

### When to Rotate

- Every 90 days (recommended)
- After security incident
- If token is compromised
- As part of security policy

### Rotation Procedure

1. **Create New Token**
   - Follow Step 1 to create new Cloudflare API token
   - Use same permissions and zone scope

2. **Update DNS Plugin**
   - **Datacenter** → **ACME** → **DNS Plugin**
   - Select `cloudflare-dns` plugin
   - Click **"Edit"**
   - Replace **API Token** with new token
   - Click **"OK"**

3. **Verify Certificates Still Work**
   - Check certificate status
   - Test HTTPS connections
   - Monitor for 24-48 hours

4. **Revoke Old Token**
   - Go to Cloudflare Dashboard
   - **My Profile** → **API Tokens**
   - Find old token
   - Click **"Revoke"**
   - Confirm revocation

### Testing New Token

Before rotating, test new token works:

```bash
# Test token can read DNS records
curl -X GET "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records" \
  -H "Authorization: Bearer NEW_TOKEN" \
  -H "Content-Type: application/json"

# Test token can create DNS records (careful!)
# This creates a test record - delete it after testing
curl -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records" \
  -H "Authorization: Bearer NEW_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"type":"TXT","name":"_acme-test.sankofa.nexus","content":"test","ttl":120}'
```

## Security Considerations

### API Token Security

- **Never commit tokens to version control**
- **Store tokens in password manager**
- **Use separate tokens for different purposes**
- **Rotate tokens regularly**
- **Monitor token usage in Cloudflare**

### Certificate Security

- **Let's Encrypt certificates are public** (anyone can view them)
- **Private keys are stored securely on Proxmox**
- **Automatic renewal prevents expiration**
- **Monitor certificate expiration dates**

### Network Security

- **Proxmox nodes should be behind firewall**
- **Only allow necessary outbound connections**
- **Monitor Cloudflare API access**
- **Use VPN or private network when possible**

### Access Control

- **Limit who can modify ACME configuration**
- **Audit certificate changes**
- **Review certificate assignments regularly**
- **Monitor for unauthorized certificate issuance**

## Compliance

### Certificate Requirements

- **Valid certificates required for compliance (SOC 2, ISO 27001)**
- **Automatic renewal ensures continuous validity**
- **Certificate audit trail in Proxmox logs**
- **Let's Encrypt certificates meet industry standards**

### Monitoring and Auditing

- **Log all certificate operations**
- **Monitor certificate expiration**
- **Audit API token usage**
- **Review certificate assignments quarterly**

## Related Documentation

- [TLS Configuration Guide](./TLS_CONFIGURATION.md) - General TLS/SSL configuration
- [DNS Configuration](./DNS_CONFIGURATION.md) - DNS setup for Proxmox
- [Cloudflare Domain Setup](./CLOUDFLARE_DOMAIN_SETUP.md) - Cloudflare integration
- [Cloudflare Tunnel Configs](../cloudflare/tunnel-configs/) - Tunnel configuration files

## Verification Checklist

After completing setup, verify:

- [ ] Cloudflare API token created and stored securely
- [ ] ACME account registered in Proxmox
- [ ] Cloudflare DNS plugin configured
- [ ] Wildcard certificate ordered and valid
- [ ] Single-host certificates ordered (if applicable)
- [ ] Certificates applied to both nodes
- [ ] HTTPS connections show valid certificates
- [ ] Auto-renewal configured and verified
- [ ] Tunnel configurations updated (skipVerify removed)
- [ ] Verification script passes
- [ ] Email notifications configured

## Support

### Proxmox Resources

- [Proxmox ACME Documentation](https://pve.proxmox.com/pve-docs/chapter-pve-certificate-management.html)
- [Proxmox Forum](https://forum.proxmox.com/)

### Let's Encrypt Resources

- [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
- [Let's Encrypt Community](https://community.letsencrypt.org/)

### Cloudflare Resources

- [Cloudflare API Documentation](https://developers.cloudflare.com/api/)
- [Cloudflare DNS API](https://developers.cloudflare.com/api/operations/dns-records-for-a-zone-list-dns-records)

## Last Updated

- **Date**: 2024-12-19
- **Status**: Complete
- **Next Review**: 2025-03-19