proxmox/reports/status/ALL_TUNNELS_DOWN.md

All Tunnels Down - Critical Issue

Status: 🔴 CRITICAL

All 6 Cloudflare tunnels are DOWN - This means no services are accessible via tunnels.

Affected Tunnels

Tunnel Name	Tunnel ID	Status	Purpose
explorer.d-bis.org	b02fe1fe-cb7d-484e-909b-7cc41298ebe8	🔴 DOWN	Explorer/Blockscout
mim4u-tunnel	f8d06879-04f8-44ef-aeda-ce84564a1792	🔴 DOWN	MIM4U Services
rpc-http-pub.d-bis.org	10ab22da-8ea3-4e2e-a896-27ece2211a05	🔴 DOWN	RPC, API, Admin (9 hostnames)
tunnel-ml110	ccd7150a-9881-4b8c-a105-9b4ead6e69a2	🔴 DOWN	Proxmox ml110-01
tunnel-r630-01	4481af8f-b24c-4cd3-bdd5-f562f4c97df4	🔴 DOWN	Proxmox r630-01
tunnel-r630-02	0876f12b-64d7-4927-9ab3-94cb6cf48af9	🔴 DOWN	Proxmox r630-02

Root Cause Analysis

All tunnels being DOWN indicates:

1. cloudflared service not running in VMID 102
2. Network connectivity issues from container to Cloudflare
3. Authentication/credentials issues
4. Container not running (VMID 102 stopped)
5. Firewall blocking outbound connections

Impact

• ❌ No Proxmox UI access via tunnels
• ❌ No RPC endpoints accessible
• ❌ No API endpoints accessible
• ❌ No Explorer accessible
• ❌ No Admin interface accessible
• ❌ All tunnel-based services offline

Diagnostic Steps

Step 1: Check Container Status

# Check if VMID 102 is running
ssh root@192.168.11.12 "pct status 102"

# Check container details
ssh root@192.168.11.12 "pct list | grep 102"

Step 2: Check cloudflared Services

# Check all cloudflared services
ssh root@192.168.11.12 "pct exec 102 -- systemctl list-units | grep cloudflared"

# Check service status
ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"

Step 3: Check Network Connectivity

# Test outbound connectivity from container
ssh root@192.168.11.12 "pct exec 102 -- curl -I https://cloudflare.com"

# Test DNS resolution
ssh root@192.168.11.12 "pct exec 102 -- nslookup cloudflare.com"

Step 4: Check Tunnel Logs

# View recent logs
ssh root@192.168.11.12 "pct exec 102 -- journalctl -u cloudflared-* -n 50 --no-pager"

# Follow logs in real-time
ssh root@192.168.11.12 "pct exec 102 -- journalctl -u cloudflared-* -f"

Step 5: Verify Credentials

# Check if credential files exist
ssh root@192.168.11.12 "pct exec 102 -- ls -la /etc/cloudflared/credentials-*.json"

# Verify file permissions (should be 600)
ssh root@192.168.11.12 "pct exec 102 -- ls -l /etc/cloudflared/credentials-*.json"

Quick Fix Attempts

Fix 1: Restart All Tunnel Services

ssh root@192.168.11.12 "pct exec 102 -- systemctl restart cloudflared-*"
sleep 5
ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"

Fix 2: Restart Container

ssh root@192.168.11.12 "pct stop 102"
sleep 2
ssh root@192.168.11.12 "pct start 102"
sleep 10
ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"

Fix 3: Check and Fix cloudflared Installation

# Check if cloudflared is installed
ssh root@192.168.11.12 "pct exec 102 -- which cloudflared"

# Check version
ssh root@192.168.11.12 "pct exec 102 -- cloudflared --version"

# Reinstall if needed
ssh root@192.168.11.12 "pct exec 102 -- apt update && apt install -y cloudflared"

Common Issues & Solutions

Issue 1: Container Not Running

Solution: Start container

ssh root@192.168.11.12 "pct start 102"

Issue 2: Services Not Enabled

Solution: Enable and start services

ssh root@192.168.11.12 "pct exec 102 -- systemctl enable cloudflared-*"
ssh root@192.168.11.12 "pct exec 102 -- systemctl start cloudflared-*"

Issue 3: Network Issues

Solution: Check container network configuration

ssh root@192.168.11.12 "pct exec 102 -- ip addr"
ssh root@192.168.11.12 "pct exec 102 -- ping -c 3 8.8.8.8"

Issue 4: Credentials Missing/Invalid

Solution: Re-download credentials from Cloudflare Dashboard

• Go to: Zero Trust → Networks → Tunnels
• Click on each tunnel → Configure → Download credentials
• Copy to container: /etc/cloudflared/credentials-<tunnel-name>.json

Issue 5: Firewall Blocking

Solution: Check firewall rules on Proxmox host

ssh root@192.168.11.12 "iptables -L -n | grep -i cloudflare"

Recovery Procedure

Full Recovery Steps

1. Verify Container Status

   ssh root@192.168.11.12 "pct status 102"
   

2. Start Container if Stopped

   ssh root@192.168.11.12 "pct start 102"
   

3. Check cloudflared Installation

   ssh root@192.168.11.12 "pct exec 102 -- cloudflared --version"
   

4. Verify Credentials Exist

   ssh root@192.168.11.12 "pct exec 102 -- ls -la /etc/cloudflared/credentials-*.json"
   

5. Restart All Services

   ssh root@192.168.11.12 "pct exec 102 -- systemctl restart cloudflared-*"
   

6. Check Service Status

   ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"
   

7. Monitor Logs

   ssh root@192.168.11.12 "pct exec 102 -- journalctl -u cloudflared-* -f"
   

8. Verify in Cloudflare Dashboard

   • Wait 1-2 minutes
   • Check tunnel status in dashboard
   • Should change from DOWN to HEALTHY

Prevention

1. Monitor Tunnel Health

   • Set up alerts in Cloudflare
   • Monitor service status regularly

2. Automated Restart

   • Use systemd restart policies
   • Set up health checks

3. Backup Credentials

   • Store credentials securely
   • Document tunnel configurations

4. Network Monitoring

   • Monitor container network connectivity
   • Alert on connectivity issues

Summary

Status: 🔴 All tunnels DOWN
Priority: 🔴 CRITICAL - Immediate action required
Impact: All tunnel-based services offline
Next Steps: Run diagnostic script, identify root cause, apply fix