proxmox/reports/status/ALL_TUNNELS_DOWN.md

# All Tunnels Down - Critical Issue

## Status: 🔴 CRITICAL

**All 6 Cloudflare tunnels are DOWN** - This means no services are accessible via tunnels.

## Affected Tunnels

| Tunnel Name | Tunnel ID | Status | Purpose |
|-------------|-----------|--------|---------|
| explorer.d-bis.org | b02fe1fe-cb7d-484e-909b-7cc41298ebe8 | 🔴 DOWN | Explorer/Blockscout |
| mim4u-tunnel | f8d06879-04f8-44ef-aeda-ce84564a1792 | 🔴 DOWN | MIM4U Services |
| rpc-http-pub.d-bis.org | 10ab22da-8ea3-4e2e-a896-27ece2211a05 | 🔴 DOWN | RPC, API, Admin (9 hostnames) |
| tunnel-ml110 | ccd7150a-9881-4b8c-a105-9b4ead6e69a2 | 🔴 DOWN | Proxmox ml110-01 |
| tunnel-r630-01 | 4481af8f-b24c-4cd3-bdd5-f562f4c97df4 | 🔴 DOWN | Proxmox r630-01 |
| tunnel-r630-02 | 0876f12b-64d7-4927-9ab3-94cb6cf48af9 | 🔴 DOWN | Proxmox r630-02 |

## Root Cause Analysis

All tunnels being DOWN indicates:

1. **cloudflared service not running** in VMID 102
2. **Network connectivity issues** from container to Cloudflare
3. **Authentication/credentials issues**
4. **Container not running** (VMID 102 stopped)
5. **Firewall blocking outbound connections**

## Impact

- ❌ No Proxmox UI access via tunnels
- ❌ No RPC endpoints accessible
- ❌ No API endpoints accessible
- ❌ No Explorer accessible
- ❌ No Admin interface accessible
- ❌ All tunnel-based services offline

## Diagnostic Steps

### Step 1: Check Container Status

```bash
# Check if VMID 102 is running
ssh root@192.168.11.12 "pct status 102"

# Check container details
ssh root@192.168.11.12 "pct list | grep 102"
```

### Step 2: Check cloudflared Services

```bash
# Check all cloudflared services
ssh root@192.168.11.12 "pct exec 102 -- systemctl list-units | grep cloudflared"

# Check service status
ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"
```

### Step 3: Check Network Connectivity

```bash
# Test outbound connectivity from container
ssh root@192.168.11.12 "pct exec 102 -- curl -I https://cloudflare.com"

# Test DNS resolution
ssh root@192.168.11.12 "pct exec 102 -- nslookup cloudflare.com"
```

### Step 4: Check Tunnel Logs

```bash
# View recent logs
ssh root@192.168.11.12 "pct exec 102 -- journalctl -u cloudflared-* -n 50 --no-pager"

# Follow logs in real-time
ssh root@192.168.11.12 "pct exec 102 -- journalctl -u cloudflared-* -f"
```

### Step 5: Verify Credentials

```bash
# Check if credential files exist
ssh root@192.168.11.12 "pct exec 102 -- ls -la /etc/cloudflared/credentials-*.json"

# Verify file permissions (should be 600)
ssh root@192.168.11.12 "pct exec 102 -- ls -l /etc/cloudflared/credentials-*.json"
```

## Quick Fix Attempts

### Fix 1: Restart All Tunnel Services

```bash
ssh root@192.168.11.12 "pct exec 102 -- systemctl restart cloudflared-*"
sleep 5
ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"
```

### Fix 2: Restart Container

```bash
ssh root@192.168.11.12 "pct stop 102"
sleep 2
ssh root@192.168.11.12 "pct start 102"
sleep 10
ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"
```

### Fix 3: Check and Fix cloudflared Installation

```bash
# Check if cloudflared is installed
ssh root@192.168.11.12 "pct exec 102 -- which cloudflared"

# Check version
ssh root@192.168.11.12 "pct exec 102 -- cloudflared --version"

# Reinstall if needed
ssh root@192.168.11.12 "pct exec 102 -- apt update && apt install -y cloudflared"
```

## Common Issues & Solutions

### Issue 1: Container Not Running
**Solution**: Start container
```bash
ssh root@192.168.11.12 "pct start 102"
```

### Issue 2: Services Not Enabled
**Solution**: Enable and start services
```bash
ssh root@192.168.11.12 "pct exec 102 -- systemctl enable cloudflared-*"
ssh root@192.168.11.12 "pct exec 102 -- systemctl start cloudflared-*"
```

### Issue 3: Network Issues
**Solution**: Check container network configuration
```bash
ssh root@192.168.11.12 "pct exec 102 -- ip addr"
ssh root@192.168.11.12 "pct exec 102 -- ping -c 3 8.8.8.8"
```

### Issue 4: Credentials Missing/Invalid
**Solution**: Re-download credentials from Cloudflare Dashboard
- Go to: Zero Trust → Networks → Tunnels
- Click on each tunnel → Configure → Download credentials
- Copy to container: `/etc/cloudflared/credentials-<tunnel-name>.json`

### Issue 5: Firewall Blocking
**Solution**: Check firewall rules on Proxmox host
```bash
ssh root@192.168.11.12 "iptables -L -n | grep -i cloudflare"
```

## Recovery Procedure

### Full Recovery Steps

1. **Verify Container Status**
   ```bash
   ssh root@192.168.11.12 "pct status 102"
   ```

2. **Start Container if Stopped**
   ```bash
   ssh root@192.168.11.12 "pct start 102"
   ```

3. **Check cloudflared Installation**
   ```bash
   ssh root@192.168.11.12 "pct exec 102 -- cloudflared --version"
   ```

4. **Verify Credentials Exist**
   ```bash
   ssh root@192.168.11.12 "pct exec 102 -- ls -la /etc/cloudflared/credentials-*.json"
   ```

5. **Restart All Services**
   ```bash
   ssh root@192.168.11.12 "pct exec 102 -- systemctl restart cloudflared-*"
   ```

6. **Check Service Status**
   ```bash
   ssh root@192.168.11.12 "pct exec 102 -- systemctl status cloudflared-* --no-pager"
   ```

7. **Monitor Logs**
   ```bash
   ssh root@192.168.11.12 "pct exec 102 -- journalctl -u cloudflared-* -f"
   ```

8. **Verify in Cloudflare Dashboard**
   - Wait 1-2 minutes
   - Check tunnel status in dashboard
   - Should change from DOWN to HEALTHY

## Prevention

1. **Monitor Tunnel Health**
   - Set up alerts in Cloudflare
   - Monitor service status regularly

2. **Automated Restart**
   - Use systemd restart policies
   - Set up health checks

3. **Backup Credentials**
   - Store credentials securely
   - Document tunnel configurations

4. **Network Monitoring**
   - Monitor container network connectivity
   - Alert on connectivity issues

## Summary

**Status**: 🔴 All tunnels DOWN  
**Priority**: 🔴 CRITICAL - Immediate action required  
**Impact**: All tunnel-based services offline  
**Next Steps**: Run diagnostic script, identify root cause, apply fix