Complete markdown files cleanup and organization
- Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
This commit is contained in:
322
scripts/cloudflare-tunnels/docs/CLOUDFLARE_ACCESS_SETUP.md
Normal file
322
scripts/cloudflare-tunnels/docs/CLOUDFLARE_ACCESS_SETUP.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# Cloudflare Access Setup Guide
|
||||
|
||||
This guide walks you through setting up Cloudflare Access (Zero Trust) to protect your Proxmox UI endpoints with SSO/MFA.
|
||||
|
||||
## Overview
|
||||
|
||||
Cloudflare Access provides:
|
||||
- ✅ **Single Sign-On (SSO)** - Use your existing identity provider
|
||||
- ✅ **Multi-Factor Authentication (MFA)** - Additional security layer
|
||||
- ✅ **Device Posture Checks** - Require managed devices
|
||||
- ✅ **Audit Logs** - Track all access attempts
|
||||
- ✅ **Session Management** - Control session duration
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. ✅ Cloudflare account with Zero Trust enabled
|
||||
2. ✅ Domain `d-bis.org` managed by Cloudflare
|
||||
3. ✅ Tunnels created and configured (see main README)
|
||||
4. ✅ DNS records created (CNAME pointing to tunnels)
|
||||
|
||||
## Step 1: Enable Cloudflare Zero Trust
|
||||
|
||||
1. **Navigate to Cloudflare Zero Trust:**
|
||||
- Go to: https://one.dash.cloudflare.com
|
||||
- Sign in with your Cloudflare account
|
||||
|
||||
2. **Verify Zero Trust is enabled:**
|
||||
- If not enabled, you'll be prompted to enable it
|
||||
- This is free for up to 50 users
|
||||
|
||||
## Step 2: Create Tunnels in Cloudflare Dashboard
|
||||
|
||||
For each Proxmox host, create a separate tunnel:
|
||||
|
||||
### 2.1 Create Tunnel for ml110-01
|
||||
|
||||
1. **Go to Zero Trust → Networks → Tunnels**
|
||||
2. **Click "Create a tunnel"**
|
||||
3. **Select "Cloudflared"**
|
||||
4. **Enter tunnel name:** `tunnel-ml110`
|
||||
5. **Click "Save tunnel"**
|
||||
6. **Copy the tunnel token** (starts with `eyJ...`)
|
||||
- Save this securely - you'll need it for VMID 102
|
||||
|
||||
### 2.2 Create Tunnel for r630-01
|
||||
|
||||
Repeat the same process:
|
||||
- **Tunnel name:** `tunnel-r630-01`
|
||||
- **Copy tunnel token**
|
||||
|
||||
### 2.3 Create Tunnel for r630-02
|
||||
|
||||
Repeat the same process:
|
||||
- **Tunnel name:** `tunnel-r630-02`
|
||||
- **Copy tunnel token**
|
||||
|
||||
## Step 3: Configure Tunnel Public Hostnames
|
||||
|
||||
For each tunnel, configure the public hostname:
|
||||
|
||||
### 3.1 Configure ml110-01 Tunnel
|
||||
|
||||
1. **Click on tunnel `tunnel-ml110`**
|
||||
2. **Click "Configure"**
|
||||
3. **Go to "Public Hostnames" tab**
|
||||
4. **Click "Add a public hostname"**
|
||||
5. **Configure:**
|
||||
- **Subdomain:** `ml110-01`
|
||||
- **Domain:** `d-bis.org`
|
||||
- **Service:** `https://192.168.11.10:8006`
|
||||
- **Type:** HTTP
|
||||
6. **Click "Save hostname"**
|
||||
|
||||
### 3.2 Configure r630-01 Tunnel
|
||||
|
||||
Repeat for r630-01:
|
||||
- **Subdomain:** `r630-01`
|
||||
- **Domain:** `d-bis.org`
|
||||
- **Service:** `https://192.168.11.11:8006`
|
||||
|
||||
### 3.3 Configure r630-02 Tunnel
|
||||
|
||||
Repeat for r630-02:
|
||||
- **Subdomain:** `r630-02`
|
||||
- **Domain:** `d-bis.org`
|
||||
- **Service:** `https://192.168.11.12:8006`
|
||||
|
||||
## Step 4: Create DNS Records
|
||||
|
||||
Create CNAME records in Cloudflare DNS:
|
||||
|
||||
1. **Go to Cloudflare Dashboard → DNS → Records**
|
||||
2. **Add records:**
|
||||
|
||||
| Type | Name | Target | Proxy | TTL |
|
||||
|------|------|--------|-------|-----|
|
||||
| CNAME | `ml110-01` | `<tunnel-id-ml110>.cfargotunnel.com` | 🟠 Proxied | Auto |
|
||||
| CNAME | `r630-01` | `<tunnel-id-r630-01>.cfargotunnel.com` | 🟠 Proxied | Auto |
|
||||
| CNAME | `r630-02` | `<tunnel-id-r630-02>.cfargotunnel.com` | 🟠 Proxied | Auto |
|
||||
|
||||
**Important:**
|
||||
- ✅ Use CNAME (not A records)
|
||||
- ✅ Enable proxy (orange cloud)
|
||||
- ✅ Replace `<tunnel-id-*>` with actual tunnel IDs from Step 2
|
||||
|
||||
## Step 5: Configure Cloudflare Access Applications
|
||||
|
||||
For each Proxmox host, create an Access application:
|
||||
|
||||
### 5.1 Create Application for ml110-01
|
||||
|
||||
1. **Go to Zero Trust → Access → Applications**
|
||||
2. **Click "Add an application"**
|
||||
3. **Select "Self-hosted"**
|
||||
4. **Configure Application:**
|
||||
- **Application name:** `Proxmox ml110-01`
|
||||
- **Application domain:** `ml110-01.d-bis.org`
|
||||
- **Session duration:** `8 hours` (or your preference)
|
||||
5. **Click "Next"**
|
||||
|
||||
### 5.2 Configure Access Policy
|
||||
|
||||
1. **Click "Add a policy"**
|
||||
2. **Policy name:** `Allow Team Access`
|
||||
3. **Action:** `Allow`
|
||||
4. **Include:**
|
||||
- **Select:** `Emails`
|
||||
- **Value:** `@yourdomain.com` (or specific emails)
|
||||
- **OR** select `Country` and choose your country
|
||||
5. **Require:**
|
||||
- ✅ **Multi-factor authentication** (MFA)
|
||||
- ✅ **Email verification** (optional but recommended)
|
||||
6. **Click "Next"**
|
||||
|
||||
### 5.3 Configure Additional Settings
|
||||
|
||||
1. **CORS settings:** Leave default (not needed for Proxmox UI)
|
||||
2. **Cookie settings:** Leave default
|
||||
3. **Click "Add application"**
|
||||
|
||||
### 5.4 Repeat for Other Hosts
|
||||
|
||||
Repeat Steps 5.1-5.3 for:
|
||||
- **r630-01** → `r630-01.d-bis.org`
|
||||
- **r630-02** → `r630-02.d-bis.org`
|
||||
|
||||
## Step 6: Configure Identity Providers (Optional but Recommended)
|
||||
|
||||
If you want to use SSO instead of email-based auth:
|
||||
|
||||
### 6.1 Add Identity Provider
|
||||
|
||||
1. **Go to Zero Trust → Access → Authentication**
|
||||
2. **Click "Add new" under Identity Providers**
|
||||
3. **Select your provider:**
|
||||
- Google Workspace
|
||||
- Microsoft Azure AD
|
||||
- Okta
|
||||
- Generic OIDC
|
||||
- Generic SAML
|
||||
- etc.
|
||||
|
||||
4. **Follow provider-specific setup instructions**
|
||||
|
||||
### 6.2 Update Access Policies
|
||||
|
||||
1. **Go back to Applications**
|
||||
2. **Edit each application policy**
|
||||
3. **Change "Include" to use your identity provider**
|
||||
4. **Save changes**
|
||||
|
||||
## Step 7: Advanced Security Settings (Recommended)
|
||||
|
||||
### 7.1 Device Posture Checks
|
||||
|
||||
Require managed devices:
|
||||
|
||||
1. **Go to Zero Trust → Settings → WARP**
|
||||
2. **Enable WARP for your organization**
|
||||
3. **Go to Zero Trust → Access → Applications**
|
||||
4. **Edit application policy**
|
||||
5. **Add "Require" condition:**
|
||||
- **Select:** `Device Posture`
|
||||
- **Require:** `Managed device` or `WARP client`
|
||||
|
||||
### 7.2 Country Blocking
|
||||
|
||||
Block access from specific countries:
|
||||
|
||||
1. **Edit application policy**
|
||||
2. **Add "Exclude" condition:**
|
||||
- **Select:** `Country`
|
||||
- **Value:** Select countries to block
|
||||
|
||||
### 7.3 IP Allowlisting
|
||||
|
||||
Restrict to specific IPs:
|
||||
|
||||
1. **Edit application policy**
|
||||
2. **Add "Include" condition:**
|
||||
- **Select:** `IP Address`
|
||||
- **Value:** Your office/home IP ranges
|
||||
|
||||
## Step 8: Test Access
|
||||
|
||||
### 8.1 Test DNS Resolution
|
||||
|
||||
```bash
|
||||
dig ml110-01.d-bis.org
|
||||
dig r630-01.d-bis.org
|
||||
dig r630-02.d-bis.org
|
||||
```
|
||||
|
||||
Should resolve to Cloudflare IPs.
|
||||
|
||||
### 8.2 Test HTTPS Access
|
||||
|
||||
```bash
|
||||
# Should redirect to Cloudflare Access login
|
||||
curl -I https://ml110-01.d-bis.org
|
||||
```
|
||||
|
||||
### 8.3 Test Browser Access
|
||||
|
||||
1. **Open browser**
|
||||
2. **Navigate to:** `https://ml110-01.d-bis.org`
|
||||
3. **Should see Cloudflare Access login page**
|
||||
4. **Login with your credentials**
|
||||
5. **Complete MFA if required**
|
||||
6. **Should redirect to Proxmox UI**
|
||||
|
||||
## Step 9: Monitor Access
|
||||
|
||||
### 9.1 View Access Logs
|
||||
|
||||
1. **Go to Zero Trust → Access → Logs**
|
||||
2. **View authentication attempts**
|
||||
3. **Check for failed login attempts**
|
||||
|
||||
### 9.2 Set Up Alerts
|
||||
|
||||
1. **Go to Zero Trust → Settings → Notifications**
|
||||
2. **Configure email alerts for:**
|
||||
- Failed authentication attempts
|
||||
- Suspicious activity
|
||||
- Policy violations
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Access Page Not Showing
|
||||
|
||||
**Problem:** Direct access to Proxmox UI, no Cloudflare Access page
|
||||
|
||||
**Solutions:**
|
||||
1. Verify DNS record has proxy enabled (orange cloud)
|
||||
2. Check tunnel is running: `systemctl status cloudflared-ml110`
|
||||
3. Verify application is configured correctly
|
||||
4. Check Cloudflare dashboard for tunnel status
|
||||
|
||||
### MFA Not Working
|
||||
|
||||
**Problem:** MFA prompt not appearing
|
||||
|
||||
**Solutions:**
|
||||
1. Verify MFA is enabled in policy
|
||||
2. Check identity provider settings
|
||||
3. Verify user has MFA configured
|
||||
|
||||
### Can't Access After Login
|
||||
|
||||
**Problem:** Login successful but can't reach Proxmox UI
|
||||
|
||||
**Solutions:**
|
||||
1. Check tunnel is running
|
||||
2. Verify tunnel configuration points to correct IP:port
|
||||
3. Check Proxmox UI is accessible internally
|
||||
4. Review tunnel logs: `journalctl -u cloudflared-ml110 -f`
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. ✅ **Always enable MFA** - Required for admin interfaces
|
||||
2. ✅ **Use short session durations** - 4-8 hours for admin access
|
||||
3. ✅ **Enable device posture checks** - Require managed devices
|
||||
4. ✅ **Monitor access logs** - Review regularly for suspicious activity
|
||||
5. ✅ **Use IP allowlisting** - If you have static IPs
|
||||
6. ✅ **Enable email verification** - Additional security layer
|
||||
7. ✅ **Set up alerts** - Get notified of failed attempts
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Application URLs
|
||||
- ml110-01: `https://ml110-01.d-bis.org`
|
||||
- r630-01: `https://r630-01.d-bis.org`
|
||||
- r630-02: `https://r630-02.d-bis.org`
|
||||
|
||||
### Tunnel Names
|
||||
- `tunnel-ml110`
|
||||
- `tunnel-r630-01`
|
||||
- `tunnel-r630-02`
|
||||
|
||||
### Service Names
|
||||
- `cloudflared-ml110.service`
|
||||
- `cloudflared-r630-01.service`
|
||||
- `cloudflared-r630-02.service`
|
||||
|
||||
## Next Steps
|
||||
|
||||
After completing this setup:
|
||||
|
||||
1. ✅ Test access to all three Proxmox hosts
|
||||
2. ✅ Configure monitoring (see `MONITORING_GUIDE.md`)
|
||||
3. ✅ Set up alerting (see `MONITORING_GUIDE.md`)
|
||||
4. ✅ Review access logs regularly
|
||||
5. ✅ Update policies as needed
|
||||
|
||||
## Support
|
||||
|
||||
For issues:
|
||||
1. Check [Troubleshooting Guide](TROUBLESHOOTING.md)
|
||||
2. Review Cloudflare Zero Trust documentation
|
||||
3. Check tunnel logs: `journalctl -u cloudflared-*`
|
||||
|
||||
363
scripts/cloudflare-tunnels/docs/MONITORING_GUIDE.md
Normal file
363
scripts/cloudflare-tunnels/docs/MONITORING_GUIDE.md
Normal file
@@ -0,0 +1,363 @@
|
||||
# Monitoring Guide
|
||||
|
||||
Complete guide for monitoring Cloudflare tunnels.
|
||||
|
||||
## Overview
|
||||
|
||||
Monitoring ensures your tunnels are healthy and alerts you to issues before they impact users.
|
||||
|
||||
## Monitoring Components
|
||||
|
||||
1. **Health Checks** - Verify tunnels are running
|
||||
2. **Connectivity Tests** - Verify DNS and HTTPS work
|
||||
3. **Log Monitoring** - Watch for errors
|
||||
4. **Alerting** - Notify on failures
|
||||
|
||||
## Quick Start
|
||||
|
||||
### One-Time Health Check
|
||||
|
||||
```bash
|
||||
./scripts/check-tunnel-health.sh
|
||||
```
|
||||
|
||||
### Continuous Monitoring
|
||||
|
||||
```bash
|
||||
# Foreground (see output)
|
||||
./scripts/monitor-tunnels.sh
|
||||
|
||||
# Background (daemon mode)
|
||||
./scripts/monitor-tunnels.sh --daemon
|
||||
```
|
||||
|
||||
## Health Check Script
|
||||
|
||||
The `check-tunnel-health.sh` script performs comprehensive checks:
|
||||
|
||||
### Checks Performed
|
||||
|
||||
1. **Service Status** - Is the systemd service running?
|
||||
2. **Log Errors** - Are there recent errors in logs?
|
||||
3. **DNS Resolution** - Does DNS resolve correctly?
|
||||
4. **HTTPS Connectivity** - Can we connect via HTTPS?
|
||||
5. **Internal Connectivity** - Can VMID 102 reach Proxmox hosts?
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Run health check
|
||||
./scripts/check-tunnel-health.sh
|
||||
|
||||
# Output shows:
|
||||
# - Service status for each tunnel
|
||||
# - DNS resolution status
|
||||
# - HTTPS connectivity
|
||||
# - Internal connectivity
|
||||
# - Recent errors
|
||||
```
|
||||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
Tunnel: ml110 (ml110-01.d-bis.org)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
[✓] Service is running
|
||||
[✓] No recent errors in logs
|
||||
[✓] DNS resolution: OK
|
||||
→ 104.16.132.229
|
||||
[✓] HTTPS connectivity: OK
|
||||
[✓] Internal connectivity to 192.168.11.10:8006: OK
|
||||
```
|
||||
|
||||
## Monitoring Script
|
||||
|
||||
The `monitor-tunnels.sh` script provides continuous monitoring:
|
||||
|
||||
### Features
|
||||
|
||||
- ✅ Continuous health checks
|
||||
- ✅ Automatic restart on failure
|
||||
- ✅ Alerting on failures
|
||||
- ✅ Logging to file
|
||||
- ✅ Daemon mode support
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Foreground mode (see output)
|
||||
./scripts/monitor-tunnels.sh
|
||||
|
||||
# Daemon mode (background)
|
||||
./scripts/monitor-tunnels.sh --daemon
|
||||
|
||||
# Check if daemon is running
|
||||
ps aux | grep monitor-tunnels
|
||||
|
||||
# Stop daemon
|
||||
kill $(cat /tmp/cloudflared-monitor.pid)
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Edit the script to customize:
|
||||
|
||||
```bash
|
||||
CHECK_INTERVAL=60 # Check every 60 seconds
|
||||
LOG_FILE="/var/log/cloudflared-monitor.log"
|
||||
ALERT_SCRIPT="./scripts/alert-tunnel-failure.sh"
|
||||
```
|
||||
|
||||
## Alerting
|
||||
|
||||
### Email Alerts
|
||||
|
||||
Configure email alerts in `alert-tunnel-failure.sh`:
|
||||
|
||||
```bash
|
||||
# Set email address
|
||||
export ALERT_EMAIL="admin@yourdomain.com"
|
||||
|
||||
# Ensure mail/sendmail is installed
|
||||
apt-get install -y mailutils
|
||||
```
|
||||
|
||||
### Webhook Alerts
|
||||
|
||||
Configure webhook alerts (Slack, Discord, etc.):
|
||||
|
||||
```bash
|
||||
# Set webhook URL
|
||||
export ALERT_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
|
||||
```
|
||||
|
||||
### Test Alerts
|
||||
|
||||
```bash
|
||||
# Test alert script
|
||||
./scripts/alert-tunnel-failure.sh ml110 service_down
|
||||
```
|
||||
|
||||
## Log Monitoring
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# All tunnels
|
||||
journalctl -u cloudflared-* -f
|
||||
|
||||
# Specific tunnel
|
||||
journalctl -u cloudflared-ml110 -f
|
||||
|
||||
# Last 100 lines
|
||||
journalctl -u cloudflared-ml110 -n 100
|
||||
|
||||
# Since specific time
|
||||
journalctl -u cloudflared-ml110 --since "1 hour ago"
|
||||
```
|
||||
|
||||
### Log Rotation
|
||||
|
||||
Systemd handles log rotation automatically. To customize:
|
||||
|
||||
```bash
|
||||
# Edit logrotate config
|
||||
sudo nano /etc/logrotate.d/cloudflared
|
||||
|
||||
# Add:
|
||||
/var/log/cloudflared/*.log {
|
||||
daily
|
||||
rotate 7
|
||||
compress
|
||||
delaycompress
|
||||
missingok
|
||||
notifempty
|
||||
}
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
### Cloudflare Dashboard
|
||||
|
||||
View tunnel metrics in Cloudflare dashboard:
|
||||
|
||||
1. **Go to:** Zero Trust → Networks → Tunnels
|
||||
2. **Click on tunnel** to view:
|
||||
- Connection status
|
||||
- Uptime
|
||||
- Traffic statistics
|
||||
- Error rates
|
||||
|
||||
### Local Metrics
|
||||
|
||||
Tunnels expose metrics endpoints (if configured):
|
||||
|
||||
```bash
|
||||
# ml110 tunnel metrics
|
||||
curl http://127.0.0.1:9091/metrics
|
||||
|
||||
# r630-01 tunnel metrics
|
||||
curl http://127.0.0.1:9092/metrics
|
||||
|
||||
# r630-02 tunnel metrics
|
||||
curl http://127.0.0.1:9093/metrics
|
||||
```
|
||||
|
||||
## Automated Monitoring Setup
|
||||
|
||||
### Systemd Timer (Recommended)
|
||||
|
||||
Create a systemd timer for automated health checks:
|
||||
|
||||
```bash
|
||||
# Create timer unit
|
||||
sudo nano /etc/systemd/system/cloudflared-healthcheck.timer
|
||||
|
||||
# Add:
|
||||
[Unit]
|
||||
Description=Cloudflare Tunnel Health Check Timer
|
||||
Requires=cloudflared-healthcheck.service
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitActiveSec=5min
|
||||
Unit=cloudflared-healthcheck.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
```bash
|
||||
# Create service unit
|
||||
sudo nano /etc/systemd/system/cloudflared-healthcheck.service
|
||||
|
||||
# Add:
|
||||
[Unit]
|
||||
Description=Cloudflare Tunnel Health Check
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/path/to/scripts/check-tunnel-health.sh
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
```
|
||||
|
||||
```bash
|
||||
# Enable and start
|
||||
sudo systemctl enable cloudflared-healthcheck.timer
|
||||
sudo systemctl start cloudflared-healthcheck.timer
|
||||
```
|
||||
|
||||
### Cron Job (Alternative)
|
||||
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Add (check every 5 minutes):
|
||||
*/5 * * * * /path/to/scripts/check-tunnel-health.sh >> /var/log/tunnel-health.log 2>&1
|
||||
```
|
||||
|
||||
## Monitoring Best Practices
|
||||
|
||||
1. ✅ **Run health checks regularly** - At least every 5 minutes
|
||||
2. ✅ **Monitor logs** - Watch for errors
|
||||
3. ✅ **Set up alerts** - Get notified immediately on failures
|
||||
4. ✅ **Review metrics** - Track trends over time
|
||||
5. ✅ **Test alerts** - Verify alerting works
|
||||
6. ✅ **Document incidents** - Keep track of issues
|
||||
|
||||
## Integration with Monitoring Systems
|
||||
|
||||
### Prometheus
|
||||
|
||||
If using Prometheus, you can scrape tunnel metrics:
|
||||
|
||||
```yaml
|
||||
# prometheus.yml
|
||||
scrape_configs:
|
||||
- job_name: 'cloudflared'
|
||||
static_configs:
|
||||
- targets: ['127.0.0.1:9091', '127.0.0.1:9092', '127.0.0.1:9093']
|
||||
```
|
||||
|
||||
### Grafana
|
||||
|
||||
Create dashboards in Grafana:
|
||||
- Tunnel uptime
|
||||
- Connection status
|
||||
- Error rates
|
||||
- Response times
|
||||
|
||||
### Nagios/Icinga
|
||||
|
||||
Create service checks:
|
||||
```bash
|
||||
# Check service status
|
||||
check_nrpe -H localhost -c check_cloudflared_ml110
|
||||
|
||||
# Check connectivity
|
||||
check_http -H ml110-01.d-bis.org -S
|
||||
```
|
||||
|
||||
## Troubleshooting Monitoring
|
||||
|
||||
### Health Check Fails
|
||||
|
||||
```bash
|
||||
# Run manually with verbose output
|
||||
bash -x ./scripts/check-tunnel-health.sh
|
||||
|
||||
# Check individual components
|
||||
systemctl status cloudflared-ml110
|
||||
dig ml110-01.d-bis.org
|
||||
curl -I https://ml110-01.d-bis.org
|
||||
```
|
||||
|
||||
### Monitor Script Not Working
|
||||
|
||||
```bash
|
||||
# Check if daemon is running
|
||||
ps aux | grep monitor-tunnels
|
||||
|
||||
# Check log file
|
||||
tail -f /var/log/cloudflared-monitor.log
|
||||
|
||||
# Run in foreground to see errors
|
||||
./scripts/monitor-tunnels.sh
|
||||
```
|
||||
|
||||
### Alerts Not Sending
|
||||
|
||||
```bash
|
||||
# Test alert script
|
||||
./scripts/alert-tunnel-failure.sh ml110 service_down
|
||||
|
||||
# Check email configuration
|
||||
echo "Test" | mail -s "Test" admin@yourdomain.com
|
||||
|
||||
# Check webhook
|
||||
curl -X POST -H "Content-Type: application/json" \
|
||||
-d '{"text":"test"}' $ALERT_WEBHOOK
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
After setting up monitoring:
|
||||
|
||||
1. ✅ Verify health checks run successfully
|
||||
2. ✅ Test alerting (trigger a test failure)
|
||||
3. ✅ Set up log aggregation (if needed)
|
||||
4. ✅ Create dashboards (if using Grafana)
|
||||
5. ✅ Document monitoring procedures
|
||||
|
||||
## Support
|
||||
|
||||
For monitoring issues:
|
||||
1. Check [Troubleshooting Guide](TROUBLESHOOTING.md)
|
||||
2. Review script logs
|
||||
3. Test components individually
|
||||
4. Check systemd service status
|
||||
|
||||
353
scripts/cloudflare-tunnels/docs/TROUBLESHOOTING.md
Normal file
353
scripts/cloudflare-tunnels/docs/TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,353 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
Common issues and solutions for Cloudflare Tunnel setup.
|
||||
|
||||
## Quick Diagnostics
|
||||
|
||||
Run the health check script first:
|
||||
```bash
|
||||
./scripts/check-tunnel-health.sh
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
### 1. Tunnel Service Not Running
|
||||
|
||||
**Symptoms:**
|
||||
- `systemctl status cloudflared-ml110` shows inactive
|
||||
- DNS resolves but connection times out
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Check service status
|
||||
systemctl status cloudflared-ml110
|
||||
|
||||
# Check logs for errors
|
||||
journalctl -u cloudflared-ml110 -n 50
|
||||
|
||||
# Restart service
|
||||
systemctl restart cloudflared-ml110
|
||||
|
||||
# Check if credentials file exists
|
||||
ls -la /etc/cloudflared/tunnel-ml110.json
|
||||
|
||||
# Verify configuration
|
||||
cat /etc/cloudflared/tunnel-ml110.yml
|
||||
```
|
||||
|
||||
**Common Causes:**
|
||||
- Missing credentials file
|
||||
- Invalid tunnel ID in config
|
||||
- Network connectivity issues
|
||||
- Incorrect configuration syntax
|
||||
|
||||
### 2. DNS Not Resolving
|
||||
|
||||
**Symptoms:**
|
||||
- `dig ml110-01.d-bis.org` returns no results
|
||||
- Browser shows "DNS_PROBE_FINISHED_NXDOMAIN"
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Verify DNS record exists:**
|
||||
- Go to Cloudflare Dashboard → DNS → Records
|
||||
- Check CNAME record exists for `ml110-01`
|
||||
- Verify proxy is enabled (orange cloud)
|
||||
|
||||
2. **Verify DNS record target:**
|
||||
- Target should be: `<tunnel-id>.cfargotunnel.com`
|
||||
- Not an IP address
|
||||
|
||||
3. **Wait for DNS propagation:**
|
||||
- DNS changes can take up to 5 minutes
|
||||
- Clear DNS cache: `sudo systemd-resolve --flush-caches`
|
||||
|
||||
4. **Test DNS resolution:**
|
||||
```bash
|
||||
dig ml110-01.d-bis.org
|
||||
nslookup ml110-01.d-bis.org
|
||||
```
|
||||
|
||||
### 3. Connection Timeout
|
||||
|
||||
**Symptoms:**
|
||||
- DNS resolves correctly
|
||||
- Connection times out when accessing URL
|
||||
- Browser shows "ERR_CONNECTION_TIMED_OUT"
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check tunnel is running:**
|
||||
```bash
|
||||
systemctl status cloudflared-ml110
|
||||
```
|
||||
|
||||
2. **Check tunnel logs:**
|
||||
```bash
|
||||
journalctl -u cloudflared-ml110 -f
|
||||
```
|
||||
|
||||
3. **Verify internal connectivity:**
|
||||
```bash
|
||||
# From VMID 102, test connection to Proxmox host
|
||||
curl -k https://192.168.11.10:8006
|
||||
```
|
||||
|
||||
4. **Check tunnel configuration:**
|
||||
- Verify service URL is correct: `https://192.168.11.10:8006`
|
||||
- Check tunnel ID matches Cloudflare dashboard
|
||||
|
||||
5. **Verify Cloudflare tunnel status:**
|
||||
- Go to Cloudflare Zero Trust → Networks → Tunnels
|
||||
- Check tunnel shows "Healthy" status
|
||||
|
||||
### 4. SSL Certificate Errors
|
||||
|
||||
**Symptoms:**
|
||||
- Browser shows SSL certificate warnings
|
||||
- "NET::ERR_CERT_AUTHORITY_INVALID"
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **This is expected** - Proxmox uses self-signed certificates
|
||||
2. **Cloudflare handles SSL termination** at the edge
|
||||
3. **Verify tunnel config has:**
|
||||
```yaml
|
||||
originRequest:
|
||||
noTLSVerify: true
|
||||
```
|
||||
|
||||
4. **If issues persist, try HTTP instead:**
|
||||
```yaml
|
||||
service: http://192.168.11.10:8006
|
||||
```
|
||||
(Note: Less secure, but may work if HTTPS issues persist)
|
||||
|
||||
### 5. Cloudflare Access Not Showing
|
||||
|
||||
**Symptoms:**
|
||||
- Direct access to Proxmox UI
|
||||
- No Cloudflare Access login page
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Verify DNS proxy is enabled:**
|
||||
- Cloudflare Dashboard → DNS → Records
|
||||
- Ensure orange cloud is ON (proxied)
|
||||
|
||||
2. **Verify Access application exists:**
|
||||
- Zero Trust → Access → Applications
|
||||
- Check application for `ml110-01.d-bis.org` exists
|
||||
|
||||
3. **Verify Access policy:**
|
||||
- Check policy is configured correctly
|
||||
- Ensure policy is active
|
||||
|
||||
4. **Check SSL/TLS settings:**
|
||||
- Cloudflare Dashboard → SSL/TLS
|
||||
- Set to "Full" or "Full (strict)"
|
||||
|
||||
5. **Clear browser cache:**
|
||||
- Hard refresh: Ctrl+Shift+R (or Cmd+Shift+R on Mac)
|
||||
|
||||
### 6. MFA Not Working
|
||||
|
||||
**Symptoms:**
|
||||
- Login works but MFA prompt doesn't appear
|
||||
- Can't complete authentication
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Verify MFA is enabled in policy:**
|
||||
- Zero Trust → Access → Applications
|
||||
- Edit application → Edit policy
|
||||
- Check "Require" includes MFA
|
||||
|
||||
2. **Check identity provider settings:**
|
||||
- Verify MFA is configured in your identity provider
|
||||
- Test MFA with another application
|
||||
|
||||
3. **Check user MFA status:**
|
||||
- Zero Trust → My Team → Users
|
||||
- Verify user has MFA enabled
|
||||
|
||||
### 7. Service Fails to Start
|
||||
|
||||
**Symptoms:**
|
||||
- `systemctl start cloudflared-ml110` fails
|
||||
- Service immediately stops after starting
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check service logs:**
|
||||
```bash
|
||||
journalctl -u cloudflared-ml110 -n 100
|
||||
```
|
||||
|
||||
2. **Verify credentials file:**
|
||||
```bash
|
||||
ls -la /etc/cloudflared/tunnel-ml110.json
|
||||
cat /etc/cloudflared/tunnel-ml110.json
|
||||
```
|
||||
|
||||
3. **Verify configuration syntax:**
|
||||
```bash
|
||||
cloudflared tunnel --config /etc/cloudflared/tunnel-ml110.yml validate
|
||||
```
|
||||
|
||||
4. **Check file permissions:**
|
||||
```bash
|
||||
chmod 600 /etc/cloudflared/tunnel-ml110.json
|
||||
chmod 644 /etc/cloudflared/tunnel-ml110.yml
|
||||
```
|
||||
|
||||
5. **Test tunnel manually:**
|
||||
```bash
|
||||
cloudflared tunnel --config /etc/cloudflared/tunnel-ml110.yml run
|
||||
```
|
||||
|
||||
### 8. Multiple Tunnels Conflict
|
||||
|
||||
**Symptoms:**
|
||||
- Only one tunnel works
|
||||
- Other tunnels fail to start
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check for port conflicts:**
|
||||
- Each tunnel uses different metrics port
|
||||
- Verify ports 9091, 9092, 9093 are not in use
|
||||
|
||||
2. **Verify separate credentials:**
|
||||
- Each tunnel needs its own credentials file
|
||||
- Verify files are separate: `tunnel-ml110.json`, `tunnel-r630-01.json`, etc.
|
||||
|
||||
3. **Check systemd services:**
|
||||
```bash
|
||||
systemctl status cloudflared-*
|
||||
```
|
||||
|
||||
4. **Verify separate config files:**
|
||||
- Each tunnel has its own config file
|
||||
- Tunnel IDs are different in each config
|
||||
|
||||
## Advanced Troubleshooting
|
||||
|
||||
### Check Tunnel Connectivity
|
||||
|
||||
```bash
|
||||
# From VMID 102, test each Proxmox host
|
||||
curl -k -I https://192.168.11.10:8006
|
||||
curl -k -I https://192.168.11.11:8006
|
||||
curl -k -I https://192.168.11.12:8006
|
||||
```
|
||||
|
||||
### Verify Tunnel Configuration
|
||||
|
||||
```bash
|
||||
# Validate config file
|
||||
cloudflared tunnel --config /etc/cloudflared/tunnel-ml110.yml validate
|
||||
|
||||
# List tunnels
|
||||
cloudflared tunnel list
|
||||
|
||||
# Get tunnel info
|
||||
cloudflared tunnel info <tunnel-id>
|
||||
```
|
||||
|
||||
### Check Network Connectivity
|
||||
|
||||
```bash
|
||||
# Test DNS resolution
|
||||
dig ml110-01.d-bis.org
|
||||
|
||||
# Test HTTPS connectivity
|
||||
curl -v https://ml110-01.d-bis.org
|
||||
|
||||
# Check Cloudflare IPs
|
||||
nslookup ml110-01.d-bis.org
|
||||
```
|
||||
|
||||
### View Real-time Logs
|
||||
|
||||
```bash
|
||||
# Follow logs for all tunnels
|
||||
journalctl -u cloudflared-ml110 -f
|
||||
journalctl -u cloudflared-r630-01 -f
|
||||
journalctl -u cloudflared-r630-02 -f
|
||||
```
|
||||
|
||||
## Diagnostic Commands
|
||||
|
||||
### Complete Health Check
|
||||
|
||||
```bash
|
||||
# Run comprehensive health check
|
||||
./scripts/check-tunnel-health.sh
|
||||
|
||||
# Check all services
|
||||
systemctl status cloudflared-ml110 cloudflared-r630-01 cloudflared-r630-02
|
||||
|
||||
# Check tunnel connectivity
|
||||
for tunnel in ml110 r630-01 r630-02; do
|
||||
echo "Checking $tunnel..."
|
||||
systemctl is-active cloudflared-$tunnel && echo "✓ Running" || echo "✗ Not running"
|
||||
done
|
||||
```
|
||||
|
||||
### Reset Tunnel
|
||||
|
||||
If a tunnel is completely broken:
|
||||
|
||||
```bash
|
||||
# Stop service
|
||||
systemctl stop cloudflared-ml110
|
||||
|
||||
# Remove old credentials (backup first!)
|
||||
cp /etc/cloudflared/tunnel-ml110.json /etc/cloudflared/tunnel-ml110.json.backup
|
||||
|
||||
# Re-authenticate (if needed)
|
||||
cloudflared tunnel login
|
||||
|
||||
# Re-create tunnel (if needed)
|
||||
cloudflared tunnel create tunnel-ml110
|
||||
|
||||
# Start service
|
||||
systemctl start cloudflared-ml110
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
If issues persist:
|
||||
|
||||
1. **Collect diagnostic information:**
|
||||
```bash
|
||||
./scripts/check-tunnel-health.sh > /tmp/tunnel-health.txt
|
||||
journalctl -u cloudflared-* > /tmp/tunnel-logs.txt
|
||||
```
|
||||
|
||||
2. **Check Cloudflare dashboard:**
|
||||
- Zero Trust → Networks → Tunnels → Status
|
||||
- Access → Logs → Recent authentication attempts
|
||||
|
||||
3. **Review documentation:**
|
||||
- [Cloudflare Tunnel Docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/)
|
||||
- [Cloudflare Access Docs](https://developers.cloudflare.com/cloudflare-one/policies/access/)
|
||||
|
||||
4. **Common issues:**
|
||||
- Invalid tunnel token
|
||||
- Network connectivity issues
|
||||
- Incorrect configuration
|
||||
- DNS propagation delays
|
||||
|
||||
## Prevention
|
||||
|
||||
To avoid issues:
|
||||
|
||||
1. ✅ **Regular health checks** - Run `check-tunnel-health.sh` daily
|
||||
2. ✅ **Monitor logs** - Set up log monitoring
|
||||
3. ✅ **Backup configs** - Keep backups of config files
|
||||
4. ✅ **Test after changes** - Test after any configuration changes
|
||||
5. ✅ **Document changes** - Keep track of what was changed
|
||||
|
||||
Reference in New Issue
Block a user