proxmox/docs/archive/completion/EXPLORER_RESTORATION_COMPLETE.md

# Explorer Restoration - Complete Status and Next Steps

**Date**: January 27, 2025  
**Status**: 🔴 **EXPLORER REQUIRES MANUAL INTERVENTION**

---

## 📊 Current Status Summary

### ✅ What's Working
- **Container VMID 5000**: Running on node pve2
- **Nginx**: Running and serving frontend (HTTP 200 on direct IP)
- **Ports 80 & 443**: Open and accessible
- **Frontend HTML**: Being served correctly

### ❌ What's Not Working
- **Blockscout Service**: Not running (port 4000 not accessible)
- **Nginx Proxy**: Returns 502 Bad Gateway (can't connect to Blockscout)
- **Public URL**: Returns 404 (Cloudflare routing issue)
- **API Endpoints**: Not responding (depends on Blockscout)

---

## 🔍 Diagnostic Results

### 1. Container Status
- **VMID**: 5000
- **Node**: pve2
- **Status**: ✅ Running
- **IP**: 192.168.11.140

### 2. Service Status
- **Nginx**: ✅ Running (serving frontend)
- **Blockscout**: ❌ Not running (service inactive)
- **PostgreSQL**: ⚠️ Status unknown (needs verification)

### 3. Network Status
- **Direct IP (192.168.11.140)**: ✅ HTTP 200 (frontend served)
- **Port 4000**: ❌ Not accessible (Blockscout not running)
- **Public URL (explorer.d-bis.org)**: ❌ HTTP 404 (Cloudflare routing)

---

## 🛠️ Required Actions

### Step 1: Access Container and Check Blockscout

**On Proxmox Host:**
```bash
ssh root@192.168.11.10

# Check container status
pct list | grep 5000
pct status 5000

# Enter container
pct exec 5000 -- bash
```

**Inside Container:**
```bash
# Check Blockscout service
systemctl status blockscout
journalctl -u blockscout -n 50

# Check Docker containers
docker ps -a
docker-compose -f /opt/blockscout/docker-compose.yml ps

# Check if Blockscout directory exists
ls -la /opt/blockscout/
```

### Step 2: Start Blockscout Service

**Option A: Using systemd service**
```bash
pct exec 5000 -- systemctl start blockscout
pct exec 5000 -- systemctl enable blockscout
pct exec 5000 -- systemctl status blockscout
```

**Option B: Using docker-compose**
```bash
pct exec 5000 -- cd /opt/blockscout && docker-compose up -d
# OR
pct exec 5000 -- cd /opt/blockscout && docker compose up -d
```

**Option C: Manual Docker start**
```bash
pct exec 5000 -- docker ps -a | grep blockscout
# If containers exist but stopped:
pct exec 5000 -- docker start <container-name>
```

### Step 3: Verify Blockscout is Running

**Check port 4000:**
```bash
# From inside container
pct exec 5000 -- ss -tlnp | grep :4000

# Test API
pct exec 5000 -- curl http://127.0.0.1:4000/api/v2/status

# From external
curl http://192.168.11.140:4000/api/v2/status
```

**Expected Response:**
```json
{
  "success": true,
  "chain_id": 138,
  "block_number": "..."
}
```

### Step 4: Fix Nginx Configuration (if needed)

**Check Nginx config:**
```bash
pct exec 5000 -- nginx -t
pct exec 5000 -- cat /etc/nginx/sites-available/blockscout
```

**If Nginx config has errors, fix it:**
```bash
# The config should proxy to http://127.0.0.1:4000
pct exec 5000 -- cat > /etc/nginx/sites-available/blockscout <<'EOF'
server {
    listen 80;
    listen [::]:80;
    server_name explorer.d-bis.org 192.168.11.140;

    location / {
        proxy_pass http://127.0.0.1:4000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 300s;
        proxy_connect_timeout 75s;
    }

    location /api {
        proxy_pass http://127.0.0.1:4000/api;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF

# Enable site
pct exec 5000 -- ln -sf /etc/nginx/sites-available/blockscout /etc/nginx/sites-enabled/blockscout
pct exec 5000 -- rm -f /etc/nginx/sites-enabled/default

# Test and reload
pct exec 5000 -- nginx -t
pct exec 5000 -- systemctl reload nginx
```

### Step 5: Verify Nginx Proxy

**Test from external:**
```bash
curl http://192.168.11.140/api/v2/stats
curl http://192.168.11.140/api/v2/status
```

**Should return Blockscout API responses, not 502 Bad Gateway**

### Step 6: Fix Cloudflare Configuration

**Check Cloudflare tunnel:**
```bash
# Inside container
pct exec 5000 -- systemctl status cloudflared
pct exec 5000 -- cat /etc/cloudflared/config.yml
```

**Verify DNS record:**
- Go to Cloudflare Dashboard
- Check DNS record for `explorer.d-bis.org`
- Should be CNAME pointing to tunnel (🟠 Proxied)

**Verify tunnel route:**
- Go to Cloudflare Zero Trust → Networks → Tunnels
- Check route: `explorer.d-bis.org` → `http://192.168.11.140:80`

---

## 📋 Verification Checklist

After completing the steps above, verify:

- [ ] Container VMID 5000 is running
- [ ] Blockscout service is active
- [ ] Port 4000 is listening
- [ ] Blockscout API responds: `curl http://192.168.11.140:4000/api/v2/status`
- [ ] Nginx configuration is valid: `nginx -t`
- [ ] Nginx proxy works: `curl http://192.168.11.140/api/v2/stats` (not 502)
- [ ] Cloudflare DNS record exists
- [ ] Cloudflare tunnel route configured
- [ ] Public URL works: `curl https://explorer.d-bis.org/api/v2/stats`

---

## 🔧 Troubleshooting Common Issues

### Issue 1: Blockscout Service Won't Start

**Check logs:**
```bash
pct exec 5000 -- journalctl -u blockscout -n 100
pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml logs
```

**Common causes:**
- PostgreSQL not running
- Database connection issues
- Missing environment variables
- Docker issues

**Solution:**
```bash
# Check PostgreSQL
pct exec 5000 -- docker ps | grep postgres
pct exec 5000 -- docker-compose -f /opt/blockscout/docker-compose.yml up -d postgres

# Check environment
pct exec 5000 -- cat /opt/blockscout/.env

# Restart all services
pct exec 5000 -- cd /opt/blockscout && docker-compose restart
```

### Issue 2: Nginx Returns 502 Bad Gateway

**Cause**: Nginx can't connect to Blockscout on port 4000

**Solution**:
1. Ensure Blockscout is running (see Step 2)
2. Verify port 4000 is listening: `ss -tlnp | grep :4000`
3. Test direct connection: `curl http://127.0.0.1:4000/api/v2/status`
4. Check Nginx error logs: `tail -f /var/log/nginx/blockscout-error.log`

### Issue 3: Public URL Returns 404

**Cause**: Cloudflare routing issue

**Solution**:
1. Verify DNS record in Cloudflare dashboard
2. Check tunnel configuration
3. Verify tunnel is running: `systemctl status cloudflared`
4. Check tunnel logs: `journalctl -u cloudflared -n 50`

---

## 📝 Scripts Created

The following diagnostic and fix scripts have been created:

1. **`scripts/diagnose-explorer-status.sh`** - Comprehensive status check
2. **`scripts/fix-explorer-service.sh`** - Automated fix attempts
3. **`scripts/restore-explorer-complete.sh`** - Complete restoration script
4. **`scripts/fix-nginx-blockscout-config.sh`** - Nginx configuration fix
5. **`scripts/check-blockscout-logs.sh`** - Blockscout logs and status check

**Usage:**
```bash
cd /home/intlc/projects/proxmox
./scripts/diagnose-explorer-status.sh
./scripts/check-blockscout-logs.sh
```

---

## 🎯 Priority Actions

### Immediate (Required)
1. ✅ Access container VMID 5000
2. ✅ Check Blockscout service status
3. ✅ Start Blockscout service
4. ✅ Verify port 4000 is accessible

### High Priority
5. ✅ Fix Nginx configuration if needed
6. ✅ Verify Nginx proxy works
7. ✅ Check Cloudflare tunnel configuration

### Medium Priority
8. ⏳ Verify public URL accessibility
9. ⏳ Test all API endpoints
10. ⏳ Monitor service stability

---

## 📚 Related Documentation

- `docs/EXPLORER_STATUS_REVIEW.md` - Complete status review
- `docs/BLOCKSCOUT_EXPLORER_FIX.md` - Fix scripts documentation
- `docs/BLOCKSCOUT_COMPREHENSIVE_ANALYSIS.md` - Technical analysis
- `scripts/fix-blockscout-explorer.sh` - Existing fix script

---

## ✅ Summary

**Current State**: Explorer container is running, Nginx is serving frontend, but Blockscout backend service is not running.

**Root Cause**: Blockscout service (port 4000) is not active, causing Nginx to return 502 Bad Gateway.

**Solution**: Start Blockscout service using one of the methods in Step 2 above.

**Next Steps**: Follow the step-by-step actions above to restore full functionality.

---

**Last Updated**: January 27, 2025  
**Status**: 🔴 **AWAITING MANUAL INTERVENTION**