explorer-monorepo/deployment/DEPLOYMENT_GUIDE.md

# Complete Deployment Guide - LXC + Nginx + Cloudflare

This guide provides step-by-step instructions for deploying the ChainID 138 Explorer Platform using LXC containers, Nginx reverse proxy, Cloudflare DNS, SSL, and Cloudflare Tunnel.

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [LXC Container Setup](#lxc-container-setup)
3. [Application Installation](#application-installation)
4. [Database Setup](#database-setup)
5. [Nginx Configuration](#nginx-configuration)
6. [Cloudflare DNS Configuration](#cloudflare-dns-configuration)
7. [SSL Certificate Setup](#ssl-certificate-setup)
8. [Cloudflare Tunnel Setup](#cloudflare-tunnel-setup)
9. [Security Hardening](#security-hardening)
10. [Monitoring & Maintenance](#monitoring--maintenance)

---

## Prerequisites

### Required Tools
- ✅ Proxmox VE with LXC support
- ✅ Cloudflare account with domain
- ✅ SSH access to Proxmox host
- ✅ Cloudflare API token (with DNS edit permissions)

### Domain Requirements
- ✅ Domain registered with Cloudflare
- ✅ DNS managed by Cloudflare

### System Requirements
- **LXC Container**: Ubuntu 22.04 LTS (recommended)
- **Resources**: 
  - CPU: 4+ cores
  - RAM: 8GB minimum (16GB recommended)
  - Storage: 100GB+ SSD
  - Network: Public IP or Cloudflare Tunnel

---

## Task List Overview

### Phase 1: LXC Container Setup
- [ ] Create LXC container
- [ ] Configure container resources
- [ ] Set up networking
- [ ] Install base packages
- [ ] Configure firewall

### Phase 2: Application Installation
- [ ] Install Go 1.21+
- [ ] Install Node.js 20+
- [ ] Install Docker & Docker Compose
- [ ] Clone repository
- [ ] Install dependencies
- [ ] Build applications

### Phase 3: Database Setup
- [ ] Install PostgreSQL with TimescaleDB
- [ ] Create database and user
- [ ] Run migrations
- [ ] Configure backups

### Phase 4: Infrastructure Services
- [ ] Deploy Elasticsearch/OpenSearch
- [ ] Deploy Redis
- [ ] Configure message queue (optional)

### Phase 5: Application Services
- [ ] Configure environment variables
- [ ] Set up systemd services
- [ ] Start indexer service
- [ ] Start API service
- [ ] Start frontend service

### Phase 6: Nginx Reverse Proxy
- [ ] Install Nginx
- [ ] Configure SSL certificates
- [ ] Set up reverse proxy config
- [ ] Configure rate limiting
- [ ] Set up caching
- [ ] Enable security headers

### Phase 7: Cloudflare Configuration
- [ ] Set up Cloudflare DNS records
- [ ] Configure Cloudflare Tunnel
- [ ] Set up SSL/TLS mode
- [ ] Configure WAF rules
- [ ] Set up DDoS protection
- [ ] Configure caching rules

### Phase 8: Security Hardening
- [ ] Configure firewall rules
- [ ] Set up fail2ban
- [ ] Configure automatic updates
- [ ] Set up log rotation
- [ ] Configure backup strategy

### Phase 9: Monitoring
- [ ] Set up health checks
- [ ] Configure log aggregation
- [ ] Set up alerting
- [ ] Configure uptime monitoring

---

## Phase 1: LXC Container Setup

### Task 1.1: Create LXC Container

```bash
# On Proxmox host
pct create 100 \
  local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
  --hostname explorer-prod \
  --memory 16384 \
  --cores 4 \
  --swap 4096 \
  --storage local-lvm \
  --rootfs local-lvm:100 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 0 \
  --features nesting=1
```

**Parameters:**
- Container ID: 100 (change as needed)
- Template: Ubuntu 22.04
- Memory: 16GB
- CPU Cores: 4
- Storage: 100GB on local-lvm
- Network: DHCP on vmbr0
- Features: Enable nesting for Docker

### Task 1.2: Start Container

```bash
pct start 100
pct enter 100
```

### Task 1.3: Initial Container Configuration

```bash
# Update system
apt update && apt upgrade -y

# Install essential packages
apt install -y curl wget git vim net-tools ufw fail2ban \
  unattended-upgrades apt-transport-https ca-certificates \
  gnupg lsb-release

# Set timezone
timedatectl set-timezone UTC

# Configure hostname
hostnamectl set-hostname explorer-prod
```

### Task 1.4: Create Deployment User

```bash
# Create deployment user
adduser explorer
usermod -aG sudo explorer
usermod -aG docker explorer

# Configure SSH (disable root login)
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
systemctl restart sshd
```

---

## Phase 2: Application Installation

### Task 2.1: Install Go 1.21+

```bash
# Download Go
cd /tmp
wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz

# Install Go
rm -rf /usr/local/go
tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz

# Add to PATH
echo 'export PATH=$PATH:/usr/local/go/bin' >> /etc/profile
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc

# Verify
go version
```

### Task 2.2: Install Node.js 20+

```bash
# Install Node.js via NodeSource
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt install -y nodejs

# Verify
node --version
npm --version
```

### Task 2.3: Install Docker & Docker Compose

```bash
# Add Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Start Docker
systemctl enable docker
systemctl start docker

# Verify
docker --version
docker compose version
```

### Task 2.4: Clone Repository

```bash
# Switch to deployment user
su - explorer

# Clone repository
cd /home/explorer
git clone <repository-url> explorer-monorepo
cd explorer-monorepo
```

### Task 2.5: Install Dependencies

```bash
# Backend dependencies
cd backend
go mod download

# Frontend dependencies
cd ../frontend
npm ci --production
```

### Task 2.6: Build Applications

```bash
# Build backend
cd /home/explorer/explorer-monorepo/backend
go build -o /usr/local/bin/explorer-indexer ./indexer/main.go
go build -o /usr/local/bin/explorer-api ./api/rest/main.go
go build -o /usr/local/bin/explorer-gateway ./api/gateway/main.go
go build -o /usr/local/bin/explorer-search ./api/search/main.go

# Build frontend
cd /home/explorer/explorer-monorepo/frontend
npm run build
```

---

## Phase 3: Database Setup

### Task 3.1: Install PostgreSQL with TimescaleDB

```bash
# Add PostgreSQL repository
sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
apt update

# Install PostgreSQL 16
apt install -y postgresql-16 postgresql-contrib-16

# Add TimescaleDB repository
echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" > /etc/apt/sources.list.d/timescaledb.list
wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | apt-key add -
apt update

# Install TimescaleDB
apt install -y timescaledb-2-postgresql-16

# Tune PostgreSQL for TimescaleDB
timescaledb-tune --quiet --yes

# Restart PostgreSQL
systemctl restart postgresql
```

### Task 3.2: Create Database and User

```bash
# Switch to postgres user
su - postgres

# Create database and user
psql << EOF
CREATE USER explorer WITH PASSWORD 'CHANGE_THIS_PASSWORD';
CREATE DATABASE explorer OWNER explorer;
\c explorer
CREATE EXTENSION IF NOT EXISTS timescaledb;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
GRANT ALL PRIVILEGES ON DATABASE explorer TO explorer;
EOF

exit
```

### Task 3.3: Run Migrations

```bash
cd /home/explorer/explorer-monorepo/backend
go run database/migrations/migrate.go
```

### Task 3.4: Configure PostgreSQL

```bash
# Edit postgresql.conf
vim /etc/postgresql/16/main/postgresql.conf

# Recommended settings:
# max_connections = 100
# shared_buffers = 4GB
# effective_cache_size = 12GB
# maintenance_work_mem = 1GB
# checkpoint_completion_target = 0.9
# wal_buffers = 16MB
# default_statistics_target = 100
# random_page_cost = 1.1
# effective_io_concurrency = 200
# work_mem = 20MB
# min_wal_size = 1GB
# max_wal_size = 4GB

# Edit pg_hba.conf for local connections
vim /etc/postgresql/16/main/pg_hba.conf

# Restart PostgreSQL
systemctl restart postgresql
```

---

## Phase 4: Infrastructure Services

### Task 4.1: Deploy Elasticsearch/OpenSearch

```bash
# Create docker-compose for infrastructure
cd /home/explorer/explorer-monorepo/deployment
docker compose -f docker-compose.yml up -d elasticsearch redis
```

### Task 4.2: Verify Services

```bash
# Check Elasticsearch
curl http://localhost:9200

# Check Redis
redis-cli ping
```

---

## Phase 5: Application Services

### Task 5.1: Create Environment Configuration

```bash
# Create production .env file
cd /home/explorer/explorer-monorepo
cp .env.example .env
vim .env
```

**Required Environment Variables:**
```env
# Database
DB_HOST=localhost
DB_PORT=5432
DB_USER=explorer
DB_PASSWORD=<SECURE_PASSWORD>
DB_NAME=explorer
DB_MAX_CONNECTIONS=50

# RPC
# Public RPC Endpoints (ChainID 138) - Internal IP Addresses
# Using internal IP for direct connection (no proxy overhead)
RPC_URL=http://192.168.11.221:8545
WS_URL=ws://192.168.11.221:8546
CHAIN_ID=138
# Alternative: Private RPC endpoints available at http://192.168.11.211:8545 (internal) or https://rpc-http-prv.d-bis.org (public)

# Search
SEARCH_URL=http://localhost:9200
SEARCH_INDEX_PREFIX=explorer-prod

# API
PORT=8080
API_GATEWAY_PORT=8081
CHAIN_ID=138

# Frontend
NEXT_PUBLIC_API_URL=https://explorer.d-bis.org/api
NEXT_PUBLIC_CHAIN_ID=138
```

### Task 5.2: Create Systemd Service Files

#### Indexer Service

```bash
cat > /etc/systemd/system/explorer-indexer.service << 'EOF'
[Unit]
Description=Explorer Indexer Service
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=explorer
Group=explorer
WorkingDirectory=/home/explorer/explorer-monorepo/backend
EnvironmentFile=/home/explorer/explorer-monorepo/.env
ExecStart=/usr/local/bin/explorer-indexer
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=explorer-indexer

[Install]
WantedBy=multi-user.target
EOF
```

#### API Service

```bash
cat > /etc/systemd/system/explorer-api.service << 'EOF'
[Unit]
Description=Explorer API Service
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=explorer
Group=explorer
WorkingDirectory=/home/explorer/explorer-monorepo/backend
EnvironmentFile=/home/explorer/explorer-monorepo/.env
ExecStart=/usr/local/bin/explorer-api
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=explorer-api

[Install]
WantedBy=multi-user.target
EOF
```

#### Frontend Service

```bash
cat > /etc/systemd/system/solacescanscout-frontend.service << 'EOF'
[Unit]
Description=SolaceScan Next Frontend Service
After=network.target explorer-api.service
Requires=explorer-api.service

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/solacescanscout/frontend/current
Environment=NODE_ENV=production
Environment=HOSTNAME=127.0.0.1
Environment=PORT=3000
ExecStart=/usr/bin/node /opt/solacescanscout/frontend/current/server.js
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=solacescanscout-frontend

[Install]
WantedBy=multi-user.target
EOF
```

### Task 5.3: Enable and Start Services

```bash
# Reload systemd
systemctl daemon-reload

# Enable services
systemctl enable explorer-indexer
systemctl enable explorer-api
systemctl enable solacescanscout-frontend

# Start services
systemctl start explorer-indexer
systemctl start explorer-api
systemctl start solacescanscout-frontend

# Check status
systemctl status explorer-indexer
systemctl status explorer-api
systemctl status solacescanscout-frontend
```

---

## Phase 6: Nginx Reverse Proxy

### Task 6.1: Install Nginx

```bash
apt install -y nginx
```

### Task 6.2: Install Certbot for SSL

```bash
apt install -y certbot python3-certbot-nginx
```

### Task 6.3: Configure Nginx

```bash
cat > /etc/nginx/sites-available/explorer << 'EOF'
# Rate limiting zones
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=general_limit:10m rate=50r/s;

# Upstream servers
upstream explorer_api {
    server 127.0.0.1:8080;
    keepalive 32;
}

upstream explorer_frontend {
    server 127.0.0.1:3000;
    keepalive 32;
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    listen [::]:80;
    server_name explorer.d-bis.org www.explorer.d-bis.org;
    
    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }
    
    location / {
        return 301 https://$server_name$request_uri;
    }
}

# Main HTTPS server
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name explorer.d-bis.org www.explorer.d-bis.org;

    # SSL Configuration (Cloudflare will handle SSL)
    # Certificates managed by Cloudflare Tunnel
    
    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://cdn.jsdelivr.net https://unpkg.com https://cdnjs.cloudflare.com; style-src 'self' 'unsafe-inline';" always;

    # Logging
    access_log /var/log/nginx/explorer-access.log;
    error_log /var/log/nginx/explorer-error.log;

    # Gzip compression
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml+rss application/json application/javascript;

    # Frontend
    location / {
        limit_req zone=general_limit burst=20 nodelay;
        proxy_pass http://explorer_frontend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 300s;
        proxy_connect_timeout 75s;
    }

    # API endpoints
    location /api/ {
        limit_req zone=api_limit burst=20 nodelay;
        proxy_pass http://explorer_api;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s;
        proxy_connect_timeout 75s;
        
        # CORS headers (if needed before Cloudflare)
        add_header Access-Control-Allow-Origin "*" always;
        add_header Access-Control-Allow-Methods "GET, POST, OPTIONS" always;
        add_header Access-Control-Allow-Headers "Content-Type, X-API-Key" always;
    }

    # WebSocket support
    location /ws {
        proxy_pass http://explorer_api;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 86400s;
    }

    # Static files caching
    location ~* \.(jpg|jpeg|png|gif|ico|css|js|svg|woff|woff2|ttf|eot)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
        access_log off;
    }

    # Health check endpoint (internal)
    location /health {
        access_log off;
        proxy_pass http://explorer_api/health;
    }
}
EOF

# Enable site
ln -s /etc/nginx/sites-available/explorer /etc/nginx/sites-enabled/
rm /etc/nginx/sites-enabled/default

# Test configuration
nginx -t

# Reload Nginx
systemctl reload nginx
```

---

## Phase 7: Cloudflare Configuration

### Task 7.1: Set Up Cloudflare DNS Records

1. **Login to Cloudflare Dashboard**
   - Go to https://dash.cloudflare.com
   - Select your domain

2. **Add DNS Records**
   - **A Record** (if using direct connection):
     - Type: A
     - Name: explorer (or @)
     - IPv4: [Your server IP]
     - Proxy: Proxied (orange cloud)
     - TTL: Auto

   - **CNAME Record** (for www):
     - Type: CNAME
     - Name: www
     - Target: explorer.d-bis.org
     - Proxy: Proxied
     - TTL: Auto

### Task 7.2: Configure Cloudflare SSL/TLS

1. **Go to SSL/TLS Settings**
   - Dashboard → SSL/TLS → Overview

2. **Set SSL/TLS Encryption Mode**
   - Select: **Full (strict)**
   - This ensures end-to-end encryption

3. **Configure SSL/TLS Options**
   - Enable: Always Use HTTPS
   - Enable: Automatic HTTPS Rewrites
   - Enable: Opportunistic Encryption
   - Enable: TLS 1.3
   - Enable: Automatic Certificate Management

### Task 7.3: Set Up Cloudflare Tunnel

#### Option A: Cloudflare Tunnel (Recommended for no public IP)

```bash
# Install cloudflared
cd /tmp
wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb
dpkg -i cloudflared-linux-amd64.deb

# Authenticate with Cloudflare
cloudflared tunnel login

# Create tunnel
cloudflared tunnel create explorer-tunnel

# Get tunnel ID
cloudflared tunnel list

# Create config file
mkdir -p /etc/cloudflared
cat > /etc/cloudflared/config.yml << EOF
tunnel: <TUNNEL_ID>
credentials-file: /etc/cloudflared/<TUNNEL_ID>.json

ingress:
  - hostname: explorer.d-bis.org
    service: http://localhost:80
  - hostname: www.explorer.d-bis.org
    service: http://localhost:80
  - service: http_status:404
EOF

# Run tunnel
cloudflared tunnel --config /etc/cloudflared/config.yml run

# Create systemd service
cloudflared service install

# Start tunnel service
systemctl enable cloudflared
systemctl start cloudflared
```

#### Option B: Direct Connection (If public IP available)

1. **Update DNS Records** (from Task 7.1)
   - Set A record to your public IP
   - Ensure proxy is enabled (orange cloud)

2. **Configure Cloudflare Page Rules** (Optional)
   - Cache static assets
   - Bypass cache for API endpoints

### Task 7.4: Configure Cloudflare WAF

1. **Go to Security → WAF**
   - Enable Cloudflare Managed Ruleset
   - Enable OWASP Core Ruleset
   - Configure custom rules as needed

2. **Rate Limiting Rules**
   - Create rule: API rate limiting
   - Action: Challenge or Block
   - Rate: 100 requests per minute per IP

### Task 7.5: Configure Cloudflare Caching

1. **Go to Caching → Configuration**
   - Caching Level: Standard
   - Browser Cache TTL: Respect Existing Headers

2. **Create Cache Rules**
   - **Static Assets**: Cache everything, Edge TTL: 1 year
   - **API Endpoints**: Bypass cache
   - **Frontend Pages**: Cache HTML for 5 minutes

---

## Phase 8: Security Hardening

### Task 8.1: Configure Firewall (UFW)

```bash
# Enable UFW
ufw --force enable

# Allow SSH
ufw allow 22/tcp

# Allow HTTP/HTTPS (if direct connection)
ufw allow 80/tcp
ufw allow 443/tcp

# Allow Cloudflare Tunnel (if using)
ufw allow from 173.245.48.0/20
ufw allow from 103.21.244.0/22
ufw allow from 103.22.200.0/22
ufw allow from 103.31.4.0/22
ufw allow from 141.101.64.0/18
ufw allow from 108.162.192.0/18
ufw allow from 190.93.240.0/20
ufw allow from 188.114.96.0/20
ufw allow from 197.234.240.0/22
ufw allow from 198.41.128.0/17
ufw allow from 162.158.0.0/15
ufw allow from 104.16.0.0/13
ufw allow from 104.24.0.0/14
ufw allow from 172.64.0.0/13
ufw allow from 131.0.72.0/22

# Check status
ufw status verbose
```

### Task 8.2: Configure Fail2ban

```bash
# Create Nginx jail
cat > /etc/fail2ban/jail.d/nginx.conf << 'EOF'
[nginx-limit-req]
enabled = true
port = http,https
logpath = /var/log/nginx/explorer-error.log
maxretry = 10
findtime = 600
bantime = 3600

[nginx-botsearch]
enabled = true
port = http,https
logpath = /var/log/nginx/explorer-access.log
maxretry = 2
findtime = 600
bantime = 86400
EOF

# Restart fail2ban
systemctl restart fail2ban
fail2ban-client status
```

### Task 8.3: Configure Automatic Updates

```bash
# Configure unattended-upgrades
cat > /etc/apt/apt.conf.d/50unattended-upgrades << 'EOF'
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
};
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";
EOF

# Enable automatic updates
systemctl enable unattended-upgrades
systemctl start unattended-upgrades
```

### Task 8.4: Configure Log Rotation

```bash
# Configure logrotate for application logs
cat > /etc/logrotate.d/explorer << 'EOF'
/var/log/explorer/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    missingok
    create 0640 explorer explorer
    sharedscripts
    postrotate
        systemctl reload explorer-indexer explorer-api solacescanscout-frontend > /dev/null 2>&1 || true
    endscript
}
EOF
```

### Task 8.5: Set Up Backup Strategy

```bash
# Create backup script
cat > /usr/local/bin/explorer-backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/backups/explorer"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR

# Backup database
pg_dump -U explorer explorer | gzip > $BACKUP_DIR/db_$DATE.sql.gz

# Backup configuration
tar -czf $BACKUP_DIR/config_$DATE.tar.gz \
    /home/explorer/explorer-monorepo/.env \
    /etc/nginx/sites-available/explorer \
    /etc/systemd/system/explorer-*.service

# Cleanup old backups (keep 30 days)
find $BACKUP_DIR -type f -mtime +30 -delete
EOF

chmod +x /usr/local/bin/explorer-backup.sh

# Add to crontab (daily at 2 AM)
(crontab -l 2>/dev/null; echo "0 2 * * * /usr/local/bin/explorer-backup.sh") | crontab -
```

---

## Phase 9: Monitoring & Maintenance

### Task 9.1: Set Up Health Checks

```bash
# Create health check script
cat > /usr/local/bin/explorer-health-check.sh << 'EOF'
#!/bin/bash
API_URL="http://localhost:8080/health"
STATUS=$(curl -s -o /dev/null -w "%{http_code}" $API_URL)

if [ $STATUS -ne 200 ]; then
    systemctl restart explorer-api
    # Send alert (configure email/Slack/etc)
fi
EOF

chmod +x /usr/local/bin/explorer-health-check.sh

# Add to crontab (every 5 minutes)
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/explorer-health-check.sh") | crontab -
```

### Task 9.2: Configure Log Monitoring

```bash
# Install log monitoring tools
apt install -y logwatch

# Configure logwatch
vim /etc/logwatch/conf/logwatch.conf
```

### Task 9.3: Set Up Cloudflare Analytics

1. **Go to Analytics → Web Traffic**
   - Monitor request rates
   - Track error rates
   - Monitor cache hit ratios

2. **Set Up Alerts**
   - High error rate alerts
   - DDoS detection alerts
   - Certificate expiration alerts

---

## Post-Deployment Checklist

- [ ] All services running and healthy
- [ ] DNS resolving correctly
- [ ] SSL certificates active
- [ ] Cloudflare Tunnel connected (if using)
- [ ] Nginx proxying correctly
- [ ] API endpoints responding
- [ ] Frontend loading correctly
- [ ] Database migrations complete
- [ ] Indexer processing blocks
- [ ] Firewall rules configured
- [ ] Backups configured and tested
- [ ] Monitoring and alerts configured
- [ ] Documentation updated

---

## Troubleshooting

### Service Not Starting
```bash
# Check service status
systemctl status explorer-api
journalctl -u explorer-api -f

# Check logs
journalctl -u explorer-api --since "1 hour ago"
```

### Database Connection Issues
```bash
# Test connection
psql -U explorer -d explorer -h localhost

# Check PostgreSQL logs
tail -f /var/log/postgresql/postgresql-16-main.log
```

### Nginx Issues
```bash
# Test configuration
nginx -t

# Check error logs
tail -f /var/log/nginx/explorer-error.log
```

### Cloudflare Tunnel Issues
```bash
# Check tunnel status
systemctl status cloudflared
cloudflared tunnel info explorer-tunnel

# View tunnel logs
journalctl -u cloudflared -f
```

---

## Maintenance Tasks

### Daily
- Monitor service status
- Check error logs
- Review Cloudflare analytics

### Weekly
- Review security logs
- Check disk space
- Verify backups

### Monthly
- Update system packages
- Review and optimize database
- Update application dependencies
- Review and adjust resource allocation

---

## Security Notes

1. **Never commit .env files** with real credentials
2. **Rotate passwords** regularly
3. **Keep system updated** with security patches
4. **Monitor logs** for suspicious activity
5. **Review Cloudflare WAF** logs regularly
6. **Backup database** daily
7. **Test disaster recovery** procedures quarterly

---

## Support & Resources

- **Cloudflare Docs**: https://developers.cloudflare.com/
- **Nginx Docs**: https://nginx.org/en/docs/
- **Proxmox Docs**: https://pve.proxmox.com/pve-docs/
- **Project Docs**: See `docs/` directory

---

**Last Updated**: 2024-12-23
**Version**: 1.0.0