proxmox/docs/04-configuration/NPMPLUS_BACKUP_RESTORE.md

# NPMplus Backup and Restore Guide

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Date**: 2026-01-20  
**Status**: Complete Backup/Restore Procedures  
**Purpose**: Backup and restore procedures for NPMplus configuration

---

## Overview

This guide provides backup and restore procedures for NPMplus (VMID 10233) configuration, including:

- NPMplus database export (SQLite)
- Proxy host configuration export (JSON via API)
- Certificate export (file copy)
- Docker volume backup

---

## NPMplus Container Information

| Property | Value |
|----------|-------|
| **VMID** | 10233 |
| **Host** | r630-01 (192.168.11.11) |
| **Internal IP (eth0)** | 192.168.11.166 |
| **Internal IP (eth1)** | 192.168.11.167 |
| **Management UI** | `https://192.168.11.166:81` |
| **Database Location** | `/data/database.sqlite` (inside container) |
| **Certificate Location** | `/data/tls/certbot/live/` (inside container) |
| **Docker Container** | `npmplus` |

---

## Backup Procedures

### Automated Backup Script

**Script Location**: `scripts/verify/backup-npmplus.sh` ✅ **CREATED**

**Manual Backup Steps**:

#### 1. NPMplus Database Export (SQLite)

**Method 1: Direct SQLite Export**:

```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11
BACKUP_DIR="/tmp/npmplus-backup-$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Export database
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c '
    cd /app
    if [ -f /data/database.sqlite ]; then
        sqlite3 /data/database.sqlite \".dump\" > /tmp/npm-database.sql 2>/dev/null || echo \"Database export may have issues\"
    fi
'"

# Copy database export
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /tmp/npm-database.sql" > "$BACKUP_DIR/database.sql"
```

**Method 2: Copy Database File**:

```bash
# Copy entire database file
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /data/database.sqlite" > "$BACKUP_DIR/database.sqlite"
```

#### 2. Proxy Host Configuration Export (JSON via API)

**Using Export Script**:
```bash
bash scripts/verify/export-npmplus-config.sh
```

**Manual Export via API**:

```bash
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file
BACKUP_DIR="/tmp/npmplus-backup-$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Authenticate
TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Export proxy hosts
curl -s -k -X GET "$NPM_URL/api/nginx/proxy-hosts" \
    -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/proxy_hosts.json"

# Export certificates
curl -s -k -X GET "$NPM_URL/api/nginx/certificates" \
    -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/certificates.json"
```

#### 3. Certificate Export (File Copy)

```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11
BACKUP_DIR="/tmp/npmplus-backup-$(date +%Y%m%d_%H%M%S)/certs"
mkdir -p "$BACKUP_DIR"

# List all certificates
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- ls -1 /data/tls/certbot/live/" > "$BACKUP_DIR/cert_list.txt"

# Copy all certificate directories
while IFS= read -r cert_dir; do
    if [ -n "$cert_dir" ] && [ "$cert_dir" != "lost+found" ]; then
        mkdir -p "$BACKUP_DIR/$cert_dir"
        
        # Copy fullchain.pem
        ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /data/tls/certbot/live/$cert_dir/fullchain.pem" > "$BACKUP_DIR/$cert_dir/fullchain.pem"
        
        # Copy privkey.pem
        ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /data/tls/certbot/live/$cert_dir/privkey.pem" > "$BACKUP_DIR/$cert_dir/privkey.pem"
    fi
done < "$BACKUP_DIR/cert_list.txt"
```

#### 4. Docker Volume Backup

```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11
BACKUP_DIR="/tmp/npmplus-backup-$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Get Docker volume name
VOLUME_NAME=$(ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- docker inspect npmplus --format '{{ range .Mounts }}{{ if eq .Destination \"/data\" }}{{ .Name }}{{ end }}{{ end }}'" 2>/dev/null || echo "")

# Create volume backup (if using Docker volumes)
if [ -n "$VOLUME_NAME" ]; then
    ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- docker run --rm -v $VOLUME_NAME:/data -v $(pwd)/$BACKUP_DIR:/backup alpine tar czf /backup/npmplus-data.tar.gz -C /data ."
fi
```

---

## Complete Backup Script

**Create**: `scripts/verify/backup-npmplus.sh`

```bash
#!/usr/bin/env bash
# Complete NPMplus backup script
# Backs up database, proxy hosts, certificates, and Docker volumes

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"

# Source .env
if [ -f "$PROJECT_ROOT/.env" ]; then
    set +euo pipefail
    source "$PROJECT_ROOT/.env" 2>/dev/null || true
    set -euo pipefail
fi

NPMPLUS_VMID="${NPMPLUS_VMID:-10233}"
NPMPLUS_HOST="${NPMPLUS_HOST:-192.168.11.11}"
NPM_URL="${NPM_URL:-https://192.168.11.166:81}"
NPM_EMAIL="${NPM_EMAIL:-nsatoshi2007@hotmail.com}"
NPM_PASSWORD="${NPM_PASSWORD:-}"

BACKUP_DIR="/tmp/npmplus-backup-$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "📦 NPMplus Complete Backup"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""
echo "Backup directory: $BACKUP_DIR"
echo ""

# 1. Database backup
echo "1. Backing up database..."
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c '
    if [ -f /data/database.sqlite ]; then
        sqlite3 /data/database.sqlite \".dump\" > /tmp/npm-database.sql 2>/dev/null || echo \"Database export may have issues\"
        cp /data/database.sqlite /tmp/database.sqlite 2>/dev/null || true
    fi
'"
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /tmp/npm-database.sql" > "$BACKUP_DIR/database.sql" 2>/dev/null || true
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /tmp/database.sqlite" > "$BACKUP_DIR/database.sqlite" 2>/dev/null || true
echo "✅ Database backup complete"
echo ""

# 2. Proxy hosts and certificates via API
if [ -n "$NPM_PASSWORD" ]; then
    echo "2. Backing up proxy hosts and certificates via API..."
    
    TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
        -H "Content-Type: application/json" \
        -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")
    
    TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token // empty' 2>/dev/null || echo "")
    
    if [ -n "$TOKEN" ] && [ "$TOKEN" != "null" ]; then
        curl -s -k -X GET "$NPM_URL/api/nginx/proxy-hosts" \
            -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/proxy_hosts.json"
        
        curl -s -k -X GET "$NPM_URL/api/nginx/certificates" \
            -H "Authorization: Bearer $TOKEN" | jq '.' > "$BACKUP_DIR/certificates.json"
        
        echo "✅ API backup complete"
    else
        echo "⚠️  API backup failed (authentication error)"
    fi
else
    echo "⚠️  Skipping API backup (NPM_PASSWORD not set)"
fi
echo ""

# 3. Certificate files
echo "3. Backing up certificate files..."
CERT_DIR="$BACKUP_DIR/certs"
mkdir -p "$CERT_DIR"

CERT_LIST=$(ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- ls -1 /data/tls/certbot/live/ 2>/dev/null" | grep -v "lost+found" || echo "")

if [ -n "$CERT_LIST" ]; then
    while IFS= read -r cert_dir; do
        if [ -n "$cert_dir" ]; then
            mkdir -p "$CERT_DIR/$cert_dir"
            
            ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /data/tls/certbot/live/$cert_dir/fullchain.pem" > "$CERT_DIR/$cert_dir/fullchain.pem" 2>/dev/null || true
            ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- cat /data/tls/certbot/live/$cert_dir/privkey.pem" > "$CERT_DIR/$cert_dir/privkey.pem" 2>/dev/null || true
        fi
    done <<< "$CERT_LIST"
    echo "✅ Certificate backup complete"
else
    echo "⚠️  No certificates found"
fi
echo ""

# 4. Create backup archive
echo "4. Creating backup archive..."
cd "$(dirname "$BACKUP_DIR")"
tar czf "$(basename "$BACKUP_DIR").tar.gz" "$(basename "$BACKUP_DIR")" 2>/dev/null || true
BACKUP_ARCHIVE="$(dirname "$BACKUP_DIR")/$(basename "$BACKUP_DIR").tar.gz"

if [ -f "$BACKUP_ARCHIVE" ]; then
    echo "✅ Backup archive created: $BACKUP_ARCHIVE"
    echo ""
    echo "Backup complete!"
    echo "Directory: $BACKUP_DIR"
    echo "Archive: $BACKUP_ARCHIVE"
else
    echo "⚠️  Archive creation failed, backup directory available: $BACKUP_DIR"
fi
```

---

## Restore Procedures

### 1. Database Restore

**Restore SQLite Database**:

```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11
BACKUP_DIR="/tmp/npmplus-backup-20260120_120000"

# Stop NPMplus (if running)
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- docker stop npmplus"

# Restore database from SQL dump
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c '
    cd /app
    if [ -f /data/database.sqlite ]; then
        mv /data/database.sqlite /data/database.sqlite.bak
    fi
'"

cat "$BACKUP_DIR/database.sql" | ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c 'sqlite3 /data/database.sqlite'"

# OR restore from file copy
# ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c 'cp /tmp/database.sqlite /data/database.sqlite'"
# cat "$BACKUP_DIR/database.sqlite" | ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c 'cat > /data/database.sqlite'"

# Restart NPMplus
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- docker start npmplus"
```

### 2. Configuration Restore (via API)

**Restore Proxy Hosts**:

```bash
NPM_URL="https://192.168.11.166:81"
NPM_EMAIL="nsatoshi2007@hotmail.com"
# Note: Use .env file for credentials in production
# NPM_PASSWORD="your-password"  # Set in .env file
BACKUP_DIR="/tmp/npmplus-backup-20260120_120000"

# Authenticate
TOKEN_RESPONSE=$(curl -s -k -X POST "$NPM_URL/api/tokens" \
    -H "Content-Type: application/json" \
    -d "{\"identity\":\"$NPM_EMAIL\",\"secret\":\"$NPM_PASSWORD\"}")

TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.token')

# Create proxy hosts from backup
cat "$BACKUP_DIR/proxy_hosts.json" | jq -c '.[]' | while read -r proxy_host; do
    curl -s -k -X POST "$NPM_URL/api/nginx/proxy-hosts" \
        -H "Authorization: Bearer $TOKEN" \
        -H "Content-Type: application/json" \
        -d "$proxy_host"
done

# Request certificates (they will auto-renew)
cat "$BACKUP_DIR/certificates.json" | jq -c '.[]' | while read -r cert; do
    domains=$(echo "$cert" | jq -r '.domain_names | join(",")')
    # Request new certificate via API or UI
done
```

### 3. Certificate Restore

```bash
NPMPLUS_VMID=10233
NPMPLUS_HOST=192.168.11.11
BACKUP_DIR="/tmp/npmplus-backup-20260120_120000"

# Stop NPMplus
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- docker stop npmplus"

# Restore certificates
CERT_DIR="$BACKUP_DIR/certs"
if [ -d "$CERT_DIR" ]; then
    for cert_dir in "$CERT_DIR"/*; do
        if [ -d "$cert_dir" ]; then
            cert_name=$(basename "$cert_dir")
            mkdir -p /tmp/cert_restore
            cp "$cert_dir/fullchain.pem" /tmp/cert_restore/fullchain.pem
            cp "$cert_dir/privkey.pem" /tmp/cert_restore/privkey.pem
            
            ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- mkdir -p /data/tls/certbot/live/$cert_name"
            cat /tmp/cert_restore/fullchain.pem | ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c 'cat > /data/tls/certbot/live/$cert_name/fullchain.pem'"
            cat /tmp/cert_restore/privkey.pem | ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- bash -c 'cat > /data/tls/certbot/live/$cert_name/privkey.pem'"
        fi
    done
fi

# Restart NPMplus
ssh root@"$NPMPLUS_HOST" "pct exec $NPMPLUS_VMID -- docker start npmplus"
```

---

## Disaster Recovery Scenarios

### Scenario 1: NPMplus Container Lost (VMID Recreated)

**Restore Steps**:

1. **Recreate Container**:
   - Install NPMplus on new container (VMID)
   - Get new IP address
   - Update UDM Pro port forwarding (if IP changed)

2. **Restore Database**:
   - Copy database backup to new container
   - Restore as described above

3. **Restore Certificates**:
   - Copy certificate files to new container
   - Restore as described above

4. **Verify Configuration**:
   - Run verification scripts
   - Test all domains

### Scenario 2: NPMplus Database Corrupted

**Restore Steps**:

1. **Stop NPMplus**:
```bash
ssh root@192.168.11.11 "pct exec 10233 -- docker stop npmplus"
```

2. **Backup Current Database**:
```bash
ssh root@192.168.11.11 "pct exec 10233 -- cp /data/database.sqlite /data/database.sqlite.corrupted"
```

3. **Restore from Backup**:
   - Restore database as described above

4. **Verify and Test**:
   - Start NPMplus
   - Verify all proxy hosts exist
   - Test domains

### Scenario 3: Certificate Files Lost

**Restore Steps**:

1. **If Backup Available**:
   - Restore certificate files as described above

2. **If No Backup Available**:
   - Re-request certificates via NPMplus UI or API
   - Certificates will auto-renew from Let's Encrypt

---

## Backup Schedule Recommendations

### Recommended Backup Schedule

| Backup Type | Frequency | Retention | Location |
|-------------|-----------|-----------|----------|
| Complete Backup | Weekly | 4 weeks | `/tmp/npmplus-backup-*` |
| Database Export | Daily | 7 days | Backup server or cloud storage |
| Configuration Export (API) | After each change | 30 days | Version control or config management |
| Certificate Export | Monthly | 90 days | Secure storage (encrypted) |

### Automated Backup Script

**Create cron job for daily backups**:

```bash
# Add to crontab (crontab -e)
0 2 * * * /home/intlc/projects/proxmox/scripts/verify/backup-npmplus.sh >> /var/log/npmplus-backup.log 2>&1
```

### Backup Retention Policy

- **Daily backups**: Keep 7 days
- **Weekly backups**: Keep 4 weeks
- **Monthly backups**: Keep 3 months
- **Before major changes**: Keep indefinitely

---

## Verification After Restore

After restore, verify:

1. **NPMplus Container Status**:
```bash
ssh root@192.168.11.11 "pct exec 10233 -- docker ps | grep npmplus"
```

2. **Proxy Hosts**:
```bash
bash scripts/verify/export-npmplus-config.sh
```

3. **Certificates**:
```bash
# Check certificate files exist
ssh root@192.168.11.11 "pct exec 10233 -- ls -la /data/tls/certbot/live/"
```

4. **End-to-End Tests**:
```bash
bash scripts/verify/verify-end-to-end-routing.sh
```

---

## Related Documentation

- **Verification Runbook**: `docs/04-configuration/INGRESS_VERIFICATION_RUNBOOK.md`
- **NPMplus Setup**: `docs/04-configuration/NPMPLUS_COMPLETE_SETUP_SUMMARY.md`
- **Backup Scripts**: `scripts/verify/backup-npmplus.sh` (to be created)

---

**Last Updated**: 2026-01-20  
**Maintained By**: Infrastructure Team  
**Status**: Complete Backup/Restore Procedures