proxmox/docs/04-configuration/VMID5000_STORAGE_AND_LOGS_RUNBOOK.md

# VMID 5000 (Explorer) — Storage and Logs Runbook

**Last updated:** 2026-02-21  
**Host:** 192.168.11.12 (r630-02)  
**Purpose:** Free disk space, maintain ≥70% free where possible, and keep logs on a separate volume.

---

## 1. Current layout

| Mount        | Size  | Purpose                    | Notes |
|-------------|-------|----------------------------|--------|
| `/` (rootfs)| 196G  | thin5 (LVM thin)           | Root + Docker (Blockscout/Postgres). Docker container logs are **capped** (see §2.3). |
| `/var/log-remote` | 220G (rpool) | Host bind mount `/var/lib/vz/logs-vmid5000` | **Log volume** — nginx and future logs; 193G free |

- **Docker container logs** were the main cause of root filling (Blockscout `*-json.log` grew to ~178G). This is now limited by `daemon.json` (max-size 100m, max-file 3). The maintenance script also truncates any container log over 500M.
- **Log volume** ensures nginx and other service logs grow on `rpool`, not on root.

---

## 2. Free space on root (quick wins)

Run from **Proxmox host** (or via SSH to 192.168.11.12) so `pct exec 5000` is available.

### 2.1 One-shot cleanup (script)

From **proxmox repo** (with SSH to 192.168.11.12):

```bash
# Option A: from repo
PROXMOX_HOST_R630_02=192.168.11.12 ./scripts/maintenance/vmid5000-free-disk-and-logs.sh

# Option B: from explorer-monorepo
EXPLORER_VM_HOST=root@192.168.11.12 bash scripts/free-disk-vmid5000.sh
```

### 2.2 Docker container log limit (prevents root filling)

**Configured on VM 5000:** `/etc/docker/daemon.json`:

```json
{"log-driver": "json-file", "log-opts": {"max-size": "100m", "max-file": "3"}}
```

- Restart Docker after changing: `pct exec 5000 -- systemctl restart docker`
- Existing containers keep the limit after restart. New containers use it by default.

### 2.3 Emergency: root full (e.g. one container log grew before limit)

If root is 100% and nginx/config updates fail, truncate the large container log from the host:

```bash
ssh root@192.168.11.12 "pct exec 5000 -- bash -c '
  for f in /var/lib/docker/containers/*/*-json.log; do
    [ -f \"\$f\" ] && [ \$(stat -c%s \"\$f\" 2>/dev/null) -gt 536870912 ] && truncate -s 0 \"\$f\"
  done
'"
```

(536870912 = 512M; adjust if needed.) Then run **§2.1** and ensure **§2.2** is applied.

### 2.4 Manual steps (if you need to free more)

```bash
# On Proxmox host
HOST=192.168.11.12
VMID=5000
ssh root@$HOST "pct exec $VMID -- bash -c '
  # Journal: keep 1 day or 100M
  journalctl --vacuum-time=1d
  journalctl --vacuum-size=100M

  # Old backups (keep last 2)
  ls -t /var/www/html/index.html.backup.* 2>/dev/null | tail -n +3 | xargs -r rm -f
  ls -t /etc/nginx/sites-available/blockscout.backup.* 2>/dev/null | tail -n +3 | xargs -r rm -f

  # Truncate syslog and force logrotate
  : > /var/log/syslog
  logrotate -f /etc/logrotate.conf

  # Docker (safe: no container/volume prune)
  docker image prune -af
  docker builder prune -af
'"
```

---

## 3. Log volume (`/var/log-remote`)

### 3.1 How it was set up

- **Host path:** `/var/lib/vz/logs-vmid5000` (on rpool; 193G free).
- **In CT 5000:** Bind mount as `/var/log-remote`.
- **CT config** (on host): `mp1: /var/lib/vz/logs-vmid5000,mp=/var/log-remote`

Nginx is configured to log to `/var/log-remote/nginx/`. Logrotate for that directory: `/etc/logrotate.d/nginx-remote` inside the container (daily, rotate 7, compress).

### 3.2 Journal size limit

To prevent journal from refilling root:

- **Config:** `/etc/systemd/journald.conf.d/size-limit.conf` in the container:
  - `SystemMaxUse=100M`
  - `SystemMaxFileSize=20M`
- Restart journald to apply: `systemctl restart systemd-journald` (inside CT).

### 3.3 Add more services to log volume

Point other services to `/var/log-remote/<service>/` and add logrotate entries under `/etc/logrotate.d/` so logs stay on the log volume.

---

## 4. Reaching ~70% free on root (optional, future)

With **Docker log limits** (§2.2), root should stay well below 100%. If you need more headroom:

1. **Expand rootfs:** On the Proxmox host, grow the LV and then inside the CT: `resize2fs /dev/mapper/...` (or the appropriate command for the root device). Requires free space in the thin pool.
2. **Second disk for Docker:** Add a bind mount (e.g. new LV) to the container and move `/var/lib/docker` there (stop Docker, move, adjust fstab or mount, start Docker).
3. **Migrate Blockscout/Postgres** to a dedicated data volume.

This runbook focuses on freeing space and capping Docker logs; expansion is optional.

---

## 5. Checks

**Daily/weekly:** `scripts/maintenance/daily-weekly-checks.sh daily` runs `check-disk-all-vmids.sh`, which reports VMID 5000 root usage (WARN ≥85%, CRIT ≥95%).

```bash
# Disk usage
ssh root@192.168.11.12 "pct exec 5000 -- df -h / /var/log-remote"

# Docker container log sizes (should be small with daemon.json limit)
ssh root@192.168.11.12 "pct exec 5000 -- du -sh /var/lib/docker/containers/*"

# Log volume usage
ssh root@192.168.11.12 "pct exec 5000 -- du -sh /var/log /var/log-remote"

# Journal size
ssh root@192.168.11.12 "pct exec 5000 -- journalctl --disk-usage"
```

---

## 6. Related

- **LOGROTATE_AUDIT_RUNBOOK.md** — logrotate for other VMIDs  
- **STORAGE_GROWTH_AND_HEALTH.md** — growth factors and thresholds  
- **explorer-monorepo/scripts/free-disk-vmid5000.sh** — lightweight Docker prune only  
- **scripts/maintenance/vmid5000-free-disk-and-logs.sh** — full cleanup (journal, backups, logs, Docker container log truncation >500M, image/builder prune)
- **Docker daemon.json** on VM 5000: `log-opts` max-size 100m, max-file 3 (§2.2)