29 lines
1.7 KiB
Markdown
29 lines
1.7 KiB
Markdown
|
# Maintenance Scripts
|
||
|
|
|
||
|
|
**daily-weekly-checks.sh** — Daily (explorer, indexer lag, RPC) and weekly (config API, thin pool, log reminder).
|
||
|
|
**schedule-daily-weekly-cron.sh** — Install cron: daily 08:00, weekly Sun 09:00.
|
||
|
|
|
||
|
|
**check-and-fix-explorer-lag.sh** — Checks RPC vs Blockscout block; if lag > threshold (default 500), runs `fix-explorer-indexer-lag.sh` (restart Blockscout).
|
||
|
|
**schedule-explorer-lag-cron.sh** — Install cron for lag check-and-fix: every 6 hours (0, 6, 12, 18). Log: `logs/explorer-lag-fix.log`. Use `--show` to print the line, `--install` to add to crontab, `--remove` to remove.
|
||
|
|
|
||
|
|
## Optional: Alerting on failures
|
||
|
|
|
||
|
|
The daily/weekly script writes a **metric file** when run (if `MAINTENANCE_METRIC_FILE` is set or default `logs/maintenance-checks.metric`):
|
||
|
|
|
||
|
|
```
|
||
|
|
maintenance_checks_failed 0
|
||
|
|
maintenance_checks_timestamp 1739123456
|
||
|
|
```
|
||
|
|
|
||
|
|
- **Use in cron:** After the check, if `maintenance_checks_failed` > 0, send alert.
|
||
|
|
- **Example wrapper (email on failure):**
|
||
|
|
```bash
|
||
|
|
cd /path/to/proxmox && bash scripts/maintenance/daily-weekly-checks.sh daily >> logs/daily-weekly-checks.log 2>&1
|
||
|
|
FAILED=$(grep '^maintenance_checks_failed' logs/maintenance-checks.metric 2>/dev/null | awk '{print $2}')
|
||
|
|
[ -n "$FAILED" ] && [ "$FAILED" -gt 0 ] && echo "Maintenance checks failed: $FAILED" | mail -s "Explorer/maintenance alert" ops@example.com
|
||
|
|
```
|
||
|
|
- **Slack:** Use a small script that reads the metric file and posts to a webhook when `maintenance_checks_failed` > 0.
|
||
|
|
- **Prometheus/Grafana:** Scrape the metric file or run a node_exporter textfile collector on `logs/maintenance-checks.metric`.
|
||
|
|
|
||
|
|
To disable the metric file, set `MAINTENANCE_METRIC_FILE=` (empty) before running the script.
|