proxmox/docs/00-meta/OPERATOR_AND_EXTERNAL_COMPLETION_CHECKLIST.md

# Operator and External Completion Checklist

**Purpose:** Items from [ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md](ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md) that **cannot** be completed from repo-only (e.g. from this workspace). They require **LAN/Proxmox access**, **operator credentials**, **external services**, or **ongoing maintenance**.

Use this checklist when you have operator or LAN access to complete the remaining recommendations.

**Full plan (Wave 0–3, required/optional/recommended):** [COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md](COMPLETE_REQUIRED_OPTIONAL_RECOMMENDED_INDEX.md).

---

## 1. Proxmox / security (LAN or host access)

| # | Action | Command or notes |
|---|--------|------------------|
| 1 | Secure .env permissions | `chmod 600 .env` (and subproject .env files); run from project root or use `scripts/security/secure-env-permissions.sh` if present |
| 2 | Secure validator key permissions | On host: `chmod 600` keys, `chown -R besu:besu` validator dirs |
| 3 | SSH key-only auth | On Proxmox/containers: set `PasswordAuthentication no`, `PubkeyAuthentication yes` in sshd_config |
| 4 | Firewall for Proxmox API (8006) | Restrict to specific IPs (e.g. iptables or UDM Pro rules) |
| 5 | Network segmentation (VLANs) | Per NETWORK_ARCHITECTURE; configure switches, Proxmox bridges, ER605 |
| 6 | Metrics (Prometheus, Besu 9545) | Deploy/configure Prometheus scrape; enable Besu metrics port |
| 7 | Health check + alerting | Configure Alertmanager / PagerDuty / Slack per MASTER_SECRETS |
| 8 | Automated backup + encrypted validator keys | Run backup script; store encrypted copies off-host |
| 9 | Backup configs + version control | Commit configs to repo or backup store from LAN |

---

## 2. Deployment and runbooks (operator)

| # | Action | When |
|---|--------|------|
| **Priority** | **Mainnet liquidity + ramps** | Fund mainnet LP and relay bridge; wire ramp API; document user flows. See [MAINNET_LIQUIDITY_AND_RAMPS_PRIORITY.md](MAINNET_LIQUIDITY_AND_RAMPS_PRIORITY.md) and [MAINNET_RAMP_USER_FLOWS.md](../04-configuration/MAINNET_RAMP_USER_FLOWS.md). |
| 10 | Integration tests for deployment scripts | Run from LAN against staging/dev |
| 11 | Runbooks: add/remove validator, upgrade Besu, key rotation, recovery, consensus | Document and run from LAN when needed |
| 12–20 | Error handling, logging, Loki, resource/network/DB tuning, CI/CD, CLI tool | Implement on infra from operator environment |

---

## 3. Infrastructure and DNS (operator / when services go live)

| # | Action | Notes |
|---|--------|-------|
| R21 | The Order / Sankofa NPMplus proxy host | When The Order portal deployed: add proxy in NPMplus; document in RPC_ENDPOINTS_MASTER, ALL_VMIDS_ENDPOINTS |
| Sankofa cutover | Replace <TARGET_IP>, <TARGET_PORT>, TBDs in SANKOFA_CUTOVER_PLAN with actual values |
| Blocks #2–#6 | Document in NETWORK_ARCHITECTURE / NETWORK_CONFIGURATION_MASTER when assigned or mark reserved |
| 75–81 | VLAN enablement, observability stack, CCIP fleet, sovereign tenants, missing containers | Per NEXT_STEPS_MASTER and deployment phases |

---

## 4. Contracts and verification (operator / after deploy)

| # | Action | Command |
|---|--------|--------|
| R1 | Verify every deployed contract on Blockscout | After each deploy when Blockscout reachable |
| R2 | Keep CONTRACT_ADDRESSES_REFERENCE and [ADDRESS_MATRIX_AND_STATUS](../11-references/ADDRESS_MATRIX_AND_STATUS.md) updated | When new contracts deployed or deprecated |
| R3 | Run on-chain check | `./scripts/verify/check-contracts-on-chain-138.sh` (set RPC_URL_138 from LAN) |
| R8 | Set RPC_URL_138; run from LAN if 192.168.11.x not reachable | Before any deploy |
| R9 | Use GAS_PRICE=1000000000 (or current min) on Chain 138 | Every forge script on 138 |
| R14 | Run verification after deploy in CI | When Blockscout reachable from runner |

---

## 5. Secrets and access (operator / security)

| # | Action | Notes |
|---|--------|-------|
| R4 | Do not use deprecated CCIPWETH9Bridge; use 0x971c... and set env | Always |
| R5 | Never commit .env or private keys; rotate exposed keys | Always |
| R7 | Restrict deployer key and RPC admin access | Access review |
| 48–52 | Secret audit, input validation, security scanning, RBAC, config validation | Run from secure env; use git-secrets, gitleaks, bandit, trivy as applicable |

---

## 6. External services and submissions

| # | Action | Where |
|---|--------|-------|
| 98 | CoinGecko submission (Chain 138) | [CoinGecko](https://www.coingecko.com/) |
| 99 | Consensys outreach (Swaps/Bridge support) | MetaMask/Consensys channels |
| 106–108 | Verify Etherlink/Jumper/LiFi for chains 138, 651940, 42793 | External APIs / docs |
| 109–121 | Tezos/Etherlink contracts and relay services | Deploy and run from operator env |

---

## 7. Maintenance (ongoing)

| # | Task | Frequency |
|---|------|-----------|
| 135 | Monitor explorer sync status | Daily |
| 136 | Monitor RPC node health (e.g. VMID 2201) | Daily |
| 137 | Check config API uptime | Weekly |
| 138 | Review explorer logs | Weekly |
| 139 | Update token list | As needed |
| R17 | Monitor critical bridge/oracle events | Ongoing |
| R18 | Ensure Blockscout (VMID 5000) up and /api reachable | Health checks |

---

## 8. Optional tools (install for full automation)

Install for scripts that recommend them:

- **shellcheck** — `apt install shellcheck` or `brew install shellcheck`. With `--optional`, run reports findings but does not fail (CI and validate pass).
- **jq** — Required on Proxmox host for genesis validation; `run-via-proxmox-ssh.sh validate` installs it when missing.
- **wscat** — `npm i -g wscat` or use `npx -y wscat` for WebSocket RPC tests
- **sqlite3, websocat, sshpass, dig, parallel** — per verification evidence NEXT_STEPS_RUN_*.md

---

## What was completed in-repo (no operator needed)

- **Validate via Proxmox SSH (2026-02-23):** `run-via-proxmox-ssh.sh validate` runs full shellcheck on `scripts/verify/*.sh` and genesis validation (smom-dbis-138); installs shellcheck and jq on the host when missing. Shellcheck with `--optional` is non-fatal so CI and validate complete successfully.
- **Mainnet liquidity & ramps priority:** [MAINNET_LIQUIDITY_AND_RAMPS_PRIORITY.md](MAINNET_LIQUIDITY_AND_RAMPS_PRIORITY.md) and [MAINNET_RAMP_USER_FLOWS.md](../04-configuration/MAINNET_RAMP_USER_FLOWS.md) — priority plan (fund LP + relay bridge, wire ramps) and user flows for on-ramp/off-ramp; linked from ALL_NEXT_STEPS_CURRENT, NEXT_STEPS_INDEX, METAMASK_BRIDGE_SWAP, and this checklist.
- **smom-dbis-138 deployment (2026-02-23):** Chain 138 verification, fund CCIP/mainnet LP dry-runs, optional G1 PMM and G2 Trustless on L2s, run-all-four-gaps and CCIPLogger documented. See [COMPLETION_RUN_20260223_ALL_STEPS.md](../../smom-dbis-138/docs/deployment/COMPLETION_RUN_20260223_ALL_STEPS.md) and [ALL_NEXT_STEPS_CURRENT.md](../../smom-dbis-138/docs/deployment/ALL_NEXT_STEPS_CURRENT.md).
- **Shebang:** Active (non-archive) scripts under `scripts/` updated from `#!/bin/bash` to `#!/usr/bin/env bash`.
- **Script header template:** [10-best-practices/SCRIPT_HEADER_TEMPLATE.md](../10-best-practices/SCRIPT_HEADER_TEMPLATE.md) added.
- **Config validation in CI:** `.github/workflows/validate-config.yml` already runs config validation and completable-tasks dry-run; **shellcheck** added as optional step.
- **Documentation:** Quick reference cards, troubleshooting FAQ, and decision tree already exist and are linked from ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md.
- **Operator checklist:** This file; R1–R24 and remaining items are in [RECOMMENDATIONS_OPERATOR_CHECKLIST.md](RECOMMENDATIONS_OPERATOR_CHECKLIST.md) and [ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md](ALL_RECOMMENDATIONS_AND_IMPROVEMENTS_LIST.md).

---

**Related:** [NEXT_STEPS_OPERATOR.md](NEXT_STEPS_OPERATOR.md), [NEXT_STEPS_AND_REMAINING_TODOS.md](NEXT_STEPS_AND_REMAINING_TODOS.md), [OPERATOR_READY_CHECKLIST.md](../04-configuration/OPERATOR_READY_CHECKLIST.md).