proxmox/scripts/README.md

# Scripts Directory

**Last Updated:** 2026-04-06

---

## Overview

This directory contains automation scripts for Proxmox VE management. Scripts have been consolidated into unified frameworks for better maintainability.

**Current Count:** 381 scripts (down from 759 - 50% reduction)

---

## Unified Frameworks

### 1. verify-all.sh
Verification framework consolidating all check/verify/validate scripts.

**Usage:**
```bash
./scripts/verify-all.sh [component] [type] [host]
```

**Examples:**
```bash
./scripts/verify-all.sh all
./scripts/verify-all.sh service status
./scripts/verify-all.sh network connectivity
```

**See:** `docs/00-meta/FRAMEWORK_USAGE_GUIDE.md` for complete documentation.

---

### 2. list.sh
Listing framework consolidating all list/show/get scripts.

**Usage:**
```bash
./scripts/list.sh [type] [filter] [host]
```

**Examples:**
```bash
./scripts/list.sh containers
./scripts/list.sh containers running
./scripts/list.sh vms r630-01
```

---

### 3. fix-all.sh
Fix framework consolidating all fix-*.sh scripts.

**Usage:**
```bash
./scripts/fix-all.sh [issue-type] [component] [host] [--dry-run]
```

**Examples:**
```bash
./scripts/fix-all.sh all
./scripts/fix-all.sh service postgresql 10100
./scripts/fix-all.sh network all --dry-run
```

---

### 4. configure.sh
Configuration framework consolidating all configure/config scripts.

**Usage:**
```bash
./scripts/configure.sh [component] [action] [host]
```

**Examples:**
```bash
./scripts/configure.sh all setup
./scripts/configure.sh network update
./scripts/configure.sh ssl validate
```

---

### 5. deploy.sh
Deployment framework consolidating all deploy/setup/install scripts.

**Usage:**
```bash
./scripts/deploy.sh [component] [options] [host]
```

**Examples:**
```bash
./scripts/deploy.sh all
./scripts/deploy.sh service postgresql
./scripts/deploy.sh all --phase=1
```

---

### 6. CCIP WETH9 Bridge (Chain 138) – router mismatch fix

Deploy and configure a new WETH9 bridge using the **working** CCIP router (fixes router mismatch where the old bridge pointed to an address with no code).

**Usage:**
```bash
# Dry-run (no PRIVATE_KEY): simulate deploy and config
./scripts/deploy-and-configure-weth9-bridge-chain138.sh --dry-run

# Real run (set in smom-dbis-138/.env or export)
# PRIVATE_KEY=0x...   # required
# RPC_URL_138=http://192.168.11.211:8545   # Chain 138 Core (admin/deploy)
./scripts/deploy-and-configure-weth9-bridge-chain138.sh
# Then: export CCIPWETH9_BRIDGE_CHAIN138=<printed address>
```

All bridge scripts use `CCIPWETH9_BRIDGE_CHAIN138` when set; otherwise they fall back to the previous bridge address. See `COMPREHENSIVE_STATUS_BRIDGE_READY.md` and `.env.example` (CCIP section).

---

### 7. Contract Verification (Blockscout)

Verify deployed contracts on Blockscout (Chain 138) using the **Forge Verification Proxy** (required for Forge/Blockscout API compatibility).

**Preferred: orchestrated script (starts proxy if needed):**
```bash
source smom-dbis-138/.env 2>/dev/null
./scripts/verify/run-contract-verification-with-proxy.sh
```

**Manual (proxy + verify):**
```bash
# 1. Start proxy (separate terminal)
BLOCKSCOUT_URL=http://192.168.11.140:4000 node forge-verification-proxy/server.js

# 2. Run verification
./scripts/verify-contracts-blockscout.sh
```

**Env:** `FORGE_VERIFY_TIMEOUT=600` (default; set to `0` for no limit). Uses `scripts/lib/load-project-env.sh` for config.

**See:** `forge-verification-proxy/README.md`, `docs/03-deployment/BLOCKSCOUT_FIX_RUNBOOK.md`

### 8. CCIP WETH9 Bridge — send ETH (WETH) to mainnet

Send WETH cross-chain via CCIP (Chain 138 → Ethereum mainnet or other destination). Uses `PRIVATE_KEY` and `CCIPWETH9_BRIDGE_CHAIN138` from env (load-project-env).

**Send to mainnet (exact command):**
```bash
cd /home/intlc/projects/proxmox
source smom-dbis-138/.env
export CCIP_DEST_CHAIN_SELECTOR=5009297550715157269   # Ethereum mainnet
./scripts/bridge/run-send-cross-chain.sh <amount_eth> [recipient]
# Example: ./scripts/bridge/run-send-cross-chain.sh 0.005
# With recipient:  ./scripts/bridge/run-send-cross-chain.sh 0.005 0xYourMainnetAddress
```

**Dry-run (simulate only):**
```bash
./scripts/bridge/run-send-cross-chain.sh <amount_eth> [recipient] --dry-run
```

Default bridge in `.env` is the **LINK-fee** bridge (pay fee in Chain 138 LINK). To pay fee in **native ETH**, set `CCIPWETH9_BRIDGE_CHAIN138=0x63cbeE010D64ab7F1760ad84482D6cC380435ab5`.

**Requirements:** Sender must have (1) WETH on Chain 138 (balance ≥ amount), (2) WETH approved to the Chain 138 bridge for at least the send amount, (3) for LINK-fee bridge: LINK on Chain 138 approved for the bridge fee amount; for native-ETH bridge: sufficient ETH for fee. For relay-backed first hops (Mainnet, BSC, Avalanche), the destination relay inventory must also already hold at least the amount being sent. Recipient defaults to sender address if omitted.

**If send reverts** (e.g. `0x9996b315` with fee-token address): the CCIP router on Chain 138 may not accept the bridge’s fee token (LINK at `0xb772...`). See [docs/07-ccip/SEND_ETH_TO_MAINNET_REVERT_TRACE.md](../docs/07-ccip/SEND_ETH_TO_MAINNET_REVERT_TRACE.md) for the revert trace and fix options.

**Env:** `CCIP_DEST_CHAIN_SELECTOR` (default: 5009297550715157269 = Ethereum mainnet); `GAS_PRICE` (default: 1000000000); `CONFIRM_ABOVE_ETH` (optional; prompt for confirmation above this amount).

**Direct first-hop guard:** This helper now only allows proven direct first hops from Chain 138 to Mainnet, BSC, or Avalanche. It also fails fast when the source-token allowance is missing or when the destination relay inventory is smaller than the requested send amount. For Gnosis, Cronos, Celo, Polygon, Arbitrum, Optimism, and Base, use the Mainnet hub unless you intentionally override with `ALLOW_UNSUPPORTED_DIRECT_FIRST_HOP=1`.

**Source quote preflight:** `smom-dbis-138/scripts/ccip/ccip-send.sh` now fails before approvals or send attempts if `calculateFee()` already reverts on the chosen source bridge. As of 2026-04-04 UTC, the active Mainnet `WETH9` public fan-out path is quote-blocked on the tracked selectors `BSC`, `Avalanche`, `Gnosis`, `Cronos`, `Celo`, `Polygon`, `Arbitrum`, `Optimism`, and `Base`, so do not assume Mainnet hub fan-out is usable until that bridge/router path is repaired.

### 9. DBIS Frontend Deploy to Container

Deploy dbis-frontend build to Proxmox container VMID 10130. Builds locally, pushes dist, reloads nginx.

**Usage:**
```bash
./scripts/dbis/deploy-dbis-frontend-to-container.sh
```

**Env:** Uses `load-project-env.sh` and `get_host_for_vmid()`. `DBIS_FRONTEND_DEPLOY_PATH` overrides container deploy path (e.g. `/opt/dbis-core/frontend/dist`).

### 10. CT 2301 Corrupted Rootfs Recovery

CT 2301 (besu-rpc-private-1) may fail to start with `lxc.hook.pre-start` due to corrupted rootfs.

**Scripts:**
- `./scripts/fix-ct-2301-corrupted-rootfs.sh` — documents recovery options
- `./scripts/recreate-ct-2301.sh` — destroys and recreates CT 2301 (data loss; use after corrupted rootfs). Uses `load-project-env.sh` for config.

### 11. Backup and Security

- **Config backup:** `./scripts/backup-proxmox-configs.sh [--dry-run]` — backs up local config and .env
- **NPMplus backup:** `./scripts/verify/backup-npmplus.sh [--dry-run]` — requires NPM_PASSWORD in .env
- **Wave 0 from LAN:** `./scripts/run-wave0-from-lan.sh [--dry-run] [--skip-backup] [--skip-rpc-fix]` — runs NPMplus RPC fix (W0-1) and NPMplus backup (W0-3); W0-2 (sendCrossChain) run separately without `--dry-run`.
- **All waves (max parallel):** `./scripts/run-all-waves-parallel.sh [--dry-run] [--skip-wave0] [--skip-wave2] [--host HOST]` — Wave 0 via SSH, Wave 1 parallel (env, cron, SSH/firewall dry-run, shellcheck, validate), Wave 2 W2-6 (create 2506/2507/2508). See `docs/00-meta/FULL_PARALLEL_EXECUTION_ORDER.md` and `FULL_PARALLEL_RUN_LOG.md`.
- **NPMplus backup cron:** `./scripts/maintenance/schedule-npmplus-backup-cron.sh [--install|--show]` — add or print daily 03:00 cron for backup-npmplus.sh. Use from a persistent host checkout, e.g. `CRON_PROJECT_ROOT=/srv/proxmox`.
- **Security:** `./scripts/security/secure-env-permissions.sh [--dry-run]` or `chmod 600 .env smom-dbis-138/.env dbis_core/.env` — secure env files. **Validator keys (W1-19):** On Proxmox host as root: `./scripts/secure-validator-keys.sh [--dry-run]` (VMIDs 1000–1004).
- **info.defi-oracle.io public smoke:** `./scripts/verify/check-info-defi-oracle-public.sh` — HTTPS SPA, `/llms.txt`, `/agent-hints.json`, same-origin `/token-aggregation/api/v1/networks` JSON. Optional `INFO_SITE_BASE=https://staging.example.com`. Wrappers: `pnpm run verify:info-defi-oracle-public`; scored browser audit: `pnpm run audit:info-defi-oracle-site` (Chromium via `pnpm exec playwright install chromium`). Deploy to dedicated LXC: `./scripts/deployment/sync-info-defi-oracle-to-vmid2400.sh` (VMID **2410**). Also run (non-fatal) from `./scripts/run-operator-tasks-from-lan.sh`, `./scripts/run-all-operator-tasks-from-lan.sh`, and after E2E in `./scripts/run-full-operator-completion-from-lan.sh`. Runbook: [INFO_DEFI_ORACLE_IO_DEPLOYMENT.md](../docs/04-configuration/INFO_DEFI_ORACLE_IO_DEPLOYMENT.md).
- **Explorer token-aggregation API:** Build bundle with `./scripts/deploy-token-aggregation-for-publication.sh`, rsync to explorer with `./scripts/deployment/push-token-aggregation-bundle-to-explorer.sh` (`EXPLORER_SSH`, `REMOTE_DIR`, optional `systemctl restart token-aggregation`). Verify: `pnpm run verify:token-aggregation-api` or `./scripts/verify/check-token-aggregation-chain138-api.sh`. Apex `/api/v1/*` vs `/token-aggregation/api/v1/*` and planner POST issues: `./scripts/fix-explorer-http-api-v1-proxy.sh`, `./scripts/fix-explorer-token-aggregation-api-v2-proxy.sh`. Runbook: [TOKEN_AGGREGATION_REPORT_API_RUNBOOK.md](../docs/04-configuration/TOKEN_AGGREGATION_REPORT_API_RUNBOOK.md).

### 12. Maintenance (135–139)

- **Daily/weekly checks:** `./scripts/maintenance/daily-weekly-checks.sh [daily|weekly|all]` — explorer sync (135), RPC health (136), config API (137). **Cron:** `./scripts/maintenance/schedule-daily-weekly-cron.sh [--install|--show]` (daily 08:00, weekly Sun 09:00). Use from a persistent host checkout, e.g. `CRON_PROJECT_ROOT=/srv/proxmox`. See [OPERATIONAL_RUNBOOKS.md](../docs/03-deployment/OPERATIONAL_RUNBOOKS.md) § Maintenance.
- **Ensure FireFly primary (6200):** `./scripts/maintenance/ensure-firefly-primary-via-ssh.sh [--dry-run]` — normalize the compose file expected by the installed `docker-compose`, install the idempotent helper-backed `firefly.service`, and verify `/api/v1/status` for the current mixed legacy-plus-compose stack.
- **Ensure Fabric sample network (6000):** `./scripts/maintenance/ensure-fabric-sample-network-via-ssh.sh [--dry-run]` — ensure nested-LXC features, install the boot-time `fabric-sample-network.service`, and verify `mychannel`.
- **Ensure legacy monitor networking (3000-3003):** `./scripts/maintenance/ensure-legacy-monitor-networkd-via-ssh.sh [--dry-run]` — host-side enable plus in-guest start for `systemd-networkd` on the legacy monitor/RPC-adjacent LXCs so their static LAN IPs actually come up.
- **Start firefly-ali-1 (6201):** `./scripts/maintenance/start-firefly-6201.sh [--dry-run] [--host HOST]` — start CT 6201 on r630-02 when needed (optional ongoing).
- **Config validation (pre-deploy):** `./scripts/validation/validate-config-files.sh` — set `VALIDATE_REQUIRED_FILES` for required paths. **CI / all validation:** `./scripts/verify/run-all-validation.sh [--skip-genesis]` — dependencies + config + optional genesis (no LAN/SSH).

### 13. Phase 2, 3 & 4 Deployment Scripts

- **Monitoring (Phase 2):** `./scripts/deployment/phase2-observability.sh [--config-only]` — writes `config/monitoring/` (prometheus.yml, alertmanager.yml).
- **Security (Phase 2):** `./scripts/security/setup-ssh-key-auth.sh [--dry-run|--apply]`, `./scripts/security/firewall-proxmox-8006.sh [--dry-run|--apply] [CIDR]`.
- **Proxmox SSH / FQDN:** `./scripts/security/ensure-proxmox-ssh-access.sh` (all five mgmt IPs; `--fqdn` for `*.sankofa.nexus`; `--copy` for `ssh-copy-id`). `./scripts/verify/check-proxmox-mgmt-fqdn.sh` (`--print-hosts` for `/etc/hosts`).
- **Backup (Phase 2):** `./scripts/backup/automated-backup.sh [--dry-run] [--with-npmplus]` — config + optional NPMplus; cron in header.
- **CCIP (Phase 3):** `./scripts/ccip/ccip-deploy-checklist.sh` — env check and deployment order from spec.
- **Sovereign tenants (Phase 4):** `./scripts/deployment/phase4-sovereign-tenants.sh [--show-steps|--dry-run]` — checklist; full runbook in OPERATIONAL_RUNBOOKS § Phase 4.
- **Full verification (6 steps):** `./scripts/verify/run-full-verification.sh` — Step 0: config validation; Steps 1–5: DNS, UDM Pro, NPMplus, backend VMs, E2E routing; Step 6: source-of-truth JSON. Run from project root.

### 14. Public Mainnet DODO cW swaps

Repeatable helper for the first public Mainnet DODO PMM `cW*` pools, including the USD bootstrap set and the first non-USD Wave 1 rows.

**Usage:**
```bash
# Dry-run, including quote-source detection and reserve fallback
bash scripts/deployment/run-mainnet-public-dodo-cw-swap.sh \
  --pair=cwusdt-usdc \
  --direction=base-to-quote \
  --amount=5000 \
  --dry-run

# Live CHF row proof
bash scripts/deployment/run-mainnet-public-dodo-cw-swap.sh \
  --pair=cwchfc-usdc \
  --direction=quote-to-base \
  --amount=1000

# Live tiny swap
bash scripts/deployment/run-mainnet-public-dodo-cw-swap.sh \
  --pair=cwusdt-usdc \
  --direction=base-to-quote \
  --amount=5000

# Dry-run the first non-USD Wave 1 Mainnet pool
bash scripts/deployment/run-mainnet-public-dodo-cw-swap.sh \
  --pair=cweurc-usdc \
  --direction=base-to-quote \
  --amount=1000 \
  --dry-run
```

**Supported pairs:** `cwusdt-usdc`, `cwusdc-usdc`, `cwusdt-usdt`, `cwusdc-usdt`, `cwusdt-cwusdc`, `cweurc-usdc`, `cwgbpc-usdc`, `cwaudc-usdc`, `cwcadc-usdc`, `cwjpyc-usdc`

**Supported directions:** `base-to-quote`, `quote-to-base`

**Important note:** the live Mainnet DODO pools can execute swaps, but the direct hosted `querySellBase` / `querySellQuote` read path may revert. This helper tries the direct pool read first and then falls back to a conservative reserve-based quote when needed, and it prints `quoteSource=pool_query` or `quoteSource=reserve_fallback` so the operator can see which path was used.

**Bootstrap verifier:**
```bash
bash scripts/verify/check-mainnet-public-dodo-cw-bootstrap-pools.sh
```
This checks that the eleven recorded Mainnet DODO cW bootstrap pools (USD rails + non-USD Wave 1 + `cWUSDT/cWUSDC`) are still mapped by the integration, have non-zero reserves, and remain dry-run routable through the repeatable swap helper.

**Current micro-support sweep (default dry-run):**
```bash
bash scripts/deployment/run-mainnet-cw-micro-support-round.sh --dry-run
```
This runs the currently validated tiny support swaps for `cwusdt-usdt`, `cwusdc-usdc`, `cwusdt-usdc`, and `cwusdc-usdt` in sequence, then points back to `check-mainnet-pmm-peg-bot-readiness.sh`.

Re-run with `--apply` only when you intentionally want to broadcast the full four-leg support round.

**Higher-level remediation sequence:**
```bash
bash scripts/deployment/run-mainnet-cw-remediation-sequence.sh --dry-run
```
Modes:
- `--mode=micro` for the four-leg support sweep only
- `--mode=flash-plan` for the `cWUSDC/USDC` flash quote-push plan only
- `--mode=full` for both

### 15. Mainnet DODO Wave 1 pool deploy helper

Repeatable helper for creating and seeding a first-tier Mainnet DODO PMM Wave 1 `cW* / USDC` pair.

**Usage:**
```bash
bash scripts/deployment/deploy-mainnet-public-dodo-wave1-pool.sh \
  --pair=cweurc-usdc \
  --initial-price=1151700000000000000 \
  --base-amount=1250000 \
  --quote-amount=1439625 \
  --mint-base-amount=1300000
```

This helper creates the pool if needed, optionally mints the base `cW*` token to the deployer when the deployer still has `MINTER_ROLE`, approves the integration, and seeds the first liquidity tranche.

### 16. Mainnet cWUSDT / cWUSDC PMM (direct wrap pair)

**Compute matched seed (e.g. 50× reference depth, optional cap):**

```bash
bash scripts/deployment/compute-mainnet-cwusdt-cwusdc-seed-amounts.sh --multiplier=50 --cap-raw=<optional_max_raw>
```

**Create (if missing) or top up 1:1 liquidity:**

```bash
bash scripts/deployment/deploy-mainnet-cwusdt-cwusdc-pool.sh \
  --initial-price=1000000000000000000 \
  --base-amount=<raw> --quote-amount=<raw> \
  --dry-run
```

**Deterministic round-trip soak (default dry-run; no RNG):**

```bash
bash scripts/deployment/run-mainnet-cwusdt-cwusdc-soak-roundtrips.sh \
  --amounts-raw=100000000,10000000000 \
  --repeat-list=10 --dry-run
```

**Routing planner (USDT↔USDC paths including direct cW leg):** `scripts/verify/plan-mainnet-usdt-usdc-via-cw-paths.sh`

---

## Utility Modules

Shared utility functions are available in `scripts/utils/`:

- `container-utils.sh` - Container operations
- `network-utils.sh` - Network operations
- `service-utils.sh` - Service operations
- `config-utils.sh` - Configuration operations
- `proxmox-utils.sh` - Proxmox operations

**Usage:**
```bash
source "$(dirname "${BASH_SOURCE[0]}")/../utils/container-utils.sh"
container_status 5000
container_restart 5000
```

---

## Shared Libraries

Core shared modules in `scripts/lib/`:

- **`load-project-env.sh`** — Load project environment (.env, config/ip-addresses.conf, smom-dbis-138/.env). **Use this** instead of hardcoding IPs or sourcing multiple files. Scripts that need config should `source "${SCRIPT_DIR}/lib/load-project-env.sh"`.
- `ip-config.sh` - Centralized IP address configuration
- `logging.sh` - Consistent logging functions
- `proxmox-api.sh` - Proxmox API helpers
- `ssh-helpers.sh` - SSH utility functions

---

## Migration

Old scripts have been archived to `scripts/archive/consolidated/`. Use the frameworks instead.

**Migration Guide:** `docs/00-meta/FRAMEWORK_MIGRATION_GUIDES.md`  
**Migration Examples:** `docs/00-meta/MIGRATION_EXAMPLES.md`  
**Migration Checklist:** `docs/00-meta/MIGRATION_CHECKLIST.md`

---

## Directory Structure

```
scripts/
├── lib/              # Shared libraries (load-project-env.sh, etc.)
├── bridge/           # CCIP bridge scripts
│   └── run-send-cross-chain.sh
├── dbis/             # DBIS Core deployment scripts
│   └── deploy-dbis-frontend-to-container.sh
├── verify/           # Verification scripts
│   ├── check-contracts-on-chain-138.sh   # On-chain bytecode check (Chain 138)
│   ├── run-contract-verification-with-proxy.sh
│   └── ...           # Other verify scripts
├── utils/            # Utility modules
├── archive/          # Archived scripts
│   ├── consolidated/ # Migrated scripts
│   ├── small-scripts/# Merged small scripts
│   ├── test/        # Test scripts
│   └── backups/     # Backup scripts
├── verify-all.sh     # Verification framework
├── list.sh           # Listing framework
├── fix-all.sh        # Fix framework
├── configure.sh      # Configuration framework
└── deploy.sh         # Deployment framework
```

---

## Documentation

- **Framework Usage:** `docs/00-meta/FRAMEWORK_USAGE_GUIDE.md`
- **Migration Guides:** `docs/00-meta/FRAMEWORK_MIGRATION_GUIDES.md`
- **Final Report:** `docs/00-meta/FINAL_REDUCTION_REPORT.md`
- **Script Inventory:** `docs/00-meta/SCRIPT_INVENTORY.md`

---

**Status:** ✅ Scripts consolidated and documented