ops: oracle publisher LXC 3500/3501, CT migrate docs, Besu/RPC maintenance

- Provision oracle-publisher on CT 3500 (quoted DATA_SOURCE URLs, dotenv).
- Host-side pct-lxc-3501-net-up for ccip-monitor eth0 after migrate.
- CoinGecko key script: avoid sed & corruption; document quoted URLs.
- Besu node list reload, fstrim/RPC scripts, storage health docs.
- Submodule smom-dbis-138: web3 v6 pin, oracle check default host r630-02.

Made-with: Cursor

docs/04-configuration/ALL_VMIDS_ENDPOINTS.md

@@ -316,8 +316,8 @@ The following VMIDs have been permanently removed:
 
 | VMID | IP Address | Hostname | Status | Endpoints | Purpose |
 |------|------------|----------|--------|-----------|---------|
-| 3500 | 192.168.11.29 | oracle-publisher-1 | ✅ Running | Oracle: Various | Oracle publisher service |
-| 3501 | 192.168.11.28 | ccip-monitor-1 | ✅ Running | Monitor: Various | CCIP monitoring service |
+| 3500 | 192.168.11.29 | oracle-publisher-1 | ✅ Running (verify on-chain) | Oracle: Various | **r630-02** `thin5`. Reprovisioned 2026-03-28 via `scripts/deployment/provision-oracle-publisher-lxc-3500.sh` (systemd `oracle-publisher`). If `updateAnswer` txs revert, set `PRIVATE_KEY` in `/opt/oracle-publisher/.env` to an EOA **authorized on the aggregator** (may differ from deployer). Metrics: `:8000/metrics`. |
+| 3501 | 192.168.11.28 | ccip-monitor-1 | ✅ Running | Monitor: Various | CCIP monitoring; **migrated 2026-03-28** to **r630-02** `thin5` (`pvesh` … `/migrate --target-storage thin5`). |
 | 5200 | 192.168.11.80 | cacti-1 | ✅ Running | Web: 80, 443 | Network monitoring (Cacti); **host r630-02** (migrated 2026-02-15) |
 
 ---

docs/04-configuration/COINGECKO_API_KEY_SETUP.md

@@ -103,13 +103,19 @@ systemctl restart token-aggregation
 
 ### For Oracle Publisher Service
 
+**Gas (Chain 138 / Besu):** In `/opt/oracle-publisher/.env`, use **`GAS_LIMIT=400000`** (not `100000`). The aggregator `updateAnswer` call can **run out of gas** at 100k (`gasUsed == gasLimit`, failed receipt) even when `isTransmitter` is true. Align with `ORACLE_UPDATE_GAS_LIMIT` in `smom-dbis-138/scripts/update-oracle-price.sh`. **`GAS_PRICE=1000000000`** (1 gwei) matches that script’s legacy defaults.
+
+**Quoted URLs in `.env`:** `DATA_SOURCE_1_URL` (and Coinbase `DATA_SOURCE_2_URL`) must be **double-quoted** when passed through **systemd `EnvironmentFile`**, because unquoted `&` in query strings can be parsed incorrectly and corrupt the value. **`scripts/update-oracle-publisher-coingecko-key.sh`** uses `grep` + `append` (not `sed` with `&` in the replacement). Do not use `sed 's|...|...&...|'` for URLs that contain `&`.
+
+**Dotenv sources for provisioning:** `scripts/lib/load-project-env.sh` loads **project root `.env`** then **`smom-dbis-138/.env`** — so `PRIVATE_KEY` / `DEPLOYER_PRIVATE_KEY`, `COINGECKO_API_KEY` (root `.env`), and `AGGREGATOR_ADDRESS` are available to `scripts/deployment/provision-oracle-publisher-lxc-3500.sh` and `scripts/update-oracle-publisher-coingecko-key.sh`.
+
 **Step 1: SSH to Proxmox host**
 
 ```bash
-ssh root@192.168.11.10
+ssh root@192.168.11.12
 ```
 
-**Step 2: Access Oracle Publisher container**
+**Step 2: Access Oracle Publisher container** (VMID 3500 runs on **r630-02**)
 
 ```bash
 pct exec 3500 -- bash
@@ -162,10 +168,10 @@ npm run test -- coingecko-adapter.test.ts
 
 ```bash
 # Check .env file
-ssh root@192.168.11.10 "pct exec 3500 -- cat /opt/oracle-publisher/.env | grep COINGECKO"
+ssh root@192.168.11.12 "pct exec 3500 -- cat /opt/oracle-publisher/.env | grep COINGECKO"
 
 # Check service logs
-ssh root@192.168.11.10 "pct exec 3500 -- journalctl -u oracle-publisher -n 50 | grep -i coingecko"
+ssh root@192.168.11.12 "pct exec 3500 -- journalctl -u oracle-publisher -n 50 | grep -i coingecko"
 
 # Should see successful price fetches without 429 rate limit errors
 ```

docs/04-configuration/STORAGE_GROWTH_AND_HEALTH.md

@@ -1,8 +1,18 @@
 # Storage Growth and Health — Predictable Growth Table & Proactive Monitoring
 
-**Last updated:** 2026-02-15  
+**Last updated:** 2026-03-28  
 **Purpose:** Real-time data collection and a predictable growth table so we can stay ahead of disk space issues on hosts and VMs.
 
+### Recent operator maintenance (2026-03-28)
+
+- **r630-01 `pve/data` (local-lvm):** Thin pool extended (+80 GiB data, +512 MiB metadata earlier); **LVM thin auto-extend** enabled in `lvm.conf` (`thin_pool_autoextend_threshold = 80`, `thin_pool_autoextend_percent = 20`); **dmeventd** must stay active.
+- **r630-01 `pve/thin1`:** Pool extended (+48 GiB data, +256 MiB metadata) to reduce pressure; metadata percent dropped accordingly.
+- **r630-01 `/var/lib/vz/dump`:** Removed obsolete **2026-02-15** vzdump archives/logs (~9 GiB); newer logs from 2026-02-28 retained.
+- **Fleet guest `fstrim`:** `scripts/maintenance/fstrim-all-running-ct.sh` supports **`FSTRIM_TIMEOUT_SEC`** and **`FSTRIM_HOSTS`** (e.g. `ml110`, `r630-01`, `r630-02`). Many CTs return FITRIM “not permitted” (guest/filesystem); others reclaim space on the thin pools (notably on **r630-02**).
+- **r630-02 `thin1`–`thin6` VGs:** Each VG is on a **single PV** with only **~124 MiB `vg_free`**; you **cannot** `lvextend` those thin pools until the underlying partition/disk is grown or a second PV is added. Monitor `pvesm status` and plan disk expansion before pools tighten.
+- **CT migration** off r630-01 for load balance remains a **planned** action when maintenance windows and target storage allow (not automated here).
+- **2026-03-28 (migration follow-up):** CT **3501** migrated to r630-02 **`thin5`** via `pvesh … lxc/3501/migrate --target-storage thin5`. CT **3500** had root LV removed after a mistaken `pct set --delete unused0` (config had `unused0: local-lvm:vm-3500-disk-0` and `rootfs: thin1:vm-3500-disk-0`); **3500** was recreated empty on r630-02 `thin5` — **reinstall Oracle Publisher** on the guest. See `MIGRATE_CT_R630_01_TO_R630_02.md`.
+
 ---
 
 ## 1. Real-time data collection