fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
209 lines
10 KiB
Markdown
209 lines
10 KiB
Markdown
# Full Fixes Prepared — Consolidated Checklist
|
||
|
||
**Last Updated:** 2026-02-07
|
||
**Purpose:** Single master list of all fixes (infra, network, optional) with copy-paste commands.
|
||
**Sources:** [FIXES_PREPARED.md](FIXES_PREPARED.md), [STUCK_TX_AND_BLOCK_STATUS_20260207.md](../08-monitoring/STUCK_TX_AND_BLOCK_STATUS_20260207.md), [SENTRY_RPC_REVIEW_20260207.md](../08-monitoring/SENTRY_RPC_REVIEW_20260207.md), [NEXT_STEPS_OPERATOR.md](../00-meta/NEXT_STEPS_OPERATOR.md).
|
||
**Applied run (2026-02-07):** [verification-evidence/APPLIED_FIXES_20260207.md](verification-evidence/APPLIED_FIXES_20260207.md).
|
||
|
||
---
|
||
|
||
## Master checklist
|
||
|
||
| # | Priority | Area | Issue | Fix | Section |
|
||
|---|----------|------|--------|-----|---------|
|
||
| 1 | **Required** | Validators | Only 1/5 active; block production stalled | Ensure validators 1000–1004 are up; re-run validator fix if needed | [§ 1](#1-validators--block-production-required) |
|
||
| 2 | **Required** | Stuck tx | Deployer nonce 13178 stuck (not in chain/txpool) | Use nonce 13178 for next send, or clear RPC DB | [§ 2](#2-stuck-transaction-required) |
|
||
| 3 | **Required** | Sentry 1504 | besu-sentry inactive; restart failed | Start service (try besu-sentry-5 or correct unit name) | [§ 3](#3-sentry-nodes-required) |
|
||
| 4 | **Required** | RPC 2301 | besu-rpc inactive; no RPC response | Start RPC service on ml110; check unit name and logs | [§ 4](#4-rpc-nodes-required) |
|
||
| 5 | **Required** | Network | UDM Pro port forward Alltra/HYBX | Add 76.53.10.38 → 192.168.11.169 (80, 81, 443) in UniFi | [§ 5](#5-udm-pro--alltrahybx-network-required) |
|
||
| 6 | **Required** | Network | Alltra/HYBX 502 (rpc-alltra*, rpc-hybx*, cacti-*) | Verify backends (2500–2502, 2503–2505, 5201, 5202); fix NPMplus or deploy | [§ 6](#6-alltrahybx-502-required) |
|
||
| 7 | Optional | Sentry 1503 | Container unknown | Confirm if 1503 exists on r630-01 or elsewhere | [§ 3](#3-sentry-nodes-required) |
|
||
| 8 | Optional | RPC 2402, 2503–2508 | Containers unknown | Create or map to correct hosts if needed | [§ 4](#4-rpc-nodes-required) |
|
||
| 9 | Optional | NPMplus certs | Remaining Alltra/HYBX hosts | Request Let's Encrypt via script or UI | [§ 7](#7-optional-fixes) |
|
||
| 10 | Optional | Explorer SSL | explorer.d-bis.org cert | NPMplus UI: SSL for explorer.d-bis.org | [§ 7](#7-optional-fixes) |
|
||
| 11 | Optional | NPMplus cert 134 | cross-all.defi-oracle.io files missing | NPMplus UI: re-save or re-request cert | [§ 7](#7-optional-fixes) |
|
||
| 12 | Optional | Shellcheck / env / verification | Code quality, permissions, E2E | Run scripts per table below | [§ 7](#7-optional-fixes) |
|
||
|
||
---
|
||
|
||
## 1. Validators & block production (required)
|
||
|
||
**Goal:** Get 4/5 validators active so QBFT quorum is met and blocks are produced.
|
||
|
||
**Hosts:** 1000, 1001, 1002 → r630-01 (192.168.11.11); 1003, 1004 → ml110 (192.168.11.10).
|
||
|
||
**Commands (from project root, SSH to hosts required):**
|
||
|
||
```bash
|
||
cd /path/to/proxmox
|
||
|
||
# 1a. Re-apply validator config + tx-pool eviction and restart (if not already done)
|
||
bash scripts/fix-all-validators-and-txpool.sh
|
||
|
||
# 1b. Check validator status (after a few minutes)
|
||
source config/ip-addresses.conf 2>/dev/null
|
||
bash scripts/monitoring/monitor-blockchain-health.sh
|
||
|
||
# 1c. Per-host: check and start validators if needed
|
||
ssh root@192.168.11.11 "for v in 1000 1001 1002; do echo \"VMID \$v:\"; pct exec \$v -- systemctl is-active besu-validator 2>/dev/null || echo 'not found'; done"
|
||
ssh root@192.168.11.10 "for v in 1003 1004; do echo \"VMID \$v:\"; pct exec \$v -- systemctl is-active besu-validator 2>/dev/null || echo 'not found'; done"
|
||
|
||
# 1d. If any are inactive, start and check logs
|
||
ssh root@<host> "pct exec <vmid> -- systemctl start besu-validator"
|
||
ssh root@<host> "pct exec <vmid> -- journalctl -u besu-validator.service -n 50 --no-pager"
|
||
```
|
||
|
||
**Docs:** [BLOCK_PRODUCTION_MONITORING.md](../08-monitoring/BLOCK_PRODUCTION_MONITORING.md), [QBFT_TROUBLESHOOTING.md](../09-troubleshooting/QBFT_TROUBLESHOOTING.md), [TXPOOL_EVICTION_PREVENT_STUCK.md](../06-besu/TXPOOL_EVICTION_PREVENT_STUCK.md).
|
||
|
||
---
|
||
|
||
## 2. Stuck transaction (required)
|
||
|
||
**Issue:** Deployer `0x4A666F96fC8764181194447A7dFdb7d471b301C8` has pending nonce 13178 not in chain or txpool.
|
||
|
||
**Fix A — Use next nonce (recommended):** Use nonce **13178** for the next transaction from this account.
|
||
|
||
```bash
|
||
cd /path/to/proxmox
|
||
source config/ip-addresses.conf 2>/dev/null
|
||
bash scripts/skip-stuck-transactions.sh
|
||
# Then in cast/forge: --nonce 13178 --gas-price 10000000000 --rpc-url http://192.168.11.211:8545
|
||
```
|
||
|
||
**Fix B — Clear RPC tx state (nuclear):** Only if you need to reset RPC’s internal state.
|
||
|
||
```bash
|
||
PROXMOX_USER=root RPC_HOST=192.168.11.11 bash scripts/clear-rpc-database-complete.sh
|
||
```
|
||
|
||
**Docs:** [STUCK_TRANSACTIONS_SOLUTION.md](../06-besu/STUCK_TRANSACTIONS_SOLUTION.md).
|
||
|
||
---
|
||
|
||
## 3. Sentry nodes (required)
|
||
|
||
**1504 (ml110):** Config already updated; service restart failed. Start with correct unit name:
|
||
|
||
```bash
|
||
ssh root@192.168.11.10 "pct exec 1504 -- systemctl list-units --type=service | grep -i besu"
|
||
# Then start the correct unit, e.g.:
|
||
ssh root@192.168.11.10 "pct exec 1504 -- systemctl start besu-sentry" # or besu-sentry-5
|
||
```
|
||
|
||
**1503:** Container status unknown on r630-01. Confirm if VMID 1503 exists:
|
||
|
||
```bash
|
||
ssh root@192.168.11.11 "pct list | grep 1503"
|
||
```
|
||
|
||
**Re-review all sentries:**
|
||
|
||
```bash
|
||
bash scripts/review-sentry-and-rpc-nodes.sh
|
||
```
|
||
|
||
---
|
||
|
||
## 4. RPC nodes (required)
|
||
|
||
**2301 (ml110, 192.168.11.232):** RPC inactive. Start service and check config:
|
||
|
||
```bash
|
||
ssh root@192.168.11.10 "pct exec 2301 -- systemctl list-units --type=service | grep -i besu"
|
||
ssh root@192.168.11.10 "pct exec 2301 -- systemctl start besu-rpc" # or besu-rpc-private
|
||
ssh root@192.168.11.10 "pct exec 2301 -- journalctl -u besu-rpc -n 30 --no-pager"
|
||
```
|
||
|
||
**2402, 2503–2508:** Containers unknown. If these nodes are required, create them per [MISSING_CONTAINERS_LIST.md](../03-deployment/MISSING_CONTAINERS_LIST.md) or confirm correct Proxmox hosts and VMIDs.
|
||
|
||
**1504 / 2301 (Besu not installed):** See [INSTALL_BESU_1504_2301_RUNBOOK.md](../06-besu/INSTALL_BESU_1504_2301_RUNBOOK.md) to install Besu and configs, then start services (or run `scripts/fix-besu-services-on-host.sh` on ml110 after install).
|
||
|
||
**Re-review all RPC nodes:**
|
||
|
||
```bash
|
||
bash scripts/review-sentry-and-rpc-nodes.sh --apply-txpool
|
||
```
|
||
|
||
---
|
||
|
||
## 5. UDM Pro & Alltra/HYBX network (required)
|
||
|
||
**UDM Pro port forward:** Add rules in UniFi Network → Firewall & Security (or Port Forwarding):
|
||
|
||
| Rule Name | Dest IP | Dest Port | Forward to IP | Forward to Port |
|
||
|-----------|---------|-----------|---------------|-----------------|
|
||
| NPMplus Alltra/HYBX HTTP | 76.53.10.38 | 80 | 192.168.11.169 | 80 |
|
||
| NPMplus Alltra/HYBX HTTPS | 76.53.10.38 | 443 | 192.168.11.169 | 443 |
|
||
| NPMplus Alltra/HYBX Admin | 76.53.10.38 | 81 | 192.168.11.169 | 81 |
|
||
|
||
**Verify from LAN:**
|
||
|
||
```bash
|
||
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.169:80/
|
||
curl -s -o /dev/null -w "%{http_code}" -k https://192.168.11.169:81/
|
||
```
|
||
|
||
**Doc:** [UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md](UDM_PRO_NPMPLUS_ALLTRA_HYBX_PORT_FORWARD.md).
|
||
|
||
---
|
||
|
||
## 6. Alltra/HYBX 502 (required)
|
||
|
||
**Domains returning 502:** rpc-alltra*.d-bis.org, rpc-hybx*.d-bis.org, cacti-alltra.d-bis.org, cacti-hybx.d-bis.org.
|
||
|
||
**Step 1 — Verify backends from LAN:**
|
||
|
||
```bash
|
||
# Alltra RPC
|
||
curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.172:8545
|
||
# HYBX RPC
|
||
curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' http://192.168.11.246:8545
|
||
# Cacti
|
||
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.177:80/
|
||
curl -s -o /dev/null -w "%{http_code}" http://192.168.11.251:80/
|
||
```
|
||
|
||
**Step 2:** If backends respond → in NPMplus (https://192.168.11.167:81) fix Proxy Hosts (forward to correct IP:port). If backends do not respond → deploy/start VMIDs 2500–2502, 2503–2505, 5201, 5202 per [NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md](NPMPLUS_ALLTRA_HYBX_MASTER_PLAN.md).
|
||
|
||
---
|
||
|
||
## 7. Optional fixes
|
||
|
||
| Goal | Command |
|
||
|------|---------|
|
||
| NPMplus certs (first only) | `FIRST_ONLY=1 NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh` |
|
||
| NPMplus certs (all remaining) | `NPM_URL=https://192.168.11.167:81 bash scripts/request-npmplus-certificates.sh` |
|
||
| Explorer SSL | NPMplus UI → SSL Certificates → explorer.d-bis.org; Proxy Hosts → SSL tab |
|
||
| Cert 134 (cross-all.defi-oracle.io) | NPMplus UI → SSL Certificates → re-save or re-request |
|
||
| Shellcheck | `bash scripts/verify/run-shellcheck.sh --optional` |
|
||
| Env permissions | `bash scripts/security/secure-env-permissions.sh` |
|
||
| Full verification | `bash scripts/verify/run-full-verification.sh` |
|
||
| NPMplus backup | `bash scripts/verify/backup-npmplus.sh` |
|
||
|
||
---
|
||
|
||
## Execution order (recommended)
|
||
|
||
1. **Validators & block production** — fix validators, confirm 4/5 active, block production resuming.
|
||
2. **Stuck transaction** — use nonce 13178 for next deployer tx (or clear RPC if required).
|
||
3. **Sentry 1504** — start besu-sentry (or correct unit) on ml110.
|
||
4. **RPC 2301** — start besu-rpc on ml110; fix config if needed.
|
||
5. **UDM Pro** — add port forward 76.53.10.38 → 192.168.11.169 if direct Alltra/HYBX access is needed.
|
||
6. **Alltra/HYBX 502** — verify backends, then fix NPMplus or deploy containers.
|
||
7. **Optional** — certs, Explorer SSL, shellcheck, env permissions, full verification, backup.
|
||
|
||
---
|
||
|
||
## Quick script index
|
||
|
||
| Purpose | Script |
|
||
|--------|--------|
|
||
| Validators + tx-pool eviction | `scripts/fix-all-validators-and-txpool.sh` |
|
||
| Sentry & RPC review (+ tx-pool) | `scripts/review-sentry-and-rpc-nodes.sh` [--apply-txpool] |
|
||
| Blockchain health | `scripts/monitoring/monitor-blockchain-health.sh` |
|
||
| Stuck tx / next nonce | `scripts/skip-stuck-transactions.sh` |
|
||
| Stuck tx investigation | `scripts/investigate-transaction-persistence.sh` |
|
||
| Full verification (6 steps) | `scripts/verify/run-full-verification.sh` |
|
||
|
||
All scripts run from **project root**; source `config/ip-addresses.conf` (and optionally `.env`) where the script does not do so.
|