d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b7eebb87b33ea4af3efe689c5a0469df4a134313
proxmox/docs/04-configuration/E2E_ENDPOINTS_LIST.md
defiQUG dbd517b279 Sync workspace: config, docs, scripts, CI, operator rules, and submodule pointers. 
- Update dbis_core, cross-chain-pmm-lps, explorer-monorepo, metamask-integration, pr-workspace/chains
- Omit embedded publish git dirs and empty placeholders from index

Made-with: Cursor
2026-04-12 06:12:20 -07:00

242 lines
23 KiB
Markdown
Raw Blame History

# E2E verification — endpoint inventory and profiles
**Source:** `scripts/verify/verify-end-to-end-routing.sh` (DOMAIN_TYPES).
**List from CLI (public):** `./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=public`
**List from CLI (private/admin):** `./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=private`
**Run E2E (public profile recommended):** `./scripts/verify/verify-end-to-end-routing.sh --profile=public` (from LAN with DNS or use `E2E_USE_SYSTEM_RESOLVER=1` and `/etc/hosts` per [E2E_DNS_FROM_LAN_RUNBOOK.md](E2E_DNS_FROM_LAN_RUNBOOK.md)).
**Run E2E (private/admin):** `./scripts/verify/verify-end-to-end-routing.sh --profile=private`.
**Gitea Actions (umbrella / cc-*):** no stable unauthenticated REST for all Gitea versions — print UI URLs with `./scripts/verify/print-gitea-actions-urls.sh` and confirm jobs in the browser after push.
**What each hostname should present (operator narrative):** [FQDN_EXPECTED_CONTENT.md](FQDN_EXPECTED_CONTENT.md).
**Latest verified public transport/TLS pass:** `2026-04-02` via `bash scripts/verify/verify-end-to-end-routing.sh --profile=public` with report at [verification_report.md](verification-evidence/e2e-verification-20260402_130259/verification_report.md). Result: exit `0`, `DNS passed: 60`, `Failed: 0`, `HTTPS passed: 44` — includes the repaired DBIS, Keycloak, Studio, `info.defi-oracle.io`, `data.d-bis.org /v1/health`, and both Cacti hostnames.
**Resolved public regression snapshot:** the earlier `2026-04-02` regression run at [verification_report.md](verification-evidence/e2e-verification-20260402_074223/verification_report.md) is now historical only; its DBIS, Keycloak, Cacti, Studio, and `info.defi-oracle.io` warnings were resolved later the same day.
**2026-04-01 verifier tightening:** `verify-end-to-end-routing.sh` rejects placeholder directory listings for **`core.d-bis.org`**, **`dbis-api.d-bis.org`**, and **`dbis-api-2.d-bis.org`**. The current `2026-04-02` pass succeeds because those hosts now serve the real DBIS API/runtime, not static placeholder content.
**2026-04-04 explorer E2E tightening:** the canonical explorer **`explorer.d-bis.org`** is no longer treated as `optional-when-fail` in the public profile. The explorer-specific verifier now also checks the static **Visual Command Center** and the live **Mission Control** stream, bridge trace, and liquidity endpoints.
**Previous:** `2026-03-29` — [verification_report.md](verification-evidence/e2e-verification-20260329_045318/verification_report.md); older: [20260329_045210](verification-evidence/e2e-verification-20260329_045210/verification_report.md), [20260327](verification-evidence/e2e-verification-20260327_134032/verification_report.md).
**Latest verified private/admin pass:** `2026-03-27` via `bash scripts/verify/verify-end-to-end-routing.sh --profile=private` with report at [verification_report.md](verification-evidence/e2e-verification-20260327_134137/verification_report.md). Result: exit `0`, `DNS passed: 4`, `Failed: 0`.
**Evidence folders:** Each run creates `verification-evidence/e2e-verification-YYYYMMDD_HHMMSS/`. Commit the runs you want on record; older dirs can be removed locally to reduce noise (`scripts/maintenance/prune-e2e-verification-evidence.sh --dry-run` lists candidates). Routing truth is **not** inferred from old reports—use [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md).
## Verification profiles
- **Public profile (default for routine E2E):** web, api, public RPC endpoints.
- **Private/admin profile:** private RPC and Fireblocks RPC endpoints. Run separately for internal operations.
## Full endpoint inventory (combined)
| Endpoint | Type | URL | Description (content provided) |
|----------|------|-----|--------------------------------|
| explorer.d-bis.org | web | https://explorer.d-bis.org | Blockscout-style blockchain explorer for Chain 138: blocks, transactions, addresses, contracts, tokens, verification. |
| explorer.d-bis.org | web | https://explorer.d-bis.org/chain138-command-center.html | Chain 138 deployment topology — interactive Mermaid command center (tabs, keyboard, `?tab=` / `?tab=mission-control` slugs); static asset with **More → Visual Command Center** entry point. |
| d-bis.org | web | https://d-bis.org | **Public** DBIS web presence — institutional portal (Gov Portals Next app when deployed behind NPM). |
| admin.d-bis.org | web | https://admin.d-bis.org | **Admin** console for DBIS operations staff; typical upstream VMID **10130**. |
| dbis-admin.d-bis.org | web | https://dbis-admin.d-bis.org | **Legacy** admin hostname; same upstream intent as **admin.d-bis.org** if still in DNS. |
| secure.d-bis.org | web | https://secure.d-bis.org | **Member** secure portal (authenticated institutions); path-based routing on **10130** per [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md). |
| core.d-bis.org | web | https://core.d-bis.org | Current DBIS Core service root on VMID **10150**. Public root returns service metadata JSON while the dedicated client UI cutover remains separate work. |
| dbis-api.d-bis.org | api | https://dbis-api.d-bis.org | Primary DBIS core API host on VMID **10150**. Root `/`, `/health`, and `/v1/health` return live JSON responses. |
| dbis-api.d-bis.org | api | `https://dbis-api.d-bis.org/api/v1/gateway/rails` | **Authenticated** — SolaceNet gateway rail adapter list (`maintainer`, `adapters[]`). Internal smoke: `scripts/verify/check-dbis-core-gateway-rails.sh` (`DBIS_CORE_API_BASE`, `DBIS_CORE_BEARER_TOKEN`). |
| dbis-api-2.d-bis.org | api | https://dbis-api-2.d-bis.org | Secondary DBIS core API host on VMID **10151** with the same root and health responses. |
| mim4u.org | web | https://mim4u.org | MIM4U main site. |
| www.mim4u.org | web | https://www.mim4u.org | MIM4U www. |
| secure.mim4u.org | web | https://secure.mim4u.org | MIM4U secure portal. |
| training.mim4u.org | web | https://training.mim4u.org | MIM4U training site. |
| sankofa.nexus | web | https://sankofa.nexus | Sankofa Nexus root / web. |
| www.sankofa.nexus | web | https://www.sankofa.nexus | **301** to `https://sankofa.nexus` (canonical apex; NPM `advanced_config`). |
| phoenix.sankofa.nexus | web | https://phoenix.sankofa.nexus | Phoenix API (7800); E2E uses `/health` for HTTPS check. |
| www.phoenix.sankofa.nexus | web | https://www.phoenix.sankofa.nexus | **301** to `https://phoenix.sankofa.nexus` (canonical apex; NPM `advanced_config`). |
| the-order.sankofa.nexus | web | https://the-order.sankofa.nexus | OSJ management portal (secure auth); app **the_order** at `~/projects/the_order`. NPM upstream default: **order-haproxy** VMID **10210** `http://192.168.11.39:80` → portal **192.168.11.51:3000** (`provision-order-haproxy-10210.sh`). Override with `THE_ORDER_UPSTREAM_*` for direct portal if 10210 is down. |
| www.the-order.sankofa.nexus | web | https://www.the-order.sankofa.nexus | **301** to `https://the-order.sankofa.nexus` (canonical apex; NPM `advanced_config`). |
| studio.sankofa.nexus | web | https://studio.sankofa.nexus | Sankofa Studio (FusionAI Creator) at VMID 7805; app-owned **302** at `/` to `/studio/`. |
| keycloak.sankofa.nexus | web | https://keycloak.sankofa.nexus | Keycloak IdP (VMID 7802); client SSO for admin/portal. |
| admin.sankofa.nexus | web | https://admin.sankofa.nexus | Client SSO: access administration (hostname intent; NPM upstream TBD). |
| portal.sankofa.nexus | web | https://portal.sankofa.nexus | Client SSO: portal / marketplace (typical upstream VMID 7801). Add DNS + NPM row via `update-npmplus-proxy-hosts-api.sh`; NextAuth public URL `https://portal.sankofa.nexus`. |
| dash.sankofa.nexus | web | https://dash.sankofa.nexus | Operator systems dashboard (IP allowlist + MFA intent; upstream TBD). |
| docs.d-bis.org | web | https://docs.d-bis.org | Docs on explorer nginx where configured. |
| blockscout.defi-oracle.io | web | https://blockscout.defi-oracle.io | Generic Blockscout hostname (often VMID 5000); not canonical Chain 138 **explorer.d-bis.org**. |
| cacti-alltra.d-bis.org | web | https://cacti-alltra.d-bis.org | Cacti monitoring UI for Alltra. |
| cacti-hybx.d-bis.org | web | https://cacti-hybx.d-bis.org | Cacti monitoring UI for HYBX. |
| mifos.d-bis.org | web | https://mifos.d-bis.org | Mifos X / Fineract banking and microfinance platform (VMID 5800). |
| dapp.d-bis.org | web | https://dapp.d-bis.org | DApp frontend for Chain 138 bridge (VMID 5801). |
| gitea.d-bis.org | web | https://gitea.d-bis.org | Gitea org forge; NPM fourth upstream defaults to **VMID 104** (`IP_GITEA_INFRA`, HTTP **:80**). Optional: route hostname to dev VM **:3000** via `GITEA_PUBLIC_UPSTREAM_*` when running `update-npmplus-fourth-proxy-hosts.sh`. |
| dev.d-bis.org | web | https://dev.d-bis.org | Dev VM web / Codespaces entry. |
| codespaces.d-bis.org | web | https://codespaces.d-bis.org | Codespaces / dev environment entry. |
| rpc-http-pub.d-bis.org | rpc-http | https://rpc-http-pub.d-bis.org | Chain 138 public JSON-RPC HTTP (VMID 2201). |
| rpc-ws-pub.d-bis.org | rpc-ws | wss://rpc-ws-pub.d-bis.org | Chain 138 public JSON-RPC WebSocket. |
| rpc.d-bis.org | rpc-http | https://rpc.d-bis.org | Chain 138 RPC HTTP (alias). |
| rpc2.d-bis.org | rpc-http | https://rpc2.d-bis.org | Chain 138 RPC HTTP (second). |
| ws.rpc.d-bis.org | rpc-ws | wss://ws.rpc.d-bis.org | Chain 138 RPC WebSocket. |
| ws.rpc2.d-bis.org | rpc-ws | wss://ws.rpc2.d-bis.org | Chain 138 RPC WebSocket (second). |
| rpc-http-prv.d-bis.org | rpc-http | https://rpc-http-prv.d-bis.org | Chain 138 private/admin RPC HTTP (VMID 2101). |
| rpc-ws-prv.d-bis.org | rpc-ws | wss://rpc-ws-prv.d-bis.org | Chain 138 private RPC WebSocket. |
| rpc-fireblocks.d-bis.org | rpc-http | https://rpc-fireblocks.d-bis.org | Chain 138 RPC for Fireblocks Web3 (VMID 2301). |
| ws.rpc-fireblocks.d-bis.org | rpc-ws | wss://ws.rpc-fireblocks.d-bis.org | Chain 138 RPC WebSocket for Fireblocks. |
| rpc.public-0138.defi-oracle.io | rpc-http | https://rpc.public-0138.defi-oracle.io | Defi Oracle Chain 138 public RPC. |
| rpc.defi-oracle.io | rpc-http | https://rpc.defi-oracle.io | Defi Oracle RPC. |
| wss.defi-oracle.io | rpc-ws | wss://wss.defi-oracle.io | Defi Oracle RPC WebSocket. |
| info.defi-oracle.io | web | https://info.defi-oracle.io | Chain 138 info hub SPA (`/`, `/tokens`, `/pools`, `/swap`, `/routing`, `/governance`, `/ecosystem`, `/documentation`, `/solacenet`, `/agents`, `/disclosures`, `llms.txt`, `agent-hints.json`). **VMID 2410** (`192.168.11.218:80`); NPM `IP_INFO_DEFI_ORACLE_WEB`. Nginx **`/token-aggregation/`** → Blockscout. Publish: `provision-info-defi-oracle-web-lxc.sh` + `sync-info-defi-oracle-to-vmid2400.sh`. Verify: `pnpm run verify:info-defi-oracle-public`. |
| rpc-alltra.d-bis.org | rpc-http | https://rpc-alltra.d-bis.org | Alltra chain RPC HTTP. |
| rpc-alltra-2.d-bis.org | rpc-http | https://rpc-alltra-2.d-bis.org | Alltra chain RPC HTTP (2). |
| rpc-alltra-3.d-bis.org | rpc-http | https://rpc-alltra-3.d-bis.org | Alltra chain RPC HTTP (3). |
| rpc-hybx.d-bis.org | rpc-http | https://rpc-hybx.d-bis.org | HYBX chain RPC HTTP. |
| rpc-hybx-2.d-bis.org | rpc-http | https://rpc-hybx-2.d-bis.org | HYBX chain RPC HTTP (2). |
| rpc-hybx-3.d-bis.org | rpc-http | https://rpc-hybx-3.d-bis.org | HYBX chain RPC HTTP (3). |
### Planned DBIS institutional subdomains (multi-portal program)
Registered in `verify-end-to-end-routing.sh` as **optional-when-fail** until DNS and upstreams are live. Detail: [DBIS_INSTITUTIONAL_SUBDOMAINS.md](DBIS_INSTITUTIONAL_SUBDOMAINS.md), blueprint: [DBIS_WEB_AND_INSTITUTION_MASTER_BLUEPRINT.md](../02-architecture/DBIS_WEB_AND_INSTITUTION_MASTER_BLUEPRINT.md).
| Endpoint | Type | URL | Description |
|----------|------|-----|---------------|
| www.d-bis.org | web | https://www.d-bis.org | Optional **www** → apex **d-bis.org** redirect. |
| members.d-bis.org | web | https://members.d-bis.org | Member institution portal (OIDC BFF). |
| developers.d-bis.org | web | https://developers.d-bis.org | Developer hub; links to Gitea + OpenAPI. |
| data.d-bis.org | api | https://data.d-bis.org | Public data/API surface. Currently routed to the primary DBIS API node on VMID **10150** with `/v1/health` live. |
| research.d-bis.org | web | https://research.d-bis.org | Research and working papers. |
| policy.d-bis.org | web | https://policy.d-bis.org | Policy publications + manifests. |
| ops.d-bis.org | web | https://ops.d-bis.org | Staff operations (SSO). |
| identity.d-bis.org | web | https://identity.d-bis.org | Trust anchors + DID registry documentation/API. |
| status.d-bis.org | web | https://status.d-bis.org | Public status / SLOs. |
| sandbox.d-bis.org | web | https://sandbox.d-bis.org | Sandbox console (isolated test). |
| interop.d-bis.org | web | https://interop.d-bis.org | Interoperability lab (CBDC / cross-chain). |
## Endpoints by type
### Web
| Domain | URL |
|--------|-----|
| explorer.d-bis.org | https://explorer.d-bis.org |
| d-bis.org | https://d-bis.org |
| admin.d-bis.org | https://admin.d-bis.org |
| dbis-admin.d-bis.org | https://dbis-admin.d-bis.org |
| secure.d-bis.org | https://secure.d-bis.org |
| core.d-bis.org | https://core.d-bis.org |
| mim4u.org | https://mim4u.org |
| www.mim4u.org | https://www.mim4u.org |
| secure.mim4u.org | https://secure.mim4u.org |
| training.mim4u.org | https://training.mim4u.org |
| sankofa.nexus | https://sankofa.nexus |
| www.sankofa.nexus | https://www.sankofa.nexus |
| phoenix.sankofa.nexus | https://phoenix.sankofa.nexus |
| www.phoenix.sankofa.nexus | https://www.phoenix.sankofa.nexus |
| the-order.sankofa.nexus | https://the-order.sankofa.nexus |
| www.the-order.sankofa.nexus | https://www.the-order.sankofa.nexus |
| studio.sankofa.nexus | https://studio.sankofa.nexus |
| keycloak.sankofa.nexus | https://keycloak.sankofa.nexus |
| admin.sankofa.nexus | https://admin.sankofa.nexus |
| portal.sankofa.nexus | https://portal.sankofa.nexus |
| dash.sankofa.nexus | https://dash.sankofa.nexus |
| docs.d-bis.org | https://docs.d-bis.org |
| blockscout.defi-oracle.io | https://blockscout.defi-oracle.io |
| info.defi-oracle.io | https://info.defi-oracle.io |
| cacti-alltra.d-bis.org | https://cacti-alltra.d-bis.org |
| cacti-hybx.d-bis.org | https://cacti-hybx.d-bis.org |
| mifos.d-bis.org | https://mifos.d-bis.org |
| dapp.d-bis.org | https://dapp.d-bis.org |
| gitea.d-bis.org | https://gitea.d-bis.org |
| dev.d-bis.org | https://dev.d-bis.org |
| codespaces.d-bis.org | https://codespaces.d-bis.org |
### API
| Domain | URL |
|--------|-----|
| dbis-api.d-bis.org | https://dbis-api.d-bis.org |
| dbis-api-2.d-bis.org | https://dbis-api-2.d-bis.org |
| info.defi-oracle.io (token-aggregation) | https://info.defi-oracle.io/token-aggregation/api/v1/ (same-origin proxy to explorer token-aggregation service; SPA default API base) |
### RPC HTTP (public)
| Domain | URL |
|--------|-----|
| rpc-http-pub.d-bis.org | https://rpc-http-pub.d-bis.org |
| rpc.d-bis.org | https://rpc.d-bis.org |
| rpc2.d-bis.org | https://rpc2.d-bis.org |
| rpc.public-0138.defi-oracle.io | https://rpc.public-0138.defi-oracle.io |
| rpc.defi-oracle.io | https://rpc.defi-oracle.io |
| rpc-alltra.d-bis.org | https://rpc-alltra.d-bis.org |
| rpc-alltra-2.d-bis.org | https://rpc-alltra-2.d-bis.org |
| rpc-alltra-3.d-bis.org | https://rpc-alltra-3.d-bis.org |
| rpc-hybx.d-bis.org | https://rpc-hybx.d-bis.org |
| rpc-hybx-2.d-bis.org | https://rpc-hybx-2.d-bis.org |
| rpc-hybx-3.d-bis.org | https://rpc-hybx-3.d-bis.org |
### RPC WebSocket (public)
| Domain | URL |
|--------|-----|
| rpc-ws-pub.d-bis.org | wss://rpc-ws-pub.d-bis.org |
| ws.rpc.d-bis.org | wss://ws.rpc.d-bis.org |
| ws.rpc2.d-bis.org | wss://ws.rpc2.d-bis.org |
| wss.defi-oracle.io | wss://wss.defi-oracle.io |
### RPC HTTP (private/admin profile)
| Domain | URL |
|--------|-----|
| rpc-http-prv.d-bis.org | https://rpc-http-prv.d-bis.org |
| rpc-fireblocks.d-bis.org | https://rpc-fireblocks.d-bis.org |
### RPC WebSocket (private/admin profile)
| Domain | URL |
|--------|-----|
| rpc-ws-prv.d-bis.org | wss://rpc-ws-prv.d-bis.org |
| ws.rpc-fireblocks.d-bis.org | wss://ws.rpc-fireblocks.d-bis.org |
## Report content
After each run, the verification report includes:
1. **All endpoints** — table of every domain, type, and URL.
2. **Summary** — counts (DNS pass, HTTPS pass, failed, skipped) and average response time.
3. **Results overview** — table of each domain with DNS | SSL | HTTPS | RPC status.
4. **Test Results by Domain** — per-domain detail (DNS, SSL, HTTPS, Blockscout API, RPC).
Output directory: `docs/04-configuration/verification-evidence/e2e-verification-<timestamp>/`
Files: `verification_report.md`, `all_e2e_results.json`, `*_https_headers.txt`, `*_rpc_response.txt`.
## Known E2E warnings (public profile)
When running from outside LAN or when backends are down, the following endpoints commonly show **HTTPS warn** (not fail, due to `E2E_OPTIONAL_WHEN_FAIL`).
**Current status:** the latest `2026-04-02` public verifier passed with `DNS passed: 60`, `HTTPS passed: 44`, and `Failed: 0`. The table below is now a historical troubleshooting guide for regressions rather than an active failure list.
**2026-03-26 note:** after recovering NPMplus CT `10233` and re-running `update-npmplus-proxy-hosts-api.sh`, the latest public profile passed for all currently tested public domains, including Sankofa, Phoenix, Studio, The Order, DBIS, Mifos, and MIM4U.
**2026-03-29 update:** public profile passed again with `Failed: 0` after fixing the explorer `/api/v1` proxy, removing the stale `192.168.11.52` address from CT `10232`, and moving VMID `10092` off `192.168.11.37` so MIM4U owns that IP exclusively. Current evidence: `docs/04-configuration/verification-evidence/e2e-verification-20260329_170619/`.
| Endpoint | Typical cause |
|----------|----------------|
| admin.d-bis.org, dbis-admin.d-bis.org | Historical 502 when the DBIS frontend on VMID **10130** is down. Current fix path: restart `nginx` on 10130. |
| core.d-bis.org | Historical warning when 10150 served placeholder content. Current host is live and returns DBIS service metadata JSON from the primary API node. |
| dbis-api.d-bis.org, dbis-api-2.d-bis.org | Historical warning when 10150/10151 were placeholder servers or down. Current fix path: restart `dbis-api.service` on those CTs. |
| secure.d-bis.org | Historical 502 when the DBIS frontend on VMID **10130** is unreachable from public. |
| data.d-bis.org | Historical warning until `/v1/health` was implemented on `2026-04-02`. Current upstream is VMID **10150**. |
| mifos.d-bis.org | 502 — Mifos (VMID 5800) unreachable from public |
| mim4u.org, www.mim4u.org, secure.mim4u.org, training.mim4u.org | Resolved on 2026-03-29. If these regress to 502, first check for IP ownership conflicts on `192.168.11.37` before debugging nginx. |
| studio.sankofa.nexus | Historically 404/502 when the proxy misses `/studio/` or backend `192.168.11.72:8000`; current `2026-04-02` pass is clean. |
| phoenix.sankofa.nexus, www.phoenix.sankofa.nexus | (Resolved in verifier) Phoenix API (7800) is API-first; `verify-end-to-end-routing.sh` checks `https://…/health` (200), not `/`. A separate **marketing** site on the apex hostname (if desired) needs another upstream or app routes—NPM still points `phoenix.sankofa.nexus` at the Fastify API today. |
| the-order.sankofa.nexus | 502 if **10210** HAProxy or backend portal is down. NPM defaults upstream to **192.168.11.39:80** (order-haproxy). Fallback: `THE_ORDER_UPSTREAM_IP` / `THE_ORDER_UPSTREAM_PORT` = portal **192.168.11.51:3000** |
| keycloak.sankofa.nexus, admin.sankofa.nexus, portal.sankofa.nexus | Resolved again on `2026-04-02` after removing the duplicate `192.168.11.52` address from CT `10232` and validating the restart path. If these regress, verify ARP ownership of `192.168.11.52` first. |
| dash.sankofa.nexus | Still optional / unprovisioned. DNS/SSL/HTTPS may warn or skip until `IP_SANKOFA_DASH` and its app upstream are intentionally wired. |
| docs.d-bis.org, blockscout.defi-oracle.io | Same optional-when-fail behavior; **blockscout.defi-oracle.io** also runs optional `/api/v2/stats` like **explorer.d-bis.org**. |
| info.defi-oracle.io | **Origin:** dedicated **VMID 2410** (`192.168.11.218`). If the public hostname regresses, run `sync-info-defi-oracle-to-vmid2400.sh`, `update-npmplus-proxy-hosts-api.sh` (upstream `.218`), then optional DNS/tunnel helpers `set-info-defi-oracle-dns-to-vmid2400-tunnel.sh`. Do **not** deploy the SPA to VMID **2400** (ThirdWeb RPC). |
**Verifier behavior (2026-03):** `openssl s_client` is wrapped with `timeout` (`E2E_OPENSSL_TIMEOUT` default 15s, `E2E_OPENSSL_X509_TIMEOUT` default 5s) so `--profile=private` / `--profile=all` cannot hang. **`--profile=all`** merges private and public `E2E_OPTIONAL_WHEN_FAIL` lists for temporary regressions. Install **`wscat`** (`npm install -g wscat`) for full WSS JSON-RPC checks; the script uses `wscat -n` to match `curl -k`, and now treats a clean `wscat` exit as a successful full WebSocket check even when the tool prints no JSON output.
**Canonical www redirects (2026-03):** For `www.sankofa.nexus`, `www.phoenix.sankofa.nexus`, and `www.the-order.sankofa.nexus`, HTTP **301**/**308** must include a **`Location`** whose host matches the expected apex (`E2E_WWW_CANONICAL_BASE` in `verify-end-to-end-routing.sh`). Wrong apex → HTTPS **fail**. Missing `Location` → **warn**.
**Cloudflare bulk DNS:** `scripts/update-all-dns-to-public-ip.sh` supports **`--dry-run`** (no API calls) and **`--zone-only=sankofa.nexus`** (or `d-bis.org` | `mim4u.org` | `defi-oracle.io`) to limit blast radius. Env: `CLOUDFLARE_DNS_DRY_RUN=1`, `DNS_ZONE_ONLY=…`.
**WebSocket test-format warnings:** Older runs may show "connection established but RPC test failed" when `wscat` is used: the upgrade succeeded but the verifier expected printable `"result"` output. The script now accepts either explicit JSON output or a clean `wscat` exit, so current runs treat those WS checks as pass when the connection completes successfully. The script also accepts Chain 138 chainId `0x8a` in output.
### Remediation (when you want these to pass from public)
| Goal | Action |
|------|--------|
| **502s (dbis-admin, dbis-api, secure, mifos)** | From LAN: `./scripts/maintenance/address-all-remaining-502s.sh [--run-besu-fix] [--e2e]` or `./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e`. If NPMplus API is unreachable: `./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh`. Runbook: [502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md](../00-meta/502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md). |
| **404 studio.sankofa.nexus** | Ensure backend (VMID 7805, 192.168.11.72:8000) is up and NPMplus proxy for `studio.sankofa.nexus` points to it. See [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md), [SANKOFA_STUDIO_E2E_FLOW.md](../03-deployment/SANKOFA_STUDIO_E2E_FLOW.md), [SANKOFA_STUDIO_DEPLOYMENT.md](../03-deployment/SANKOFA_STUDIO_DEPLOYMENT.md). |
| **the-order 502** | Check **10210** HAProxy (`curl http://192.168.11.39:80/` with `Host: the-order.sankofa.nexus`) and portal **192.168.11.51:3000**. Re-provision: `bash scripts/deployment/provision-order-haproxy-10210.sh`. NPM refresh: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`. Direct portal bypass: `THE_ORDER_UPSTREAM_IP=192.168.11.51 THE_ORDER_UPSTREAM_PORT=3000` for that run. |
| **Historical April 2026 public regressions** | Generate a fresh domain-by-domain plan with `bash scripts/verify/generate-public-surface-remediation-plan.sh --print` if the public sweep regresses again. Canonical matrix: [PUBLIC_SURFACE_502_AND_DNS_REMEDIATION_MATRIX.md](PUBLIC_SURFACE_502_AND_DNS_REMEDIATION_MATRIX.md). |
Reference in New Issue View Git Blame Copy Permalink