# E2E verification — endpoint inventory and profiles

**Source:** `scripts/verify/verify-end-to-end-routing.sh` (DOMAIN_TYPES).  
**List from CLI (public):** `./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=public`  
**List from CLI (private/admin):** `./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=private`  
**Run E2E (public profile recommended):** `./scripts/verify/verify-end-to-end-routing.sh --profile=public` (from LAN with DNS or use `E2E_USE_SYSTEM_RESOLVER=1` and `/etc/hosts` per [E2E_DNS_FROM_LAN_RUNBOOK.md](E2E_DNS_FROM_LAN_RUNBOOK.md)).  
**Run E2E (private/admin):** `./scripts/verify/verify-end-to-end-routing.sh --profile=private`.  
**Gitea Actions (umbrella / cc-*):** no stable unauthenticated REST for all Gitea versions — print UI URLs with `./scripts/verify/print-gitea-actions-urls.sh` and confirm jobs in the browser after push.

**What each hostname should present (operator narrative):** [FQDN_EXPECTED_CONTENT.md](FQDN_EXPECTED_CONTENT.md).

**Latest verified public pass:** `2026-03-27` via `bash scripts/verify/verify-end-to-end-routing.sh --profile=public` with report at [verification_report.md](verification-evidence/e2e-verification-20260327_134032/verification_report.md). Result: exit `0`, `DNS passed: 38`, `Failed: 0`, `HTTPS passed: 19`, `Skipped / optional: 1` (after `run-all-operator-tasks-from-lan.sh` NPM sync; `rpc.defi-oracle.io` may log HTTP 405 on the verifier probe but stays non-failing for the profile).
**Latest verified private/admin pass:** `2026-03-27` via `bash scripts/verify/verify-end-to-end-routing.sh --profile=private` with report at [verification_report.md](verification-evidence/e2e-verification-20260327_134137/verification_report.md). Result: exit `0`, `DNS passed: 4`, `Failed: 0`.

**Evidence folders:** Each run creates `verification-evidence/e2e-verification-YYYYMMDD_HHMMSS/`. Commit the runs you want on record; older dirs can be removed locally to reduce noise (`scripts/maintenance/prune-e2e-verification-evidence.sh --dry-run` lists candidates). Routing truth is **not** inferred from old reports—use [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md).

## Verification profiles

- **Public profile (default for routine E2E):** web, api, public RPC endpoints.
- **Private/admin profile:** private RPC and Fireblocks RPC endpoints. Run separately for internal operations.

## Full endpoint inventory (combined)

| Endpoint | Type | URL | Description (content provided) |
|----------|------|-----|--------------------------------|
| explorer.d-bis.org | web | https://explorer.d-bis.org | Blockscout-style blockchain explorer for Chain 138: blocks, transactions, addresses, contracts, tokens, verification. |
| dbis-admin.d-bis.org | web | https://dbis-admin.d-bis.org | DBIS admin dashboard and frontend (VMID 10130). |
| secure.d-bis.org | web | https://secure.d-bis.org | Secure DBIS frontend / authenticated portal. |
| dbis-api.d-bis.org | api | https://dbis-api.d-bis.org | DBIS core API: token aggregation, Crypto.com OTC, exchange endpoints (VMID 10150). |
| dbis-api-2.d-bis.org | api | https://dbis-api-2.d-bis.org | DBIS API secondary instance (VMID 10151). |
| mim4u.org | web | https://mim4u.org | MIM4U main site. |
| www.mim4u.org | web | https://www.mim4u.org | MIM4U www. |
| secure.mim4u.org | web | https://secure.mim4u.org | MIM4U secure portal. |
| training.mim4u.org | web | https://training.mim4u.org | MIM4U training site. |
| sankofa.nexus | web | https://sankofa.nexus | Sankofa Nexus root / web. |
| www.sankofa.nexus | web | https://www.sankofa.nexus | **301** to `https://sankofa.nexus` (canonical apex; NPM `advanced_config`). |
| phoenix.sankofa.nexus | web | https://phoenix.sankofa.nexus | Phoenix API (7800); E2E uses `/health` for HTTPS check. |
| www.phoenix.sankofa.nexus | web | https://www.phoenix.sankofa.nexus | **301** to `https://phoenix.sankofa.nexus` (canonical apex; NPM `advanced_config`). |
| the-order.sankofa.nexus | web | https://the-order.sankofa.nexus | OSJ management portal (secure auth); app **the_order** at `~/projects/the_order`. NPM upstream default: **order-haproxy** VMID **10210** `http://192.168.11.39:80` → portal **192.168.11.51:3000** (`provision-order-haproxy-10210.sh`). Override with `THE_ORDER_UPSTREAM_*` for direct portal if 10210 is down. |
| www.the-order.sankofa.nexus | web | https://www.the-order.sankofa.nexus | **301** to `https://the-order.sankofa.nexus` (canonical apex; NPM `advanced_config`). |
| studio.sankofa.nexus | web | https://studio.sankofa.nexus | Sankofa Studio (FusionAI Creator) at VMID 7805. |
| keycloak.sankofa.nexus | web | https://keycloak.sankofa.nexus | Keycloak IdP (VMID 7802); client SSO for admin/portal. |
| admin.sankofa.nexus | web | https://admin.sankofa.nexus | Client SSO: access administration (hostname intent; NPM upstream TBD). |
| portal.sankofa.nexus | web | https://portal.sankofa.nexus | Client SSO: portal / marketplace (typical upstream VMID 7801). |
| dash.sankofa.nexus | web | https://dash.sankofa.nexus | Operator systems dashboard (IP allowlist + MFA intent; upstream TBD). |
| docs.d-bis.org | web | https://docs.d-bis.org | Docs on explorer nginx where configured. |
| blockscout.defi-oracle.io | web | https://blockscout.defi-oracle.io | Generic Blockscout hostname (often VMID 5000); not canonical Chain 138 **explorer.d-bis.org**. |
| cacti-alltra.d-bis.org | web | https://cacti-alltra.d-bis.org | Cacti monitoring UI for Alltra. |
| cacti-hybx.d-bis.org | web | https://cacti-hybx.d-bis.org | Cacti monitoring UI for HYBX. |
| mifos.d-bis.org | web | https://mifos.d-bis.org | Mifos X / Fineract banking and microfinance platform (VMID 5800). |
| dapp.d-bis.org | web | https://dapp.d-bis.org | DApp frontend for Chain 138 bridge (VMID 5801). |
| gitea.d-bis.org | web | https://gitea.d-bis.org | Gitea git repository and CI (Dev VM 5700). |
| dev.d-bis.org | web | https://dev.d-bis.org | Dev VM web / Codespaces entry. |
| codespaces.d-bis.org | web | https://codespaces.d-bis.org | Codespaces / dev environment entry. |
| rpc-http-pub.d-bis.org | rpc-http | https://rpc-http-pub.d-bis.org | Chain 138 public JSON-RPC HTTP (VMID 2201). |
| rpc-ws-pub.d-bis.org | rpc-ws | wss://rpc-ws-pub.d-bis.org | Chain 138 public JSON-RPC WebSocket. |
| rpc.d-bis.org | rpc-http | https://rpc.d-bis.org | Chain 138 RPC HTTP (alias). |
| rpc2.d-bis.org | rpc-http | https://rpc2.d-bis.org | Chain 138 RPC HTTP (second). |
| ws.rpc.d-bis.org | rpc-ws | wss://ws.rpc.d-bis.org | Chain 138 RPC WebSocket. |
| ws.rpc2.d-bis.org | rpc-ws | wss://ws.rpc2.d-bis.org | Chain 138 RPC WebSocket (second). |
| rpc-http-prv.d-bis.org | rpc-http | https://rpc-http-prv.d-bis.org | Chain 138 private/admin RPC HTTP (VMID 2101). |
| rpc-ws-prv.d-bis.org | rpc-ws | wss://rpc-ws-prv.d-bis.org | Chain 138 private RPC WebSocket. |
| rpc-fireblocks.d-bis.org | rpc-http | https://rpc-fireblocks.d-bis.org | Chain 138 RPC for Fireblocks Web3 (VMID 2301). |
| ws.rpc-fireblocks.d-bis.org | rpc-ws | wss://ws.rpc-fireblocks.d-bis.org | Chain 138 RPC WebSocket for Fireblocks. |
| rpc.public-0138.defi-oracle.io | rpc-http | https://rpc.public-0138.defi-oracle.io | Defi Oracle Chain 138 public RPC. |
| rpc.defi-oracle.io | rpc-http | https://rpc.defi-oracle.io | Defi Oracle RPC. |
| wss.defi-oracle.io | rpc-ws | wss://wss.defi-oracle.io | Defi Oracle RPC WebSocket. |
| rpc-alltra.d-bis.org | rpc-http | https://rpc-alltra.d-bis.org | Alltra chain RPC HTTP. |
| rpc-alltra-2.d-bis.org | rpc-http | https://rpc-alltra-2.d-bis.org | Alltra chain RPC HTTP (2). |
| rpc-alltra-3.d-bis.org | rpc-http | https://rpc-alltra-3.d-bis.org | Alltra chain RPC HTTP (3). |
| rpc-hybx.d-bis.org | rpc-http | https://rpc-hybx.d-bis.org | HYBX chain RPC HTTP. |
| rpc-hybx-2.d-bis.org | rpc-http | https://rpc-hybx-2.d-bis.org | HYBX chain RPC HTTP (2). |
| rpc-hybx-3.d-bis.org | rpc-http | https://rpc-hybx-3.d-bis.org | HYBX chain RPC HTTP (3). |

## Endpoints by type

### Web

| Domain | URL |
|--------|-----|
| explorer.d-bis.org | https://explorer.d-bis.org |
| dbis-admin.d-bis.org | https://dbis-admin.d-bis.org |
| secure.d-bis.org | https://secure.d-bis.org |
| mim4u.org | https://mim4u.org |
| www.mim4u.org | https://www.mim4u.org |
| secure.mim4u.org | https://secure.mim4u.org |
| training.mim4u.org | https://training.mim4u.org |
| sankofa.nexus | https://sankofa.nexus |
| www.sankofa.nexus | https://www.sankofa.nexus |
| phoenix.sankofa.nexus | https://phoenix.sankofa.nexus |
| www.phoenix.sankofa.nexus | https://www.phoenix.sankofa.nexus |
| the-order.sankofa.nexus | https://the-order.sankofa.nexus |
| www.the-order.sankofa.nexus | https://www.the-order.sankofa.nexus |
| studio.sankofa.nexus | https://studio.sankofa.nexus |
| keycloak.sankofa.nexus | https://keycloak.sankofa.nexus |
| admin.sankofa.nexus | https://admin.sankofa.nexus |
| portal.sankofa.nexus | https://portal.sankofa.nexus |
| dash.sankofa.nexus | https://dash.sankofa.nexus |
| docs.d-bis.org | https://docs.d-bis.org |
| blockscout.defi-oracle.io | https://blockscout.defi-oracle.io |
| cacti-alltra.d-bis.org | https://cacti-alltra.d-bis.org |
| cacti-hybx.d-bis.org | https://cacti-hybx.d-bis.org |
| mifos.d-bis.org | https://mifos.d-bis.org |
| dapp.d-bis.org | https://dapp.d-bis.org |
| gitea.d-bis.org | https://gitea.d-bis.org |
| dev.d-bis.org | https://dev.d-bis.org |
| codespaces.d-bis.org | https://codespaces.d-bis.org |

### API

| Domain | URL |
|--------|-----|
| dbis-api.d-bis.org | https://dbis-api.d-bis.org |
| dbis-api-2.d-bis.org | https://dbis-api-2.d-bis.org |

### RPC HTTP (public)

| Domain | URL |
|--------|-----|
| rpc-http-pub.d-bis.org | https://rpc-http-pub.d-bis.org |
| rpc.d-bis.org | https://rpc.d-bis.org |
| rpc2.d-bis.org | https://rpc2.d-bis.org |
| rpc.public-0138.defi-oracle.io | https://rpc.public-0138.defi-oracle.io |
| rpc.defi-oracle.io | https://rpc.defi-oracle.io |
| rpc-alltra.d-bis.org | https://rpc-alltra.d-bis.org |
| rpc-alltra-2.d-bis.org | https://rpc-alltra-2.d-bis.org |
| rpc-alltra-3.d-bis.org | https://rpc-alltra-3.d-bis.org |
| rpc-hybx.d-bis.org | https://rpc-hybx.d-bis.org |
| rpc-hybx-2.d-bis.org | https://rpc-hybx-2.d-bis.org |
| rpc-hybx-3.d-bis.org | https://rpc-hybx-3.d-bis.org |

### RPC WebSocket (public)

| Domain | URL |
|--------|-----|
| rpc-ws-pub.d-bis.org | wss://rpc-ws-pub.d-bis.org |
| ws.rpc.d-bis.org | wss://ws.rpc.d-bis.org |
| ws.rpc2.d-bis.org | wss://ws.rpc2.d-bis.org |
| wss.defi-oracle.io | wss://wss.defi-oracle.io |

### RPC HTTP (private/admin profile)

| Domain | URL |
|--------|-----|
| rpc-http-prv.d-bis.org | https://rpc-http-prv.d-bis.org |
| rpc-fireblocks.d-bis.org | https://rpc-fireblocks.d-bis.org |

### RPC WebSocket (private/admin profile)

| Domain | URL |
|--------|-----|
| rpc-ws-prv.d-bis.org | wss://rpc-ws-prv.d-bis.org |
| ws.rpc-fireblocks.d-bis.org | wss://ws.rpc-fireblocks.d-bis.org |

## Report content

After each run, the verification report includes:

1. **All endpoints** — table of every domain, type, and URL.
2. **Summary** — counts (DNS pass, HTTPS pass, failed, skipped) and average response time.
3. **Results overview** — table of each domain with DNS | SSL | HTTPS | RPC status.
4. **Test Results by Domain** — per-domain detail (DNS, SSL, HTTPS, Blockscout API, RPC).

Output directory: `docs/04-configuration/verification-evidence/e2e-verification-<timestamp>/`  
Files: `verification_report.md`, `all_e2e_results.json`, `*_https_headers.txt`, `*_rpc_response.txt`.

## Known E2E warnings (public profile)

When running from outside LAN or when backends are down, the following endpoints commonly show **HTTPS warn** (not fail, due to `E2E_OPTIONAL_WHEN_FAIL`).

**These known items do not block contract or pool completion.** Fix when convenient; E2E still passes when they are in `E2E_OPTIONAL_WHEN_FAIL`.

**2026-03-26 note:** after recovering NPMplus CT `10233` and re-running `update-npmplus-proxy-hosts-api.sh`, the latest public profile passed for all currently tested public domains, including Sankofa, Phoenix, Studio, The Order, DBIS, Mifos, and MIM4U.

| Endpoint | Typical cause |
|----------|----------------|
| dbis-admin.d-bis.org | 502 — backend (VMID 10130) unreachable from public |
| dbis-api.d-bis.org, dbis-api-2.d-bis.org | 502 — API backends (10150/10151) unreachable |
| secure.d-bis.org | 502 — secure portal backend unreachable |
| mifos.d-bis.org | 502 — Mifos (VMID 5800) unreachable from public |
| mim4u.org, www.mim4u.org, secure.mim4u.org, training.mim4u.org | 502 — MIM4U web backends (192.168.11.37:80); non-blocking for contract/pool |
| studio.sankofa.nexus | Historically 404 when the proxy misses `/studio/` or backend `192.168.11.72:8000`; verifier checks `/studio/`. Passed on 2026-03-26 after the NPMplus host update |
| phoenix.sankofa.nexus, www.phoenix.sankofa.nexus | (Resolved in verifier) Phoenix API (7800) is API-first; `verify-end-to-end-routing.sh` checks `https://…/health` (200), not `/`. A separate **marketing** site on the apex hostname (if desired) needs another upstream or app routes—NPM still points `phoenix.sankofa.nexus` at the Fastify API today. |
| the-order.sankofa.nexus | 502 if **10210** HAProxy or backend portal is down. NPM defaults upstream to **192.168.11.39:80** (order-haproxy). Fallback: `THE_ORDER_UPSTREAM_IP` / `THE_ORDER_UPSTREAM_PORT` = portal **192.168.11.51:3000** |
| keycloak.sankofa.nexus, admin.sankofa.nexus, portal.sankofa.nexus, dash.sankofa.nexus | DNS/SSL/HTTPS **warn** or **skip** when NPM or backends are unwired; listed in `E2E_OPTIONAL_WHEN_FAIL` so the public profile still exits **0**. |
| docs.d-bis.org, blockscout.defi-oracle.io | Same optional-when-fail behavior; **blockscout.defi-oracle.io** also runs optional `/api/v2/stats` like **explorer.d-bis.org**. |

**Verifier behavior (2026-03):** `openssl s_client` is wrapped with `timeout` (`E2E_OPENSSL_TIMEOUT` default 15s, `E2E_OPENSSL_X509_TIMEOUT` default 5s) so `--profile=private` / `--profile=all` cannot hang. **`--profile=all`** merges private and public `E2E_OPTIONAL_WHEN_FAIL` lists for temporary regressions. Install **`wscat`** (`npm install -g wscat`) for full WSS JSON-RPC checks; the script uses `wscat -n` to match `curl -k`, and now treats a clean `wscat` exit as a successful full WebSocket check even when the tool prints no JSON output.

**Canonical www redirects (2026-03):** For `www.sankofa.nexus`, `www.phoenix.sankofa.nexus`, and `www.the-order.sankofa.nexus`, HTTP **301**/**308** must include a **`Location`** whose host matches the expected apex (`E2E_WWW_CANONICAL_BASE` in `verify-end-to-end-routing.sh`). Wrong apex → HTTPS **fail**. Missing `Location` → **warn**.

**Cloudflare bulk DNS:** `scripts/update-all-dns-to-public-ip.sh` supports **`--dry-run`** (no API calls) and **`--zone-only=sankofa.nexus`** (or `d-bis.org` | `mim4u.org` | `defi-oracle.io`) to limit blast radius. Env: `CLOUDFLARE_DNS_DRY_RUN=1`, `DNS_ZONE_ONLY=…`.

**WebSocket test-format warnings:** Older runs may show "connection established but RPC test failed" when `wscat` is used: the upgrade succeeded but the verifier expected printable `"result"` output. The script now accepts either explicit JSON output or a clean `wscat` exit, so current runs treat those WS checks as pass when the connection completes successfully. The script also accepts Chain 138 chainId `0x8a` in output.

### Remediation (when you want these to pass from public)

| Goal | Action |
|------|--------|
| **502s (dbis-admin, dbis-api, secure, mifos)** | From LAN: `./scripts/maintenance/address-all-remaining-502s.sh [--run-besu-fix] [--e2e]` or `./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e`. If NPMplus API is unreachable: `./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh`. Runbook: [502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md](../00-meta/502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md). |
| **404 studio.sankofa.nexus** | Ensure backend (VMID 7805, 192.168.11.72:8000) is up and NPMplus proxy for `studio.sankofa.nexus` points to it. See [ALL_VMIDS_ENDPOINTS.md](ALL_VMIDS_ENDPOINTS.md), [SANKOFA_STUDIO_E2E_FLOW.md](../03-deployment/SANKOFA_STUDIO_E2E_FLOW.md), [SANKOFA_STUDIO_DEPLOYMENT.md](../03-deployment/SANKOFA_STUDIO_DEPLOYMENT.md). |
| **the-order 502** | Check **10210** HAProxy (`curl http://192.168.11.39:80/` with `Host: the-order.sankofa.nexus`) and portal **192.168.11.51:3000**. Re-provision: `bash scripts/deployment/provision-order-haproxy-10210.sh`. NPM refresh: `bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh`. Direct portal bypass: `THE_ORDER_UPSTREAM_IP=192.168.11.51 THE_ORDER_UPSTREAM_PORT=3000` for that run. |