dbd517b279
- Update dbis_core, cross-chain-pmm-lps, explorer-monorepo, metamask-integration, pr-workspace/chains - Omit embedded publish git dirs and empty placeholders from index Made-with: Cursor
23 KiB
E2E verification — endpoint inventory and profiles
Source: scripts/verify/verify-end-to-end-routing.sh (DOMAIN_TYPES).
List from CLI (public): ./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=public
List from CLI (private/admin): ./scripts/verify/verify-end-to-end-routing.sh --list-endpoints --profile=private
Run E2E (public profile recommended): ./scripts/verify/verify-end-to-end-routing.sh --profile=public (from LAN with DNS or use E2E_USE_SYSTEM_RESOLVER=1 and /etc/hosts per E2E_DNS_FROM_LAN_RUNBOOK.md).
Run E2E (private/admin): ./scripts/verify/verify-end-to-end-routing.sh --profile=private.
Gitea Actions (umbrella / cc-*): no stable unauthenticated REST for all Gitea versions — print UI URLs with ./scripts/verify/print-gitea-actions-urls.sh and confirm jobs in the browser after push.
What each hostname should present (operator narrative): FQDN_EXPECTED_CONTENT.md.
Latest verified public transport/TLS pass: 2026-04-02 via bash scripts/verify/verify-end-to-end-routing.sh --profile=public with report at verification_report.md. Result: exit 0, DNS passed: 60, Failed: 0, HTTPS passed: 44 — includes the repaired DBIS, Keycloak, Studio, info.defi-oracle.io, data.d-bis.org /v1/health, and both Cacti hostnames.
Resolved public regression snapshot: the earlier 2026-04-02 regression run at verification_report.md is now historical only; its DBIS, Keycloak, Cacti, Studio, and info.defi-oracle.io warnings were resolved later the same day.
2026-04-01 verifier tightening: verify-end-to-end-routing.sh rejects placeholder directory listings for core.d-bis.org, dbis-api.d-bis.org, and dbis-api-2.d-bis.org. The current 2026-04-02 pass succeeds because those hosts now serve the real DBIS API/runtime, not static placeholder content.
2026-04-04 explorer E2E tightening: the canonical explorer explorer.d-bis.org is no longer treated as optional-when-fail in the public profile. The explorer-specific verifier now also checks the static Visual Command Center and the live Mission Control stream, bridge trace, and liquidity endpoints.
Previous: 2026-03-29 — verification_report.md; older: 20260329_045210, 20260327.
Latest verified private/admin pass: 2026-03-27 via bash scripts/verify/verify-end-to-end-routing.sh --profile=private with report at verification_report.md. Result: exit 0, DNS passed: 4, Failed: 0.
Evidence folders: Each run creates verification-evidence/e2e-verification-YYYYMMDD_HHMMSS/. Commit the runs you want on record; older dirs can be removed locally to reduce noise (scripts/maintenance/prune-e2e-verification-evidence.sh --dry-run lists candidates). Routing truth is not inferred from old reports—use ALL_VMIDS_ENDPOINTS.md.
Verification profiles
- Public profile (default for routine E2E): web, api, public RPC endpoints.
- Private/admin profile: private RPC and Fireblocks RPC endpoints. Run separately for internal operations.
Full endpoint inventory (combined)
| Endpoint | Type | URL | Description (content provided) |
|---|---|---|---|
| explorer.d-bis.org | web | https://explorer.d-bis.org | Blockscout-style blockchain explorer for Chain 138: blocks, transactions, addresses, contracts, tokens, verification. |
| explorer.d-bis.org | web | https://explorer.d-bis.org/chain138-command-center.html | Chain 138 deployment topology — interactive Mermaid command center (tabs, keyboard, ?tab= / ?tab=mission-control slugs); static asset with More → Visual Command Center entry point. |
| d-bis.org | web | https://d-bis.org | Public DBIS web presence — institutional portal (Gov Portals Next app when deployed behind NPM). |
| admin.d-bis.org | web | https://admin.d-bis.org | Admin console for DBIS operations staff; typical upstream VMID 10130. |
| dbis-admin.d-bis.org | web | https://dbis-admin.d-bis.org | Legacy admin hostname; same upstream intent as admin.d-bis.org if still in DNS. |
| secure.d-bis.org | web | https://secure.d-bis.org | Member secure portal (authenticated institutions); path-based routing on 10130 per ALL_VMIDS_ENDPOINTS.md. |
| core.d-bis.org | web | https://core.d-bis.org | Current DBIS Core service root on VMID 10150. Public root returns service metadata JSON while the dedicated client UI cutover remains separate work. |
| dbis-api.d-bis.org | api | https://dbis-api.d-bis.org | Primary DBIS core API host on VMID 10150. Root /, /health, and /v1/health return live JSON responses. |
| dbis-api.d-bis.org | api | https://dbis-api.d-bis.org/api/v1/gateway/rails |
Authenticated — SolaceNet gateway rail adapter list (maintainer, adapters[]). Internal smoke: scripts/verify/check-dbis-core-gateway-rails.sh (DBIS_CORE_API_BASE, DBIS_CORE_BEARER_TOKEN). |
| dbis-api-2.d-bis.org | api | https://dbis-api-2.d-bis.org | Secondary DBIS core API host on VMID 10151 with the same root and health responses. |
| mim4u.org | web | https://mim4u.org | MIM4U main site. |
| www.mim4u.org | web | https://www.mim4u.org | MIM4U www. |
| secure.mim4u.org | web | https://secure.mim4u.org | MIM4U secure portal. |
| training.mim4u.org | web | https://training.mim4u.org | MIM4U training site. |
| sankofa.nexus | web | https://sankofa.nexus | Sankofa Nexus root / web. |
| www.sankofa.nexus | web | https://www.sankofa.nexus | 301 to https://sankofa.nexus (canonical apex; NPM advanced_config). |
| phoenix.sankofa.nexus | web | https://phoenix.sankofa.nexus | Phoenix API (7800); E2E uses /health for HTTPS check. |
| www.phoenix.sankofa.nexus | web | https://www.phoenix.sankofa.nexus | 301 to https://phoenix.sankofa.nexus (canonical apex; NPM advanced_config). |
| the-order.sankofa.nexus | web | https://the-order.sankofa.nexus | OSJ management portal (secure auth); app the_order at ~/projects/the_order. NPM upstream default: order-haproxy VMID 10210 http://192.168.11.39:80 → portal 192.168.11.51:3000 (provision-order-haproxy-10210.sh). Override with THE_ORDER_UPSTREAM_* for direct portal if 10210 is down. |
| www.the-order.sankofa.nexus | web | https://www.the-order.sankofa.nexus | 301 to https://the-order.sankofa.nexus (canonical apex; NPM advanced_config). |
| studio.sankofa.nexus | web | https://studio.sankofa.nexus | Sankofa Studio (FusionAI Creator) at VMID 7805; app-owned 302 at / to /studio/. |
| keycloak.sankofa.nexus | web | https://keycloak.sankofa.nexus | Keycloak IdP (VMID 7802); client SSO for admin/portal. |
| admin.sankofa.nexus | web | https://admin.sankofa.nexus | Client SSO: access administration (hostname intent; NPM upstream TBD). |
| portal.sankofa.nexus | web | https://portal.sankofa.nexus | Client SSO: portal / marketplace (typical upstream VMID 7801). Add DNS + NPM row via update-npmplus-proxy-hosts-api.sh; NextAuth public URL https://portal.sankofa.nexus. |
| dash.sankofa.nexus | web | https://dash.sankofa.nexus | Operator systems dashboard (IP allowlist + MFA intent; upstream TBD). |
| docs.d-bis.org | web | https://docs.d-bis.org | Docs on explorer nginx where configured. |
| blockscout.defi-oracle.io | web | https://blockscout.defi-oracle.io | Generic Blockscout hostname (often VMID 5000); not canonical Chain 138 explorer.d-bis.org. |
| cacti-alltra.d-bis.org | web | https://cacti-alltra.d-bis.org | Cacti monitoring UI for Alltra. |
| cacti-hybx.d-bis.org | web | https://cacti-hybx.d-bis.org | Cacti monitoring UI for HYBX. |
| mifos.d-bis.org | web | https://mifos.d-bis.org | Mifos X / Fineract banking and microfinance platform (VMID 5800). |
| dapp.d-bis.org | web | https://dapp.d-bis.org | DApp frontend for Chain 138 bridge (VMID 5801). |
| gitea.d-bis.org | web | https://gitea.d-bis.org | Gitea org forge; NPM fourth upstream defaults to VMID 104 (IP_GITEA_INFRA, HTTP :80). Optional: route hostname to dev VM :3000 via GITEA_PUBLIC_UPSTREAM_* when running update-npmplus-fourth-proxy-hosts.sh. |
| dev.d-bis.org | web | https://dev.d-bis.org | Dev VM web / Codespaces entry. |
| codespaces.d-bis.org | web | https://codespaces.d-bis.org | Codespaces / dev environment entry. |
| rpc-http-pub.d-bis.org | rpc-http | https://rpc-http-pub.d-bis.org | Chain 138 public JSON-RPC HTTP (VMID 2201). |
| rpc-ws-pub.d-bis.org | rpc-ws | wss://rpc-ws-pub.d-bis.org | Chain 138 public JSON-RPC WebSocket. |
| rpc.d-bis.org | rpc-http | https://rpc.d-bis.org | Chain 138 RPC HTTP (alias). |
| rpc2.d-bis.org | rpc-http | https://rpc2.d-bis.org | Chain 138 RPC HTTP (second). |
| ws.rpc.d-bis.org | rpc-ws | wss://ws.rpc.d-bis.org | Chain 138 RPC WebSocket. |
| ws.rpc2.d-bis.org | rpc-ws | wss://ws.rpc2.d-bis.org | Chain 138 RPC WebSocket (second). |
| rpc-http-prv.d-bis.org | rpc-http | https://rpc-http-prv.d-bis.org | Chain 138 private/admin RPC HTTP (VMID 2101). |
| rpc-ws-prv.d-bis.org | rpc-ws | wss://rpc-ws-prv.d-bis.org | Chain 138 private RPC WebSocket. |
| rpc-fireblocks.d-bis.org | rpc-http | https://rpc-fireblocks.d-bis.org | Chain 138 RPC for Fireblocks Web3 (VMID 2301). |
| ws.rpc-fireblocks.d-bis.org | rpc-ws | wss://ws.rpc-fireblocks.d-bis.org | Chain 138 RPC WebSocket for Fireblocks. |
| rpc.public-0138.defi-oracle.io | rpc-http | https://rpc.public-0138.defi-oracle.io | Defi Oracle Chain 138 public RPC. |
| rpc.defi-oracle.io | rpc-http | https://rpc.defi-oracle.io | Defi Oracle RPC. |
| wss.defi-oracle.io | rpc-ws | wss://wss.defi-oracle.io | Defi Oracle RPC WebSocket. |
| info.defi-oracle.io | web | https://info.defi-oracle.io | Chain 138 info hub SPA (/, /tokens, /pools, /swap, /routing, /governance, /ecosystem, /documentation, /solacenet, /agents, /disclosures, llms.txt, agent-hints.json). VMID 2410 (192.168.11.218:80); NPM IP_INFO_DEFI_ORACLE_WEB. Nginx /token-aggregation/ → Blockscout. Publish: provision-info-defi-oracle-web-lxc.sh + sync-info-defi-oracle-to-vmid2400.sh. Verify: pnpm run verify:info-defi-oracle-public. |
| rpc-alltra.d-bis.org | rpc-http | https://rpc-alltra.d-bis.org | Alltra chain RPC HTTP. |
| rpc-alltra-2.d-bis.org | rpc-http | https://rpc-alltra-2.d-bis.org | Alltra chain RPC HTTP (2). |
| rpc-alltra-3.d-bis.org | rpc-http | https://rpc-alltra-3.d-bis.org | Alltra chain RPC HTTP (3). |
| rpc-hybx.d-bis.org | rpc-http | https://rpc-hybx.d-bis.org | HYBX chain RPC HTTP. |
| rpc-hybx-2.d-bis.org | rpc-http | https://rpc-hybx-2.d-bis.org | HYBX chain RPC HTTP (2). |
| rpc-hybx-3.d-bis.org | rpc-http | https://rpc-hybx-3.d-bis.org | HYBX chain RPC HTTP (3). |
Planned DBIS institutional subdomains (multi-portal program)
Registered in verify-end-to-end-routing.sh as optional-when-fail until DNS and upstreams are live. Detail: DBIS_INSTITUTIONAL_SUBDOMAINS.md, blueprint: DBIS_WEB_AND_INSTITUTION_MASTER_BLUEPRINT.md.
| Endpoint | Type | URL | Description |
|---|---|---|---|
| www.d-bis.org | web | https://www.d-bis.org | Optional www → apex d-bis.org redirect. |
| members.d-bis.org | web | https://members.d-bis.org | Member institution portal (OIDC BFF). |
| developers.d-bis.org | web | https://developers.d-bis.org | Developer hub; links to Gitea + OpenAPI. |
| data.d-bis.org | api | https://data.d-bis.org | Public data/API surface. Currently routed to the primary DBIS API node on VMID 10150 with /v1/health live. |
| research.d-bis.org | web | https://research.d-bis.org | Research and working papers. |
| policy.d-bis.org | web | https://policy.d-bis.org | Policy publications + manifests. |
| ops.d-bis.org | web | https://ops.d-bis.org | Staff operations (SSO). |
| identity.d-bis.org | web | https://identity.d-bis.org | Trust anchors + DID registry documentation/API. |
| status.d-bis.org | web | https://status.d-bis.org | Public status / SLOs. |
| sandbox.d-bis.org | web | https://sandbox.d-bis.org | Sandbox console (isolated test). |
| interop.d-bis.org | web | https://interop.d-bis.org | Interoperability lab (CBDC / cross-chain). |
Endpoints by type
Web
API
| Domain | URL |
|---|---|
| dbis-api.d-bis.org | https://dbis-api.d-bis.org |
| dbis-api-2.d-bis.org | https://dbis-api-2.d-bis.org |
| info.defi-oracle.io (token-aggregation) | https://info.defi-oracle.io/token-aggregation/api/v1/ (same-origin proxy to explorer token-aggregation service; SPA default API base) |
RPC HTTP (public)
| Domain | URL |
|---|---|
| rpc-http-pub.d-bis.org | https://rpc-http-pub.d-bis.org |
| rpc.d-bis.org | https://rpc.d-bis.org |
| rpc2.d-bis.org | https://rpc2.d-bis.org |
| rpc.public-0138.defi-oracle.io | https://rpc.public-0138.defi-oracle.io |
| rpc.defi-oracle.io | https://rpc.defi-oracle.io |
| rpc-alltra.d-bis.org | https://rpc-alltra.d-bis.org |
| rpc-alltra-2.d-bis.org | https://rpc-alltra-2.d-bis.org |
| rpc-alltra-3.d-bis.org | https://rpc-alltra-3.d-bis.org |
| rpc-hybx.d-bis.org | https://rpc-hybx.d-bis.org |
| rpc-hybx-2.d-bis.org | https://rpc-hybx-2.d-bis.org |
| rpc-hybx-3.d-bis.org | https://rpc-hybx-3.d-bis.org |
RPC WebSocket (public)
| Domain | URL |
|---|---|
| rpc-ws-pub.d-bis.org | wss://rpc-ws-pub.d-bis.org |
| ws.rpc.d-bis.org | wss://ws.rpc.d-bis.org |
| ws.rpc2.d-bis.org | wss://ws.rpc2.d-bis.org |
| wss.defi-oracle.io | wss://wss.defi-oracle.io |
RPC HTTP (private/admin profile)
| Domain | URL |
|---|---|
| rpc-http-prv.d-bis.org | https://rpc-http-prv.d-bis.org |
| rpc-fireblocks.d-bis.org | https://rpc-fireblocks.d-bis.org |
RPC WebSocket (private/admin profile)
| Domain | URL |
|---|---|
| rpc-ws-prv.d-bis.org | wss://rpc-ws-prv.d-bis.org |
| ws.rpc-fireblocks.d-bis.org | wss://ws.rpc-fireblocks.d-bis.org |
Report content
After each run, the verification report includes:
- All endpoints — table of every domain, type, and URL.
- Summary — counts (DNS pass, HTTPS pass, failed, skipped) and average response time.
- Results overview — table of each domain with DNS | SSL | HTTPS | RPC status.
- Test Results by Domain — per-domain detail (DNS, SSL, HTTPS, Blockscout API, RPC).
Output directory: docs/04-configuration/verification-evidence/e2e-verification-<timestamp>/
Files: verification_report.md, all_e2e_results.json, *_https_headers.txt, *_rpc_response.txt.
Known E2E warnings (public profile)
When running from outside LAN or when backends are down, the following endpoints commonly show HTTPS warn (not fail, due to E2E_OPTIONAL_WHEN_FAIL).
Current status: the latest 2026-04-02 public verifier passed with DNS passed: 60, HTTPS passed: 44, and Failed: 0. The table below is now a historical troubleshooting guide for regressions rather than an active failure list.
2026-03-26 note: after recovering NPMplus CT 10233 and re-running update-npmplus-proxy-hosts-api.sh, the latest public profile passed for all currently tested public domains, including Sankofa, Phoenix, Studio, The Order, DBIS, Mifos, and MIM4U.
2026-03-29 update: public profile passed again with Failed: 0 after fixing the explorer /api/v1 proxy, removing the stale 192.168.11.52 address from CT 10232, and moving VMID 10092 off 192.168.11.37 so MIM4U owns that IP exclusively. Current evidence: docs/04-configuration/verification-evidence/e2e-verification-20260329_170619/.
| Endpoint | Typical cause |
|---|---|
| admin.d-bis.org, dbis-admin.d-bis.org | Historical 502 when the DBIS frontend on VMID 10130 is down. Current fix path: restart nginx on 10130. |
| core.d-bis.org | Historical warning when 10150 served placeholder content. Current host is live and returns DBIS service metadata JSON from the primary API node. |
| dbis-api.d-bis.org, dbis-api-2.d-bis.org | Historical warning when 10150/10151 were placeholder servers or down. Current fix path: restart dbis-api.service on those CTs. |
| secure.d-bis.org | Historical 502 when the DBIS frontend on VMID 10130 is unreachable from public. |
| data.d-bis.org | Historical warning until /v1/health was implemented on 2026-04-02. Current upstream is VMID 10150. |
| mifos.d-bis.org | 502 — Mifos (VMID 5800) unreachable from public |
| mim4u.org, www.mim4u.org, secure.mim4u.org, training.mim4u.org | Resolved on 2026-03-29. If these regress to 502, first check for IP ownership conflicts on 192.168.11.37 before debugging nginx. |
| studio.sankofa.nexus | Historically 404/502 when the proxy misses /studio/ or backend 192.168.11.72:8000; current 2026-04-02 pass is clean. |
| phoenix.sankofa.nexus, www.phoenix.sankofa.nexus | (Resolved in verifier) Phoenix API (7800) is API-first; verify-end-to-end-routing.sh checks https://…/health (200), not /. A separate marketing site on the apex hostname (if desired) needs another upstream or app routes—NPM still points phoenix.sankofa.nexus at the Fastify API today. |
| the-order.sankofa.nexus | 502 if 10210 HAProxy or backend portal is down. NPM defaults upstream to 192.168.11.39:80 (order-haproxy). Fallback: THE_ORDER_UPSTREAM_IP / THE_ORDER_UPSTREAM_PORT = portal 192.168.11.51:3000 |
| keycloak.sankofa.nexus, admin.sankofa.nexus, portal.sankofa.nexus | Resolved again on 2026-04-02 after removing the duplicate 192.168.11.52 address from CT 10232 and validating the restart path. If these regress, verify ARP ownership of 192.168.11.52 first. |
| dash.sankofa.nexus | Still optional / unprovisioned. DNS/SSL/HTTPS may warn or skip until IP_SANKOFA_DASH and its app upstream are intentionally wired. |
| docs.d-bis.org, blockscout.defi-oracle.io | Same optional-when-fail behavior; blockscout.defi-oracle.io also runs optional /api/v2/stats like explorer.d-bis.org. |
| info.defi-oracle.io | Origin: dedicated VMID 2410 (192.168.11.218). If the public hostname regresses, run sync-info-defi-oracle-to-vmid2400.sh, update-npmplus-proxy-hosts-api.sh (upstream .218), then optional DNS/tunnel helpers set-info-defi-oracle-dns-to-vmid2400-tunnel.sh. Do not deploy the SPA to VMID 2400 (ThirdWeb RPC). |
Verifier behavior (2026-03): openssl s_client is wrapped with timeout (E2E_OPENSSL_TIMEOUT default 15s, E2E_OPENSSL_X509_TIMEOUT default 5s) so --profile=private / --profile=all cannot hang. --profile=all merges private and public E2E_OPTIONAL_WHEN_FAIL lists for temporary regressions. Install wscat (npm install -g wscat) for full WSS JSON-RPC checks; the script uses wscat -n to match curl -k, and now treats a clean wscat exit as a successful full WebSocket check even when the tool prints no JSON output.
Canonical www redirects (2026-03): For www.sankofa.nexus, www.phoenix.sankofa.nexus, and www.the-order.sankofa.nexus, HTTP 301/308 must include a Location whose host matches the expected apex (E2E_WWW_CANONICAL_BASE in verify-end-to-end-routing.sh). Wrong apex → HTTPS fail. Missing Location → warn.
Cloudflare bulk DNS: scripts/update-all-dns-to-public-ip.sh supports --dry-run (no API calls) and --zone-only=sankofa.nexus (or d-bis.org | mim4u.org | defi-oracle.io) to limit blast radius. Env: CLOUDFLARE_DNS_DRY_RUN=1, DNS_ZONE_ONLY=….
WebSocket test-format warnings: Older runs may show "connection established but RPC test failed" when wscat is used: the upgrade succeeded but the verifier expected printable "result" output. The script now accepts either explicit JSON output or a clean wscat exit, so current runs treat those WS checks as pass when the connection completes successfully. The script also accepts Chain 138 chainId 0x8a in output.
Remediation (when you want these to pass from public)
| Goal | Action |
|---|---|
| 502s (dbis-admin, dbis-api, secure, mifos) | From LAN: ./scripts/maintenance/address-all-remaining-502s.sh [--run-besu-fix] [--e2e] or ./scripts/maintenance/run-all-maintenance-via-proxmox-ssh.sh --e2e. If NPMplus API is unreachable: ./scripts/maintenance/fix-npmplus-services-via-proxmox-ssh.sh. Runbook: 502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md. |
| 404 studio.sankofa.nexus | Ensure backend (VMID 7805, 192.168.11.72:8000) is up and NPMplus proxy for studio.sankofa.nexus points to it. See ALL_VMIDS_ENDPOINTS.md, SANKOFA_STUDIO_E2E_FLOW.md, SANKOFA_STUDIO_DEPLOYMENT.md. |
| the-order 502 | Check 10210 HAProxy (curl http://192.168.11.39:80/ with Host: the-order.sankofa.nexus) and portal 192.168.11.51:3000. Re-provision: bash scripts/deployment/provision-order-haproxy-10210.sh. NPM refresh: bash scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh. Direct portal bypass: THE_ORDER_UPSTREAM_IP=192.168.11.51 THE_ORDER_UPSTREAM_PORT=3000 for that run. |
| Historical April 2026 public regressions | Generate a fresh domain-by-domain plan with bash scripts/verify/generate-public-surface-remediation-plan.sh --print if the public sweep regresses again. Canonical matrix: PUBLIC_SURFACE_502_AND_DNS_REMEDIATION_MATRIX.md. |