CurrenciCombo/scripts/deployment/README.md

# CurrenciCombo — Phoenix / systemd deployment

This directory holds everything needed to deploy CurrenciCombo onto a
systemd host — starting with Phoenix CT 8604 on `r630-01`, but any
Debian/Ubuntu (or Alpine) host with Postgres + Redis available works.

The files here are **target-agnostic**. They hardcode no IPs, hostnames,
or VLANs. Environment-specific values — `curucombo.曼李.com`, the
`10.160.0.14` VIP, the NPMplus reverse proxy — are applied at the
edge (NPMplus) and at `/etc/currencicombo/orchestrator.env`, never in
the repo.

## Architecture on CT 8604

```
                                   ┌────────────────────┐
   curucombo.曼李.com  ──▶ NPMplus │192.168.11.167      │
   (Cloudflare-proxied)            │ TLS terminates here│
                                   └─────────┬──────────┘
                                             │
                      ┌──────────────────────┴──────────────────────┐
                      │                                             │
                      ▼                                             ▼
       curucombo.曼李.com/* (default)                   curucombo.曼李.com/api/*
                                                (incl. SSE /api/plans/*/events/stream)
                      │                                             │
             CT 8604  │10.160.0.14:3000                    CT 8604  │10.160.0.14:8080
                      ▼                                             ▼
       ┌─────────────────────────────┐               ┌─────────────────────────────┐
       │ currencicombo-webapp.service │               │ currencicombo-orchestrator  │
       │  nginx → /opt/currencicombo/ │               │  .service (systemd)          │
       │          webapp/dist/        │               │  node dist/index.js          │
       └─────────────────────────────┘               │  env /etc/currencicombo/     │
                                                      │       orchestrator.env       │
                                                      └──────────────┬──────────────┘
                                                                     │
                                                                     ▼
                                                 postgresql + redis (same CT, local)
```

## Files

| path | purpose |
|---|---|
| `systemd/currencicombo-orchestrator.service` | Node orchestrator, reads `/etc/currencicombo/orchestrator.env` |
| `systemd/currencicombo-webapp.service` | nginx serving the Vite SPA on `:3000` |
| `webapp-nginx.conf` | full nginx.conf for the webapp unit |
| `.env.prod.example` | env template installed to `/etc/currencicombo/orchestrator.env` |
| `install.sh` | one-shot host setup: user / dirs / DB role / systemd units / first-run key handoff file |
| `install-prune-cron.sh` | opt-in daily cron that prunes `/var/lib/currencicombo/backups/` (30-day retention, keep-min 5) |
| `deploy-currencicombo-8604.sh` | build-and-swap deploy driver (the script Phoenix/proxmox deploy-api calls) |
| `README.md` | you're reading it |

## First-time setup on CT 8604

All commands run as **root** inside the CT.

1. Ensure Postgres + Redis are installed and running:
   ```
   apt-get install -y postgresql redis-server
   systemctl enable --now postgresql redis-server
   ```
2. Clone the repo into its staging location (once):
   ```
   install -d -o root -g root /var/lib/currencicombo
   git clone https://gitea.d-bis.org/d-bis/CurrenciCombo.git /var/lib/currencicombo/repo
   ```
3. Run `install.sh` (creates user, DB, systemd units, env file):
   ```
   bash /var/lib/currencicombo/repo/scripts/deployment/install.sh
   ```
   On success you'll see:
   ```
   [install] generated EVENT_SIGNING_SECRET (64 hex)
   [install] generated 3 API keys (initiator/settler/auditor)
   [install] initial secrets written to /root/currencicombo-first-keys.txt (0600) — record in password manager, then 'shred -u /root/currencicombo-first-keys.txt'
   [install] install complete.
   ```
   `install.sh` writes the three API keys + `EVENT_SIGNING_SECRET` to **two** places:
   - `/etc/currencicombo/orchestrator.env` — canonical, read by systemd (`0640`, owned by `currencicombo`).
   - `/root/currencicombo-first-keys.txt` — **root-only handoff file** (`0600`). Grab it once, record the values in your password manager, then `shred -u` it.
   The handoff file is **not** regenerated on re-run — if `orchestrator.env` already exists, `install.sh` does not produce new secrets.
4. (Optional) Install the backup-pruning cron:
   ```
   bash /var/lib/currencicombo/repo/scripts/deployment/install-prune-cron.sh
   ```
   Drops a `/etc/cron.daily/currencicombo-prune-backups` that deletes anything under `/var/lib/currencicombo/backups/` older than 30 days while **always keeping the newest 5** regardless of age. Safe on re-run; opt out with `sudo rm /etc/cron.daily/currencicombo-prune-backups`.
5. If you need to resolve any `EXT-*` blocker (e.g. point at a real dbis_core), edit `/etc/currencicombo/orchestrator.env` before the first deploy.
6. First build-and-start:
   ```
   bash /var/lib/currencicombo/repo/scripts/deployment/deploy-currencicombo-8604.sh
   ```
   Expected tail:
   ```
   [deploy] orchestrator ready: {"ready":true}
   [deploy] portal OK (HTTP 200)
   [deploy] EXT-* blocker summary from orchestrator boot log:
              [ExternalBlockers] 6 active, 1 resolved
              id: EXT-DBIS-CORE
              id: EXT-CC-PAYMENT-ADAPTERS
              ...
              id: EXT-CHAIN138-CI-RPC  (resolved)
   [deploy] deploy complete. ref=main sha=<short> ts=<timestamp>
   ```

## NPMplus ingress changes required at cutover

`curucombo.曼李.com` today proxies 100% to `10.160.0.14:3000`. After
cutover it must become a **single-origin path-routed proxy** with **two**
rules (the SSE endpoint lives at `/api/plans/:id/events/stream`, so it's
already under `/api/*` — no separate `/events/*` rule is needed):

| location | upstream | proxy settings |
|---|---|---|
| `/api/*` | `http://10.160.0.14:8080` | **SSE-friendly settings apply here because the SSE route `/api/plans/:id/events/stream` is under /api/**. Set: `proxy_http_version 1.1;`, `proxy_set_header Connection "";`, `proxy_buffering off;`, `proxy_cache off;`, `proxy_read_timeout 24h;`, `proxy_send_timeout 24h;`. Standard forwarding: `proxy_set_header Host $host;`, `X-Real-IP $remote_addr;`, `X-Forwarded-For $proxy_add_x_forwarded_for;`, `X-Forwarded-Proto $scheme;`. The slight overhead of `proxy_buffering off` on plain REST calls is negligible for this workload. |
| `/` | `http://10.160.0.14:3000` | Vite SPA. Default upstream. No special settings. |

If you skip the `/api/*` rule, the nginx in `webapp-nginx.conf`
intentionally returns `HTTP 421` for that path — a clean "upstream is
misconfigured" signal instead of silently returning `index.html` and
breaking the browser with a JSON parse error.

## Subsequent deploys

Every deploy after the first is just:

```
sudo /var/lib/currencicombo/repo/scripts/deployment/deploy-currencicombo-8604.sh
```

Flags:
- `--ref=<branch-or-sha>` — deploy something other than `main`.
- `--dry-run` — print what would happen, don't touch anything.
- `--skip-migrate` — hotfix deploys that don't change the schema.
- `--skip-build` — reuse the build from the previous run (debugging only).
- `--rollback` — restore the most recent `/var/lib/currencicombo/backups/<ts>/` and restart units. Does **not** git-pull or rebuild.

Every deploy writes a timestamped backup to
`/var/lib/currencicombo/backups/<YYYYmmdd-HHMMSS>/` before swapping. Pruning is opt-in via `install-prune-cron.sh` (30-day retention, keep-min 5). Without the cron, backups accumulate forever — quietly filling `/var/lib` is how the next outage starts.

## Failure handling on deploy

**Rollback is manual.** `deploy-currencicombo-8604.sh` **does not** auto-restore the previous backup if the orchestrator fails to become ready. First cutovers typically fail because of env typos or migration mistakes, and auto-restoring hides the failure state ops needs.

Instead, on a readiness timeout the deploy script prints:
- last 40 lines of `journalctl -u currencicombo-orchestrator`
- last 20 lines of `journalctl -u currencicombo-webapp`
- **the exact `--rollback` command with the specific backup path filled in**

Example tail on failure:
```
================================================================
DEPLOY FAILED: orchestrator did not become ready after 60s
================================================================

## currencicombo-orchestrator (last 40 lines):
... env validation error: EVENT_SIGNING_SECRET is required ...

## Units are in whatever state deploy left them. To restore
## the previous build (does NOT revert DB migrations):

    sudo /var/lib/currencicombo/repo/scripts/deployment/deploy-currencicombo-8604.sh --rollback
    # (will restore /var/lib/currencicombo/backups/20260423-140215)

================================================================
```

Rollback one-liner (when ops has decided to restore):
```
sudo /var/lib/currencicombo/repo/scripts/deployment/deploy-currencicombo-8604.sh --rollback
```

Rollback restores the most recent backup and restarts both units. It **does not** touch the DB. If the failed deploy applied a new migration, DB rollback is a manual `psql` task — the orchestrator's migration runner only emits `up()` paths.

## Post-cutover smoke checks through NPMplus

Once the NPMplus `/api/*` rule is live, from a workstation (not the CT):

```
# 1. Front-door TLS is healthy
curl -skI https://curucombo.xn--vov0g.com/ | head -3
#   expect: HTTP/2 200
#   expect: NO 'x-nextjs-prerender' header (that was the old Next.js build)

# 2. SPA is the new Vite portal
curl -sk https://curucombo.xn--vov0g.com/ | grep -oE '<title>[^<]+</title>'
#   expect: <title>Solace Bank Group PLC — Treasury Management Portal</title>

# 3. Orchestrator ready through NPMplus
curl -sk https://curucombo.xn--vov0g.com/api/ready | head -1
#   expect: {"ready":true}   (not HTML)

# 4. Orchestrator blocker log (through CT shell, not NPMplus)
ssh root@10.160.0.14 'journalctl -u currencicombo-orchestrator -n 200 | grep -E "ExternalBlockers|EXT-"'
#   expect: [ExternalBlockers] 6 active, 1 resolved
#   expect: one line per EXT-* id

# 5. SSE actually streams (catches silent NPMplus proxy_buffering=on misconfig)
curl -sk -N --max-time 5 -H 'Accept: text/event-stream' \
  https://curucombo.xn--vov0g.com/api/plans/demo-pay-014/events/stream \
  | head -20 || true
#   expect: HTTP/2 200 with Content-Type: text/event-stream
#   expect: at least one 'data: {...}\n\n' frame to arrive WITHIN ~1s
#   if you see nothing for 3-5s and then everything dumps at once:
#     NPMplus has proxy_buffering=on. Fix: proxy_buffering off; proxy_http_version 1.1; proxy_set_header Connection "";
#   if the ping is 401/403: expected — SSE is auth-gated; the point is to
#     prove the request REACHED the orchestrator (content-type header +
#     chunked response headers) rather than hitting the Vite SPA.
```

A plain `HTTP/2 200` with a `Content-Type: text/html` body on `/api/ready` means NPMplus is silently falling back to the `/` rule — the `/api/*` rule is missing or ordered wrong. The `webapp-nginx.conf` in this repo returns `HTTP 421` for `/api/*` to make that case obvious when debugging CT-locally, but at the NPMplus edge nginx serves whatever NPMplus routes to it.

## Troubleshooting

| symptom | cause / check |
|---|---|
| `/api/*` returns `421 NPMplus is misconfigured` | NPMplus `/api/*` rule missing or wrong upstream. |
| `/events/*` connects then disconnects after ~60s | NPMplus forgot `proxy_buffering off` + high `proxy_read_timeout`. |
| orchestrator unit enters `activating (auto-restart)` loop | `journalctl -u currencicombo-orchestrator -n 80` — usually a zod env-validation error. The boot-time assertion message names the missing/invalid var. |
| orchestrator boot log says `[ExternalBlockers] N active` where N > 6 | you added an `EXT-*` env var without also updating the central registry in `orchestrator/src/config/externalBlockers.ts`. |
| `/health` returns 503 but `/ready` is 200 | memory `critical` is a separate signal from readiness. Inspect CT memory; this happens on constrained builders and is not a deploy bug. |
| portal page loads but MetaMask login does nothing | the portal couldn't reach `/api/auth/*`. Walk back up the NPMplus rule chain. |

## Cutting over from the pre-existing Next.js build

Phoenix previously had an older Next.js "ISO-20022 Combo Flow" app in
`/opt/currencicombo/webapp`. The cutover sequence on CT 8604 is:

1. **Backup the old install** out-of-band:
   ```
   tar czf /root/currencicombo-preRepo-$(date +%s).tgz /opt/currencicombo /etc/currencicombo 2>/dev/null || true
   ```
2. **Disable the pre-existing systemd units** (they're the same names but point at the old tree):
   ```
   systemctl stop currencicombo-webapp currencicombo-orchestrator
   systemctl disable currencicombo-webapp currencicombo-orchestrator
   ```
3. Run `install.sh` (writes the new units, new nginx, new env). On an already-set-up host this is idempotent: it preserves `/etc/currencicombo/orchestrator.env` if it already exists.
4. Run `deploy-currencicombo-8604.sh`.
5. Apply the NPMplus `/api` + `/events` path rules.
6. Smoke from outside the CT: `curl -skI https://curucombo.xn--vov0g.com/ && curl -sk https://curucombo.xn--vov0g.com/api/ready`.

## Proxmox-side follow-up (not in this PR)

After this PR merges and the above cutover runs cleanly, the
`/home/intlc/projects/proxmox` repo needs a separate commit to:

- Update `phoenix-deploy-api/deploy-targets.json` to point at:
  - repo: `d-bis/CurrenciCombo`
  - branch: `main`
  - target: `default`
  - deploy entrypoint: `scripts/deployment/deploy-currencicombo-8604.sh`
- Remove any stale `/opt/currencicombo/webapp` Next.js references.
- Drop any description of `ignoreBuildErrors: true` in `webapp/next.config.ts` — the new webapp is Vite+tsc-strict, no build-error suppression.