smom-dbis-138/services/token-aggregation/README.md

# Token Aggregation Service

A comprehensive token aggregation service that indexes token info, volume, liquidity, and market signals from on-chain data and enriches with CoinGecko, CoinMarketCap, and DexScreener APIs for ChainID 138 (DeFi Oracle Meta Mainnet) and ChainID 651940 (ALL Mainnet).

**REST API reference:** [docs/REST_API_REFERENCE.md](docs/REST_API_REFERENCE.md) — tokens, pools, prices, volume, OHLCV for dApps and MetaMask Snap discovery.

**Chain 138 Snap:** The MetaMask Chain 138 Snap (companion site at e.g. https://explorer.d-bis.org/snap/) calls this service for market data, swap quotes, and bridge routes. If the Snap is built with `GATSBY_SNAP_API_BASE_URL=https://explorer.d-bis.org`, then explorer.d-bis.org must serve this API (e.g. proxy `/api/v1/*` to this service). Otherwise build the Snap site with `GATSBY_SNAP_API_BASE_URL` set to this service’s public URL. See [metamask-integration/chain138-snap/docs/CHAIN138_SNAP_TROUBLESHOOTING.md](~/projects/metamask-integration/chain138-snap/docs/CHAIN138_SNAP_TROUBLESHOOTING.md). **CORS:** The service uses `cors()` (all origins allowed by default) so MetaMask Snap and browser clients can fetch token list and networks.

## Features

- **Chain-Native Indexing**: Indexes tokens, DEX pools, and swap events directly from blockchain
- **External API Enrichment**: Enriches data with CoinGecko, CoinMarketCap, and DexScreener
- **Multi-DEX Support**: Supports UniswapV2, UniswapV3, and DODO PMM protocols
- **Chain 138 PMM quotes**: `GET /api/v1/quote` runs on-chain `querySellBase` / `querySellQuote` when RPC is configured (`RPC_URL_138` or `TOKEN_AGGREGATION_*` in `.env.example`); JSON includes `quoteEngine` (`pmm-onchain` vs `constant-product`)
- **OHLCV Data**: Generates Open, High, Low, Close, Volume data for price charts
- **Volume Analytics**: Calculates 5m, 1h, 24h, 7d, 30d volume metrics
- **REST API**: Unified REST API for all token data

## Architecture

### Three-Layer Design

1. **Layer 1: Chain-Native Indexer** - Source of truth from on-chain data
2. **Layer 2: External Enrichment Adapters** - Best-effort enrichment from external APIs
3. **Layer 3: Unified REST API** - Single API endpoint for consumption

## Prerequisites

- Node.js 20+
- PostgreSQL 14+ with TimescaleDB extension
- Access to RPC endpoints for ChainID 138 and 651940
- (Optional) API keys for CoinGecko, CoinMarketCap, DexScreener

## Installation

1. Clone the repository and navigate to the service directory:
```bash
cd smom-dbis-138/services/token-aggregation
```

2. Install dependencies:
```bash
npm install
```

3. Copy environment file:
```bash
cp .env.example .env
```

4. Configure environment variables in `.env` (see `.env.example` for full list):
```bash
# Chain RPCs
# Explorer-side/LAN deployment: use the public RPC node directly.
CHAIN_138_RPC_URL=http://192.168.11.221:8545
# External-only deployments may use: https://rpc-http-pub.d-bis.org
CHAIN_651940_RPC_URL=https://mainnet-rpc.alltra.global

# Database
DATABASE_URL=postgresql://user:password@localhost:5432/explorer_db

# External APIs (optional) — use placeholders in .env.example; get keys from provider dashboards
COINGECKO_API_KEY=your_key_here
COINMARKETCAP_API_KEY=your_key_here
DEXSCREENER_API_KEY=your_key_here
```

For Chain 138 there is an important split:
- Explorer, Blockscout, token-aggregation, and other read-mostly services should use the public RPC node on `192.168.11.221:8545` when they run on the LAN.
- External wallets and dApps should use `https://rpc-http-pub.d-bis.org`.
- Operator/deploy workflows use the core RPC `http://192.168.11.211:8545`, but that core endpoint is not for explorer/read traffic.

**Canonical token addresses (report API):** Set per-chain env vars for tokens you want in `/api/v1/report/*`. Required/minimal for report: `CUSDC_ADDRESS_138`, `CUSDT_ADDRESS_138` (Chain 138); optionally `CUSDC_ADDRESS_651940`, `CUSDT_ADDRESS_651940` for Chain 651940. **Chain 138 compliant fiat** (cEURC, cEURT, cGBPC, cGBPT, cAUDC, cJPYC, cCHFC, cCADC, cXAUC, cXAUT) have **fallback addresses** in `src/config/canonical-tokens.ts` (DeployCompliantFiatTokens 2026-02-27); they are included in the report without env. The live **ALL Mainnet AUSDT corridor** is also surfaced there as `cAUSDT` on Chain 138 plus `cWAUSDT` mirrors on BSC, Polygon, Avalanche, and Celo. The planned **ALL Mainnet gold corridor** is env-gated under `CAXAUC_ADDRESS_651940`, `CAXAUT_ADDRESS_651940`, `CWAXAUC_ADDRESS_651940`, and `CWAXAUT_ADDRESS_651940`. Other symbols (USDW, acUSDC, vdcUSDC, sdcUSDC, etc.) — see `.env.example`. Unset tokens (with no fallback) are omitted from the report.

### Required environment variables (canonical tokens — Blitzkrieg Step 1/9)

The canonical token list is defined in `src/config/canonical-tokens.ts` and is the single source of truth for the Token Aggregation API and report endpoints. Addresses are read from environment variables; unset tokens are omitted from `getCanonicalTokensByChain` and report output.

| Purpose | Env var pattern | Example |
|--------|------------------|--------|
| Chain 138 | `{SYMBOL}_ADDRESS_138` (symbol with `-` → `_`, uppercase) | `CUSDC_ADDRESS_138`, `CUSDT_ADDRESS_138`, `CAUSDT_ADDRESS_138`, `USDW_ADDRESS_138`, `ACUSDC_ADDRESS_138`, `VDCUSDC_ADDRESS_138`, `SDCUSDC_ADDRESS_138` |
| Chain 651940 (ALL Mainnet) | `{SYMBOL}_ADDRESS_651940` | `CUSDC_ADDRESS_651940`, `CUSDT_ADDRESS_651940`, `AUSDT_ADDRESS_651940`, `CAXAUC_ADDRESS_651940`, `CWAXAUC_ADDRESS_651940` |

**Minimum for report API:** `CUSDC_ADDRESS_138`, `CUSDT_ADDRESS_138`. For full GRU M1 + W-tokens + ac*/vdc*/sdc* coverage, set the corresponding `*_ADDRESS_138` and `*_ADDRESS_651940` vars. See `.env.example` for the full commented list. Refs: [BLITZKRIEG_SUPER_PRO_MAX_MASTER_PLAN](../../../docs/00-meta/BLITZKRIEG_SUPER_PRO_MAX_MASTER_PLAN.md) §2–§3, [PLACEHOLDERS_AND_COMPLETION_MASTER_LIST](../../../docs/00-meta/PLACEHOLDERS_AND_COMPLETION_MASTER_LIST.md).

**CoinGecko/CMC chain support:** ChainID 138 and 651940 are not yet supported by CoinGecko/CMC; external price/volume for these chains will be empty in the report until the platforms add support or you use another source. The report API still returns chain-native token/pool data.

**Bridge routes (CCIP + Trustless):** `GET /api/v1/bridge/routes` returns CCIP (WETH9/WETH10) and **Trustless** bridge data. Trustless is **enabled in production by default**: the deployed Lockbox138 address on Chain 138 is included when `LOCKBOX_138` is not set. Optional env (restart after change):

- `LOCKBOX_138` — Override Lockbox contract address on Chain 138 (default: deployed Lockbox138).
- `INBOX_ETH` — (Optional) InboxETH contract on Ethereum Mainnet; when set, the Snap shows "Trustless → Ethereum Mainnet" with this address.

The MetaMask Snap bridge dialog always shows the Trustless (Lockbox) route when using this API. **Full Trustless operations** (user lock/claim) also require the Trustless stack on Ethereum (InboxETH, BondManager, LiquidityPool, etc.) to be deployed and operational; see `smom-dbis-138/docs/bridge/trustless`.

**Token mapping (multichain):** When run from the monorepo (proxmox), `GET /api/v1/token-mapping?fromChain=138&toChain=651940` (and `?fromChain=&toChain=` for any pair), `GET /api/v1/token-mapping/pairs`, and `GET /api/v1/token-mapping/resolve?fromChain=&toChain=&address=` expose `config/token-mapping-multichain.json` for bridge UIs and cross-chain address resolution.

5. Run database migrations:
```bash
# Ensure the migration has been run in the explorer database
# Migration file: ~/projects/explorer-monorepo/backend/database/migrations/0011_token_aggregation_schema.up.sql
```

6. Build the project:
```bash
npm run build
```

7. Start the service:
```bash
npm start
```

## Development

```bash
# Run in development mode with hot reload
npm run dev

# Run tests
npm test

# Lint code
npm run lint
```

## Docker Deployment

### Build and run with Docker Compose

```bash
docker-compose up -d
```

### Build Docker image

```bash
docker build -t token-aggregation-service .
docker run -p 3000:3000 --env-file .env token-aggregation-service
```

## API Endpoints

### Health Check
```
GET /health
```

### List Chains
```
GET /api/v1/chains
```

### List Tokens
```
GET /api/v1/tokens?chainId=138&limit=50&offset=0
```

### Get Token Details
```
GET /api/v1/tokens/:address?chainId=138
```

### Get Token Pools
```
GET /api/v1/tokens/:address/pools?chainId=138
```

### Get OHLCV Data
```
GET /api/v1/tokens/:address/ohlcv?chainId=138&interval=1h&from=timestamp&to=timestamp
```

### Get Token Signals
```
GET /api/v1/tokens/:address/signals?chainId=138
```

### Search Tokens
```
GET /api/v1/search?q=USDT&chainId=138
```

### Get Pool Details
```
GET /api/v1/pools/:poolAddress?chainId=138
```

### Route Decision Tree
```
GET /api/v1/routes/tree?chainId=138&tokenIn=0x...&tokenOut=0x...&amountIn=1000000&destinationChainId=1
```
Live route tree with current pool depth, freshness, bridge fallback, and destination-swap legs.

### Route Depth Summary
```
GET /api/v1/routes/depth?chainId=138&tokenIn=0x...&tokenOut=0x...&amountIn=1000000&destinationChainId=1
```
Compact summary of the same route tree for routing UIs.

### CMC and CoinGecko Reporting (all tokens, liquidity, volume)

Full API for all coins, tokens, liquidity, and reportable data in CMC/CoinGecko-friendly formats:

| Endpoint | Description |
|----------|-------------|
| `GET /api/v1/report/all` | All tokens, pools, liquidity, volume, and summary by chain |
| `GET /api/v1/report/coingecko?chainId=138` | Token list in CoinGecko submission format |
| `GET /api/v1/report/cmc?chainId=138` | Token list and DEX pairs in CoinMarketCap format |
| `GET /api/v1/report/token-list` | Flat canonical token list, including explicit Chain 138 V1/V2 GRU deployments where available |
| `GET /api/v1/report/canonical` | Raw canonical token spec with version/family metadata and addresses |

See [docs/CMC_COINGECKO_REPORTING.md](docs/CMC_COINGECKO_REPORTING.md) for usage and listing submission.

### Token mapping (multichain)

When the service runs from the monorepo (with `config/token-mapping-multichain.json` available):

| Endpoint | Description |
|----------|-------------|
| `GET /api/v1/token-mapping?fromChain=138&toChain=651940` | Token mapping for a chain pair (tokens, addressMapFromTo, addressMapToFrom) |
| `GET /api/v1/token-mapping/pairs` | All defined chain pairs |
| `GET /api/v1/token-mapping/resolve?fromChain=&toChain=&address=` | Resolve token address on target chain |

## Configuration

### DEX Factory Configuration

Configure DEX factory addresses in `src/config/dex-factories.ts` or via environment variables:

```bash
# ChainID 138
CHAIN_138_DODO_POOL_MANAGER=0x...
CHAIN_138_UNISWAP_V2_FACTORY=0x0C30F6e67Ab3667fCc2f5CEA8e274ef1FB920279
CHAIN_138_UNISWAP_V2_ROUTER=0x3019A7fDc76ba7F64F18d78e66842760037ee638
CHAIN_138_UNISWAP_V2_START_BLOCK=4041370
CHAIN_138_SUSHISWAP_FACTORY=0x2871207ff0d56089D70c0134d33f1291B6Fce0BE
CHAIN_138_SUSHISWAP_ROUTER=0xB37b93D38559f53b62ab020A14919f2630a1aE34
CHAIN_138_SUSHISWAP_START_BLOCK=4041495
CHAIN_138_UNISWAP_V3_FACTORY=0x2f7219276e3ce367dB9ec74C1196a8ecEe67841C
CHAIN_138_UNISWAP_V3_ROUTER=0xde9cD8ee2811E6E64a41D5F68Be315d33995975E

# ChainID 651940
CHAIN_651940_UNISWAP_V2_FACTORY=0x...
CHAIN_651940_UNISWAP_V2_ROUTER=0x...
CHAIN_651940_UNISWAP_V2_START_BLOCK=0
CHAIN_651940_UNISWAP_V3_FACTORY=0x...
CHAIN_651940_UNISWAP_V3_ROUTER=0x...
CHAIN_651940_UNISWAP_V3_START_BLOCK=0
CHAIN_651940_HYDX_FACTORY=0x...
CHAIN_651940_HYDX_ROUTER=0x...
CHAIN_651940_HYDX_START_BLOCK=0
```

For ALL Mainnet non-DODO discovery, the repo now treats `HYDX` as the canonical custom venue surface when factory/router details are known. The broader `651940` non-DODO inventory is tracked in `config/allmainnet-non-dodo-protocol-surface.json`.

## Monitoring

The service includes:
- Health check endpoint at `/health`
- Structured logging with Winston
- Request rate limiting
- Response caching

## License

MIT