d-bis/explorer-monorepo
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add full monorepo: virtual-banker, backend, frontend, docs, scripts, deployment

Browse Source 
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
2026-02-10 11:32:49 -08:00
parent aafcd913c2
commit 88bc76da91
815 changed files with 125522 additions and 264 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
docs/specs/README.md Normal file
View File

@@ -0,0 +1,61 @@
# Technical Specifications Index
This directory contains comprehensive technical specifications for all major components of the ChainID 138 Explorer+ and Virtual Banking VTM Platform.
## Directory Structure
- [infrastructure/](./infrastructure/) - Infrastructure and node layer specifications
- [indexing/](./indexing/) - Indexing pipeline specifications
- [database/](./database/) - Database architecture specifications
- [api/](./api/) - API layer specifications
- [mempool/](./mempool/) - Mempool and real-time specifications
- [multichain/](./multichain/) - Multi-chain architecture specifications
- [ccip/](./ccip/) - CCIP integration specifications
- [actions/](./actions/) - Action layer specifications
- [banking/](./banking/) - Banking layer specifications
- [vtm/](./vtm/) - Virtual Teller Machine specifications
- [frontend/](./frontend/) - Frontend architecture specifications
- [xr/](./xr/) - XR experience specifications
- [security/](./security/) - Security and privacy specifications
- [observability/](./observability/) - Observability specifications
- [deployment/](./deployment/) - Deployment and operations specifications
## Documentation Standards
Each specification document follows a standard structure:
1. **Overview**: High-level description
2. **Architecture**: System design with diagrams
3. **Data Models**: Schemas and interfaces
4. **API Contracts**: Request/response formats
5. **Algorithms**: Key algorithms and data structures
6. **Integration Points**: How components integrate
7. **Security Considerations**: Security requirements
8. **Performance Requirements**: Latency, throughput, scalability targets
9. **Implementation Guidelines**: Code structure and patterns
10. **Testing Strategy**: How to test the component
11. **Monitoring & Observability**: What to monitor
## Quick Links
### Phase 0 (Foundations)
- [Infrastructure Specifications](./infrastructure/)
- [Indexing Pipeline](./indexing/)
- [Database Schema](./database/)
- [API Gateway](./api/api-gateway.md)
- [REST API](./api/rest-api.md)
### Phase 1 (Blockscout+ Parity)
- [Contract Verification](./indexing/verification-pipeline.md)
- [Search Index](./database/search-index-schema.md)
- [GraphQL API](./api/graphql-api.md)
- [WebSocket API](./api/websocket-api.md)
### Phase 2-6 (Advanced Features)
- See respective directories for multi-chain, CCIP, actions, banking, VTM, and XR specifications
## References
- [Platform Plan](../../chain_id_138_explorer_virtual_banking_vtm_platform_plan.md) - High-level platform plan
- Existing Blockscout Integration: See `../../docs/BLOCKSCOUT_COMPLETE_SUMMARY.md`

159
docs/specs/actions/bridge-engine.md Normal file
View File

@@ -0,0 +1,159 @@
# Bridge Engine Specification
## Overview
This document specifies the bridge engine that provides cross-chain asset transfers using CCIP and third-party bridge providers.
## Provider Abstraction
### Bridge Providers
**CCIP**: Chainlink CCIP (primary for supported chains)
**Third-Party**:
- Stargate
- Hop Protocol
- Across Protocol
- Others as needed
### Provider Interface
```typescript
interface BridgeProvider {
getName(): string;
getSupportedChains(): number[];
getQuote(request: BridgeQuoteRequest): Promise<BridgeQuote>;
executeBridge(request: BridgeRequest): Promise<Transaction>;
getStatus(txHash: string): Promise<BridgeStatus>;
}
```
## Quote Comparison
### Quote Structure
```json
{
"provider": "CCIP",
"from_chain": 138,
"to_chain": 1,
"from_token": "0x...",
"to_token": "0x...",
"amount_in": "1000000000000000000",
"amount_out": "990000000000000000",
"fee": "10000000000000000",
"estimated_time": 300, // seconds
"trust_score": 0.95,
"route": {
"steps": [...]
}
}
```
### Comparison Factors
**1. Fees**: Lower is better
**2. ETA**: Faster is better
**3. Trust Score**: Higher is better (security, audits, track record)
**4. Liquidity**: Sufficient liquidity available
**5. Supported Amounts**: Amount within limits
### Trust Scores
**Factors**:
- Security audits
- Total value locked (TVL)
- Incident history
- Team reputation
- Code maturity
## CCIP Routing Integration
### CCIP as Primary
**When to Use**: Chains supported by CCIP
**Benefits**: Secure, audited, Chainlink-backed
### CCIP Quote Flow
1. Check CCIP router for quote
2. Calculate fees (message fee + token fee)
3. Estimate execution time
4. Return quote
### CCIP Execution
1. Approve token spending (if needed)
2. Call CCIP router send function
3. Monitor source transaction
4. Track CCIP message
5. Monitor destination execution
## Failover Routing
### Strategy
**Priority Order**:
1. CCIP (if available and competitive)
2. Trusted third-party bridges
3. Alternative routes
### Failover Triggers
- CCIP unavailable
- CCIP quote significantly worse
- User preference
- Amount limits exceeded
## Proof and Receipt Tracking
### Tracking Flow
1. Store source transaction hash
2. Monitor CCIP message status
3. Link to destination transaction
4. Track execution status
5. Provide end-to-end visibility
### Status Updates
**States**:
- `pending`: Source transaction pending
- `sent`: Message sent to CCIP
- `delivered`: Message delivered
- `executing`: Executing on destination
- `completed`: Bridge completed
- `failed`: Bridge failed
## API Endpoints
### Get Bridge Quote
`GET /api/v1/bridge/quote`
**Parameters**:
- `from_chain`: Source chain ID
- `to_chain`: Destination chain ID
- `from_token`: Source token address
- `to_token`: Destination token address
- `amount`: Amount to bridge
**Response**: Best quote from all providers
### Execute Bridge
`POST /api/v1/bridge/execute`
**Body**: Bridge request parameters
**Response**: Source transaction hash
### Get Bridge Status
`GET /api/v1/bridge/status/{tx_hash}`
**Response**: Bridge status with source and destination transaction details
## References
- CCIP Tracking: See `../ccip/ccip-tracking.md`
- Wallet Connectivity: See `wallet-connectivity.md`

179
docs/specs/actions/safety-controls.md Normal file
View File

@@ -0,0 +1,179 @@
# Safety Controls Specification
## Overview
This document specifies safety controls including phishing detection, contract risk scoring, address screening, and transaction simulation warnings.
## Phishing/Contract Risk Scoring
### Risk Factors
**1. Contract Verification**:
- Unverified: Higher risk
- Verified: Lower risk
**2. Reputation**:
- Known malicious addresses: High risk
- Known legitimate addresses: Low risk
- Unknown: Medium risk
**3. Code Analysis**:
- Suspicious functions (self-destruct, delegatecall abuse)
- Honeypot patterns
- Proxy contracts (verify implementation)
**4. Historical Data**:
- Previous scam reports
- User reports
- Security audit results
### Risk Score
**Calculation**:
```
risk_score = (verification_risk * 0.3) +
(reputation_risk * 0.3) +
(code_risk * 0.2) +
(historical_risk * 0.2)
```
**Score Ranges**:
- 0.0-0.3: Low risk (green)
- 0.3-0.6: Medium risk (yellow)
- 0.6-1.0: High risk (red)
## Address Screening
### Screening Sources
**1. Blocklists**:
- Known scam addresses
- Phishing addresses
- Mixer/tumbler addresses
**2. Reputation Services**:
- Etherscan labels
- Community reports
- Security databases
**3. Pattern Detection**:
- Address similarity (homoglyph attacks)
- Known scam patterns
### Screening Result
```json
{
"address": "0x...",
"risk_level": "high",
"reasons": [
"Address on known phishing list",
"Unverified contract",
"Suspicious transaction patterns"
],
"sources": ["etherscan", "community_reports"],
"confidence": 0.95
}
```
## Transaction Simulation and Warnings
### Simulation Checks
**1. Balance Checks**:
- Sufficient token balance
- Sufficient native balance for gas
**2. Approval Checks**:
- Current allowance
- Approval amount vs transaction amount
**3. State Changes**:
- Simulate transaction
- Detect unexpected state changes
- Check for reentrancy risks
**4. Token Transfers**:
- Unexpected token transfers
- High-value transfers
- Transfers to unknown addresses
### Warning Types
**Critical Warnings** (Block transaction):
- Insufficient balance
- Contract self-destruct
- Known malicious address
**High Warnings** (Strong recommendation to cancel):
- Unverified contract
- High slippage
- Large value transfer to new address
**Medium Warnings** (Informational):
- First interaction with contract
- Unusual transaction pattern
- High gas cost
### Warning Display
**UI Elements**:
- Warning banner with risk level
- Detailed explanation
- Option to proceed or cancel
- Educational content
## User Consent Workflow
### Consent Steps
**1. Transaction Review**:
- Display transaction details
- Show risk assessment
- Highlight warnings
**2. Confirmation**:
- User acknowledges risks
- Explicit confirmation required
- Optional: Additional confirmation for high-risk
**3. Execution**:
- Proceed with transaction
- Log consent for audit
### Consent Logging
**Fields**:
- Transaction hash
- User address
- Risk level
- Warnings shown
- Consent timestamp
- User acknowledgment
## Integration Points
### Transaction Flow Integration
1. User initiates transaction
2. Screen address/contract
3. Simulate transaction
4. Calculate risk score
5. Display warnings
6. Request consent
7. Execute if approved
### API Integration
**Risk Assessment API**:
`POST /api/v1/safety/assess`
**Request**: Transaction details
**Response**: Risk score and warnings
## References
- Swap Engine: See `swap-engine.md`
- Bridge Engine: See `bridge-engine.md`
- Security: See `../security/security-architecture.md`

181
docs/specs/actions/swap-engine.md Normal file
View File

@@ -0,0 +1,181 @@
# Swap Engine Specification
## Overview
This document specifies the swap engine that integrates with DEX aggregators to provide optimal swap routing and execution.
## DEX Aggregator Integration
### Supported Aggregators
**1inch**: 1inch API
**0x**: 0x API
**Paraswap**: Paraswap API
**Others**: Additional aggregators as needed
### Integration Pattern
**Abstraction Layer**:
- Unified interface for all aggregators
- Quote comparison
- Best route selection
## Quote Comparison
### Quote Structure
```json
{
"from_token": "0x...",
"to_token": "0x...",
"amount_in": "1000000000000000000",
"amount_out": "2000000000000000000",
"routes": [
{
"protocol": "Uniswap V3",
"portion": 0.6
},
{
"protocol": "Curve",
"portion": 0.4
}
],
"gas_estimate": 150000,
"slippage": 0.005,
"price_impact": 0.01
}
```
### Comparison Algorithm
**Factors**:
1. **Output Amount**: Higher is better
2. **Gas Cost**: Lower is better
3. **Slippage**: Lower is better
4. **Price Impact**: Lower is better
5. **Execution Time**: Faster is better
**Scoring**:
```
score = (output_amount * 0.5) - (gas_cost * 0.2) - (slippage * 0.2) - (price_impact * 0.1)
```
## Routing Optimization
### Multi-Path Routing
**Strategy**: Split trade across multiple DEXs for better rates
**Implementation**:
- Get quotes from multiple aggregators
- Compare split vs single-path routes
- Select optimal route
### Slippage Calculation
**Formula**: `(expected_output - actual_output) / expected_output`
**Protection**:
- User-specified slippage tolerance
- Default: 0.5% (0.005)
- Warn if slippage high
## Approval Management
### Token Approvals
**Process**:
1. Check current allowance
2. Request approval if needed
3. Wait for approval confirmation
4. Execute swap
### Approval Optimization
**Strategies**:
- Approve max amount (one-time approval)
- Approve exact amount (safer, more transactions)
- Use permit2 (EIP-2612) if supported
### Approval Revocation
**Feature**: Allow users to revoke approvals
**UI**: Show all active approvals
**Action**: Revoke individual or all approvals
## Transaction Simulation
### Pre-Flight Simulation
**Purpose**: Verify transaction will succeed before signing
**Checks**:
- Sufficient balance
- Sufficient allowance
- Valid route
- Price impact within limits
- Gas estimation
### Simulation Result
```json
{
"success": true,
"simulated_output": "2000000000000000000",
"gas_estimate": 150000,
"warnings": [],
"errors": []
}
```
## Execution
### Transaction Building
**Steps**:
1. Get best quote
2. Build transaction with aggregator
3. Simulate transaction
4. Present to user for approval
5. Sign and broadcast
6. Monitor confirmation
### Error Handling
**Errors**:
- Insufficient balance
- Insufficient allowance
- Slippage exceeded
- Route failure
**Recovery**:
- Retry with adjusted parameters
- Suggest alternative routes
- Provide error explanations
## API Endpoints
### Get Quote
`GET /api/v1/swap/quote`
**Parameters**:
- `from_token`: Source token address
- `to_token`: Destination token address
- `amount_in`: Input amount
- `slippage`: Slippage tolerance (optional)
**Response**: Best quote with route details
### Execute Swap
`POST /api/v1/swap/execute`
**Body**: Transaction parameters
**Response**: Transaction hash
## References
- Wallet Connectivity: See `wallet-connectivity.md`
- Safety Controls: See `safety-controls.md`

116
docs/specs/actions/wallet-connectivity.md Normal file
View File

@@ -0,0 +1,116 @@
# Wallet Connectivity Specification
## Overview
This document specifies wallet connectivity options including WalletConnect v2, hardware wallet support, and embedded wallet architecture.
## WalletConnect v2 Integration
### Implementation
**Library**: `@walletconnect/web3wallet` or `@web3modal/wagmi`
**Features**:
- Multi-wallet support
- Session management
- Chain switching
- Transaction signing
- Message signing
### Connection Flow
1. User initiates connection
2. QR code displayed or deep link generated
3. User approves in wallet app
4. Session established
5. Ready for transactions
### Session Management
**Storage**: Persist sessions in localStorage
**Expiration**: Handle session expiration gracefully
**Reconnection**: Auto-reconnect on page reload
## Hardware Wallet Support
### Supported Wallets
**Ledger**: Via Ledger Live or browser extension
**Trezor**: Via Trezor Connect
**Other**: Via Web3 provider standard
### Integration
**Method**: Use Web3 provider interface
**Security**: Never expose private keys
**UX**: Clear instructions for hardware wallet usage
## Embedded Wallet
### Architecture
**Options**:
1. **Non-Custodial**: User controls keys (Web3Auth, Magic)
2. **Custodial**: Platform manages keys (Fireblocks, Circle)
**Recommendation**: Start with non-custodial, add custodial for banking features
### Non-Custodial Embedded Wallet
**Technology**: Web3Auth or similar
**Features**:
- Social login (Google, Twitter, etc.)
- Passwordless authentication
- Key management via MPC or smart contract wallets
- Recovery options
### Custodial Embedded Wallet
**Use Cases**: Banking features requiring custody
**Requirements**:
- Regulatory compliance
- Secure key storage (HSM)
- Insurance
- Audit trails
**Policy Gating**: Only enable for users meeting compliance requirements
## Key Management
### Non-Custodial Keys
**Storage**: User's device or MPC network
**Recovery**: Social recovery or seed phrase
**Security**: Never transmitted to server
### Custodial Keys
**Storage**: Hardware Security Module (HSM)
**Access**: Multi-signature approval
**Audit**: All key operations logged
## API Integration
### Wallet Connection
**Methods**:
- `connect()`: Initiate connection
- `disconnect()`: Close connection
- `getAccount()`: Get connected account
- `switchChain()`: Switch to different chain
### Transaction Signing
**Flow**:
1. Build transaction
2. Request user approval
3. Sign transaction
4. Broadcast transaction
5. Monitor confirmation
## References
- Swap Engine: See `swap-engine.md`
- Bridge Engine: See `bridge-engine.md`
- Security: See `../security/security-architecture.md`

366
docs/specs/api/api-gateway.md Normal file
View File

@@ -0,0 +1,366 @@
# API Gateway Architecture Specification
## Overview
This document specifies the API Gateway architecture that provides unified edge API access, authentication, authorization, rate limiting, and request routing for all explorer platform services.
## Architecture
```mermaid
flowchart TB
subgraph Clients
Web[Web Clients]
Mobile[Mobile Apps]
API_Client[API Clients]
end
subgraph Gateway[API Gateway]
Router[Request Router]
Auth[Authentication]
RateLimit[Rate Limiter]
Transform[Request Transformer]
Cache[Response Cache]
LB[Load Balancer]
end
subgraph Services[Backend Services]
Explorer[Explorer API]
Search[Search Service]
Analytics[Analytics Service]
Actions[Action Service]
Banking[Banking API]
end
Clients --> Router
Router --> Auth
Auth --> RateLimit
RateLimit --> Transform
Transform --> Cache
Cache --> LB
LB --> Explorer
LB --> Search
LB --> Analytics
LB --> Actions
LB --> Banking
```
## Gateway Components
### Request Router
**Purpose**: Route requests to appropriate backend services based on path, method, and headers.
**Routing Rules**:
- `/api/v1/blocks/*` → Explorer API
- `/api/v1/transactions/*` → Explorer API
- `/api/v1/addresses/*` → Explorer API
- `/api/v1/search/*` → Search Service
- `/api/v1/analytics/*` → Analytics Service
- `/api/v1/swap/*` → Action Service
- `/api/v1/bridge/*` → Action Service
- `/api/v1/banking/*` → Banking API
- `/graphql` → Explorer API (GraphQL endpoint)
**Implementation**: NGINX, Kong, AWS API Gateway, or custom router
### Authentication
**Purpose**: Authenticate requests and extract user/API key information.
**Methods**:
1. **API Key Authentication**:
- Header: `X-API-Key: <key>`
- Query parameter: `?api_key=<key>` (less secure)
- Validate key hash against database
- Extract user_id and tier from key
2. **OAuth 2.0**:
- Bearer token: `Authorization: Bearer <token>`
- Validate JWT token
- Extract user claims
3. **Session Authentication**:
- Cookie-based for web clients
- Validate session token
**Anonymous Access**: Allow unauthenticated requests with lower rate limits
### Authorization
**Purpose**: Authorize requests based on user permissions and API key tier.
**Authorization Checks**:
- API key tier (free, pro, enterprise)
- User roles and permissions
- Resource-level permissions (user's own data)
- IP whitelist restrictions
**Enforcement**:
- Block unauthorized requests (403 Forbidden)
- Filter responses based on permissions
- Log authorization failures
### Rate Limiting
**Purpose**: Prevent abuse and ensure fair usage.
**Rate Limiting Strategy**:
**Tiers**:
1. **Anonymous**: 10 req/s, 100 req/min
2. **Free API Key**: 100 req/s, 1000 req/min
3. **Pro API Key**: 500 req/s, 5000 req/min
4. **Enterprise API Key**: 1000 req/s, unlimited
**Per-Endpoint Limits**:
- Simple queries (GET): Higher limits
- Complex queries (search, analytics): Lower limits
- Write operations (POST): Strict limits
**Implementation**:
- Token bucket algorithm
- Redis for distributed rate limiting
- Sliding window for smooth rate limiting
**Rate Limit Headers**:
```
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1640995200
```
**Response on Limit Exceeded**:
```json
{
"error": "rate_limit_exceeded",
"message": "Rate limit exceeded. Please try again later.",
"retry_after": 60
}
```
HTTP Status: 429 Too Many Requests
### Request Transformation
**Purpose**: Transform requests before forwarding to backend services.
**Transformations**:
- Add user context headers (user_id, tier)
- Normalize request format
- Add tracing headers (request_id, trace_id)
- Validate and sanitize input
- Add default parameters
**Example Headers Added**:
```
X-User-ID: <uuid>
X-API-Tier: pro
X-Request-ID: <uuid>
X-Trace-ID: <uuid>
```
### Response Caching
**Purpose**: Cache responses to reduce backend load and improve latency.
**Cacheable Endpoints**:
- Block data (cache for 10 seconds)
- Token metadata (cache for 1 hour)
- Contract ABIs (cache for 1 hour)
- Analytics aggregates (cache for 5 minutes)
**Cache Keys**:
- Include path, query parameters, API key tier
- Exclude user-specific parameters
**Cache Headers**:
- `Cache-Control: public, max-age=60`
- `ETag` for conditional requests
- `Vary: X-API-Key` to vary by tier
**Cache Invalidation**:
- Time-based expiration
- Manual invalidation on data updates
- Event-driven invalidation
### Load Balancing
**Purpose**: Distribute requests across backend service instances.
**Strategies**:
- Round-robin: Even distribution
- Least connections: Send to least loaded instance
- Health-aware: Skip unhealthy instances
**Health Checks**:
- HTTP health check endpoint: `/health`
- Check interval: 10 seconds
- Unhealthy threshold: 3 consecutive failures
- Recovery threshold: 2 successful checks
## Request/Response Flow
### Request Flow
1. **Client Request** → Gateway
2. **Routing** → Determine target service
3. **Authentication** → Validate credentials
4. **Authorization** → Check permissions
5. **Rate Limiting** → Check limits
6. **Cache Check** → Return cached if available
7. **Request Transformation** → Add headers, normalize
8. **Load Balancing** → Select backend instance
9. **Forward Request** → Send to backend service
10. **Response Handling** → Transform, cache, return
### Response Flow
1. **Backend Response** → Gateway
2. **Response Transformation** → Add headers, format
3. **Cache Storage** → Store if cacheable
4. **Error Handling** → Format errors consistently
5. **Response to Client** → Return response
## Error Handling
### Error Format
**Standard Error Response**:
```json
{
"error": {
"code": "rate_limit_exceeded",
"message": "Rate limit exceeded. Please try again later.",
"details": {},
"request_id": "uuid",
"timestamp": "2024-01-01T00:00:00Z"
}
}
```
### Error Codes
- `400 Bad Request`: Invalid request
- `401 Unauthorized`: Authentication required
- `403 Forbidden`: Authorization failed
- `404 Not Found`: Resource not found
- `429 Too Many Requests`: Rate limit exceeded
- `500 Internal Server Error`: Server error
- `502 Bad Gateway`: Backend service unavailable
- `503 Service Unavailable`: Service temporarily unavailable
- `504 Gateway Timeout`: Backend timeout
## Security Considerations
### Request Validation
- Validate request size (max 10MB)
- Validate content-type
- Sanitize input to prevent injection
- Validate API key format
### DDoS Protection
- Rate limiting per IP
- Connection limits
- Request size limits
- WAF integration (Cloudflare)
### TLS/SSL
- Require HTTPS for all requests
- TLS 1.2+ only
- Strong cipher suites
- Certificate management
## Monitoring and Observability
### Metrics
**Key Metrics**:
- Request rate (requests/second)
- Response time (p50, p95, p99)
- Error rate (by error type)
- Cache hit rate
- Rate limit hits
- Backend service health
### Logging
**Log Fields**:
- Request ID
- User ID / API Key
- Method, path, query parameters
- Response status, latency
- Backend service called
- Error messages
**Log Level**: INFO for normal requests, ERROR for failures
### Tracing
- Add trace ID to requests
- Propagate trace ID to backend services
- Correlate requests across services
## Configuration
### Gateway Configuration
```yaml
gateway:
port: 443
health_check_path: /health
rate_limiting:
redis_url: "redis://..."
default_limit: 100
default_period: 60
cache:
redis_url: "redis://..."
default_ttl: 60
services:
explorer_api:
base_url: "http://explorer-api:8080"
health_check: "/health"
search_service:
base_url: "http://search-service:8080"
health_check: "/health"
```
## Implementation Options
### Option 1: NGINX + Lua
- High performance
- Custom logic with Lua
- Good caching support
### Option 2: Kong
- API Gateway features out of the box
- Plugin ecosystem
- Good rate limiting
### Option 3: AWS API Gateway
- Managed service
- Good integration with AWS services
- Cost considerations
### Option 4: Custom Gateway (Go/Node.js)
- Full control
- Custom features
- More maintenance
**Recommendation**: Start with Kong or NGINX, migrate to custom if needed.
## References
- REST API: See `rest-api.md`
- Authentication: See `../security/auth-spec.md`
- Rate Limiting: See `../security/ddos-protection.md`

322
docs/specs/api/etherscan-compatible-api.md Normal file
View File

@@ -0,0 +1,322 @@
# Etherscan-Compatible API Specification
## Overview
This document specifies the Etherscan-compatible API layer that provides an API surface matching Blockscout/Etherscan APIs for tool compatibility and easier migration.
**Base URL**: `https://api.explorer.d-bis.org/api`
**Note**: This is a compatibility layer that translates Etherscan API calls to our internal API.
## API Surface Mapping
### Account Endpoints
#### Get Account Balance
**Etherscan**: `GET /api?module=account&action=balance&address={address}&tag=latest`
**Our Implementation**:
- Maps to: `GET /v1/addresses/{chain_id}/{address}` (extract balance)
**Response Format** (matches Etherscan):
```json
{
"status": "1",
"message": "OK",
"result": "1000000000000000000"
}
```
#### Get Token Balance
**Etherscan**: `GET /api?module=account&action=tokenbalance&contractaddress={token}&address={address}&tag=latest`
**Our Implementation**:
- Maps to: `GET /v1/addresses/{chain_id}/{address}/tokens` (filter by token address)
**Response Format**:
```json
{
"status": "1",
"message": "OK",
"result": "1000000000000000000"
}
```
#### Get Transaction List
**Etherscan**: `GET /api?module=account&action=txlist&address={address}&startblock=0&endblock=99999999&page=1&offset=10&sort=desc`
**Our Implementation**:
- Maps to: `GET /v1/addresses/{chain_id}/{address}/transactions` (with pagination)
**Response Format**:
```json
{
"status": "1",
"message": "OK",
"result": [
{
"blockNumber": "12345",
"timeStamp": "1640995200",
"hash": "0x...",
"from": "0x...",
"to": "0x...",
"value": "1000000000000000000",
"gas": "21000",
"gasPrice": "20000000000",
"gasUsed": "21000",
"isError": "0",
"txreceipt_status": "1"
}
]
}
```
### Transaction Endpoints
#### Get Transaction Status
**Etherscan**: `GET /api?module=transaction&action=getstatus&txhash={hash}`
**Our Implementation**:
- Maps to: `GET /v1/transactions/{chain_id}/{hash}` (extract status)
**Response Format**:
```json
{
"status": "1",
"message": "OK",
"result": {
"isError": "0",
"errDescription": ""
}
}
```
#### Get Transaction Receipt Status
**Etherscan**: `GET /api?module=transaction&action=gettxreceiptstatus&txhash={hash}`
**Response Format**:
```json
{
"status": "1",
"message": "OK",
"result": {
"status": "1"
}
}
```
### Block Endpoints
#### Get Block by Number
**Etherscan**: `GET /api?module=proxy&action=eth_getBlockByNumber&tag={number}&boolean=true`
**Our Implementation**:
- Maps to: `GET /v1/blocks/{chain_id}/{number}` (transform format)
**Response Format** (Ethereum JSON-RPC format):
```json
{
"jsonrpc": "2.0",
"result": {
"number": "0x3039",
"hash": "0x...",
"transactions": [...]
}
}
```
### Contract Endpoints
#### Get Contract ABI
**Etherscan**: `GET /api?module=contract&action=getabi&address={address}`
**Our Implementation**:
- Maps to: `GET /v1/contracts/{chain_id}/{address}/abi`
**Response Format**:
```json
{
"status": "1",
"message": "OK",
"result": "[{...}]"
}
```
#### Verify Contract
**Etherscan**: `POST /api?module=contract&action=verifysourcecode`
**Our Implementation**:
- Maps to: `POST /v1/contracts/{chain_id}/{address}/verify`
**Parameter Mapping**:
- `sourceCode` → `source_code`
- `codeformat` → `verification_method`
- `contractaddress` → `address`
- `compilerversion` → `compiler_version`
### Token Endpoints
#### Get Token Total Supply
**Etherscan**: `GET /api?module=stats&action=tokensupply&contractaddress={address}`
**Our Implementation**:
- Maps to: `GET /v1/tokens/{chain_id}/{address}` (extract total_supply)
### Event Logs
#### Get Event Logs
**Etherscan**: `GET /api?module=logs&action=getLogs&address={address}&topic0={topic0}&fromBlock={fromBlock}&toBlock={toBlock}`
**Our Implementation**:
- Maps to: `GET /v1/logs` (with filters)
**Response Format**:
```json
{
"status": "1",
"message": "OK",
"result": [
{
"address": "0x...",
"topics": ["0x..."],
"data": "0x...",
"blockNumber": "0x3039",
"transactionHash": "0x...",
"logIndex": "0x0"
}
]
}
```
## Parameter Translation
### Common Parameters
| Etherscan | Our API | Notes |
|-----------|---------|-------|
| `module` | N/A | Determines endpoint category |
| `action` | N/A | Determines specific action |
| `address` | `address` | Same |
| `tag` | `block_number` | Convert "latest" to current block |
| `startblock` | `from_block` | Same |
| `endblock` | `to_block` | Same |
| `page` | `page` | Same |
| `offset` | `page_size` | Same |
| `sort` | `order` | Same (asc/desc) |
### Chain ID Handling
**Etherscan**: No chain_id parameter (single chain)
**Our API**: Require `chain_id` parameter or use default from context
**Mapping**: Add `chain_id` parameter to all requests
## Response Format Compatibility
### Status Codes
**Etherscan Format**:
- `status: "1"` = Success
- `status: "0"` = Error
**Our Format**:
- HTTP 200 = Success (map to status "1")
- HTTP 4xx/5xx = Error (map to status "0")
### Error Messages
**Etherscan Format**:
```json
{
"status": "0",
"message": "NOTOK",
"result": "Error message here"
}
```
**Our Mapping**:
- Map our error codes to Etherscan error messages
- Preserve error details in result field
## Migration Guide
### For Tools Using Etherscan API
**Step 1**: Update base URL
- Old: `https://api.etherscan.io/api`
- New: `https://api.explorer.d-bis.org/api`
**Step 2**: Add chain_id parameter (if needed)
- Add `&chainid=138` to requests
- Or use default chain from connection
**Step 3**: Test compatibility
- Verify response formats match
- Check error handling
- Validate data accuracy
### For New Integrations
**Recommendation**: Use native REST API instead of Etherscan-compatible API
- More features
- Better performance
- Clearer API design
## Limitations
### Not All Endpoints Supported
**Supported**:
- Account operations
- Transaction queries
- Block queries
- Contract verification
- Token operations
- Event logs
**Not Supported** (Etherscan-specific):
- Gas tracker (use our analytics API)
- Historical stats (use our analytics API)
- Token list (use our token API)
### Response Format Differences
**Some differences may exist**:
- Field naming (camelCase vs snake_case)
- Data precision
- Additional fields in our responses
## Implementation
### Translation Layer
**Architecture**:
- Request interceptor: Parse Etherscan-style requests
- Parameter mapper: Convert to our API format
- Response transformer: Convert our responses to Etherscan format
**Code Structure**:
```
compatibility/
├── etherscan/
│ ├── request_parser.py # Parse Etherscan requests
│ ├── parameter_mapper.py # Map parameters
│ └── response_transformer.py # Transform responses
```
## References
- REST API: See `rest-api.md`
- API Gateway: See `api-gateway.md`

351
docs/specs/api/graphql-api.md Normal file
View File

@@ -0,0 +1,351 @@
# GraphQL API Specification
## Overview
This document specifies the GraphQL API for the explorer platform, providing flexible queries, mutations for user data, and subscriptions for real-time updates.
**Endpoint**: `https://api.explorer.d-bis.org/graphql`
## Schema Overview
### Core Types
#### Block
```graphql
type Block {
chainId: Int!
number: BigInt!
hash: String!
parentHash: String!
timestamp: DateTime!
miner: Address
transactionCount: Int!
gasUsed: BigInt!
gasLimit: BigInt!
transactions: [Transaction!]!
}
```
#### Transaction
```graphql
type Transaction {
chainId: Int!
hash: String!
block: Block
blockNumber: BigInt
from: Address!
to: Address
value: BigInt!
gasPrice: BigInt
gasUsed: BigInt
status: TransactionStatus!
logs: [Log!]!
traces: [Trace!]!
}
```
#### Address
```graphql
type Address {
address: String!
chainId: Int!
balance: BigInt!
balanceUSD: Float
transactionCount: Int!
transactions: TransactionConnection!
tokens: [TokenBalance!]!
nfts: [NFT!]!
label: String
tags: [String!]!
isContract: Boolean!
contract: Contract
}
```
#### Token
```graphql
type Token {
address: String!
chainId: Int!
name: String
symbol: String
type: TokenType!
decimals: Int
totalSupply: BigInt
holderCount: Int!
transfers: TokenTransferConnection!
holders: HolderConnection!
}
```
#### Contract
```graphql
type Contract {
address: String!
chainId: Int!
name: String
verificationStatus: VerificationStatus!
sourceCode: String
abi: JSON
compilerVersion: String
createdAt: DateTime
}
```
## Queries
### Block Queries
```graphql
query GetBlock($chainId: Int!, $number: BigInt!) {
block(chainId: $chainId, number: $number) {
number
hash
timestamp
transactionCount
transactions {
hash
from
to
value
}
}
}
```
### Transaction Queries
```graphql
query GetTransaction($chainId: Int!, $hash: String!) {
transaction(chainId: $chainId, hash: $hash) {
hash
from
to
value
status
logs {
address
topics
data
}
traces {
type
from
to
value
}
}
}
```
### Address Queries
```graphql
query GetAddress($chainId: Int!, $address: String!) {
address(chainId: $chainId, address: $address) {
address
balance
balanceUSD
transactionCount
transactions(first: 10) {
edges {
node {
hash
blockNumber
value
}
}
}
tokens {
token {
symbol
name
}
balance
}
}
}
```
### Search Query
```graphql
query Search($query: String!, $chainId: Int) {
search(query: $query, chainId: $chainId) {
... on Block {
number
hash
}
... on Transaction {
hash
from
to
}
... on Address {
address
label
}
... on Token {
address
symbol
name
}
}
}
```
## Mutations
### User Preferences
```graphql
mutation UpdateWatchlist($chainId: Int!, $address: String!, $label: String) {
addToWatchlist(chainId: $chainId, address: $address, label: $label) {
success
watchlistItem {
address
label
}
}
}
mutation RemoveFromWatchlist($chainId: Int!, $address: String!) {
removeFromWatchlist(chainId: $chainId, address: $address) {
success
}
}
```
### Address Labels
```graphql
mutation AddAddressLabel($chainId: Int!, $address: String!, $label: String!) {
addAddressLabel(chainId: $chainId, address: $address, label: $label) {
success
label {
address
label
}
}
}
```
## Subscriptions
### New Blocks
```graphql
subscription NewBlocks($chainId: Int!) {
newBlock(chainId: $chainId) {
number
hash
timestamp
transactionCount
}
}
```
### Address Transactions
```graphql
subscription AddressTransactions($chainId: Int!, $address: String!) {
addressTransaction(chainId: $chainId, address: $address) {
hash
from
to
value
status
}
}
```
### Pending Transactions
```graphql
subscription PendingTransactions($chainId: Int!) {
pendingTransaction(chainId: $chainId) {
hash
from
to
value
gasPrice
}
}
```
## Resolver Architecture
### DataLoader Pattern
**Purpose**: Prevent N+1 query problems.
**Implementation**:
- Batch load related entities
- Cache within request context
- Example: Batch load all blocks for transactions in a query
### Field Resolvers
**Strategy**: Lazy loading of related fields.
**Example**:
```javascript
Block: {
transactions: (block, args, context) => {
return context.loaders.transactions.load({
chainId: block.chainId,
blockNumber: block.number
});
}
}
```
## Performance Considerations
### Query Complexity
**Limits**:
- Max depth: 10 levels
- Max complexity score: 1000
- Rate limit based on complexity
### Caching
**Strategy**:
- Cache frequently accessed fields (blocks, tokens)
- Cache duration: 1-60 seconds depending on data freshness needs
- Invalidate on updates
## Authentication
Same as REST API: API Key via header or OAuth Bearer token.
## Error Handling
**GraphQL Error Format**:
```json
{
"errors": [
{
"message": "Resource not found",
"extensions": {
"code": "NOT_FOUND",
"requestId": "uuid"
},
"path": ["block"]
}
],
"data": null
}
```
## References
- REST API: See `rest-api.md`
- API Gateway: See `api-gateway.md`

399
docs/specs/api/rest-api.md Normal file
View File

@@ -0,0 +1,399 @@
# REST API Specification
## Overview
This document specifies the REST API for the explorer platform. The API follows RESTful principles and provides comprehensive access to blockchain data, search, and analytics.
**Base URL**: `https://api.explorer.d-bis.org/v1`
**Authentication**: API Key via `X-API-Key` header (optional for read-only endpoints with rate limits)
## API Design Principles
1. **RESTful**: Use HTTP methods appropriately (GET, POST, etc.)
2. **Consistent**: Uniform response format
3. **Versioned**: `/v1` in path, version in response headers
4. **Paginated**: All list endpoints support pagination
5. **Filtered**: Support filtering and sorting
6. **Documented**: OpenAPI 3.0 specification
## Response Format
### Success Response
```json
{
"data": { ... },
"meta": {
"pagination": {
"page": 1,
"page_size": 20,
"total": 100,
"total_pages": 5
}
}
}
```
### Error Response
```json
{
"error": {
"code": "not_found",
"message": "Resource not found",
"details": {},
"request_id": "uuid"
}
}
```
## Endpoints
### Blocks
#### List Blocks
`GET /blocks`
**Query Parameters**:
- `chain_id` (integer, required): Chain ID
- `page` (integer, default: 1): Page number
- `page_size` (integer, default: 20, max: 100): Items per page
- `min_block` (integer): Minimum block number
- `max_block` (integer): Maximum block number
- `miner` (string): Filter by miner address
- `sort` (string, default: "number"): Sort field (number, timestamp, gas_used)
- `order` (string, default: "desc"): Sort order (asc, desc)
**Response**:
```json
{
"data": [
{
"chain_id": 138,
"number": 12345,
"hash": "0x...",
"timestamp": "2024-01-01T00:00:00Z",
"miner": "0x...",
"transaction_count": 100,
"gas_used": 15000000,
"gas_limit": 20000000
}
],
"meta": { "pagination": {...} }
}
```
#### Get Block by Number
`GET /blocks/{chain_id}/{number}`
**Response**: Single block object
#### Get Block by Hash
`GET /blocks/{chain_id}/hash/{hash}`
**Response**: Single block object
### Transactions
#### List Transactions
`GET /transactions`
**Query Parameters**:
- `chain_id` (integer, required)
- `block_number` (integer): Filter by block
- `from_address` (string): Filter by sender
- `to_address` (string): Filter by recipient
- `status` (string): Filter by status (success, failed)
- `min_value` (string): Minimum value
- `page`, `page_size`, `sort`, `order`
**Response**: Array of transaction objects
#### Get Transaction by Hash
`GET /transactions/{chain_id}/{hash}`
**Response**: Single transaction object with full details
**Includes**:
- Transaction data
- Receipt data
- Logs
- Traces (if available)
#### Get Transaction Receipt
`GET /transactions/{chain_id}/{hash}/receipt`
**Response**: Transaction receipt object
### Addresses
#### Get Address Info
`GET /addresses/{chain_id}/{address}`
**Response**:
```json
{
"address": "0x...",
"chain_id": 138,
"balance": "1000000000000000000",
"balance_usd": "3000.00",
"transaction_count": 500,
"token_count": 10,
"label": "My Wallet",
"tags": ["wallet"],
"is_contract": false
}
```
#### Get Address Transactions
`GET /addresses/{chain_id}/{address}/transactions`
**Query Parameters**: Standard pagination and filtering
**Response**: Array of transactions
#### Get Address Token Holdings
`GET /addresses/{chain_id}/{address}/tokens`
**Query Parameters**:
- `type` (string): Filter by token type (ERC20, ERC721, ERC1155)
- `page`, `page_size`
**Response**: Array of token holdings with balances
#### Get Address NFTs
`GET /addresses/{chain_id}/{address}/nfts`
**Response**: Array of NFT holdings (ERC-721/1155)
### Tokens
#### List Tokens
`GET /tokens`
**Query Parameters**:
- `chain_id` (integer, required)
- `type` (string): Filter by type
- `verified` (boolean): Filter verified tokens
- `search` (string): Search by name or symbol
- `page`, `page_size`, `sort`, `order`
**Response**: Array of token objects
#### Get Token Info
`GET /tokens/{chain_id}/{address}`
**Response**: Token details including metadata
#### Get Token Holders
`GET /tokens/{chain_id}/{address}/holders`
**Query Parameters**: Pagination
**Response**: Array of holder addresses with balances
#### Get Token Transfers
`GET /tokens/{chain_id}/{address}/transfers`
**Query Parameters**: Pagination, `from_address`, `to_address`
**Response**: Array of token transfers
### Contracts
#### Get Contract Info
`GET /contracts/{chain_id}/{address}`
**Response**: Contract details including verification status, source code, ABI
#### Verify Contract
`POST /contracts/{chain_id}/{address}/verify`
**Request Body**:
```json
{
"compiler_version": "0.8.19",
"optimization_enabled": true,
"optimization_runs": 200,
"source_code": "...",
"constructor_arguments": "0x...",
"verification_method": "standard_json"
}
```
**Response**: Verification status
#### Get Contract Source Code
`GET /contracts/{chain_id}/{address}/source`
**Response**: Source code and metadata
#### Get Contract ABI
`GET /contracts/{chain_id}/{address}/abi`
**Response**: Contract ABI JSON
### Logs
#### Get Logs
`GET /logs`
**Query Parameters**:
- `chain_id` (integer, required)
- `address` (string): Filter by contract address
- `topic0` (string): Filter by event signature
- `from_block` (integer): Start block
- `to_block` (integer): End block
- `page`, `page_size`
**Response**: Array of log objects
### Traces
#### Get Transaction Traces
`GET /transactions/{chain_id}/{hash}/traces`
**Response**: Array of trace objects
#### Get Block Traces
`GET /blocks/{chain_id}/{number}/traces`
**Response**: Array of trace objects for all transactions in block
### Search
#### Unified Search
`GET /search`
**Query Parameters**:
- `chain_id` (integer, optional): Filter by chain
- `q` (string, required): Search query
- `type` (string): Filter by type (block, transaction, address, token, contract)
- `page`, `page_size`
**Response**: Mixed array of results with type indicators
### Analytics
#### Network Stats
`GET /analytics/{chain_id}/network/stats`
**Response**:
```json
{
"current_block": 12345,
"tps": 15.5,
"gps": 30000000,
"avg_gas_price": 20000000000,
"pending_transactions": 50
}
```
#### Gas Price Statistics
`GET /analytics/{chain_id}/gas/price`
**Query Parameters**:
- `period` (string): Time period (1h, 24h, 7d, 30d)
**Response**: Gas price statistics (min, max, avg, percentiles)
#### Top Contracts
`GET /analytics/{chain_id}/contracts/top`
**Query Parameters**:
- `by` (string): Sort by (transactions, gas_used, value)
- `limit` (integer, default: 100)
**Response**: Array of top contracts
## Pagination
All list endpoints support pagination:
**Query Parameters**:
- `page` (integer, default: 1): Page number (1-indexed)
- `page_size` (integer, default: 20, max: 100): Items per page
**Response Metadata**:
```json
{
"meta": {
"pagination": {
"page": 1,
"page_size": 20,
"total": 500,
"total_pages": 25,
"has_next": true,
"has_prev": false
}
}
}
```
## Filtering and Sorting
**Common Filter Parameters**:
- Date ranges: `from_date`, `to_date`
- Block ranges: `from_block`, `to_block`
- Value ranges: `min_value`, `max_value`
- Address filters: `from_address`, `to_address`
**Sorting**:
- `sort` (string): Field to sort by
- `order` (string): `asc` or `desc`
## Rate Limiting
See API Gateway specification for rate limiting details.
**Rate Limit Headers**:
- `X-RateLimit-Limit`: Request limit
- `X-RateLimit-Remaining`: Remaining requests
- `X-RateLimit-Reset`: Reset timestamp
## OpenAPI Specification
Full OpenAPI 3.0 specification available at:
- `/api-docs/openapi.json`
- Interactive docs: `/api-docs` (Swagger UI)
## Versioning
**Current Version**: v1
**Version Header**: `API-Version: 1`
**Deprecation**: Deprecated endpoints return `Deprecation` header with deprecation date
## References
- API Gateway: See `api-gateway.md`
- GraphQL API: See `graphql-api.md`
- Etherscan-Compatible API: See `etherscan-compatible-api.md`

337
docs/specs/api/websocket-api.md Normal file
View File

@@ -0,0 +1,337 @@
# WebSocket API Specification
## Overview
This document specifies the WebSocket API for real-time blockchain data subscriptions, including new blocks, pending transactions, and address activity.
**Endpoint**: `wss://api.explorer.d-bis.org/ws`
## Connection Lifecycle
### Connection
**URL**: `wss://api.explorer.d-bis.org/ws?chain_id=138`
**Query Parameters**:
- `chain_id` (optional): Default chain ID for subscriptions
- `api_key` (optional): API key for authenticated connections
**Headers**:
- `Authorization: Bearer <token>` (optional)
### Authentication
**Methods**:
1. Query parameter: `?api_key=<key>`
2. Header: `Authorization: Bearer <token>`
3. Message: Send auth message after connection
**Auth Message**:
```json
{
"type": "auth",
"api_key": "your-api-key"
}
```
**Response**:
```json
{
"type": "auth_success",
"user_id": "uuid",
"tier": "pro"
}
```
### Heartbeat
**Purpose**: Keep connection alive and detect disconnections.
**Client Ping**: Send every 30 seconds
```json
{
"type": "ping"
}
```
**Server Pong**: Response within 1 second
```json
{
"type": "pong",
"timestamp": 1640995200
}
```
**Timeout**: Connection closed if no pong received within 60 seconds
## Subscription Model
### Subscribe to New Blocks
**Request**:
```json
{
"type": "subscribe",
"channel": "blocks",
"chain_id": 138,
"params": {
"include_transactions": false
}
}
```
**Response**:
```json
{
"type": "subscribed",
"subscription_id": "sub_123",
"channel": "blocks"
}
```
**Updates**:
```json
{
"type": "update",
"subscription_id": "sub_123",
"data": {
"number": 12345,
"hash": "0x...",
"timestamp": "2024-01-01T00:00:00Z",
"transaction_count": 100
}
}
```
### Subscribe to Pending Transactions
**Request**:
```json
{
"type": "subscribe",
"channel": "pending_transactions",
"chain_id": 138,
"params": {
"from_address": "0x...", // Optional filter
"min_value": "1000000000000000000" // Optional filter
}
}
```
**Updates**: Transaction objects as they enter mempool
### Subscribe to Address Activity
**Request**:
```json
{
"type": "subscribe",
"channel": "address",
"chain_id": 138,
"params": {
"address": "0x..."
}
}
```
**Updates**: Transactions involving the address
### Subscribe to Logs/Events
**Request**:
```json
{
"type": "subscribe",
"channel": "logs",
"chain_id": 138,
"params": {
"address": "0x...",
"topics": ["0xddf252..."] // Event signatures
}
}
```
**Updates**: Matching log entries
### Unsubscribe
**Request**:
```json
{
"type": "unsubscribe",
"subscription_id": "sub_123"
}
```
**Response**:
```json
{
"type": "unsubscribed",
"subscription_id": "sub_123"
}
```
## Message Formats
### Client Messages
**Subscribe**:
```json
{
"type": "subscribe",
"channel": "<channel_name>",
"chain_id": 138,
"params": { ... }
}
```
**Unsubscribe**:
```json
{
"type": "unsubscribe",
"subscription_id": "<id>"
}
```
**Ping**:
```json
{
"type": "ping"
}
```
### Server Messages
**Update**:
```json
{
"type": "update",
"subscription_id": "<id>",
"data": { ... }
}
```
**Error**:
```json
{
"type": "error",
"code": "invalid_subscription",
"message": "Invalid subscription parameters",
"subscription_id": "<id>"
}
```
**Notification**:
```json
{
"type": "notification",
"level": "info", // info, warning, error
"message": "Subscription limit reached"
}
```
## Subscription Limits
**Per Connection**:
- Anonymous: 5 subscriptions
- Free API Key: 20 subscriptions
- Pro API Key: 100 subscriptions
- Enterprise: Unlimited
**Per Chain**:
- Limit subscriptions per chain to prevent abuse
## Reconnection Strategy
### Automatic Reconnection
**Strategy**: Exponential backoff
**Backoff Schedule**:
- Initial: 1 second
- Max: 60 seconds
- Multiplier: 2x
**Behavior**:
- Reconnect automatically
- Resubscribe to all active subscriptions
- Resume from last received update
### Manual Reconnection
**Steps**:
1. Close connection gracefully
2. Reconnect with same parameters
3. Resubscribe to desired channels
## Rate Limiting
**Limits**:
- Message rate: 10 messages/second per connection
- Subscription rate: 5 subscriptions/minute per connection
- Update rate: Throttled per subscription type
**Throttling**: Server may batch or skip updates if client cannot keep up
## Error Codes
- `invalid_message`: Malformed message
- `invalid_subscription`: Invalid subscription parameters
- `subscription_limit`: Too many subscriptions
- `rate_limit`: Rate limit exceeded
- `unauthorized`: Authentication required
- `chain_not_supported`: Chain ID not supported
- `internal_error`: Server error
## Security Considerations
### Connection Security
- WSS (WebSocket Secure) required
- TLS 1.2+ only
- Certificate validation
### Authentication
- API key validation
- User permission checks
- Subscription authorization
### Message Validation
- Validate all incoming messages
- Sanitize parameters
- Reject invalid requests
## Performance Considerations
### Message Batching
**Strategy**: Batch multiple updates into single message when possible
**Example**:
```json
{
"type": "batch_update",
"updates": [
{ "subscription_id": "sub_1", "data": {...} },
{ "subscription_id": "sub_2", "data": {...} }
]
}
```
### Backpressure
**Strategy**: Drop updates if client cannot process them
**Indicators**:
- Client buffer full
- No ping/pong responses
- Slow message processing
## References
- REST API: See `rest-api.md`
- GraphQL API: See `graphql-api.md`
- Mempool Service: See `../mempool/mempool-service.md`

167
docs/specs/banking/account-ledger.md Normal file
View File

@@ -0,0 +1,167 @@
# Account & Ledger Specification
## Overview
This document specifies the customer ledger system using double-entry bookkeeping principles for banking operations.
## Double-Entry Ledger Schema
### Ledger Entry
```sql
CREATE TABLE ledger_entries (
id UUID PRIMARY KEY,
customer_id UUID NOT NULL,
transaction_id UUID NOT NULL,
entry_type VARCHAR(20) NOT NULL, -- 'debit', 'credit'
account_type VARCHAR(50) NOT NULL, -- 'cash', 'crypto', 'fiat', etc.
amount NUMERIC(78, 0) NOT NULL,
currency VARCHAR(10) NOT NULL,
description TEXT,
reference_id VARCHAR(255), -- External transaction reference
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (customer_id) REFERENCES customers(id),
FOREIGN KEY (transaction_id) REFERENCES transactions(id)
);
```
### Transaction Table
```sql
CREATE TABLE transactions (
id UUID PRIMARY KEY,
customer_id UUID NOT NULL,
transaction_type VARCHAR(50) NOT NULL, -- 'deposit', 'withdrawal', 'transfer', etc.
status VARCHAR(20) NOT NULL, -- 'pending', 'completed', 'failed'
amount NUMERIC(78, 0) NOT NULL,
currency VARCHAR(10) NOT NULL,
from_account UUID,
to_account UUID,
blockchain_tx_hash VARCHAR(66), -- If blockchain transaction
created_at TIMESTAMP DEFAULT NOW(),
completed_at TIMESTAMP,
FOREIGN KEY (customer_id) REFERENCES customers(id)
);
```
### Double-Entry Rule
**Constraint**: Sum of debits = Sum of credits for each transaction
**Implementation**: Two entries per transaction (debit and credit)
## Wallet Mapping
### Customer-Wallet Mapping
```sql
CREATE TABLE customer_wallets (
id UUID PRIMARY KEY,
customer_id UUID NOT NULL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
wallet_type VARCHAR(20) NOT NULL, -- 'external', 'custodial'
verified BOOLEAN DEFAULT false,
verified_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE (customer_id, chain_id, address),
FOREIGN KEY (customer_id) REFERENCES customers(id)
);
```
### Wallet Verification
**Process**:
1. Customer provides address
2. Request signature verification
3. Verify signature matches address
4. Link address to customer account
5. Mark as verified
## Reconciliation Jobs
### Reconciliation Process
**Purpose**: Ensure ledger matches actual blockchain/custodial balances
**Frequency**: Daily or real-time (depending on account type)
**Steps**:
1. Query actual balances (blockchain or custodial wallet)
2. Calculate expected balance from ledger
3. Compare and identify discrepancies
4. Create reconciliation entries if needed
5. Alert on discrepancies
### Reconciliation Schema
```sql
CREATE TABLE reconciliations (
id UUID PRIMARY KEY,
customer_id UUID NOT NULL,
account_type VARCHAR(50) NOT NULL,
currency VARCHAR(10) NOT NULL,
expected_balance NUMERIC(78, 0) NOT NULL,
actual_balance NUMERIC(78, 0) NOT NULL,
discrepancy NUMERIC(78, 0) NOT NULL,
status VARCHAR(20) NOT NULL, -- 'matched', 'discrepancy', 'resolved'
reconciled_at TIMESTAMP DEFAULT NOW(),
resolved_at TIMESTAMP,
FOREIGN KEY (customer_id) REFERENCES customers(id)
);
```
## Audit Trails
### Audit Log Schema
```sql
CREATE TABLE audit_logs (
id UUID PRIMARY KEY,
customer_id UUID,
action VARCHAR(100) NOT NULL,
resource_type VARCHAR(50) NOT NULL,
resource_id VARCHAR(255),
old_value JSONB,
new_value JSONB,
actor_id UUID NOT NULL, -- User/system that performed action
ip_address VARCHAR(45),
user_agent TEXT,
created_at TIMESTAMP DEFAULT NOW()
);
```
### Immutable Log Storage
**Storage**: Append-only log (immutable)
**Retention**: Per regulatory requirements (typically 7 years)
**Access**: Audit team only
**Integrity**: Cryptographic hashing for tamper detection
## Balance Calculations
### Account Balance
**Formula**: Sum of credits - Sum of debits for account
**Implementation**:
```sql
SELECT
SUM(CASE WHEN entry_type = 'credit' THEN amount ELSE 0 END) -
SUM(CASE WHEN entry_type = 'debit' THEN amount ELSE 0 END) AS balance
FROM ledger_entries
WHERE customer_id = ? AND account_type = ? AND currency = ?;
```
### Real-Time Balance
**Caching**: Cache balances with TTL
**Updates**: Invalidate on new ledger entries
**Accuracy**: Reconcile with actual balances periodically
## References
- Identity & Compliance: See `identity-compliance.md`
- Payment Rails: See `payment-rails.md`
- Database Schema: See `../database/postgres-schema.md`

87
docs/specs/banking/compliance-dashboards.md Normal file
View File

@@ -0,0 +1,87 @@
# Compliance Dashboards Specification
## Overview
Compliance dashboards for case management, SAR/STR workflows, and reporting.
## Case Management
### Case Schema
```sql
CREATE TABLE compliance_cases (
id UUID PRIMARY KEY,
case_type VARCHAR(50) NOT NULL, -- 'suspicious_activity', 'sanctions_match', etc.
customer_id UUID,
severity VARCHAR(20) NOT NULL, -- 'low', 'medium', 'high', 'critical'
status VARCHAR(20) NOT NULL, -- 'open', 'under_review', 'resolved', 'escalated'
assigned_to UUID, -- Compliance officer
description TEXT,
evidence JSONB,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
resolved_at TIMESTAMP
);
```
### Case Workflow
1. Case created (automated or manual)
2. Assigned to compliance officer
3. Investigation
4. Resolution or escalation
5. Documentation and closure
## SAR/STR Workflows
### Suspicious Activity Reports (SAR)
**Trigger Conditions**:
- Unusual transaction patterns
- High-value transactions
- Sanctions/PEP matches
- Manual flagging
**Workflow**:
1. Detect suspicious activity
2. Create case
3. Investigate
4. File SAR if confirmed
5. Monitor for follow-up
### Suspicious Transaction Reports (STR)
Similar to SAR, jurisdiction-specific
## Evidence Export
### Export Format
**Package Contents**:
- Case details
- Transaction history
- Customer information
- Screening results
- Investigation notes
- Supporting documents
**Format**: PDF report + supporting data files
## Reporting APIs
### Case Management API
`GET /api/v1/compliance/cases`
`POST /api/v1/compliance/cases`
`GET /api/v1/compliance/cases/{id}`
`PUT /api/v1/compliance/cases/{id}`
### Reporting API
`GET /api/v1/compliance/reports`
`POST /api/v1/compliance/reports/generate`
## References
- Identity & Compliance: See `identity-compliance.md`

197
docs/specs/banking/identity-compliance.md Normal file
View File

@@ -0,0 +1,197 @@
# Identity & Compliance Specification
## Overview
This document specifies the identity verification (KYC/KYB) and compliance orchestration system for banking features.
## KYC/KYB Workflow Orchestration
### Workflow Stages
**1. Initial Registration**:
- User registration
- Basic information collection
- Terms acceptance
**2. Identity Verification**:
- Document upload (ID, proof of address)
- Biometric verification (if required)
- Liveness check
**3. Risk Assessment**:
- Sanctions screening
- PEP screening
- Risk scoring
**4. Approval/Rejection**:
- Automated approval (low risk)
- Manual review (medium/high risk)
- Rejection with reasons
### Workflow State Machine
```
[Registered] → [Identity Verification] → [Risk Assessment] → [Approved/Rejected]
↓
[Manual Review]
```
## Sanctions/PEP Screening Integration
### Screening Providers
**Options**:
- WorldCheck
- Dow Jones Risk & Compliance
- Chainalysis
- Others
### Screening Process
**1. Data Collection**:
- Name, date of birth, nationality
- Address information
- Associated addresses (blockchain addresses)
**2. Screening Check**:
- Sanctions lists (OFAC, UN, EU, etc.)
- PEP lists (politically exposed persons)
- Adverse media screening
**3. Match Resolution**:
- Automated false positive filtering
- Manual review for potential matches
- Risk scoring based on match confidence
### Screening Result
```json
{
"user_id": "uuid",
"screening_status": "cleared",
"matches": [],
"risk_score": 0.1,
"screened_at": "2024-01-01T00:00:00Z",
"next_screening": "2025-01-01T00:00:00Z"
}
```
## Risk Tier Assignment
### Risk Tiers
**Tier 1 - Low Risk**:
- Verified identity
- No sanctions/PEP matches
- Low transaction volume
- Limits: Standard limits
**Tier 2 - Medium Risk**:
- Verified identity
- Minor concerns (e.g., high-risk country)
- Medium transaction volume
- Limits: Reduced limits, additional monitoring
**Tier 3 - High Risk**:
- Unverified or incomplete verification
- Sanctions/PEP matches
- High transaction volume
- Limits: Very restricted or blocked
### Risk Scoring
**Factors**:
- Identity verification status
- Sanctions/PEP screening results
- Transaction patterns
- Geographic risk
- Source of funds
**Score Range**: 0.0 (low risk) to 1.0 (high risk)
## Limit Management
### Limit Types
**Transaction Limits**:
- Daily transaction limit
- Monthly transaction limit
- Single transaction limit
**Account Limits**:
- Maximum balance
- Withdrawal limits
### Limit Enforcement
**Real-time Checks**:
- Check limits before transaction
- Reject if limit exceeded
- Provide limit status to user
**Dynamic Limits**:
- Adjust limits based on risk tier
- Increase limits with step-up verification
- Temporary limit increases (pending approval)
## Step-Up Verification
### Trigger Conditions
**Triggers**:
- Transaction exceeds current tier limits
- Suspicious activity detected
- User request
- Regulatory requirement
### Verification Levels
**Level 1**: Basic KYC (standard)
**Level 2**: Enhanced due diligence (EDD)
**Level 3**: Institutional/KYB verification
### Step-Up Process
1. Notify user of requirement
2. Collect additional documentation
3. Enhanced screening
4. Review and approval
5. Update risk tier and limits
## Integration Points
### Identity Provider Integration
**Providers**:
- Jumio
- Onfido
- Sumsub
- Others
**Integration Pattern**:
- API integration
- Webhook callbacks for status updates
- Document storage
### Compliance System Integration
**Systems**:
- Transaction monitoring
- Reporting systems
- Audit systems
## Data Privacy
### PII Handling
**Storage**: Encrypted storage
**Access**: Role-based access control
**Retention**: Per regulatory requirements
**Deletion**: Right to deletion support
## References
- Account & Ledger: See `account-ledger.md`
- Compliance Dashboards: See `compliance-dashboards.md`
- Security: See `../security/privacy-controls.md`

95
docs/specs/banking/payment-rails.md Normal file
View File

@@ -0,0 +1,95 @@
# Payment Rails Specification
## Overview
Payment rails for on-ramp/off-ramp, ACH, wire, and card processing integration.
## On-Ramp Integration
### Providers
**Crypto On-Ramp**:
- MoonPay
- Ramp
- Transak
- Others
### Integration Pattern
**Flow**:
1. User initiates on-ramp
2. Redirect to provider or embed widget
3. User completes purchase
4. Provider webhook notifies completion
5. Credit customer account
## Off-Ramp Integration
### Providers
Similar to on-ramp providers
### Flow
1. User initiates withdrawal
2. Verify balance and limits
3. Initiate withdrawal with provider
4. Provider processes withdrawal
5. Debit customer account
6. Monitor completion
## ACH/Wire/Card Rails
### ACH Processing
**Integration**: Banking partner or payment processor
**Use Cases**: Fiat deposits/withdrawals
**Processing Time**: 1-3 business days
### Wire Transfers
**Integration**: Banking partner
**Use Cases**: Large fiat transfers
**Processing Time**: Same day or next day
### Card Processing
**Integration**: Payment processor (Stripe, etc.)
**Use Cases**: Card purchases
**Processing Time**: Instant (authorization), 1-3 days (settlement)
## Settlement Monitoring
### Monitoring Process
**Track**:
- Transaction status
- Settlement status
- Failed transactions
- Disputes/chargebacks
### Alerts
**Trigger Conditions**:
- Settlement failures
- Unusual delays
- Disputes
- Chargebacks
## Payment Status Tracking
### Status States
- `initiated`: Payment initiated
- `pending`: Pending processing
- `processing`: Being processed
- `completed`: Successfully completed
- `failed`: Processing failed
- `refunded`: Refunded
- `disputed`: Under dispute
## References
- Account & Ledger: See `account-ledger.md`
- Compliance: See `compliance-dashboards.md`

138
docs/specs/ccip/ccip-event-schema.md Normal file
View File

@@ -0,0 +1,138 @@
# CCIP Event Schema Specification
## Overview
This document specifies the event schemas for CCIP messages on source and destination chains.
## Source Chain Events
### MessageSent Event
**Event Signature**: `MessageSent(address indexed sender, uint64 indexed destinationChainSelector, bytes receiver, bytes data, address feeToken, uint256 fee)`
**Decoded Structure**:
```json
{
"event": "MessageSent",
"messageId": "0x...", // Derived from tx hash + log index
"sender": "0x...",
"destinationChainSelector": "866240039685049171407962509760789466724431933144813155647626",
"receiver": "0x...",
"data": "0x...",
"feeToken": "0x...",
"fee": "1000000000000000000",
"transactionHash": "0x...",
"blockNumber": 12345,
"logIndex": 5,
"timestamp": "2024-01-01T00:00:00Z"
}
```
### MessageSentWithTransfer Event (if applicable)
**For token transfers via CCIP**:
```json
{
"event": "MessageSentWithTransfer",
"messageId": "0x...",
"token": "0x...",
"amount": "1000000000000000000",
// ... other MessageSent fields
}
```
## Destination Chain Events
### MessageExecuted Event
**Event Signature**: `MessageExecuted(bytes32 indexed messageId, uint64 indexed sourceChainSelector, bytes sender, bytes data)`
**Decoded Structure**:
```json
{
"event": "MessageExecuted",
"messageId": "0x...",
"sourceChainSelector": "866240039685049171407962509760789466724431933144813155647626",
"sender": "0x...",
"data": "0x...",
"transactionHash": "0x...",
"blockNumber": 19000000,
"logIndex": 10,
"timestamp": "2024-01-01T00:05:00Z",
"status": "success" // or "failed" if execution reverted
}
```
### MessageExecutionFailed Event (if applicable)
**For failed executions**:
```json
{
"event": "MessageExecutionFailed",
"messageId": "0x...",
"reason": "execution reverted",
// ... other fields
}
```
## Message Metadata Schema
### Stored Message Structure
```json
{
"message_id": "0x...",
"source": {
"chain_id": 138,
"tx_hash": "0x...",
"block_number": 12345,
"sender": "0x...",
"receiver": "0x...",
"destination_chain_selector": "866240039685049171407962509760789466724431933144813155647626",
"data": "0x...",
"fee_token": "0x...",
"fee": "1000000000000000000",
"timestamp": "2024-01-01T00:00:00Z"
},
"destination": {
"chain_id": 1,
"tx_hash": "0x...",
"block_number": 19000000,
"sender": "0x...",
"receiver": "0x...",
"status": "executed",
"timestamp": "2024-01-01T00:05:00Z"
},
"status": "executed",
"retry_count": 0,
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:05:00Z"
}
```
## Status Transition Model
### State Machine
```
[Sent] → [Delivered] → [Executing] → [Executed]
↓
[Failed]
↓
[Expired] (if timeout)
```
### Status Definitions
- **sent**: Message sent on source chain, event indexed
- **delivered**: Message delivered to CCIP DON (if detectable)
- **executing**: Message execution started on destination
- **executed**: Message executed successfully
- **failed**: Message execution failed/reverted
- **expired**: Message timeout exceeded
## References
- CCIP Tracking: See `ccip-tracking.md`
- Database Schema: See `../database/postgres-schema.md`

110
docs/specs/ccip/ccip-observability.md Normal file
View File

@@ -0,0 +1,110 @@
# CCIP Observability Dashboard Specification
## Overview
This document specifies the observability dashboards and analytics for CCIP message lifecycle visualization and monitoring.
## Dashboard Components
### Message Lifecycle Visualization
**Components**:
- Timeline view showing source → CCIP → destination flow
- Status indicators at each stage
- Time metrics (send time, delivery time, execution time)
- Transaction links (click to view source/destination transactions)
**Visualization**:
```
[Source Chain] → [CCIP DON] → [Destination Chain]
✓ ✓ ✓
Sent Delivered Executed
00:00 00:02 00:05
```
### Status Aggregation
**Metrics**:
- Total messages by status
- Success rate (% executed successfully)
- Average execution time
- Failed message count
- Expired message count
**Time Series**:
- Messages per hour/day
- Success rate over time
- Average execution time over time
### Failure Analysis
**Failure Categories**:
- Execution failures (revert on destination)
- Timeouts (expired messages)
- Delivery failures (CCIP DON issues)
- Invalid messages
**Analysis**:
- Failure rate by category
- Common failure reasons
- Failure patterns (time, chain pairs, etc.)
### Performance Metrics
**Metrics**:
- Average delivery time (source → CCIP)
- Average execution time (CCIP → destination)
- Total end-to-end time
- P50, P95, P99 latencies
**Charts**:
- Latency distribution
- Latency trends over time
- Latency by chain pair
## Cross-Chain Analytics
### Chain Pair Analysis
**Metrics per Chain Pair**:
- Message volume
- Success rate
- Average latency
- Popular routes
**Visualization**:
- Network graph showing chain connections
- Edge weights showing message volume
- Color coding for success rates
### Token Flow Analysis
**Tracking**:
- Token transfers via CCIP
- Volume by token
- Volume by chain pair
- Cumulative volume over time
## Real-Time Monitoring
### Live Message Stream
**Features**:
- Real-time updates of new messages
- Status change notifications
- Alert on failures
- Filter by chain, status, etc.
### Alerts
**Alert Conditions**:
- High failure rate (> 5%)
- Message timeout (> 30 minutes)
- Unusual message volume spike
- Chain connectivity issues
## References
- CCIP Tracking: See `ccip-tracking.md`
- Observability: See `../observability/`

203
docs/specs/ccip/ccip-tracking.md Normal file
View File

@@ -0,0 +1,203 @@
# CCIP Message Tracking Specification
## Overview
This document specifies how CCIP (Cross-Chain Interoperability Protocol) messages are tracked from source chain to destination chain, including status monitoring and execution receipt linking.
## CCIP Message Lifecycle
```mermaid
sequenceDiagram
participant Source as Source Chain
participant CCIP as CCIP DON
participant Dest as Destination Chain
participant Indexer as Indexer
Source->>Indexer: Emit CCIP MessageSent event
Indexer->>Indexer: Index source tx + messageId
CCIP->>Dest: Deliver message
Dest->>Indexer: Emit CCIP MessageExecuted event
Indexer->>Indexer: Link messageId to dest tx
Indexer->>Indexer: Update message status
```
## Message Ingestion
### Source Chain Events
**Event**: `MessageSent(address indexed sender, uint64 indexed destinationChainSelector, bytes receiver, bytes data, address feeToken, uint256 fee)`
**Data Extracted**:
- `messageId`: Unique message identifier
- `sender`: Source address
- `destinationChainSelector`: Destination chain selector
- `receiver`: Destination receiver address
- `data`: Message payload
- `feeToken`: Fee token address
- `fee`: Fee amount
- `transactionHash`: Source transaction hash
- `blockNumber`: Source block number
- `timestamp`: Source transaction timestamp
### Destination Chain Events
**Event**: `MessageExecuted(bytes32 indexed messageId, uint64 indexed sourceChainSelector, bytes sender, bytes data)`
**Data Extracted**:
- `messageId`: Message identifier (matches source)
- `sourceChainSelector`: Source chain selector
- `sender`: Source sender address
- `data`: Message payload
- `transactionHash`: Destination transaction hash
- `blockNumber`: Destination block number
- `timestamp`: Destination transaction timestamp
- `status`: Execution status (success/failure)
## Message ID Normalization
### Message ID Format
**Source**: Generated from transaction hash and event log index
**Format**: `keccak256(transactionHash, logIndex)` or CCIP-provided messageId
### Normalization Strategy
**Goal**: Ensure consistent messageId across source and destination
**Method**:
1. Extract messageId from source event
2. Store with source transaction
3. Match with destination event by messageId
4. Link source and destination transactions
## Delivery Status Tracking
### Status States
**States**:
- `sent`: Message sent on source chain
- `delivered`: Message delivered to CCIP DON
- `executing`: Message executing on destination chain
- `executed`: Message executed successfully
- `failed`: Message execution failed
- `expired`: Message expired (timeout)
### Status Transitions
```
sent → delivered → executing → executed
↓
failed
↓
expired (if timeout)
```
### Status Updates
**Tracking**:
- Monitor destination chain for execution events
- Poll CCIP router for message status (if API available)
- Track timeout periods
- Update status in database
## Execution Receipt Linking
### Receipt Storage
**Database Schema**:
```sql
CREATE TABLE ccip_messages (
id UUID PRIMARY KEY,
message_id VARCHAR(66) NOT NULL UNIQUE,
source_chain_id INTEGER NOT NULL,
source_tx_hash VARCHAR(66) NOT NULL,
source_block_number BIGINT NOT NULL,
destination_chain_id INTEGER NOT NULL,
destination_tx_hash VARCHAR(66),
destination_block_number BIGINT,
status VARCHAR(20) NOT NULL,
sent_at TIMESTAMP NOT NULL,
executed_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
```
### Linking Process
**Steps**:
1. Index source transaction with messageId
2. Wait for destination chain event
3. Match messageId from destination event
4. Link destination transaction to source
5. Update status to "executed" or "failed"
## Retry Tracking
### Retry Detection
**Detection**:
- Multiple execution attempts with same messageId
- Track retry count
- Store retry timestamps
### Retry Data
**Fields**:
- `retry_count`: Number of retry attempts
- `retry_timestamps`: Array of retry attempt times
- `last_retry_at`: Last retry timestamp
## API Endpoints
### Get Message by ID
`GET /api/v1/ccip/messages/{message_id}`
**Response**:
```json
{
"message_id": "0x...",
"source": {
"chain_id": 138,
"tx_hash": "0x...",
"block_number": 12345,
"sender": "0x...",
"timestamp": "2024-01-01T00:00:00Z"
},
"destination": {
"chain_id": 1,
"tx_hash": "0x...",
"block_number": 19000000,
"receiver": "0x...",
"timestamp": "2024-01-01T00:05:00Z"
},
"status": "executed",
"sent_at": "2024-01-01T00:00:00Z",
"executed_at": "2024-01-01T00:05:00Z"
}
```
### Get Message by Transaction
`GET /api/v1/ccip/transactions/{chain_id}/{tx_hash}/messages`
**Response**: Array of CCIP messages for transaction
### Search Messages
`GET /api/v1/ccip/messages`
**Query Parameters**:
- `source_chain_id`: Filter by source chain
- `destination_chain_id`: Filter by destination chain
- `status`: Filter by status
- `sender`: Filter by sender address
- `receiver`: Filter by receiver address
## References
- CCIP Observability: See `ccip-observability.md`
- CCIP Event Schema: See `ccip-event-schema.md`
- Graph Database: See `../database/graph-schema.md`

294
docs/specs/database/data-lake-schema.md Normal file
View File

@@ -0,0 +1,294 @@
# Data Lake Schema Specification
## Overview
This document specifies the data lake schema for long-term storage of blockchain data in S3-compatible object storage using Parquet format for analytics, ML, and compliance purposes.
## Storage Structure
### Directory Layout
```
s3://explorer-data-lake/
├── raw/
│ ├── chain_id=138/
│ │ ├── year=2024/
│ │ │ ├── month=01/
│ │ │ │ ├── day=01/
│ │ │ │ │ ├── blocks.parquet
│ │ │ │ │ ├── transactions.parquet
│ │ │ │ │ └── logs.parquet
│ │ │ │ └── ...
│ │ │ └── ...
│ │ └── ...
│ └── ...
├── processed/
│ ├── chain_id=138/
│ │ ├── daily_aggregates/
│ │ │ ├── year=2024/
│ │ │ │ └── month=01/
│ │ │ │ └── day=01.parquet
│ │ └── ...
│ └── ...
└── archived/
└── ...
```
### Partitioning Strategy
**Partition Keys**:
- `chain_id`: Chain identifier
- `year`: Year (YYYY)
- `month`: Month (MM)
- `day`: Day (DD)
**Benefits**:
- Efficient query pruning
- Parallel processing
- Easy data management (delete by partition)
## Parquet Schema
### Blocks Parquet Schema
```json
{
"type": "struct",
"fields": [
{"name": "chain_id", "type": "integer", "nullable": false},
{"name": "number", "type": "long", "nullable": false},
{"name": "hash", "type": "string", "nullable": false},
{"name": "parent_hash", "type": "string", "nullable": false},
{"name": "timestamp", "type": "timestamp", "nullable": false},
{"name": "miner", "type": "string", "nullable": true},
{"name": "gas_used", "type": "long", "nullable": true},
{"name": "gas_limit", "type": "long", "nullable": true},
{"name": "transaction_count", "type": "integer", "nullable": true},
{"name": "size", "type": "integer", "nullable": true}
]
}
```
### Transactions Parquet Schema
```json
{
"type": "struct",
"fields": [
{"name": "chain_id", "type": "integer", "nullable": false},
{"name": "hash", "type": "string", "nullable": false},
{"name": "block_number", "type": "long", "nullable": false},
{"name": "transaction_index", "type": "integer", "nullable": false},
{"name": "from_address", "type": "string", "nullable": false},
{"name": "to_address", "type": "string", "nullable": true},
{"name": "value", "type": "string", "nullable": false}, // Decimal as string
{"name": "gas_price", "type": "long", "nullable": true},
{"name": "gas_used", "type": "long", "nullable": true},
{"name": "gas_limit", "type": "long", "nullable": false},
{"name": "status", "type": "integer", "nullable": true},
{"name": "timestamp", "type": "timestamp", "nullable": false}
]
}
```
### Logs Parquet Schema
```json
{
"type": "struct",
"fields": [
{"name": "chain_id", "type": "integer", "nullable": false},
{"name": "transaction_hash", "type": "string", "nullable": false},
{"name": "block_number", "type": "long", "nullable": false},
{"name": "log_index", "type": "integer", "nullable": false},
{"name": "address", "type": "string", "nullable": false},
{"name": "topic0", "type": "string", "nullable": true},
{"name": "topic1", "type": "string", "nullable": true},
{"name": "topic2", "type": "string", "nullable": true},
{"name": "topic3", "type": "string", "nullable": true},
{"name": "data", "type": "string", "nullable": true},
{"name": "timestamp", "type": "timestamp", "nullable": false}
]
}
```
### Token Transfers Parquet Schema
```json
{
"type": "struct",
"fields": [
{"name": "chain_id", "type": "integer", "nullable": false},
{"name": "transaction_hash", "type": "string", "nullable": false},
{"name": "block_number", "type": "long", "nullable": false},
{"name": "token_address", "type": "string", "nullable": false},
{"name": "token_type", "type": "string", "nullable": false},
{"name": "from_address", "type": "string", "nullable": false},
{"name": "to_address", "type": "string", "nullable": false},
{"name": "amount", "type": "string", "nullable": true},
{"name": "token_id", "type": "string", "nullable": true},
{"name": "timestamp", "type": "timestamp", "nullable": false}
]
}
```
## Data Ingestion
### ETL Pipeline
**Process**:
1. Extract: Query PostgreSQL for daily data
2. Transform: Convert to Parquet format
3. Load: Upload to S3 with partitioning
**Schedule**: Daily batch job after day ends
**Tools**: Apache Spark, AWS Glue, or custom ETL scripts
### Compression
**Format**: Snappy compression (good balance of speed and compression ratio)
**Alternative**: Gzip (better compression, slower)
### File Sizing
**Target Size**: 100-500 MB per Parquet file
- Smaller files: Better parallelism
- Larger files: Better compression
**Strategy**: Write files of target size, or split by time ranges
## Query Interface
### AWS Athena / Presto
**Table Definition**:
```sql
CREATE EXTERNAL TABLE blocks_138 (
chain_id int,
number bigint,
hash string,
parent_hash string,
timestamp timestamp,
miner string,
gas_used bigint,
gas_limit bigint,
transaction_count int,
size int
)
STORED AS PARQUET
LOCATION 's3://explorer-data-lake/raw/chain_id=138/'
TBLPROPERTIES (
'projection.enabled' = 'true',
'projection.year.type' = 'integer',
'projection.year.range' = '2020,2030',
'projection.month.type' = 'integer',
'projection.month.range' = '1,12',
'projection.day.type' = 'integer',
'projection.day.range' = '1,31'
);
```
### Query Examples
**Daily Transaction Count**:
```sql
SELECT
DATE(timestamp) as date,
COUNT(*) as transaction_count
FROM transactions_138
WHERE year = 2024 AND month = 1
GROUP BY DATE(timestamp)
ORDER BY date;
```
**Token Transfer Analytics**:
```sql
SELECT
token_address,
COUNT(*) as transfer_count,
SUM(CAST(amount AS DECIMAL(78, 0))) as total_volume
FROM token_transfers_138
WHERE year = 2024 AND month = 1
GROUP BY token_address
ORDER BY total_volume DESC
LIMIT 100;
```
## Data Retention
### Retention Policies
**Raw Data**: 7 years (compliance requirement)
**Processed Aggregates**: Indefinite
**Archived Data**: Move to Glacier after 1 year
### Lifecycle Policies
**S3 Lifecycle Rules**:
1. Move to Infrequent Access after 30 days
2. Move to Glacier after 1 year
3. Delete after 7 years (raw data)
## Data Processing
### Aggregation Jobs
**Daily Aggregates**:
- Transaction counts by hour
- Gas usage statistics
- Token transfer volumes
- Address activity metrics
**Monthly Aggregates**:
- Network growth metrics
- Token distribution changes
- Protocol usage statistics
### ML/Analytics Workflows
**Use Cases**:
- Anomaly detection
- Fraud detection
- Market analysis
- Network health monitoring
**Tools**: Spark, Pandas, Jupyter notebooks
## Security and Access Control
### Access Control
**IAM Policies**: Restrict access to specific prefixes
**Encryption**: Server-side encryption (SSE-S3 or SSE-KMS)
**Audit Logging**: Enable S3 access logging
### Data Classification
**Public Data**: Blocks, transactions (public blockchain data)
**Sensitive Data**: User addresses, labels (requires authentication)
**Compliance Data**: Banking/transaction data (strict access control)
## Cost Optimization
### Storage Optimization
**Strategies**:
- Use appropriate storage classes (Standard, IA, Glacier)
- Compress data (Parquet + Snappy)
- Delete old data per retention policy
- Use intelligent tiering
### Query Optimization
**Strategies**:
- Partition pruning (query only relevant partitions)
- Column pruning (select only needed columns)
- Predicate pushdown (filter early)
## References
- Database Schema: See `postgres-schema.md`
- Analytics: See `../observability/metrics-monitoring.md`

300
docs/specs/database/graph-schema.md Normal file
View File

@@ -0,0 +1,300 @@
# Graph Database Schema Specification
## Overview
This document specifies the Neo4j graph database schema for storing cross-chain entity relationships, address clustering, and protocol interactions.
## Schema Design
### Node Types
#### Address Node
**Labels**: `Address`, `Chain{chain_id}` (e.g., `Chain138`)
**Properties**:
```cypher
{
address: "0x...", // Unique identifier
chainId: 138, // Chain ID
label: "My Wallet", // Optional label
isContract: false, // Is contract address
firstSeen: timestamp, // First seen timestamp
lastSeen: timestamp, // Last seen timestamp
transactionCount: 100, // Transaction count
balance: "1.5" // Current balance (string for precision)
}
```
**Constraints**:
```cypher
CREATE CONSTRAINT address_address_chain_id FOR (a:Address)
REQUIRE (a.address, a.chainId) IS UNIQUE;
```
#### Contract Node
**Labels**: `Contract`, `Address`
**Properties**: Inherits from Address, plus:
```cypher
{
name: "MyToken",
verificationStatus: "verified",
compilerVersion: "0.8.19"
}
```
#### Token Node
**Labels**: `Token`, `Contract`
**Properties**: Inherits from Contract, plus:
```cypher
{
symbol: "MTK",
decimals: 18,
totalSupply: "1000000",
type: "ERC20" // ERC20, ERC721, ERC1155
}
```
#### Protocol Node
**Labels**: `Protocol`
**Properties**:
```cypher
{
name: "Uniswap V3",
category: "DEX",
website: "https://uniswap.org"
}
```
### Relationship Types
#### TRANSFERRED_TO
**Purpose**: Track token transfers between addresses.
**Properties**:
```cypher
{
amount: "1000000000000000000",
tokenAddress: "0x...",
transactionHash: "0x...",
blockNumber: 12345,
timestamp: timestamp
}
```
**Example**:
```cypher
(a1:Address {address: "0x..."})-[r:TRANSFERRED_TO {
amount: "1000000000000000000",
tokenAddress: "0x...",
transactionHash: "0x..."
}]->(a2:Address {address: "0x..."})
```
#### CALLED
**Purpose**: Track contract calls between addresses.
**Properties**:
```cypher
{
transactionHash: "0x...",
blockNumber: 12345,
timestamp: timestamp,
gasUsed: 21000,
method: "transfer"
}
```
#### OWNS
**Purpose**: Track token ownership (current balances).
**Properties**:
```cypher
{
balance: "1000000000000000000",
tokenId: "123", // For ERC-721/1155
updatedAt: timestamp
}
```
**Example**:
```cypher
(a:Address)-[r:OWNS {
balance: "1000000000000000000",
updatedAt: timestamp
}]->(t:Token)
```
#### INTERACTS_WITH
**Purpose**: Track protocol interactions.
**Properties**:
```cypher
{
interactionType: "swap", // swap, deposit, withdraw, etc.
transactionHash: "0x...",
timestamp: timestamp
}
```
**Example**:
```cypher
(a:Address)-[r:INTERACTS_WITH {
interactionType: "swap",
transactionHash: "0x..."
}]->(p:Protocol)
```
#### CLUSTERED_WITH
**Purpose**: Link addresses that belong to the same entity (address clustering).
**Properties**:
```cypher
{
confidence: 0.95, // Clustering confidence score
method: "heuristic", // Clustering method
createdAt: timestamp
}
```
#### CCIP_MESSAGE_LINK
**Purpose**: Link transactions across chains via CCIP messages.
**Properties**:
```cypher
{
messageId: "0x...",
sourceTxHash: "0x...",
destTxHash: "0x...",
status: "delivered",
timestamp: timestamp
}
```
**Example**:
```cypher
(srcTx:Transaction)-[r:CCIP_MESSAGE_LINK {
messageId: "0x...",
status: "delivered"
}]->(destTx:Transaction)
```
## Query Patterns
### Find Token Holders
```cypher
MATCH (t:Token {address: "0x...", chainId: 138})-[r:OWNS]-(a:Address)
WHERE r.balance > "0"
RETURN a.address, r.balance
ORDER BY toFloat(r.balance) DESC
LIMIT 100;
```
### Find Transfer Path
```cypher
MATCH path = (a1:Address {address: "0x..."})-[:TRANSFERRED_TO*1..3]-(a2:Address {address: "0x..."})
WHERE ALL(r in relationships(path) WHERE r.tokenAddress = "0x...")
RETURN path
LIMIT 10;
```
### Find Protocol Users
```cypher
MATCH (a:Address)-[r:INTERACTS_WITH]->(p:Protocol {name: "Uniswap V3"})
RETURN a.address, count(r) as interactionCount
ORDER BY interactionCount DESC
LIMIT 100;
```
### Address Clustering
```cypher
MATCH (a1:Address)-[r:CLUSTERED_WITH]-(a2:Address)
WHERE a1.address = "0x..."
RETURN a2.address, r.confidence, r.method;
```
### Cross-Chain CCIP Links
```cypher
MATCH (srcTx:Transaction {hash: "0x..."})-[r:CCIP_MESSAGE_LINK]-(destTx:Transaction)
RETURN srcTx, r, destTx;
```
## Data Ingestion
### Transaction Ingestion
**Process**:
1. Process transaction from indexer
2. Create/update address nodes
3. Create TRANSFERRED_TO relationships for token transfers
4. Create CALLED relationships for contract calls
5. Update OWNS relationships for token balances
### Batch Ingestion
**Strategy**:
- Use Neo4j Batch API for bulk inserts
- Batch size: 1000-10000 operations
- Use transactions for atomicity
### Incremental Updates
**Process**:
- Update relationships as new transactions processed
- Maintain OWNS relationships (update balances)
- Add new relationships for new interactions
## Performance Optimization
### Indexing
**Indexes**:
```cypher
CREATE INDEX address_address FOR (a:Address) ON (a.address);
CREATE INDEX address_chain_id FOR (a:Address) ON (a.chainId);
CREATE INDEX transaction_hash FOR (t:Transaction) ON (t.hash);
```
### Relationship Constraints
**Uniqueness**: Use MERGE to avoid duplicate relationships
**Example**:
```cypher
MATCH (a1:Address {address: "0x...", chainId: 138})
MATCH (a2:Address {address: "0x...", chainId: 138})
MERGE (a1)-[r:TRANSFERRED_TO {
transactionHash: "0x..."
}]->(a2)
ON CREATE SET r.amount = "1000000", r.timestamp = timestamp();
```
## Data Retention
**Strategy**:
- Keep all current relationships
- Archive old relationships (older than 1 year) to separate database
- Keep aggregated statistics (interaction counts) instead of all relationships
## References
- Entity Graph: See `../multichain/entity-graph.md`
- CCIP Integration: See `../ccip/ccip-tracking.md`

517
docs/specs/database/postgres-schema.md Normal file
View File

@@ -0,0 +1,517 @@
# PostgreSQL Database Schema Specification
## Overview
This document specifies the complete PostgreSQL database schema for the explorer platform. The schema is designed to support multi-chain operation, high-performance queries, and data consistency.
## Schema Design Principles
1. **Multi-chain Support**: All tables include `chain_id` for chain isolation
2. **Normalization**: Normalized structure to avoid data duplication
3. **Performance**: Strategic indexing for common query patterns
4. **Consistency**: Foreign key constraints where appropriate
5. **Extensibility**: JSONB columns for flexible data storage
6. **Partitioning**: Large tables partitioned by `chain_id`
## Core Tables
### Blocks Table
See `../indexing/data-models.md` for detailed block schema.
**Partitioning**: Partition by `chain_id` for large deployments.
**Key Indexes**:
- Primary: `(chain_id, number)`
- Unique: `(chain_id, hash)`
- Index: `(chain_id, timestamp)` for time-range queries
### Transactions Table
See `../indexing/data-models.md` for detailed transaction schema.
**Key Indexes**:
- Primary: `(chain_id, hash)`
- Index: `(chain_id, block_number, transaction_index)` for block queries
- Index: `(chain_id, from_address)` for address queries
- Index: `(chain_id, to_address)` for address queries
- Index: `(chain_id, block_number, from_address)` for compound queries
### Logs Table
See `../indexing/data-models.md` for detailed log schema.
**Key Indexes**:
- Primary: `(chain_id, transaction_hash, log_index)`
- Index: `(chain_id, address)` for contract event queries
- Index: `(chain_id, topic0)` for event type queries
- Index: `(chain_id, address, topic0)` for filtered event queries
- Index: `(chain_id, block_number)` for block-based queries
### Traces Table
See `../indexing/data-models.md` for detailed trace schema.
**Key Indexes**:
- Primary: `(chain_id, transaction_hash, trace_address)`
- Index: `(chain_id, action_from)` for address queries
- Index: `(chain_id, action_to)` for address queries
- Index: `(chain_id, block_number)` for block queries
### Internal Transactions Table
See `../indexing/data-models.md` for detailed internal transaction schema.
**Key Indexes**:
- Primary: `(chain_id, transaction_hash, trace_address)`
- Index: `(chain_id, from_address)`
- Index: `(chain_id, to_address)`
- Index: `(chain_id, block_number)`
## Token Tables
### Tokens Table
```sql
CREATE TABLE tokens (
id BIGSERIAL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
type VARCHAR(10) NOT NULL CHECK (type IN ('ERC20', 'ERC721', 'ERC1155')),
name VARCHAR(255),
symbol VARCHAR(50),
decimals INTEGER CHECK (decimals >= 0 AND decimals <= 18),
total_supply NUMERIC(78, 0),
holder_count INTEGER DEFAULT 0,
transfer_count INTEGER DEFAULT 0,
logo_url TEXT,
website_url TEXT,
description TEXT,
verified BOOLEAN DEFAULT false,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
UNIQUE (chain_id, address)
) PARTITION BY LIST (chain_id);
CREATE INDEX idx_tokens_chain_address ON tokens(chain_id, address);
CREATE INDEX idx_tokens_chain_type ON tokens(chain_id, type);
CREATE INDEX idx_tokens_chain_symbol ON tokens(chain_id, symbol);
```
### Token Transfers Table
```sql
CREATE TABLE token_transfers (
id BIGSERIAL,
chain_id INTEGER NOT NULL,
transaction_hash VARCHAR(66) NOT NULL,
block_number BIGINT NOT NULL,
log_index INTEGER NOT NULL,
token_address VARCHAR(42) NOT NULL,
token_type VARCHAR(10) NOT NULL CHECK (token_type IN ('ERC20', 'ERC721', 'ERC1155')),
from_address VARCHAR(42) NOT NULL,
to_address VARCHAR(42) NOT NULL,
amount NUMERIC(78, 0),
token_id VARCHAR(78),
operator VARCHAR(42),
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
FOREIGN KEY (chain_id, transaction_hash) REFERENCES transactions(chain_id, hash),
FOREIGN KEY (chain_id, token_address) REFERENCES tokens(chain_id, address),
UNIQUE (chain_id, transaction_hash, log_index)
) PARTITION BY LIST (chain_id);
CREATE INDEX idx_token_transfers_chain_token ON token_transfers(chain_id, token_address);
CREATE INDEX idx_token_transfers_chain_from ON token_transfers(chain_id, from_address);
CREATE INDEX idx_token_transfers_chain_to ON token_transfers(chain_id, to_address);
CREATE INDEX idx_token_transfers_chain_tx ON token_transfers(chain_id, transaction_hash);
CREATE INDEX idx_token_transfers_chain_block ON token_transfers(chain_id, block_number);
CREATE INDEX idx_token_transfers_chain_token_from ON token_transfers(chain_id, token_address, from_address);
CREATE INDEX idx_token_transfers_chain_token_to ON token_transfers(chain_id, token_address, to_address);
```
### Token Holders Table (Optional)
**Purpose**: Maintain current token balances for efficient queries.
```sql
CREATE TABLE token_holders (
id BIGSERIAL,
chain_id INTEGER NOT NULL,
token_address VARCHAR(42) NOT NULL,
address VARCHAR(42) NOT NULL,
balance NUMERIC(78, 0) NOT NULL DEFAULT 0,
token_id VARCHAR(78), -- For ERC-721/1155
updated_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
FOREIGN KEY (chain_id, token_address) REFERENCES tokens(chain_id, address),
UNIQUE (chain_id, token_address, address, COALESCE(token_id, ''))
) PARTITION BY LIST (chain_id);
CREATE INDEX idx_token_holders_chain_token ON token_holders(chain_id, token_address);
CREATE INDEX idx_token_holders_chain_address ON token_holders(chain_id, address);
```
## Contract Tables
### Contracts Table
```sql
CREATE TABLE contracts (
id BIGSERIAL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
name VARCHAR(255),
compiler_version VARCHAR(50),
optimization_enabled BOOLEAN,
optimization_runs INTEGER,
evm_version VARCHAR(20),
source_code TEXT,
abi JSONB,
constructor_arguments TEXT,
verification_status VARCHAR(20) NOT NULL CHECK (verification_status IN ('pending', 'verified', 'failed')),
verified_at TIMESTAMP,
verification_method VARCHAR(50),
license VARCHAR(50),
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
UNIQUE (chain_id, address)
) PARTITION BY LIST (chain_id);
CREATE INDEX idx_contracts_chain_address ON contracts(chain_id, address);
CREATE INDEX idx_contracts_chain_verified ON contracts(chain_id, verification_status);
CREATE INDEX idx_contracts_abi_gin ON contracts USING GIN (abi); -- For ABI queries
```
### Contract ABIs Table
```sql
CREATE TABLE contract_abis (
id BIGSERIAL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
abi JSONB NOT NULL,
source VARCHAR(50) NOT NULL,
verified BOOLEAN DEFAULT false,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
UNIQUE (chain_id, address)
) PARTITION BY LIST (chain_id);
CREATE INDEX idx_abis_chain_address ON contract_abis(chain_id, address);
CREATE INDEX idx_abis_abi_gin ON contract_abis USING GIN (abi);
```
### Contract Verifications Table
```sql
CREATE TABLE contract_verifications (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
status VARCHAR(20) NOT NULL CHECK (status IN ('pending', 'processing', 'verified', 'failed', 'partially_verified')),
compiler_version VARCHAR(50),
optimization_enabled BOOLEAN,
optimization_runs INTEGER,
evm_version VARCHAR(20),
source_code TEXT,
abi JSONB,
constructor_arguments TEXT,
verification_method VARCHAR(50),
error_message TEXT,
verified_at TIMESTAMP,
version INTEGER DEFAULT 1,
is_active BOOLEAN DEFAULT true,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, address) REFERENCES contracts(chain_id, address)
);
CREATE INDEX idx_verifications_chain_address ON contract_verifications(chain_id, address);
CREATE INDEX idx_verifications_status ON contract_verifications(status);
```
## Address-Related Tables
### Address Labels Table
```sql
CREATE TABLE address_labels (
id BIGSERIAL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
label VARCHAR(255) NOT NULL,
label_type VARCHAR(20) NOT NULL CHECK (label_type IN ('user', 'public', 'contract_name')),
user_id UUID,
source VARCHAR(50),
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
UNIQUE (chain_id, address, label_type, user_id),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE INDEX idx_labels_chain_address ON address_labels(chain_id, address);
CREATE INDEX idx_labels_chain_user ON address_labels(chain_id, user_id);
```
### Address Tags Table
```sql
CREATE TABLE address_tags (
id BIGSERIAL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
tag VARCHAR(50) NOT NULL,
tag_type VARCHAR(20) NOT NULL CHECK (tag_type IN ('category', 'risk', 'protocol')),
user_id UUID,
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
UNIQUE (chain_id, address, tag, user_id),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE INDEX idx_tags_chain_address ON address_tags(chain_id, address);
CREATE INDEX idx_tags_chain_tag ON address_tags(chain_id, tag);
```
## User Tables
### Users Table
```sql
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email VARCHAR(255) UNIQUE,
username VARCHAR(100) UNIQUE,
password_hash TEXT,
api_key_hash TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
last_login_at TIMESTAMP
);
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_username ON users(username);
```
### Watchlists Table
```sql
CREATE TABLE watchlists (
id BIGSERIAL,
user_id UUID NOT NULL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
label VARCHAR(255),
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (id),
UNIQUE (user_id, chain_id, address),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE INDEX idx_watchlists_user ON watchlists(user_id);
CREATE INDEX idx_watchlists_chain_address ON watchlists(chain_id, address);
```
### API Keys Table
```sql
CREATE TABLE api_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL,
key_hash TEXT NOT NULL UNIQUE,
name VARCHAR(255),
tier VARCHAR(20) NOT NULL CHECK (tier IN ('free', 'pro', 'enterprise')),
rate_limit_per_second INTEGER,
rate_limit_per_minute INTEGER,
ip_whitelist TEXT[], -- Array of CIDR blocks
last_used_at TIMESTAMP,
expires_at TIMESTAMP,
revoked BOOLEAN DEFAULT false,
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE INDEX idx_api_keys_user ON api_keys(user_id);
CREATE INDEX idx_api_keys_hash ON api_keys(key_hash);
```
## Multi-Chain Partitioning
### Partitioning Strategy
**Large Tables**: Partition by `chain_id` using LIST partitioning.
**Tables to Partition**:
- `blocks`
- `transactions`
- `logs`
- `traces`
- `internal_transactions`
- `token_transfers`
- `tokens`
- `token_holders` (if used)
### Partition Creation
**Example for blocks table**:
```sql
-- Create parent table
CREATE TABLE blocks (
-- columns
) PARTITION BY LIST (chain_id);
-- Create partitions
CREATE TABLE blocks_chain_138 PARTITION OF blocks
FOR VALUES IN (138);
CREATE TABLE blocks_chain_1 PARTITION OF blocks
FOR VALUES IN (1);
-- Add indexes to partitions (inherited from parent)
```
**Benefits**:
- Faster queries (partition pruning)
- Easier maintenance (per-chain operations)
- Parallel processing
- Data isolation
## Indexing Strategy
### Index Types
1. **B-tree**: Default for most indexes (equality, range, sorting)
2. **Hash**: For exact match only (rarely used, B-tree usually better)
3. **GIN**: For JSONB columns (ABIs, decoded data)
4. **BRIN**: For large ordered columns (block numbers, timestamps)
5. **Partial**: For filtered indexes (e.g., verified contracts only)
### Index Maintenance
**Regular Maintenance**:
- `VACUUM ANALYZE` regularly (auto-vacuum enabled)
- `REINDEX` if needed (bloat, corruption)
- Monitor index usage (`pg_stat_user_indexes`)
**Index Monitoring**:
- Track index sizes
- Monitor index bloat
- Remove unused indexes
## Data Retention and Archiving
### Retention Policies
**Hot Data**: Recent data (last 1 year)
- Fast access required
- All indexes maintained
**Warm Data**: Older data (1-5 years)
- Archive to slower storage
- Reduced indexing
**Cold Data**: Very old data (5+ years)
- Archive to object storage
- Minimal indexing
### Archiving Strategy
**Approach**:
1. Partition tables by time ranges (monthly/yearly)
2. Move old partitions to archive storage
3. Query archive when needed (slower but available)
**Implementation**:
- Use PostgreSQL table partitioning by date range
- Move partitions to archive storage (S3, etc.)
- Query via foreign data wrappers if needed
## Migration Strategy
### Versioning
**Migration Tool**: Use migration tool (Flyway, Liquibase, or custom).
**Versioning Format**: `YYYYMMDDHHMMSS_description.sql`
**Example**:
```
20240101000001_initial_schema.sql
20240115000001_add_token_holders.sql
20240201000001_add_partitioning.sql
```
### Migration Best Practices
1. **Backward Compatible**: Additive changes preferred
2. **Reversible**: All migrations should be reversible
3. **Tested**: Test on staging before production
4. **Documented**: Document breaking changes
5. **Rollback Plan**: Have rollback strategy
### Schema Evolution
**Adding Columns**:
- Use `ALTER TABLE ADD COLUMN` with default values
- Avoid NOT NULL without defaults (use two-step migration)
**Removing Columns**:
- Mark as deprecated first
- Remove after migration period
**Changing Types**:
- Create new column
- Migrate data
- Drop old column
- Rename new column
## Performance Optimization
### Query Optimization
**Common Query Patterns**:
1. Get block by number: Use `(chain_id, number)` index
2. Get transaction by hash: Use `(chain_id, hash)` index
3. Get address transactions: Use `(chain_id, from_address)` or `(chain_id, to_address)` index
4. Filter logs by address and event: Use `(chain_id, address, topic0)` index
### Connection Pooling
**Configuration**:
- Use connection pooler (PgBouncer, pgpool-II)
- Pool size: 20-100 connections per application server
- Statement-level pooling for better concurrency
### Read Replicas
**Strategy**:
- Primary: Write operations
- Replicas: Read operations (load balanced)
- Async replication (small lag acceptable)
## Backup and Recovery
### Backup Strategy
**Full Backups**: Daily full database dumps
**Incremental Backups**: Continuous WAL archiving
**Point-in-Time Recovery**: Enabled via WAL archiving
### Recovery Procedures
**RTO Target**: 1 hour
**RPO Target**: 5 minutes (max data loss)
## References
- Data Models: See `../indexing/data-models.md`
- Indexer Architecture: See `../indexing/indexer-architecture.md`
- Search Index Schema: See `search-index-schema.md`
- Multi-chain Architecture: See `../multichain/multichain-indexing.md`

458
docs/specs/database/search-index-schema.md Normal file
View File

@@ -0,0 +1,458 @@
# Search Index Schema Specification
## Overview
This document specifies the Elasticsearch/OpenSearch index schema for full-text search and faceted querying across blocks, transactions, addresses, tokens, and contracts.
## Architecture
```mermaid
flowchart LR
PG[(PostgreSQL<br/>Canonical Data)]
Transform[Data Transformer]
ES[(Elasticsearch<br/>Search Index)]
PG --> Transform
Transform --> ES
Query[Search Query]
Query --> ES
ES --> Results[Search Results]
```
## Index Structure
### Blocks Index
**Index Name**: `blocks-{chain_id}` (e.g., `blocks-138`)
**Document Structure**:
```json
{
"block_number": 12345,
"hash": "0x...",
"timestamp": "2024-01-01T00:00:00Z",
"miner": "0x...",
"transaction_count": 100,
"gas_used": 15000000,
"gas_limit": 20000000,
"chain_id": 138,
"parent_hash": "0x...",
"size": 1024
}
```
**Field Mappings**:
- `block_number`: `long` (not analyzed, for sorting/filtering)
- `hash`: `keyword` (exact match)
- `timestamp`: `date`
- `miner`: `keyword` (exact match)
- `transaction_count`: `integer`
- `gas_used`: `long`
- `gas_limit`: `long`
- `chain_id`: `integer`
- `parent_hash`: `keyword`
**Searchable Fields**:
- Hash (exact match)
- Miner address (exact match)
### Transactions Index
**Index Name**: `transactions-{chain_id}`
**Document Structure**:
```json
{
"hash": "0x...",
"block_number": 12345,
"transaction_index": 5,
"from_address": "0x...",
"to_address": "0x...",
"value": "1000000000000000000",
"gas_price": "20000000000",
"gas_used": 21000,
"status": "success",
"timestamp": "2024-01-01T00:00:00Z",
"chain_id": 138,
"input_data_length": 100,
"is_contract_creation": false,
"contract_address": null
}
```
**Field Mappings**:
- `hash`: `keyword`
- `block_number`: `long`
- `transaction_index`: `integer`
- `from_address`: `keyword`
- `to_address`: `keyword`
- `value`: `text` (for full-text search on large numbers)
- `value_numeric`: `long` (for range queries)
- `gas_price`: `long`
- `gas_used`: `long`
- `status`: `keyword`
- `timestamp`: `date`
- `chain_id`: `integer`
- `input_data_length`: `integer`
- `is_contract_creation`: `boolean`
- `contract_address`: `keyword`
**Searchable Fields**:
- Hash (exact match)
- From/to addresses (exact match)
- Value (range queries)
### Addresses Index
**Index Name**: `addresses-{chain_id}`
**Document Structure**:
```json
{
"address": "0x...",
"chain_id": 138,
"label": "My Wallet",
"tags": ["wallet", "exchange"],
"token_count": 10,
"transaction_count": 500,
"first_seen": "2024-01-01T00:00:00Z",
"last_seen": "2024-01-15T00:00:00Z",
"is_contract": true,
"contract_name": "MyToken",
"balance_eth": "1.5",
"balance_usd": "3000"
}
```
**Field Mappings**:
- `address`: `keyword`
- `chain_id`: `integer`
- `label`: `text` (analyzed) + `keyword` (exact match)
- `tags`: `keyword` (array)
- `token_count`: `integer`
- `transaction_count`: `long`
- `first_seen`: `date`
- `last_seen`: `date`
- `is_contract`: `boolean`
- `contract_name`: `text` + `keyword`
- `balance_eth`: `double`
- `balance_usd`: `double`
**Searchable Fields**:
- Address (exact match, prefix match)
- Label (full-text search)
- Contract name (full-text search)
- Tags (facet filter)
### Tokens Index
**Index Name**: `tokens-{chain_id}`
**Document Structure**:
```json
{
"address": "0x...",
"chain_id": 138,
"name": "My Token",
"symbol": "MTK",
"type": "ERC20",
"decimals": 18,
"total_supply": "1000000000000000000000000",
"holder_count": 1000,
"transfer_count": 50000,
"logo_url": "https://...",
"verified": true,
"description": "A token description"
}
```
**Field Mappings**:
- `address`: `keyword`
- `chain_id`: `integer`
- `name`: `text` (analyzed) + `keyword` (exact match)
- `symbol`: `keyword` (uppercase normalized)
- `type`: `keyword`
- `decimals`: `integer`
- `total_supply`: `text` (for large numbers)
- `total_supply_numeric`: `double` (for sorting)
- `holder_count`: `integer`
- `transfer_count`: `long`
- `logo_url`: `keyword`
- `verified`: `boolean`
- `description`: `text` (analyzed)
**Searchable Fields**:
- Name (full-text search)
- Symbol (exact match, prefix match)
- Address (exact match)
### Contracts Index
**Index Name**: `contracts-{chain_id}`
**Document Structure**:
```json
{
"address": "0x...",
"chain_id": 138,
"name": "MyContract",
"verification_status": "verified",
"compiler_version": "0.8.19",
"source_code": "contract MyContract {...}",
"abi": [...],
"verified_at": "2024-01-01T00:00:00Z",
"transaction_count": 1000,
"created_at": "2024-01-01T00:00:00Z"
}
```
**Field Mappings**:
- `address`: `keyword`
- `chain_id`: `integer`
- `name`: `text` + `keyword`
- `verification_status`: `keyword`
- `compiler_version`: `keyword`
- `source_code`: `text` (analyzed, indexed but not stored in full for large contracts)
- `abi`: `object` (nested, for structured queries)
- `verified_at`: `date`
- `transaction_count`: `long`
- `created_at`: `date`
**Searchable Fields**:
- Name (full-text search)
- Address (exact match)
- Source code (full-text search, limited)
## Indexing Pipeline
### Data Transformation
**Purpose**: Transform canonical PostgreSQL data into search-optimized documents.
**Transformation Steps**:
1. **Fetch Data**: Query PostgreSQL for entities to index
2. **Enrich Data**: Add computed fields (balances, counts, etc.)
3. **Normalize Data**: Normalize addresses, format values
4. **Index Document**: Send to Elasticsearch/OpenSearch
### Indexing Strategy
**Initial Indexing**:
- Bulk index existing data
- Process in batches (1000 documents per batch)
- Use bulk API for efficiency
**Incremental Indexing**:
- Index new entities as they're created
- Update entities when changed
- Delete entities when removed
**Update Frequency**:
- Real-time: Index immediately after database insert/update
- Batch: Bulk update every N minutes for efficiency
### Index Aliases
**Purpose**: Enable zero-downtime index updates.
**Strategy**:
- Write to new index (e.g., `blocks-138-v2`)
- Build index in background
- Switch alias when ready
- Delete old index after switch
**Alias Names**:
- `blocks-{chain_id}` → points to latest version
- `transactions-{chain_id}` → points to latest version
- etc.
## Query Patterns
### Full-Text Search
**Blocks Search**:
```json
{
"query": {
"match": {
"hash": "0x123..."
}
}
}
```
**Address Search**:
```json
{
"query": {
"bool": {
"should": [
{ "match": { "label": "wallet" } },
{ "prefix": { "address": "0x123" } }
]
}
}
}
```
**Token Search**:
```json
{
"query": {
"bool": {
"should": [
{ "match": { "name": "My Token" } },
{ "match": { "symbol": "MTK" } }
]
}
}
}
```
### Faceted Search
**Filter by Multiple Criteria**:
```json
{
"query": {
"bool": {
"must": [
{ "term": { "chain_id": 138 } },
{ "term": { "type": "ERC20" } },
{ "range": { "holder_count": { "gte": 100 } } }
]
}
},
"aggs": {
"by_type": {
"terms": { "field": "type" }
}
}
}
```
### Unified Search
**Cross-Entity Search**:
- Search across blocks, transactions, addresses, tokens
- Use `_index` field to filter by entity type
- Combine results with relevance scoring
**Multi-Index Query**:
```json
{
"query": {
"multi_match": {
"query": "0x123",
"fields": ["hash", "address", "from_address", "to_address"],
"type": "best_fields"
}
}
}
```
## Index Configuration
### Analysis Settings
**Custom Analyzer**:
- Address analyzer: Lowercase, no tokenization
- Symbol analyzer: Uppercase, no tokenization
- Text analyzer: Standard analyzer with lowercase
**Example Configuration**:
```json
{
"settings": {
"analysis": {
"analyzer": {
"address_analyzer": {
"type": "custom",
"tokenizer": "keyword",
"filter": ["lowercase"]
}
}
}
}
}
```
### Sharding and Replication
**Sharding**:
- Number of shards: Based on index size
- Large indices (> 50GB): Multiple shards
- Small indices: Single shard
**Replication**:
- Replica count: 1-2 (for high availability)
- Increase replicas for read-heavy workloads
## Performance Optimization
### Index Optimization
**Refresh Interval**:
- Default: 1 second
- For bulk indexing: Increase to 30 seconds, then reset
**Bulk Indexing**:
- Batch size: 1000-5000 documents
- Use bulk API
- Disable refresh during bulk indexing
### Query Optimization
**Query Caching**:
- Enable query cache for repeated queries
- Cache filter results
**Field Data**:
- Use `doc_values` for sorting/aggregations
- Avoid `fielddata` for text fields
## Maintenance
### Index Monitoring
**Metrics**:
- Index size
- Document count
- Query performance (p50, p95, p99)
- Index lag (time behind database)
### Index Cleanup
**Strategy**:
- Delete old indices (after alias switch)
- Archive old indices to cold storage
- Compress indices for storage efficiency
## Integration with PostgreSQL
### Data Sync
**Sync Strategy**:
- Real-time: Listen to database changes (CDC, triggers, or polling)
- Batch: Periodic sync jobs
- Hybrid: Real-time for recent data, batch for historical
**Change Detection**:
- Use `updated_at` timestamp
- Use database triggers to queue changes
- Use CDC (Change Data Capture) if available
### Consistency
**Eventual Consistency**:
- Search index is eventually consistent with database
- Small lag acceptable (< 1 minute)
- Critical queries can fall back to database
## References
- Database Schema: See `postgres-schema.md`
- Indexer Architecture: See `../indexing/indexer-architecture.md`
- Unified Search: See `../multichain/unified-search.md`

239
docs/specs/database/timeseries-schema.md Normal file
View File

@@ -0,0 +1,239 @@
# Time-Series Database Schema Specification
## Overview
This document specifies the time-series database schema using ClickHouse or TimescaleDB for storing mempool data, metrics, and analytics time-series data.
## Technology Choice
**Option 1: TimescaleDB** (PostgreSQL extension)
- Pros: PostgreSQL compatibility, SQL interface, easier integration
- Cons: Less optimized for very high throughput
**Option 2: ClickHouse**
- Pros: Very high performance, columnar storage, excellent compression
- Cons: Different SQL dialect, separate infrastructure
**Recommendation**: Start with TimescaleDB for easier integration, migrate to ClickHouse if needed for scale.
## TimescaleDB Schema
### Mempool Transactions Table
**Table**: `mempool_transactions`
```sql
CREATE TABLE mempool_transactions (
time TIMESTAMPTZ NOT NULL,
chain_id INTEGER NOT NULL,
hash VARCHAR(66) NOT NULL,
from_address VARCHAR(42) NOT NULL,
to_address VARCHAR(42),
value NUMERIC(78, 0),
gas_price BIGINT,
max_fee_per_gas BIGINT,
max_priority_fee_per_gas BIGINT,
gas_limit BIGINT,
nonce BIGINT,
input_data_length INTEGER,
first_seen TIMESTAMPTZ NOT NULL,
status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'confirmed', 'dropped'
confirmed_block_number BIGINT,
confirmed_at TIMESTAMPTZ,
PRIMARY KEY (time, chain_id, hash)
);
SELECT create_hypertable('mempool_transactions', 'time');
CREATE INDEX idx_mempool_chain_hash ON mempool_transactions(chain_id, hash);
CREATE INDEX idx_mempool_chain_from ON mempool_transactions(chain_id, from_address);
CREATE INDEX idx_mempool_chain_status ON mempool_transactions(chain_id, status, time);
```
**Retention Policy**: 7 days for detailed data, aggregates for longer periods
### Network Metrics Table
**Table**: `network_metrics`
```sql
CREATE TABLE network_metrics (
time TIMESTAMPTZ NOT NULL,
chain_id INTEGER NOT NULL,
block_number BIGINT,
tps DOUBLE PRECISION, -- Transactions per second
gps DOUBLE PRECISION, -- Gas per second
avg_gas_price BIGINT,
pending_transactions INTEGER,
block_time_seconds DOUBLE PRECISION,
PRIMARY KEY (time, chain_id)
);
SELECT create_hypertable('network_metrics', 'time');
CREATE INDEX idx_network_metrics_chain_time ON network_metrics(chain_id, time DESC);
```
**Aggregation**: Pre-aggregate to 1-minute, 5-minute, 1-hour intervals
### Gas Price History Table
**Table**: `gas_price_history`
```sql
CREATE TABLE gas_price_history (
time TIMESTAMPTZ NOT NULL,
chain_id INTEGER NOT NULL,
block_number BIGINT,
min_gas_price BIGINT,
max_gas_price BIGINT,
avg_gas_price BIGINT,
p25_gas_price BIGINT, -- 25th percentile
p50_gas_price BIGINT, -- 50th percentile (median)
p75_gas_price BIGINT, -- 75th percentile
p95_gas_price BIGINT, -- 95th percentile
p99_gas_price BIGINT, -- 99th percentile
PRIMARY KEY (time, chain_id)
);
SELECT create_hypertable('gas_price_history', 'time');
```
### Address Activity Metrics Table
**Table**: `address_activity_metrics`
```sql
CREATE TABLE address_activity_metrics (
time TIMESTAMPTZ NOT NULL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
transaction_count INTEGER,
received_count INTEGER,
sent_count INTEGER,
total_received NUMERIC(78, 0),
total_sent NUMERIC(78, 0),
PRIMARY KEY (time, chain_id, address)
);
SELECT create_hypertable('address_activity_metrics', 'time',
chunk_time_interval => INTERVAL '1 day');
CREATE INDEX idx_address_activity_chain_address ON address_activity_metrics(chain_id, address, time DESC);
```
**Aggregation**: Pre-aggregate to hourly/daily for addresses
## ClickHouse Schema (Alternative)
### Mempool Transactions Table
```sql
CREATE TABLE mempool_transactions (
time DateTime('UTC') NOT NULL,
chain_id UInt32 NOT NULL,
hash String NOT NULL,
from_address String NOT NULL,
to_address Nullable(String),
value Decimal128(0),
gas_price UInt64,
max_fee_per_gas Nullable(UInt64),
max_priority_fee_per_gas Nullable(UInt64),
gas_limit UInt64,
nonce UInt64,
input_data_length UInt32,
first_seen DateTime('UTC') NOT NULL,
status String DEFAULT 'pending',
confirmed_block_number Nullable(UInt64),
confirmed_at Nullable(DateTime('UTC'))
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(time)
ORDER BY (chain_id, time, hash)
TTL time + INTERVAL 7 DAY; -- Auto-delete after 7 days
```
## Data Retention and Aggregation
### Retention Policies
**Raw Data**:
- Mempool transactions: 7 days
- Network metrics: 30 days
- Gas price history: 90 days
- Address activity: 30 days
**Aggregated Data**:
- 1-minute aggregates: 90 days
- 5-minute aggregates: 1 year
- 1-hour aggregates: 5 years
- Daily aggregates: Indefinite
### Continuous Aggregates (TimescaleDB)
```sql
-- 1-minute network metrics aggregate
CREATE MATERIALIZED VIEW network_metrics_1m
WITH (timescaledb.continuous) AS
SELECT
time_bucket('1 minute', time) AS bucket,
chain_id,
AVG(tps) AS avg_tps,
AVG(gps) AS avg_gps,
AVG(avg_gas_price) AS avg_gas_price,
AVG(pending_transactions) AS avg_pending_tx
FROM network_metrics
GROUP BY bucket, chain_id;
-- Add refresh policy
SELECT add_continuous_aggregate_policy('network_metrics_1m',
start_offset => INTERVAL '1 hour',
end_offset => INTERVAL '1 minute',
schedule_interval => INTERVAL '1 minute');
```
## Query Patterns
### Recent Mempool Transactions
```sql
SELECT * FROM mempool_transactions
WHERE chain_id = 138
AND time > NOW() - INTERVAL '1 hour'
AND status = 'pending'
ORDER BY time DESC
LIMIT 100;
```
### Gas Price Statistics
```sql
SELECT
time_bucket('5 minutes', time) AS bucket,
AVG(avg_gas_price) AS avg_gas_price,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY avg_gas_price) AS median_gas_price
FROM gas_price_history
WHERE chain_id = 138
AND time > NOW() - INTERVAL '24 hours'
GROUP BY bucket
ORDER BY bucket DESC;
```
### Network Throughput
```sql
SELECT
time_bucket('1 minute', time) AS bucket,
AVG(tps) AS avg_tps,
MAX(tps) AS max_tps
FROM network_metrics
WHERE chain_id = 138
AND time > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;
```
## References
- Mempool Service: See `../mempool/mempool-service.md`
- Observability: See `../observability/metrics-monitoring.md`

73
docs/specs/deployment/cicd.md Normal file
View File

@@ -0,0 +1,73 @@
# CI/CD Pipeline Specification
## Overview
Continuous Integration and Continuous Deployment pipeline.
## Build Pipeline
### Stages
1. **Source**: Code checkout
2. **Build**: Compile/build application
3. **Test**: Run tests
4. **Package**: Create container images
5. **Deploy**: Deploy to environment
### Tools
**CI/CD Platform**: GitHub Actions, GitLab CI, or Jenkins
**Build**: Docker build
**Registry**: Container registry (Docker Hub, ECR, GCR)
## Test Strategy
### Test Types
**Unit Tests**: Component-level tests
**Integration Tests**: Service integration tests
**E2E Tests**: End-to-end user flow tests
### Test Execution
- Run on every commit
- Block deployment on test failures
- Parallel test execution
## Deployment Strategies
### Blue-Green Deployment
**Method**: Deploy new version alongside old, switch traffic
**Benefits**: Zero downtime, instant rollback
**Use Case**: Production deployments
### Canary Deployment
**Method**: Gradually roll out new version to subset of users
**Benefits**: Risk mitigation, gradual rollout
**Use Case**: Major updates
## Rollback Procedures
### Automatic Rollback
**Triggers**:
- Health check failures
- Error rate spike
- Performance degradation
### Manual Rollback
**Process**:
1. Identify issue
2. Rollback to previous version
3. Verify rollback success
4. Investigate issue
## References
- Infrastructure: See `infrastructure.md`
- Deployment: See `disaster-recovery.md`

69
docs/specs/deployment/disaster-recovery.md Normal file
View File

@@ -0,0 +1,69 @@
# Disaster Recovery Specification
## Overview
Disaster recovery procedures and backup strategies.
## Backup Strategies
### Database Backups
**Full Backups**: Daily full database dumps
**Incremental Backups**: Continuous WAL archiving (PostgreSQL)
**Storage**: Off-site backup storage
**Retention**: 30 days full, 7 days incremental
### Application Backups
**Configuration**: Backup configuration files
**Secrets**: Secure backup of secrets
**Code**: Version control (Git)
## Recovery Procedures
### Recovery Scenarios
**1. Database Corruption**:
- Restore from latest backup
- Replay WAL logs
- Verify data integrity
**2. Service Failure**:
- Restart services
- Verify health
- Check logs
**3. Data Center Failure**:
- Failover to secondary region
- Restore from backups
- Verify functionality
### Recovery Testing
**Frequency**: Quarterly
**Tests**: Restore from backups, verify data integrity
## RTO/RPO Targets
**RTO (Recovery Time Objective)**: 1 hour
**RPO (Recovery Point Objective)**: 5 minutes (max data loss)
## Multi-Region Failover
### Failover Strategy
**Primary Region**: Active services
**Secondary Region**: Standby/replica services
**Failover**: Automatic or manual failover
### Data Replication
**Method**: Database replication, data synchronization
**Latency**: Acceptable replication lag
**Consistency**: Eventual consistency acceptable
## References
- Infrastructure: See `infrastructure.md`
- Scaling: See `scaling.md`

43
docs/specs/deployment/infrastructure.md Normal file
View File

@@ -0,0 +1,43 @@
# Infrastructure as Code Specification
## Overview
Infrastructure as Code (IaC) for containerization and orchestration.
## Containerization Strategy
**Technology**: Docker
**Images**:
- Application services
- Indexers
- APIs
- Worker processes
## Orchestration Patterns
**Technology**: Kubernetes
**Components**:
- Deployments: Application services
- StatefulSets: Database, stateful services
- Jobs: Batch processing
- CronJobs: Scheduled tasks
## Service Discovery
**Method**: Kubernetes DNS
**Service Names**: `service-name.namespace.svc.cluster.local`
**Internal Communication**: Use service names
## Configuration Management
**Method**: ConfigMaps and Secrets
**External Config**: Environment variables
**Secrets**: Kubernetes Secrets or external secret management
## References
- CI/CD: See `cicd.md`
- Scaling: See `scaling.md`

61
docs/specs/deployment/scaling.md Normal file
View File

@@ -0,0 +1,61 @@
# Scaling Strategy Specification
## Overview
Horizontal and vertical scaling strategies for the platform.
## Horizontal Scaling Patterns
### Application Scaling
**Method**: Add more service instances
**Auto-scaling**: Based on CPU, memory, request rate
**Load Balancing**: Distribute traffic across instances
### Database Scaling
**Read Replicas**: Scale read capacity
**Sharding**: Partition data for write scaling
**Connection Pooling**: Efficient connection management
## Database Scaling
### Read Replicas
**Strategy**: Multiple read replicas for read-heavy workloads
**Replication**: Async replication from primary
**Use Case**: API read operations
### Sharding
**Strategy**: Partition data by chain_id
**Implementation**: Database partitioning
**Use Case**: Very large datasets
## Caching Strategy
### Redis Cache
**Use Cases**:
- API response caching
- Session storage
- Rate limiting counters
- Frequently accessed data
### CDN Caching
**Use Cases**:
- Static assets
- API responses (short TTL)
## Load Balancing Configuration
**Method**: Kubernetes service or external load balancer
**Algorithm**: Round-robin or least connections
**Health Checks**: Regular health checks, route to healthy instances
## References
- Infrastructure: See `infrastructure.md`
- CI/CD: See `cicd.md`

86
docs/specs/frontend/action-ui.md Normal file
View File

@@ -0,0 +1,86 @@
# Action Layer UI Specification
## Overview
UI specifications for swap, bridge, and wallet interaction interfaces.
## Swap Interface
### Components
- Token selector (from/to)
- Amount input
- Swap quote display
- Slippage settings
- Gas estimate
- Execute button
### Flow
1. Select tokens
2. Enter amount
3. Display quote
4. Adjust slippage (optional)
5. Connect wallet
6. Approve token (if needed)
7. Execute swap
8. Show transaction status
## Bridge Interface
### Components
- Chain selector (from/to)
- Token selector
- Amount input
- Bridge quote display
- Estimated time
- Execute button
### Flow
Similar to swap with chain selection
## Wallet Connection UI
### Connection Options
- WalletConnect
- Browser extension (MetaMask, etc.)
- Hardware wallet
- Embedded wallet
### UI Elements
- Connection modal
- Wallet list
- Connection status
- Disconnect option
## Transaction Confirmation Flows
### Confirmation Steps
1. Review transaction details
2. Show risk warnings (if any)
3. Gas fee display
4. Final confirmation
5. Transaction pending
6. Success/failure status
## Portfolio View
### Components
- Total portfolio value
- Token list with balances
- Value breakdown
- Transaction history
- Quick actions (swap, bridge, send)
## References
- Frontend Architecture: See `frontend-architecture.md`
- Swap Engine: See `../actions/swap-engine.md`
- Bridge Engine: See `../actions/bridge-engine.md`

83
docs/specs/frontend/component-library.md Normal file
View File

@@ -0,0 +1,83 @@
# Component Library Specification
## Overview
Reusable component library for the explorer platform.
## Design System
**Framework**: Tailwind CSS or styled-components
**Design Tokens**: Colors, typography, spacing, shadows
**Theme**: Light/dark mode support
## Component Catalog
### Common Components
- Button
- Input
- Select
- Modal
- Table
- Card
- Badge
- Tooltip
- Loading spinner
### Blockchain-Specific Components
- Address display (with copy, ENS lookup)
- Transaction hash display
- Block number display
- Token amount display (with formatting)
- Gas price display
- Status badges (success/failed/pending)
### Data Display Components
- Transaction list
- Block list
- Token list
- Log viewer
- Trace viewer
- ABI viewer
## Component API Contracts
### Address Component
```typescript
interface AddressProps {
address: string;
chainId?: number;
showCopy?: boolean;
showENS?: boolean;
truncate?: boolean;
}
```
### Transaction Component
```typescript
interface TransactionProps {
hash: string;
chainId: number;
showDetails?: boolean;
compact?: boolean;
}
```
## Styling Approach
**Recommendation**: Tailwind CSS
**Benefits**:
- Utility-first CSS
- Consistent design system
- Small bundle size
- Fast development
## References
- Frontend Architecture: See `frontend-architecture.md`

122
docs/specs/frontend/explorer-ui.md Normal file
View File

@@ -0,0 +1,122 @@
# Explorer UI Structure Specification
## Overview
UI structure for blockchain explorer pages.
## Page Layouts
### Home/Dashboard
**Components**:
- Network stats (TPS, block height, gas price)
- Recent blocks
- Recent transactions
- Search bar
### Block Detail Page
**Sections**:
- Block header (number, hash, timestamp, miner)
- Block stats (gas used, transaction count)
- Transaction list
- Raw data (optional)
### Transaction Detail Page
**Sections**:
- Transaction header (hash, status, block)
- Transaction details (from, to, value, gas)
- Receipt information
- Logs
- Traces (if available)
- Internal transactions
### Address Detail Page
**Sections**:
- Address header (address, label, tags)
- Balance information
- Transaction list (sent/received/all)
- Token holdings
- NFT holdings (if applicable)
### Token Detail Page
**Sections**:
- Token information (name, symbol, supply)
- Price and market data
- Holders list
- Transfer history
- Token contract (if verified)
### Contract Detail Page
**Sections**:
- Contract address and name
- Verification status
- Source code (if verified)
- ABI
- Contract interactions
- Read functions (if verified)
## Data Tables
### Table Features
- Pagination
- Sorting
- Filtering
- Export (CSV)
- Column customization
### Performance
- Virtual scrolling for large lists
- Lazy loading
- Infinite scroll option
## Filtering and Search UI
### Search Bar
- Global search
- Autocomplete
- Search suggestions
- Recent searches
### Filters
- Chain selector
- Date range
- Transaction type
- Status filters
## Real-Time Update Integration
**Method**: WebSocket subscriptions
**Updates**:
- New blocks
- New transactions
- Balance updates
- Status changes
**UI**: Visual indicators for new data
## Responsive Design
**Breakpoints**:
- Mobile: < 768px
- Tablet: 768px - 1024px
- Desktop: > 1024px
**Adaptations**:
- Stacked layouts on mobile
- Collapsible sections
- Mobile-optimized tables
## References
- Frontend Architecture: See `frontend-architecture.md`
- Component Library: See `component-library.md`

75
docs/specs/frontend/frontend-architecture.md Normal file
View File

@@ -0,0 +1,75 @@
# Frontend Architecture Specification
## Overview
Frontend architecture for the explorer platform web application.
## Framework Selection
**Recommendation**: Next.js (React framework)
**Rationale**:
- Server-side rendering (SSR) for SEO
- API routes for backend integration
- Excellent React ecosystem
- Good performance
- TypeScript support
## Component Architecture
### Structure
```
src/
├── components/ # Reusable components
│ ├── blocks/
│ ├── transactions/
│ ├── addresses/
│ └── common/
├── pages/ # Next.js pages/routes
├── hooks/ # Custom React hooks
├── services/ # API clients
├── store/ # State management
├── types/ # TypeScript types
└── utils/ # Utility functions
```
## State Management
**Strategy**: Zustand or Redux Toolkit
**Global State**:
- User authentication
- Wallet connection
- Selected chain
- UI preferences
**Local State**: Component-level state with React hooks
## Routing Structure
**Routes**:
- `/`: Home/Dashboard
- `/blocks`: Block list
- `/blocks/[number]`: Block detail
- `/transactions/[hash]`: Transaction detail
- `/addresses/[address]`: Address detail
- `/tokens`: Token list
- `/tokens/[address]`: Token detail
- `/contracts/[address]`: Contract detail
- `/search`: Search results
- `/swap`: Swap interface
- `/bridge`: Bridge interface
- `/vtm`: Virtual Teller Machine
## Build and Deployment
**Build Tool**: Next.js built-in
**Deployment**: Vercel, AWS, or self-hosted
**Environment**: Development, staging, production
## References
- Component Library: See `component-library.md`
- Explorer UI: See `explorer-ui.md`

69
docs/specs/frontend/vtm-ui.md Normal file
View File

@@ -0,0 +1,69 @@
# VTM UI Integration Specification
## Overview
UI integration for Virtual Teller Machine (VTM) with Soul Machines digital human.
## Digital Human Embedding
### Container
**Position**: Modal overlay or side panel
**Size**: Responsive, minimum 600x400px
**Z-index**: High (overlay other content)
### Integration Points
- Session initialization
- Conversation display
- User input (text/voice)
- Workflow step indicators
## Conversation Interface
### Components
- Digital human video/avatar
- Message history
- Input field (text)
- Voice input button
- Workflow progress indicator
### Message Display
- User messages (right-aligned)
- Assistant messages (left-aligned)
- System messages (centered)
- Typing indicators
## Workflow Step Indicators
### Progress Indicator
**Display**:
- Current step
- Completed steps
- Remaining steps
- Estimated time
**Visual**: Progress bar or step list
## Human Escalation UI
### Escalation Trigger
**Button**: "Talk to Human Agent"
**Status**: Show when human agent available
**Transfer**: Smooth transition to human agent
### Escalation Status
- Waiting for agent
- Connected to agent
- Agent typing indicator
## References
- VTM Integration: See `../vtm/soul-machines-integration.md`
- Frontend Architecture: See `frontend-architecture.md`

566
docs/specs/indexing/data-models.md Normal file
View File

@@ -0,0 +1,566 @@
# Data Models Specification
## Overview
This document specifies the data models used throughout the indexing pipeline and stored in the database. All models support multi-chain operation via a `chain_id` field.
## Core Data Models
### Block Schema
**Table**: `blocks`
**Fields**:
```sql
blocks (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
number BIGINT NOT NULL,
hash VARCHAR(66) NOT NULL,
parent_hash VARCHAR(66) NOT NULL,
nonce VARCHAR(18),
sha3_uncles VARCHAR(66),
logs_bloom TEXT,
transactions_root VARCHAR(66),
state_root VARCHAR(66),
receipts_root VARCHAR(66),
miner VARCHAR(42),
difficulty NUMERIC,
total_difficulty NUMERIC,
size BIGINT,
extra_data TEXT,
gas_limit BIGINT,
gas_used BIGINT,
timestamp TIMESTAMP NOT NULL,
transaction_count INTEGER DEFAULT 0,
base_fee_per_gas BIGINT, -- EIP-1559
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(chain_id, number),
UNIQUE(chain_id, hash)
)
```
**Indexes**:
- `idx_blocks_chain_number` ON (chain_id, number)
- `idx_blocks_chain_hash` ON (chain_id, hash)
- `idx_blocks_chain_timestamp` ON (chain_id, timestamp)
**Relationships**:
- One-to-many with `transactions`
- One-to-many with `logs`
### Transaction Schema
**Table**: `transactions`
**Fields**:
```sql
transactions (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
hash VARCHAR(66) NOT NULL,
block_number BIGINT NOT NULL,
block_hash VARCHAR(66) NOT NULL,
transaction_index INTEGER NOT NULL,
from_address VARCHAR(42) NOT NULL,
to_address VARCHAR(42), -- NULL for contract creation
value NUMERIC NOT NULL DEFAULT 0,
gas_price BIGINT,
max_fee_per_gas BIGINT, -- EIP-1559
max_priority_fee_per_gas BIGINT, -- EIP-1559
gas_limit BIGINT NOT NULL,
gas_used BIGINT,
nonce BIGINT NOT NULL,
input_data TEXT, -- Contract call data
status INTEGER, -- 0 = failed, 1 = success
contract_address VARCHAR(42), -- NULL if not contract creation
cumulative_gas_used BIGINT,
effective_gas_price BIGINT, -- Actual gas price paid
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, block_number) REFERENCES blocks(chain_id, number),
UNIQUE(chain_id, hash)
)
```
**Indexes**:
- `idx_transactions_chain_hash` ON (chain_id, hash)
- `idx_transactions_chain_block` ON (chain_id, block_number, transaction_index)
- `idx_transactions_chain_from` ON (chain_id, from_address)
- `idx_transactions_chain_to` ON (chain_id, to_address)
- `idx_transactions_chain_block_from` ON (chain_id, block_number, from_address)
**Relationships**:
- Many-to-one with `blocks`
- One-to-many with `logs`
- One-to-many with `internal_transactions`
- One-to-many with `token_transfers`
### Receipt Schema
**Note**: Receipt data is stored denormalized in the `transactions` table for efficiency. If separate storage is needed:
**Table**: `transaction_receipts`
```sql
transaction_receipts (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
transaction_hash VARCHAR(66) NOT NULL,
transaction_index INTEGER NOT NULL,
block_number BIGINT NOT NULL,
block_hash VARCHAR(66) NOT NULL,
from_address VARCHAR(42) NOT NULL,
to_address VARCHAR(42),
gas_used BIGINT,
cumulative_gas_used BIGINT,
contract_address VARCHAR(42),
logs_bloom TEXT,
status INTEGER,
root VARCHAR(66), -- Pre-Byzantium
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, transaction_hash) REFERENCES transactions(chain_id, hash),
UNIQUE(chain_id, transaction_hash)
)
```
### Log Schema
**Table**: `logs`
**Fields**:
```sql
logs (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
transaction_hash VARCHAR(66) NOT NULL,
block_number BIGINT NOT NULL,
block_hash VARCHAR(66) NOT NULL,
log_index INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
topic0 VARCHAR(66), -- Event signature
topic1 VARCHAR(66), -- First indexed parameter
topic2 VARCHAR(66), -- Second indexed parameter
topic3 VARCHAR(66), -- Third indexed parameter
data TEXT, -- Non-indexed parameters
decoded_data JSONB, -- Decoded event data (if ABI available)
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, transaction_hash) REFERENCES transactions(chain_id, hash),
UNIQUE(chain_id, transaction_hash, log_index)
)
```
**Indexes**:
- `idx_logs_chain_tx` ON (chain_id, transaction_hash)
- `idx_logs_chain_address` ON (chain_id, address)
- `idx_logs_chain_topic0` ON (chain_id, topic0)
- `idx_logs_chain_block` ON (chain_id, block_number)
- `idx_logs_chain_address_topic0` ON (chain_id, address, topic0) -- For event filtering
**Relationships**:
- Many-to-one with `transactions`
### Trace Schema
**Table**: `traces`
**Fields**:
```sql
traces (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
transaction_hash VARCHAR(66) NOT NULL,
block_number BIGINT NOT NULL,
block_hash VARCHAR(66) NOT NULL,
trace_address INTEGER[], -- Array representing call hierarchy [0,1,2]
subtraces INTEGER, -- Number of child calls
action_type VARCHAR(20) NOT NULL, -- 'call', 'create', 'suicide', 'delegatecall'
action_from VARCHAR(42),
action_to VARCHAR(42),
action_value NUMERIC DEFAULT 0,
action_input TEXT,
action_gas BIGINT,
action_call_type VARCHAR(20), -- 'call', 'delegatecall', 'staticcall'
result_type VARCHAR(20), -- 'callresult', 'createresult'
result_gas_used BIGINT,
result_output TEXT,
result_address VARCHAR(42), -- For create results
result_code TEXT, -- For create results
error TEXT, -- Error message if trace failed
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, transaction_hash) REFERENCES transactions(chain_id, hash)
)
```
**Indexes**:
- `idx_traces_chain_tx` ON (chain_id, transaction_hash)
- `idx_traces_chain_block` ON (chain_id, block_number)
- `idx_traces_chain_from` ON (chain_id, action_from)
- `idx_traces_chain_to` ON (chain_id, action_to)
**Note**: Trace data can be large. Consider partitioning or separate storage for historical traces.
### Internal Transaction Schema
**Table**: `internal_transactions`
**Purpose**: Track value transfers that occur within transactions (via calls).
**Fields**:
```sql
internal_transactions (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
transaction_hash VARCHAR(66) NOT NULL,
block_number BIGINT NOT NULL,
trace_address INTEGER[] NOT NULL,
from_address VARCHAR(42) NOT NULL,
to_address VARCHAR(42) NOT NULL,
value NUMERIC NOT NULL,
call_type VARCHAR(20), -- 'call', 'delegatecall', 'staticcall', 'create'
gas_limit BIGINT,
gas_used BIGINT,
input_data TEXT,
output_data TEXT,
error TEXT,
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, transaction_hash) REFERENCES transactions(chain_id, hash)
)
```
**Indexes**:
- `idx_internal_tx_chain_tx` ON (chain_id, transaction_hash)
- `idx_internal_tx_chain_from` ON (chain_id, from_address)
- `idx_internal_tx_chain_to` ON (chain_id, to_address)
- `idx_internal_tx_chain_block` ON (chain_id, block_number)
**Relationships**:
- Many-to-one with `transactions`
### Token Transfer Schema
**Table**: `token_transfers`
**Purpose**: Track ERC-20, ERC-721, and ERC-1155 token transfers.
**Fields**:
```sql
token_transfers (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
transaction_hash VARCHAR(66) NOT NULL,
block_number BIGINT NOT NULL,
log_index INTEGER NOT NULL,
token_address VARCHAR(42) NOT NULL,
token_type VARCHAR(10) NOT NULL, -- 'ERC20', 'ERC721', 'ERC1155'
from_address VARCHAR(42) NOT NULL,
to_address VARCHAR(42) NOT NULL,
amount NUMERIC, -- For ERC-20 and ERC-1155
token_id VARCHAR(78), -- For ERC-721 and ERC-1155 (can be large)
operator VARCHAR(42), -- For ERC-1155
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, transaction_hash) REFERENCES transactions(chain_id, hash),
FOREIGN KEY (chain_id, token_address) REFERENCES tokens(chain_id, address)
)
```
**Indexes**:
- `idx_token_transfers_chain_token` ON (chain_id, token_address)
- `idx_token_transfers_chain_from` ON (chain_id, from_address)
- `idx_token_transfers_chain_to` ON (chain_id, to_address)
- `idx_token_transfers_chain_tx` ON (chain_id, transaction_hash)
- `idx_token_transfers_chain_block` ON (chain_id, block_number)
- `idx_token_transfers_chain_token_from` ON (chain_id, token_address, from_address)
- `idx_token_transfers_chain_token_to` ON (chain_id, token_address, to_address)
**Relationships**:
- Many-to-one with `transactions`
- Many-to-one with `tokens`
### Token Schema
**Table**: `tokens`
**Purpose**: Store token metadata (ERC-20, ERC-721, ERC-1155).
**Fields**:
```sql
tokens (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
type VARCHAR(10) NOT NULL, -- 'ERC20', 'ERC721', 'ERC1155'
name VARCHAR(255),
symbol VARCHAR(50),
decimals INTEGER, -- For ERC-20
total_supply NUMERIC,
holder_count INTEGER DEFAULT 0,
transfer_count INTEGER DEFAULT 0,
logo_url TEXT,
website_url TEXT,
description TEXT,
verified BOOLEAN DEFAULT false,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(chain_id, address)
)
```
**Indexes**:
- `idx_tokens_chain_address` ON (chain_id, address)
- `idx_tokens_chain_type` ON (chain_id, type)
- `idx_tokens_chain_symbol` ON (chain_id, symbol) -- For search
**Relationships**:
- One-to-many with `token_transfers`
- One-to-many with `token_holders` (if maintained)
### Contract Metadata Schema
**Table**: `contracts`
**Purpose**: Store verified contract information.
**Fields**:
```sql
contracts (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
name VARCHAR(255),
compiler_version VARCHAR(50),
optimization_enabled BOOLEAN,
optimization_runs INTEGER,
evm_version VARCHAR(20),
source_code TEXT,
abi JSONB,
constructor_arguments TEXT,
verification_status VARCHAR(20) NOT NULL, -- 'pending', 'verified', 'failed'
verified_at TIMESTAMP,
verification_method VARCHAR(50), -- 'standard_json', 'sourcify', 'multi_file'
license VARCHAR(50),
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(chain_id, address)
)
```
**Indexes**:
- `idx_contracts_chain_address` ON (chain_id, address)
- `idx_contracts_chain_verified` ON (chain_id, verification_status)
**Relationships**:
- One-to-one with `contract_abis` (if separate ABI storage)
### Contract ABI Schema
**Table**: `contract_abis`
**Purpose**: Store contract ABIs for decoding (can be separate from verification).
**Fields**:
```sql
contract_abis (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
abi JSONB NOT NULL,
source VARCHAR(50) NOT NULL, -- 'verification', 'sourcify', 'public', 'user_submitted'
verified BOOLEAN DEFAULT false,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(chain_id, address)
)
```
**Indexes**:
- `idx_abis_chain_address` ON (chain_id, address)
## Address-Related Models
### Address Labels Schema
**Table**: `address_labels`
**Purpose**: User-defined and public labels for addresses.
**Fields**:
```sql
address_labels (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
label VARCHAR(255) NOT NULL,
label_type VARCHAR(20) NOT NULL, -- 'user', 'public', 'contract_name'
user_id UUID, -- NULL for public labels
source VARCHAR(50), -- 'user', 'etherscan', 'blockscout', etc.
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(chain_id, address, label_type, user_id)
)
```
**Indexes**:
- `idx_labels_chain_address` ON (chain_id, address)
- `idx_labels_chain_user` ON (chain_id, user_id)
### Address Tags Schema
**Table**: `address_tags`
**Purpose**: Categorize addresses (e.g., "exchange", "defi", "wallet").
**Fields**:
```sql
address_tags (
id BIGSERIAL PRIMARY KEY,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
tag VARCHAR(50) NOT NULL,
tag_type VARCHAR(20) NOT NULL, -- 'category', 'risk', 'protocol'
user_id UUID, -- NULL for public tags
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE(chain_id, address, tag, user_id)
)
```
**Indexes**:
- `idx_tags_chain_address` ON (chain_id, address)
- `idx_tags_chain_tag` ON (chain_id, tag)
## User-Related Models
### User Accounts Schema
**Table**: `users`
**Purpose**: User accounts for watchlists, alerts, preferences.
**Fields**:
```sql
users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email VARCHAR(255) UNIQUE,
username VARCHAR(100) UNIQUE,
password_hash TEXT, -- If using password auth
api_key_hash TEXT, -- Hashed API key
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
last_login_at TIMESTAMP
)
```
**Indexes**:
- `idx_users_email` ON (email)
- `idx_users_username` ON (username)
### Watchlists Schema
**Table**: `watchlists`
**Purpose**: User-defined lists of addresses to monitor.
**Fields**:
```sql
watchlists (
id BIGSERIAL PRIMARY KEY,
user_id UUID NOT NULL,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
label VARCHAR(255),
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE,
UNIQUE(user_id, chain_id, address)
)
```
**Indexes**:
- `idx_watchlists_user` ON (user_id)
- `idx_watchlists_chain_address` ON (chain_id, address)
## Data Type Definitions
### Numeric Types
- **BIGINT**: Used for block numbers, gas values, nonces (64-bit integers)
- **NUMERIC**: Used for token amounts, ETH values (arbitrary precision decimals)
- Precision: 78 digits (sufficient for wei)
- Scale: 0 (integers) or configurable for token decimals
### Address Types
- **VARCHAR(42)**: Ethereum addresses (0x + 40 hex chars)
- Normalize to lowercase for consistency
### Hash Types
- **VARCHAR(66)**: Transaction/block hashes (0x + 64 hex chars)
- **TEXT**: For very long hashes or variable-length data
### JSONB Types
- Used for: ABIs, decoded event data, complex nested structures
- Benefits: Indexing, querying, efficient storage
## Multi-Chain Considerations
### Chain ID Partitioning
All tables include `chain_id` as the first column after primary key:
- Enables efficient partitioning by chain_id
- Ensures data isolation between chains
- Simplifies multi-chain queries
### Partitioning Strategy
**Recommended**: Partition large tables by `chain_id`:
- `blocks`, `transactions`, `logs` partitioned by chain_id
- Benefits: Faster queries, easier maintenance, parallel processing
**Implementation** (PostgreSQL):
```sql
-- Example partitioning
CREATE TABLE blocks (
-- columns
) PARTITION BY LIST (chain_id);
CREATE TABLE blocks_chain_138 PARTITION OF blocks FOR VALUES IN (138);
CREATE TABLE blocks_chain_1 PARTITION OF blocks FOR VALUES IN (1);
```
## Data Consistency
### Foreign Key Constraints
- Enforce referential integrity where possible
- Consider performance impact for high-throughput inserts
- May disable for initial backfill, enable after catch-up
### Unique Constraints
- Prevent duplicate blocks, transactions, logs
- Enable idempotent processing
- Use ON CONFLICT for upserts
## Indexing Strategy
### Index Types
1. **B-tree**: Default for most indexes (equality, range queries)
2. **Hash**: For exact match lookups (addresses, hashes)
3. **GIN**: For JSONB columns (ABIs, decoded data)
4. **BRIN**: For large ordered columns (block numbers, timestamps)
### Index Maintenance
- Regular VACUUM and ANALYZE
- Monitor index bloat
- Consider partial indexes for filtered queries
## References
- Indexer Architecture: See `indexer-architecture.md`
- Database Schema: See `../database/postgres-schema.md`
- Search Index Schema: See `../database/search-index-schema.md`

518
docs/specs/indexing/indexer-architecture.md Normal file
View File

@@ -0,0 +1,518 @@
# Indexer Architecture Specification
## Overview
This document specifies the architecture for the blockchain indexing pipeline that ingests, processes, and stores blockchain data from ChainID 138 and other supported chains. The indexer is responsible for maintaining a complete, queryable database of blocks, transactions, logs, traces, and token transfers.
## Architecture
```mermaid
flowchart TB
subgraph Input[Input Layer]
Node[RPC Node<br/>ChainID 138]
WS[WebSocket<br/>New Block Events]
end
subgraph Ingest[Ingestion Layer]
BL[Block Listener<br/>Real-time]
BW[Backfill Worker<br/>Historical]
Q[Message Queue<br/>Kafka/RabbitMQ]
end
subgraph Process[Processing Layer]
BP[Block Processor]
TP[Transaction Processor]
LP[Log Processor]
TrP[Trace Processor]
TokenP[Token Transfer Processor]
end
subgraph Decode[Decoding Layer]
ABI[ABI Registry]
SigDB[Signature Database]
Decoder[Event Decoder]
end
subgraph Persist[Persistence Layer]
PG[(PostgreSQL<br/>Canonical Data)]
ES[(Elasticsearch<br/>Search Index)]
TS[(TimescaleDB<br/>Metrics)]
end
subgraph Materialize[Materialization Layer]
Agg[Aggregator<br/>TPS, Gas Stats]
Cache[Cache Layer<br/>Redis]
end
Node --> BL
Node --> BW
WS --> BL
BL --> Q
BW --> Q
Q --> BP
BP --> TP
BP --> LP
BP --> TrP
TP --> TokenP
LP --> Decoder
Decoder --> ABI
Decoder --> SigDB
BP --> PG
TP --> PG
LP --> PG
TrP --> PG
TokenP --> PG
BP --> ES
TP --> ES
LP --> ES
BP --> TS
TP --> TS
PG --> Agg
Agg --> Cache
```
## Block Ingestion Pipeline
### Block Listener (Real-time)
**Purpose**: Monitor blockchain for new blocks and ingest them immediately.
**Implementation**:
- Subscribe to `newHeads` via WebSocket
- Poll `eth_blockNumber` as fallback (every 2 seconds)
- Handle WebSocket reconnection automatically
**Flow**:
1. Receive block header event
2. Fetch full block data via `eth_getBlockByNumber`
3. Enqueue block to processing queue
4. Acknowledge receipt
**Error Handling**:
- Retry on network errors (exponential backoff)
- Handle reorgs (see reorg handling section)
- Log errors for monitoring
### Backfill Worker (Historical)
**Purpose**: Index historical blocks from genesis or a specific starting point.
**Implementation**:
- Parallel workers for faster indexing
- Configurable batch size (e.g., 100 blocks per batch)
- Rate limiting to avoid overloading RPC node
- Checkpoint system for resuming interrupted backfills
**Flow**:
1. Determine starting block (checkpoint or genesis)
2. Fetch batch of blocks
3. Enqueue each block to processing queue
4. Update checkpoint
5. Repeat until caught up with chain head
**Optimization Strategies**:
- Parallel workers process different block ranges
- Skip blocks already indexed (idempotent processing)
- Batch RPC requests where possible
### Message Queue
**Purpose**: Decouple ingestion from processing, enable scaling, ensure durability.
**Technology**: Kafka or RabbitMQ
**Topics/Queues**:
- `blocks`: New blocks to process
- `transactions`: Transactions to decode
- `traces`: Traces to process (async)
**Configuration**:
- Durability: Persistent storage
- Replication: 3 replicas for high availability
- Partitioning: By chain_id and block number (for ordering)
## Transaction Processing Flow
### Block Processing
**Steps**:
1. **Validate Block**: Verify block hash, parent hash, block number
2. **Extract Transactions**: Get transaction list from block
3. **Fetch Receipts**: Get transaction receipts for all transactions
4. **Process Each Transaction**:
- Store transaction data
- Process receipt (logs, status)
- Extract token transfers (ERC-20/721/1155)
- Link to contract interactions
**Data Extracted**:
- Transaction fields (hash, from, to, value, gas, etc.)
- Receipt fields (status, gasUsed, logs, etc.)
- Contract creation detection
- Token transfer events
### Transaction Decoding
**Purpose**: Decode event logs and transaction data using ABIs.
**Process**:
1. Identify contract address (to field or created address)
2. Look up ABI in registry (verified contracts)
3. Decode function calls and events
4. Store decoded data for search and filtering
**Fallback Strategies**:
- Signature database for unknown functions/events (4-byte signatures)
- Heuristic detection for common patterns (Transfer events)
- Store raw data when decoding fails
### ABI Registry
**Purpose**: Store contract ABIs for decoding transactions and events.
**Data Sources**:
- Contract verification submissions
- Sourcify integration
- Public ABI repositories (4byte.directory, etc.)
**Storage**:
- Database table: `contract_abis`
- Cache layer: Redis for frequently accessed ABIs
- Versioning: Support multiple ABI versions per contract
**Schema**:
```sql
contract_abis (
id UUID PRIMARY KEY,
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
abi JSONB NOT NULL,
verified BOOLEAN DEFAULT false,
source VARCHAR(50), -- 'verification', 'sourcify', 'public'
created_at TIMESTAMP,
updated_at TIMESTAMP,
UNIQUE(chain_id, address)
)
```
### Signature Database
**Purpose**: Map 4-byte function signatures and 32-byte event signatures to function/event names.
**Data Sources**:
- Public signature databases (4byte.directory)
- User submissions
- Automatic extraction from verified contracts
**Usage**:
- Lookup function name from signature (e.g., `0x095ea7b3` → `approve(address,uint256)`)
- Lookup event name from topic[0] (e.g., `0xddf252...` → `Transfer(address,address,uint256)`)
- Partial decoding when full ABI unavailable
## Event Log Indexing
### Log Processing
**Purpose**: Index event logs for efficient querying and filtering.
**Process**:
1. Extract logs from transaction receipts
2. Decode log topics and data using ABI
3. Index by:
- Contract address
- Event signature (topic[0])
- Indexed parameters (topic[1..3])
- Block number and transaction hash
- Log index
**Indexing Strategy**:
- PostgreSQL table: `logs` with indexes on (address, topic0, block_number)
- Elasticsearch index: Full-text search on decoded event data
- Time-series: Aggregate log counts per contract/event
### Event Decoding
**Decoding Flow**:
1. Identify event signature from topic[0]
2. Look up event definition in ABI registry
3. Decode indexed parameters (topics 1-3)
4. Decode non-indexed parameters (data field)
5. Store decoded parameters as JSONB
**Common Events to Index**:
- ERC-20: `Transfer(address,address,uint256)`
- ERC-721: `Transfer(address,address,uint256)`
- ERC-1155: `TransferSingle`, `TransferBatch`
- Approval events: `Approval(address,address,uint256)`
## Trace Processing
### Call Trace Extraction
**Purpose**: Extract detailed call traces for transaction debugging and internal transaction tracking.
**Trace Types**:
- `call`: Contract calls
- `create`: Contract creation
- `suicide`: Contract self-destruct
- `delegatecall`: Delegate calls
**Process**:
1. Request trace via `trace_transaction` or `trace_block`
2. Parse trace result structure
3. Extract:
- Call hierarchy (parent-child relationships)
- Internal transactions (value transfers)
- Gas usage per call
- Revert information
### Internal Transaction Tracking
**Purpose**: Track value transfers that occur inside transactions (not just top-level).
**Data Extracted**:
- From address (caller)
- To address (callee)
- Value transferred
- Call type (call, delegatecall, etc.)
- Success/failure status
- Gas used
**Storage**:
- Separate table: `internal_transactions`
- Link to parent transaction via `transaction_hash`
- Link to parent call via `trace_address` array
## Token Transfer Extraction
### ERC-20 Transfer Detection
**Detection Method**:
1. Look for `Transfer(address,address,uint256)` event
2. Decode event parameters (from, to, value)
3. Store in `token_transfers` table
4. Update token holder balances
**Data Stored**:
- Token contract address
- From address
- To address
- Amount (with decimals)
- Block number
- Transaction hash
- Log index
### ERC-721 Transfer Detection
**Similar to ERC-20 but**:
- Token ID is tracked (unique NFT)
- Transfer can be from zero address (mint) or to zero address (burn)
### ERC-1155 Transfer Detection
**Events**:
- `TransferSingle`: Single token transfer
- `TransferBatch`: Batch token transfer
**Challenges**:
- Multiple token IDs and amounts per transfer
- Batch operations require array decoding
### Token Holder Tracking
**Purpose**: Maintain list of addresses holding each token.
**Strategy**:
- Real-time updates: Update on each transfer
- Periodic reconciliation: Verify balances via RPC
- Balance snapshots: Store balance at each block (for historical queries)
## Indexer Worker Scaling and Partitioning
### Horizontal Scaling
**Strategy**: Multiple indexer workers processing different blocks/chains.
**Partitioning Methods**:
1. **By Chain**: Each worker handles one chain
2. **By Block Range**: Workers split block ranges (for backfill)
3. **By Processing Stage**: Separate workers for blocks, traces, token transfers
### Worker Coordination
**Mechanisms**:
- Message queue: Workers consume from shared queue
- Database locks: Prevent duplicate processing
- Leader election: For single-worker tasks (reorg handling)
### Load Balancing
**Distribution**:
- Round-robin for backfill workers
- Sticky sessions for chain-specific workers
- Priority queuing: Real-time blocks before historical blocks
### Performance Targets
**Throughput**:
- Process 100 blocks/minute per worker
- Process 1000 transactions/minute per worker
- Process 100 traces/minute per worker (trace operations are slower)
**Latency**:
- Real-time blocks: Indexed within 5 seconds of block production
- Historical blocks: Catch up to chain head within reasonable time
## Data Consistency
### Transaction Isolation
**Strategy**: Process blocks atomically (all or nothing).
**Implementation**:
- Database transactions for block-level operations
- Idempotent processing (can safely retry)
- Checkpoint system to track last processed block
### Idempotency
**Requirements**:
- Processing same block multiple times should not create duplicates
- Use unique constraints in database
- Upsert operations where applicable
## Error Handling and Retry Logic
### Error Types
1. **Transient Errors**: Network issues, temporary RPC failures
- Retry with exponential backoff
- Max retries: 10
- Max backoff: 5 minutes
2. **Permanent Errors**: Invalid data, unsupported features
- Log error and skip
- Alert for investigation
3. **Reorg Errors**: Block replaced by different block
- Handle via reorg detection (see reorg handling spec)
### Retry Strategy
**Exponential Backoff**:
- Initial delay: 1 second
- Multiplier: 2x
- Max delay: 5 minutes
- Jitter: Random ±20% to avoid thundering herd
## Monitoring and Observability
### Key Metrics
**Throughput**:
- Blocks processed per minute
- Transactions processed per minute
- Logs indexed per minute
**Latency**:
- Time from block production to index completion
- Time to process block (p50, p95, p99)
**Lag**:
- Block height lag (current block - last indexed block)
- Time lag (current time - last indexed block time)
**Errors**:
- Error rate by type
- Retry count
- Failed blocks
### Alerting Rules
- Block lag > 10 blocks: Warning
- Block lag > 100 blocks: Critical
- Error rate > 1%: Warning
- Error rate > 5%: Critical
- Worker down: Critical
## Integration Points
### RPC Node Integration
- See `../infrastructure/node-rpc-architecture.md`
- Connection pooling
- Rate limiting awareness
- Failover handling
### Database Integration
- See `../database/postgres-schema.md`
- Connection pooling
- Batch inserts for performance
- Transaction management
### Search Integration
- See `../database/search-index-schema.md`
- Async indexing to Elasticsearch
- Bulk indexing for efficiency
## Implementation Guidelines
### Technology Stack
**Recommended**:
- **Language**: Go, Rust, or Python (performance considerations)
- **Queue**: Kafka (high throughput) or RabbitMQ (simpler setup)
- **Database**: PostgreSQL with connection pooling
- **Caching**: Redis for frequently accessed data
### Code Structure
```
indexer/
├── cmd/
│ ├── block-listener/ # Real-time block listener
│ ├── backfill-worker/ # Historical indexing worker
│ └── processor/ # Block/transaction processor
├── internal/
│ ├── ingestion/ # Ingestion logic
│ ├── processing/ # Processing logic
│ ├── decoding/ # ABI/signature decoding
│ └── persistence/ # Database operations
└── pkg/
├── abi/ # ABI registry
└── rpc/ # RPC client
```
### Testing Strategy
**Unit Tests**:
- Decoding logic
- Data transformation
- Error handling
**Integration Tests**:
- End-to-end block processing
- Database operations
- Queue integration
**Load Tests**:
- Process historical blocks
- Simulate high block production rate
- Test worker scaling
## References
- Data Models: See `data-models.md`
- Reorg Handling: See `reorg-handling.md`
- Database Schema: See `../database/postgres-schema.md`
- RPC Architecture: See `../infrastructure/node-rpc-architecture.md`

353
docs/specs/indexing/reorg-handling.md Normal file
View File

@@ -0,0 +1,353 @@
# Reorg Handling Specification
## Overview
This document specifies how the indexer handles blockchain reorganizations (reorgs), where the canonical chain changes and previously indexed blocks become invalid. Reorg handling ensures data consistency and maintains accurate blockchain state.
## Reorg Detection
### Detection Methods
**1. Block Hash Mismatch**
- Compare stored block hash with RPC node block hash
- If mismatch detected, reorg has occurred
**2. Parent Hash Validation**
- Verify each block's parent_hash matches previous block's hash
- Chain break indicates reorg point
**3. Block Height Comparison**
- Monitor chain head block number
- Sudden decrease indicates potential reorg
**4. Chain Head Monitoring**
- Poll `eth_blockNumber` periodically
- Compare with last indexed block number
- Detect rollback scenarios
### Detection Strategy
**Real-time Monitoring**:
- Check block hash after each new block ingestion
- Compare with RPC node's block hash for same block number
- Immediate detection for recent blocks
**Periodic Validation**:
- Validate last N blocks (e.g., 100 blocks) every M seconds
- Detect deep reorgs that may have been missed
- Background validation job
**Checkpoint Validation**:
- Validate checkpoint blocks (every 1000 blocks)
- Ensure checkpoint blocks still exist in canonical chain
- Detect deep reorgs quickly
## Reorg Handling Flow
```mermaid
flowchart TB
Detect[Detect Reorg]
Identify[Identify Reorg Point<br/>Find Common Ancestor]
Mark[Mark Blocks as Orphaned]
Delete[Delete Orphaned Data]
Reindex[Re-index New Chain]
Verify[Verify Data Consistency]
Detect --> Identify
Identify --> Mark
Mark --> Delete
Delete --> Reindex
Reindex --> Verify
Verify --> Done[Complete]
Verify -->|Inconsistency| Delete
```
### Step 1: Identify Reorg Point
**Goal**: Find the common ancestor block (last block that's still valid).
**Algorithm**:
1. Start from current chain head
2. Compare block hash with stored hash at each block number
3. When hash matches, that's the common ancestor
4. All blocks after common ancestor need reorg handling
**Optimization**:
- Binary search to find reorg point quickly
- Start from recent blocks and work backwards
- Cache last N block hashes for faster comparison
### Step 2: Mark Blocks as Orphaned
**Strategy**: Mark blocks as orphaned before deletion (for audit trail).
**Database Update**:
```sql
UPDATE blocks
SET orphaned = true, orphaned_at = NOW()
WHERE chain_id = ?
AND block_number > ?
AND orphaned = false;
```
**Benefits**:
- Audit trail of reorgs
- Ability to query orphaned blocks
- Easier debugging
### Step 3: Delete Orphaned Data
**Cascade Deletion Order**:
1. Token transfers (depends on transactions)
2. Logs (depends on transactions)
3. Traces (depends on transactions)
4. Internal transactions (depends on transactions)
5. Transactions (depends on blocks)
6. Blocks (orphaned blocks)
**Implementation**:
- Use database transactions for atomicity
- Cascade deletes via foreign keys (if enabled)
- Or explicit deletion in correct order
**Performance Considerations**:
- Batch deletions for large reorgs
- Index on `block_number` for efficient deletion
- Consider soft deletes (mark as deleted) vs hard deletes
### Step 4: Re-index New Chain
**Process**:
1. Fetch new blocks from RPC node (from reorg point onward)
2. Process blocks through normal indexing pipeline
3. Verify each block before marking as indexed
**Optimization**:
- Parallel processing for multiple blocks (if safe)
- Use existing indexer workers
- Priority queue: reorg blocks before new blocks
### Step 5: Verify Data Consistency
**Validation Checks**:
- Block hashes match RPC node
- Transaction counts match
- Parent hashes form valid chain
- No gaps in block numbers
**Metrics**:
- Reorg depth (number of blocks affected)
- Reorg duration (time to handle)
- Data consistency verification
## Data Consistency Guarantees
### Transaction Isolation
**Strategy**: Use database transactions to ensure atomic reorg handling.
**Implementation**:
```sql
BEGIN;
-- Mark blocks as orphaned
-- Delete orphaned data
-- Insert new blocks
-- Verify consistency
COMMIT;
```
**Rollback**: If any step fails, rollback entire operation.
### Idempotency
**Requirement**: Reorg handling must be idempotent (safe to retry).
**Mechanisms**:
- Check block hash before processing
- Skip blocks already correctly indexed
- Use unique constraints to prevent duplicates
### Finality Considerations
**Reorg Depth Limits**:
- Only handle reorgs within finality window
- For PoW: Typically 12-100 blocks deep
- For PoS: Typically 1-2 epochs (32-64 blocks)
- For BFT: Typically immediate finality
**Configuration**:
- Configurable reorg depth limit per chain
- Skip reorgs beyond finality window (log for investigation)
## Performance Optimization
### Handling Frequent Reorgs
**Problem**: Some chains have frequent small reorgs (1-2 blocks).
**Solution**:
- Batch reorg handling (wait for stability)
- Detect reorgs but delay handling for short period
- Only handle if reorg persists
**Configuration**:
- Reorg confirmation period: 30 seconds
- Maximum reorg depth to handle immediately: 5 blocks
- Deeper reorgs: Manual investigation
### Handling Deep Reorgs
**Problem**: Deep reorgs require deleting and re-indexing many blocks.
**Optimization Strategies**:
1. **Parallel Processing**: Process new chain blocks in parallel
2. **Batch Operations**: Batch database operations
3. **Incremental Updates**: Only update changed data
4. **Selective Deletion**: Only delete affected data (not entire blocks if possible)
### Index Maintenance
**During Reorg**:
- Pause index updates temporarily
- Resume after reorg handling complete
- Rebuild affected indexes if necessary
## Monitoring and Alerting
### Metrics
**Reorg Metrics**:
- Reorg count (per chain, per time period)
- Reorg depth distribution
- Reorg handling duration (p50, p95, p99)
- Data consistency check results
**Alerting Rules**:
- Reorg depth > 10 blocks: Warning (investigate)
- Reorg depth > 100 blocks: Critical (potential chain issue)
- Reorg handling duration > 5 minutes: Warning
- Data consistency check failure: Critical
### Logging
**Log Events**:
- Reorg detection (block number, depth)
- Reorg point identification (common ancestor)
- Blocks orphaned (count, block numbers)
- Re-indexing progress
- Data consistency verification results
**Log Levels**:
- INFO: Normal reorgs (< 5 blocks)
- WARN: Unusual reorgs (5-10 blocks)
- ERROR: Deep reorgs (> 10 blocks) or failures
## Edge Cases
### Multiple Reorgs in Quick Succession
**Scenario**: Chain reorgs, then reorgs again before first reorg is handled.
**Handling**:
- Cancel in-progress reorg handling
- Start new reorg handling from latest state
- Ensure idempotency
### Reorg During Backfill
**Scenario**: Historical block indexing encounters a reorg.
**Handling**:
- Pause backfill
- Handle reorg
- Resume backfill from reorg point
### Concurrent Reorg Handling
**Prevention**:
- Use database locks to prevent concurrent reorg handling
- Single reorg handler per chain
- Queue reorg events if handler is busy
## Recovery Procedures
### Manual Reorg Handling
**When to Use**:
- Automatic handling fails
- Deep reorgs beyond normal limits
- Data corruption detected
**Procedure**:
1. Identify reorg point manually
2. Verify with RPC node
3. Mark blocks as orphaned
4. Delete orphaned data
5. Trigger re-indexing
6. Verify data consistency
### Data Recovery
**Backup Strategy**:
- Regular database backups
- Point-in-time recovery capability
- Ability to restore to pre-reorg state
**Recovery Steps**:
1. Restore database to point before reorg
2. Re-run indexer from that point
3. Let automatic reorg handling process naturally
## Testing Strategy
### Unit Tests
- Reorg detection logic
- Common ancestor identification
- Orphan marking
- Data deletion logic
### Integration Tests
- Simulate reorgs (testnet with known reorgs)
- Verify data consistency after reorg
- Test concurrent reorg handling
- Test deep reorgs
### Load Tests
- Simulate frequent reorgs
- Measure performance impact
- Test reorg handling during high load
## Configuration
### Reorg Handling Configuration
```yaml
reorg:
detection:
check_interval_seconds: 10
validation_depth: 100
checkpoint_interval: 1000
handling:
max_depth: 100
confirmation_period_seconds: 30
batch_size: 1000
parallel_processing: true
finality:
chain_138:
type: "poa" # or "pos", "pow", "bft"
depth_limit: 50
finality_blocks: 12
```
## References
- Indexer Architecture: See `indexer-architecture.md`
- Data Models: See `data-models.md`
- Database Schema: See `../database/postgres-schema.md`

523
docs/specs/indexing/verification-pipeline.md Normal file
View File

@@ -0,0 +1,523 @@
# Contract Verification Pipeline Specification
## Overview
This document specifies the pipeline for verifying smart contracts on the explorer platform. Contract verification allows users to submit source code, which is compiled and compared against deployed bytecode to enable source code viewing, debugging, and ABI extraction.
## Architecture
```mermaid
flowchart TB
subgraph Submit[Submission]
User[User Submits<br/>Source Code]
UI[Verification UI]
API[Verification API]
end
subgraph Validate[Validation]
Val[Validate Input]
Check[Check Contract Exists]
Dup[Check Duplicate]
end
subgraph Compile[Compilation]
Comp[Compiler Service]
Versions[Compiler Version<br/>Registry]
Build[Build Artifacts]
end
subgraph Verify[Verification]
Match[Bytecode Matching]
Construct[Constructor Args<br/>Extraction]
MatchResult[Match Result]
end
subgraph Store[Storage]
DB[(Database)]
Artifacts[Artifact Storage<br/>S3/Immutable]
ABI[ABI Registry]
end
User --> UI
UI --> API
API --> Val
Val --> Check
Check --> Dup
Dup --> Comp
Comp --> Versions
Comp --> Build
Build --> Match
Match --> Construct
Construct --> MatchResult
MatchResult --> DB
MatchResult --> Artifacts
MatchResult --> ABI
```
## Source Code Submission Workflow
### Submission Methods
**1. Standard JSON Input** (Recommended)
- Submit Solidity compiler's standard JSON input format
- Includes source files, compiler settings, optimization
- Most reliable for complex contracts
**2. Multi-file Upload**
- Upload individual source files
- Specify compiler version and settings
- Compiler constructs standard JSON input
**3. Sourcify Integration**
- Verify via Sourcify API
- Automatic source code and metadata retrieval
- Supports verified contracts from Sourcify registry
**4. Flattened Source**
- Single flattened source file
- All imports inlined
- Simpler but less flexible
### Submission API
**Endpoint**: `POST /api/v1/contracts/{address}/verify`
**Request Body**:
```json
{
"chain_id": 138,
"address": "0x...",
"compiler_version": "v0.8.19+commit.7dd6d404",
"optimization_enabled": true,
"optimization_runs": 200,
"evm_version": "london",
"source_code": "...", // or standard_json_input
"constructor_arguments": "0x...",
"library_addresses": {
"Lib1": "0x..."
},
"verification_method": "standard_json"
}
```
**Response**:
```json
{
"status": "pending",
"verification_id": "uuid",
"message": "Verification submitted"
}
```
### Input Validation
**Validation Rules**:
1. **Contract Address**: Must be valid Ethereum address, must exist on chain
2. **Compiler Version**: Must be supported compiler version
3. **Source Code**: Must be valid Solidity/Vyper code
4. **Constructor Arguments**: Must match deployed contract (if provided)
5. **Library Addresses**: Must match deployed libraries (if provided)
**Error Handling**:
- Invalid address: 400 Bad Request
- Unsupported compiler: 400 Bad Request
- Invalid source code: 400 Bad Request
- Contract not found: 404 Not Found
## Compiler Version Management
### Compiler Registry
**Purpose**: Manage available compiler versions and their metadata.
**Storage**:
```sql
compiler_versions (
id SERIAL PRIMARY KEY,
version VARCHAR(50) UNIQUE NOT NULL,
compiler_type VARCHAR(20) NOT NULL, -- 'solidity', 'vyper'
evm_version VARCHAR(20),
optimizer_available BOOLEAN DEFAULT true,
download_url TEXT,
checksum VARCHAR(64),
installed BOOLEAN DEFAULT false,
installed_path TEXT,
created_at TIMESTAMP DEFAULT NOW()
)
```
### Compiler Installation
**Methods**:
1. **Pre-installed**: Common versions pre-installed on compilation servers
2. **On-demand**: Download and install when needed
3. **Docker**: Use compiler Docker images (isolated, reproducible)
**Recommended**: Docker-based compilation for isolation and reproducibility.
**Docker Setup**:
```dockerfile
FROM ethereum/solc:0.8.19
# Or use solc-select for version management
```
### Version Selection
**Strategy**:
- Exact match: User specifies exact version
- Pragma matching: Extract version from source code pragma
- Latest compatible: Use latest compatible version if exact not available
**Pragma Parsing**:
- Extract `pragma solidity ^0.8.0;` or `>=0.8.0 <0.9.0`
- Resolve to specific compiler version
- Handle caret (^), tilde (~), and range operators
## Compilation Process
### Standard JSON Input Format
**Structure**:
```json
{
"language": "Solidity",
"sources": {
"Contract.sol": {
"content": "pragma solidity ^0.8.0; ..."
}
},
"settings": {
"optimizer": {
"enabled": true,
"runs": 200
},
"evmVersion": "london",
"outputSelection": {
"*": {
"*": ["abi", "evm.bytecode", "evm.deployedBytecode"]
}
}
}
}
```
### Compilation Steps
1. **Prepare Input**: Construct standard JSON input from user submission
2. **Select Compiler**: Choose appropriate compiler version
3. **Resolve Imports**: Handle import statements (local files, external URLs)
4. **Compile**: Execute compiler with standard JSON input
5. **Extract Artifacts**: Extract ABI, bytecode, deployed bytecode
6. **Handle Errors**: Parse compilation errors and return to user
### Import Resolution
**Import Types**:
- **Local Files**: Included in submission
- **External URLs**: Fetch from URL (GitHub, IPFS, etc.)
- **Standard Libraries**: Known library addresses (OpenZeppelin, etc.)
**Resolution Strategy**:
1. Check local files first
2. Try external URL fetching
3. Check standard library registry
4. Fail if cannot resolve
### Optimization Settings
**Optimizer Configuration**:
- **Enabled**: Boolean flag
- **Runs**: Optimization runs (affects bytecode size vs gas cost)
- **EVN Version**: Target EVM version (affects bytecode generation)
**Matching Strategy**:
- Must match deployed contract's optimization settings exactly
- Try multiple optimization combinations if initial match fails
## Bytecode Matching
### Matching Process
**Goal**: Compare compiled bytecode with deployed bytecode.
**Steps**:
1. Fetch deployed bytecode from chain via `eth_getCode(address)`
2. Extract deployed bytecode from compilation artifacts
3. Compare bytecodes (exact match required)
4. Handle constructor arguments (trimmed from deployed bytecode)
### Bytecode Normalization
**Normalization Steps**:
1. Remove metadata hash (last 53 bytes)
2. Remove constructor arguments (if contract creation)
3. Compare remaining bytecode
**Metadata Hash**:
- Solidity appends metadata hash to bytecode
- Format: `0xa2646970667358221220...` + 43 bytes
- Should be excluded from comparison
### Constructor Arguments Extraction
**Purpose**: Extract constructor arguments from deployed bytecode.
**Process**:
1. Compiled bytecode: `creation_code + constructor_args`
2. Deployed bytecode: `runtime_code` (constructor args removed)
3. Extract constructor args: `deployed_bytecode.length - runtime_code.length`
**Validation**:
- Verify extracted constructor args match user-provided args (if provided)
- Decode constructor args if ABI available
### Library Linking
**Problem**: Contracts using libraries have placeholders in bytecode.
**Solution**:
1. Identify library placeholders in compiled bytecode
2. Replace placeholders with actual library addresses
3. Compare linked bytecode with deployed bytecode
**Library Placeholder Format**:
- `__$...$__` (Solidity)
- Must match user-provided library addresses
## Verification Status Tracking
### Status States
**States**:
1. **pending**: Verification submitted, queued for processing
2. **processing**: Compilation/verification in progress
3. **verified**: Bytecode matches, contract verified
4. **failed**: Verification failed (mismatch, compilation error, etc.)
5. **partially_verified**: Some source files verified (multi-file contracts)
### Status Updates
**Database Schema**:
```sql
contract_verifications (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
chain_id INTEGER NOT NULL,
address VARCHAR(42) NOT NULL,
status VARCHAR(20) NOT NULL,
compiler_version VARCHAR(50),
optimization_enabled BOOLEAN,
optimization_runs INTEGER,
evm_version VARCHAR(20),
source_code TEXT,
abi JSONB,
constructor_arguments TEXT,
verification_method VARCHAR(50),
error_message TEXT,
verified_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY (chain_id, address) REFERENCES contracts(chain_id, address)
)
```
**Status Transitions**:
- `pending` → `processing` → `verified` or `failed`
- Webhook/notification on status change (optional)
## Build Artifact Storage
### Artifact Types
**Artifacts to Store**:
1. **Source Code**: Original submitted source files
2. **Standard JSON Input**: Compiler input
3. **Compiler Output**: Full compiler JSON output
4. **ABI**: Extracted ABI
5. **Bytecode**: Creation and runtime bytecode
6. **Metadata**: Compiler metadata
### Storage Strategy
**Immutable Storage**:
- Use S3-compatible storage (AWS S3, MinIO, etc.)
- Immutable after verification (no updates)
- Versioned storage if updates needed
**Storage Path Structure**:
```
contracts/{chain_id}/{address}/verification_{id}/
- source_code.sol
- standard_json_input.json
- compiler_output.json
- abi.json
- bytecode.txt
- metadata.json
```
**Database Reference**:
- Store artifact storage path in database
- Link to contract record
### Access Control
**Public Access**:
- Verified contracts: Public read access
- Source code: Public read access
- Artifacts: Public read access
**Private Access**:
- Pending verifications: Owner only
- Failed verifications: Owner only (optional public)
## Sourcify Integration
### Sourcify API
**Endpoint**: `GET /api/v1/verify/{chain_id}/{address}`
**Process**:
1. Query Sourcify API for contract verification
2. Retrieve source files and metadata
3. Verify match with deployed bytecode
4. Store in our database if match
**Benefits**:
- Leverage existing verified contracts
- Automatic verification for popular contracts
- Reduces manual verification workload
### Sourcify Format
**Structure**:
```
contracts/
- {chain_id}/
- {address}/
- metadata.json
- sources/
- Contract.sol
```
**Metadata Format**:
- Compiler version
- Settings
- Source file mapping
## Multi-Compiler Version Support
### Supported Compilers
**Solidity**:
- Versions: 0.4.x through latest
- Multiple versions per contract (updates)
**Vyper**:
- Versions: 0.1.x through latest
- Similar workflow to Solidity
### Version Compatibility
**Handling**:
- Support multiple verification attempts with different versions
- Store all verification attempts (history)
- Mark latest successful verification as active
**Database Schema**:
```sql
contract_verifications (
-- ... fields ...
version INTEGER DEFAULT 1, -- Increment for each new verification
is_active BOOLEAN DEFAULT true -- Latest successful verification
)
```
## Error Handling
### Compilation Errors
**Error Types**:
- Syntax errors
- Type errors
- Import resolution errors
- Optimization errors
**Response**:
- Return detailed error messages to user
- Include file and line number
- Suggest fixes when possible
### Verification Failures
**Failure Reasons**:
- Bytecode mismatch
- Constructor arguments mismatch
- Library address mismatch
- Optimization settings mismatch
**Response**:
- Return specific mismatch reason
- Suggest correct settings if possible
- Allow retry with corrected input
## Performance Considerations
### Compilation Performance
**Optimization**:
- Cache compilation results (same source + settings)
- Parallel compilation for multiple contracts
- Compiler server pool for load distribution
### Queue Management
**Queue System**:
- Use message queue (RabbitMQ, Kafka) for verification jobs
- Priority queue: User submissions before automated checks
- Rate limiting per user/IP
**Processing Time**:
- Target: < 30 seconds for simple contracts
- Target: < 5 minutes for complex contracts
- Timeout: 10 minutes maximum
## Security Considerations
### Source Code Validation
**Validation**:
- Validate source code size (max 10MB)
- Sanitize input to prevent injection attacks
- Validate compiler version (whitelist known versions)
### Artifact Storage Security
**Access Control**:
- Verify ownership before allowing updates
- Audit log all verification submissions
- Rate limit submissions per user/IP
## API Endpoints
### Submit Verification
`POST /api/v1/contracts/{address}/verify`
### Check Status
`GET /api/v1/contracts/{address}/verification/{verification_id}`
### Get Verified Contract
`GET /api/v1/contracts/{address}`
### List Verification History
`GET /api/v1/contracts/{address}/verifications`
## References
- Indexer Architecture: See `indexer-architecture.md`
- Data Models: See `data-models.md`
- Database Schema: See `../database/postgres-schema.md`
- API Specification: See `../api/rest-api.md`

402
docs/specs/infrastructure/network-topology.md Normal file
View File

@@ -0,0 +1,402 @@
# Network Topology Specification
## Overview
This document specifies the network architecture for the ChainID 138 Explorer+ platform, including internal network structure, external access patterns, security boundaries, and CDN/WAF integration.
## Network Architecture
```mermaid
flowchart TB
subgraph Internet[Internet]
Users[End Users]
Bots[Bots/Crawlers]
end
subgraph CDN[CDN Layer]
Cloudflare[Cloudflare CDN/WAF]
end
subgraph Edge[Edge Layer]
WAF[Web Application Firewall]
LB[Load Balancer]
end
subgraph DMZ[DMZ Network<br/>192.168.11.0/24]
API_GW[API Gateway]
Web_Server[Web Server]
end
subgraph Internal[Internal Network<br/>10.0.0.0/16]
Indexer[Indexer Service]
Explorer_API[Explorer API]
Mempool[Mempool Service]
Search[Search Service]
end
subgraph Blockchain[Blockchain Network<br/>192.168.11.250:8545]
Node1[ChainID 138 Node 1]
Node2[ChainID 138 Node 2]
end
subgraph Data[Data Layer<br/>Private Network]
DB[(PostgreSQL)]
ES[(Elasticsearch)]
Redis[(Redis Cache)]
end
Users --> Cloudflare
Bots --> Cloudflare
Cloudflare --> WAF
WAF --> LB
LB --> API_GW
LB --> Web_Server
API_GW --> Explorer_API
Web_Server --> Explorer_API
Explorer_API --> Indexer
Explorer_API --> Mempool
Explorer_API --> Search
Explorer_API --> Redis
Indexer --> Node1
Indexer --> Node2
Mempool --> Node1
Explorer_API --> DB
Indexer --> DB
Search --> ES
```
## Network Segments
### External Network (Internet)
**Description**: Public internet access points for end users and bots.
**Access Patterns**:
- HTTPS requests to `https://explorer.d-bis.org`
- API requests to `https://api.explorer.d-bis.org`
- WebSocket connections for real-time updates
**Security Requirements**:
- All traffic must pass through CDN/WAF
- DDoS protection enabled
- Rate limiting at edge
- SSL/TLS termination
### CDN Layer (Cloudflare)
**Description**: Content delivery network and web application firewall.
**Configuration**:
- **CDN**: Static asset caching, global distribution
- **WAF**: DDoS protection, bot mitigation, rate limiting
- **SSL/TLS**: End-to-end encryption, certificate management
- **DNS**: DNS resolution and routing
**Rules**:
- Cache static assets (HTML, CSS, JS, images) for 1 hour
- Cache API responses for 1 minute (configurable)
- Block known malicious IPs
- Rate limit: 100 requests/minute per IP
- Challenge suspicious traffic (CAPTCHA, JavaScript challenge)
**Integration Points**:
- Origin: `192.168.11.140:443` (Blockscout/Web server)
- Tunnel: Cloudflare Zero Trust tunnel for internal services
### Edge Layer (DMZ - 192.168.11.0/24)
**Description**: Demilitarized zone containing publicly accessible services.
**Subnet**: `192.168.11.0/24`
**Components**:
- **API Gateway** (192.168.11.141)
- Reverse proxy for API requests
- Authentication/authorization
- Rate limiting per API key
- Request routing
- **Web Server** (192.168.11.140)
- Serves frontend application
- Static file hosting
- Server-side rendering (if applicable)
**Security Rules**:
- No direct database access
- Communication with internal network via firewall rules only
- All outbound connections logged
- Intrusion detection system (IDS) monitoring
### Internal Network (10.0.0.0/16)
**Description**: Private network for backend services.
**Subnet**: `10.0.0.0/16`
**Service Allocation**:
- **10.0.1.0/24**: Explorer services (API, indexer, mempool)
- **10.0.2.0/24**: Search and analytics services
- **10.0.3.0/24**: Banking and compliance services
- **10.0.4.0/24**: VTM and conversational services
**Components**:
- Indexer Service (10.0.1.10)
- Explorer API (10.0.1.20)
- Mempool Service (10.0.1.30)
- Search Service (10.0.2.10)
- Analytics Service (10.0.2.20)
**Security Rules**:
- No internet access (outbound firewall)
- Communication with DMZ only via specific ports
- Internal service-to-service communication allowed
- All traffic encrypted (mTLS recommended)
### Blockchain Network (192.168.11.250:8545)
**Description**: Network segment containing blockchain nodes.
**Nodes**:
- **Primary Node**: `192.168.11.250:8545` (RPC)
- **Primary Node WS**: `192.168.11.250:8546` (WebSocket)
- **Backup Nodes**: Additional nodes for redundancy
**Access Rules**:
- Indexer service: Full access (archive, trace)
- Mempool service: Read access (pending tx), write access (submit tx)
- External services: Rate-limited public access via RPC endpoint
- Internal services: Direct access without rate limits
### Data Layer (Private Network)
**Description**: Network segment for data storage systems.
**Subnet**: `10.0.10.0/24` (isolated)
**Components**:
- **PostgreSQL** (10.0.10.10:5432)
- Primary database
- Read replicas for scaling
- **Elasticsearch** (10.0.10.20:9200)
- Search index
- Log aggregation
- **Redis** (10.0.10.30:6379)
- Cache layer
- Session storage
- Rate limiting counters
- **TimescaleDB** (10.0.10.40:5432)
- Time-series data
- Mempool metrics
- **Neo4j** (10.0.10.50:7687)
- Graph database
- Cross-chain entity graph
**Security Rules**:
- No internet access
- Access only from internal services
- Network encryption (TLS)
- Database authentication required
- Backup network isolation
## External Access Patterns
### Public Web Access
**Flow**:
1. User → Cloudflare CDN (DNS resolution)
2. Cloudflare → WAF (security checks)
3. WAF → Load Balancer (traffic distribution)
4. Load Balancer → Web Server (content delivery)
**Ports**:
- HTTP: 80 (redirects to HTTPS)
- HTTPS: 443
### Public API Access
**Flow**:
1. Client → Cloudflare CDN
2. Cloudflare → WAF (rate limiting, bot detection)
3. WAF → API Gateway (authentication, routing)
4. API Gateway → Explorer API (request processing)
**Endpoints**:
- `https://api.explorer.d-bis.org/v1/*` (REST API)
- `https://api.explorer.d-bis.org/graphql` (GraphQL API)
- `wss://api.explorer.d-bis.org/ws` (WebSocket API)
### Internal Service Access
**Flow**:
- Direct communication within internal network
- Service discovery via DNS or service mesh
- Load balancing via service mesh or internal LB
## Security Boundaries
### Firewall Rules
**DMZ → Internal**:
- Allow: HTTP/HTTPS to API services (80, 443)
- Allow: Specific service ports (indexer, mempool)
- Deny: Database ports (5432, 9200, 6379)
- Deny: All other ports
**Internal → Data Layer**:
- Allow: Database connections from specific services
- Allow: Specific ports only (5432, 9200, 6379, 7687)
- Deny: All other access
**Internal → Blockchain Network**:
- Allow: RPC connections (8545, 8546)
- Deny: All other ports
**Internet → DMZ**:
- Allow: HTTP (80) → HTTPS redirect
- Allow: HTTPS (443) → Web/API servers
- Deny: All other ports
**Internal → Internet**:
- Deny: All outbound (except specific whitelisted services)
- Allow: Package updates (via proxy)
- Allow: External API calls (via gateway)
### Network Segmentation Benefits
1. **Isolation**: Compromise in one segment doesn't affect others
2. **Access Control**: Granular firewall rules
3. **Monitoring**: Network traffic analysis per segment
4. **Compliance**: Easier to demonstrate security controls
## CDN/WAF Integration
### Cloudflare Configuration
**DNS Records**:
- `explorer.d-bis.org` → Cloudflare proxy (orange cloud)
- `api.explorer.d-bis.org` → Cloudflare proxy
- Tunnel target: `192.168.11.140:80` (Web) or `192.168.11.141:443` (API)
**WAF Rules**:
1. **DDoS Protection**: Automatic mitigation
2. **Rate Limiting**:
- 100 requests/minute per IP (general)
- 10 requests/second per IP (API)
- 1000 requests/minute per API key
3. **Bot Management**: Challenge suspicious bots
4. **Geographic Blocking**: Optional country restrictions
5. **IP Access Rules**: Whitelist/blacklist specific IPs
**Cache Rules**:
- Static assets: Cache for 1 hour
- HTML pages: Cache for 5 minutes
- API responses: No cache (dynamic data)
- GraphQL: No cache
### Cloudflare Tunnel (Zero Trust)
**Purpose**: Secure access to internal services without exposing ports.
**Configuration**:
- Tunnel endpoint: Internal load balancer
- Routes:
- `explorer.d-bis.org/*` → `http://192.168.11.140:80`
- `api.explorer.d-bis.org/*` → `http://192.168.11.141:443`
**Benefits**:
- No open ports to internet
- Encrypted connections
- Access control via Zero Trust policies
- Automatic failover
## Monitoring and Observability
### Network Monitoring
**Metrics to Track**:
- Bandwidth utilization per segment
- Connection counts
- Latency between segments
- Packet loss
- Firewall rule hit counts
**Tools**:
- Netflow/sFlow analysis
- Network packet capture (tcpdump, Wireshark)
- Network performance monitoring (PRTG, Zabbix)
### Security Monitoring
**Events to Monitor**:
- Firewall rule violations
- Unusual traffic patterns
- DDoS attack attempts
- Port scan attempts
- Failed authentication attempts
**Tools**:
- Intrusion Detection System (IDS)
- Security Information and Event Management (SIEM)
- Cloudflare security logs
- Firewall logs aggregation
## Implementation Guidelines
### Network Configuration
**IP Address Allocation**:
- Use CIDR notation for subnets
- Reserve IP ranges for future expansion
- Document all IP assignments
**DNS Configuration**:
- Internal DNS for service discovery
- External DNS via Cloudflare
- Service names: `service-name.internal.explorer`
**Load Balancing**:
- Use HAProxy or NGINX for internal LB
- Health checks for backend services
- Session affinity where needed
### Security Hardening
1. **Disable unused services**: Close unnecessary ports
2. **Use strong encryption**: TLS 1.2+ for all connections
3. **Implement network segmentation**: Isolate sensitive systems
4. **Regular security audits**: Review firewall rules
5. **Monitor network traffic**: Detect anomalies
## Migration from Current Setup
### Current Network Configuration
- **Blockscout Container**: `192.168.11.140:4000`
- **RPC Endpoint**: `192.168.11.250:8545`
- **Cloudflare Tunnel**: Configured for `explorer.d-bis.org`
### Migration Steps
1. Deploy new network segments
2. Configure firewall rules
3. Deploy services to appropriate segments
4. Update DNS and routing
5. Test connectivity
6. Monitor for issues
7. Gradually migrate traffic
## References
- Node & RPC Architecture: See `node-rpc-architecture.md`
- API Gateway: See `../api/api-gateway.md`
- Security Architecture: See `../security/security-architecture.md`
- Existing Blockscout setup: `../../docs/BLOCKSCOUT_COMPLETE_SUMMARY.md`

387
docs/specs/infrastructure/node-rpc-architecture.md Normal file
View File

@@ -0,0 +1,387 @@
# Node & RPC Architecture Specification
## Overview
This document specifies the architecture for ChainID 138 blockchain nodes and RPC endpoints that power the explorer platform. The architecture ensures high availability, performance, and reliability for block data retrieval and transaction submission.
## Architecture
```mermaid
flowchart TB
subgraph Nodes[ChainID 138 Nodes]
AN[Archive Node<br/>Full History]
TN[Tracing Node<br/>Call Traces]
VN[Validator Node<br/>Block Production]
end
subgraph LoadBalancer[Load Balancer Layer]
LB[NGINX/HAProxy<br/>SSL Termination]
end
subgraph RPCPool[RPC Endpoint Pool]
RPC1[RPC Endpoint 1<br/>Archive + Trace]
RPC2[RPC Endpoint 2<br/>Archive + Trace]
RPC3[RPC Endpoint 3<br/>Archive Only]
end
subgraph Services[Explorer Services]
Indexer[Indexer Service]
API[API Gateway]
Mempool[Mempool Service]
end
AN --> RPC1
TN --> RPC1
AN --> RPC2
TN --> RPC2
AN --> RPC3
LB --> RPC1
LB --> RPC2
LB --> RPC3
Indexer --> LB
API --> LB
Mempool --> LB
```
## Node Setup Requirements
### Archive Node Configuration
**Purpose**: Store complete blockchain history for historical queries and data indexing.
**Requirements**:
- **Archive mode**: Full historical state access (`--sync-mode=full --gcmode=archive`)
- **Storage**: High-capacity storage with SSD for recent blocks, HDD for cold storage
- **Memory**: Minimum 32GB RAM for efficient state access
- **Database**: Fast state database (LevelDB, RocksDB)
- **Network**: High-bandwidth connection for syncing
**Configuration Example** (Hyperledger Besu):
```toml
data-path="./data"
genesis-file="./genesis.json"
network-id=138
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","WEB3","ADMIN","DEBUG","TRACE","EEA","PRIV"]
rpc-http-cors-origins=["*"]
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
rpc-ws-api=["ETH","NET","WEB3"]
sync-mode="FAST"
data-storage-format="BONSAI"
```
### Tracing Node Configuration
**Purpose**: Provide call trace capabilities for transaction debugging and internal transaction tracking.
**Requirements**:
- Archive mode enabled for historical trace access
- Debug and trace API endpoints enabled
- Additional CPU for trace computation
- Trace storage strategy (on-demand vs cached)
**Trace Types Supported**:
- `trace_call`: Single call trace
- `trace_block`: All traces in a block
- `trace_transaction`: All traces in a transaction
- `trace_replayBlockTransactions`: Replay with state
- `trace_replayTransaction`: Replay single transaction
### Node Health Monitoring
**Health Check Endpoints**:
- `eth_blockNumber`: Returns latest block number
- `eth_syncing`: Returns sync status
- `net_peerCount`: Returns connected peer count
**Health Check Criteria**:
- Latest block is within 5 blocks of network head
- Peer count > 5 (configurable)
- Response time < 500ms for simple queries
- No critical errors in logs
**Monitoring Metrics**:
- Block height lag vs network
- Peer connection count
- RPC request latency (p50, p95, p99)
- Error rate by endpoint
- CPU and memory usage
- Disk I/O and storage usage
## RPC Endpoint Architecture
### Load Balancing Strategy
**Distribution Method**: Round-robin with health-aware routing
**Failover Strategy**:
- Primary: Archive + Trace capable nodes
- Secondary: Archive-only nodes (fallback for non-trace requests)
- Automatic failover on health check failure
- Circuit breaker pattern to prevent cascading failures
**Session Affinity**: Not required for stateless RPC requests
### RPC Capabilities Matrix
| Capability | Archive Node | Tracing Node | Light Node |
|------------|--------------|--------------|------------|
| Historical blocks | ✅ Full | ✅ Full | ❌ Limited |
| Current state | ✅ Full | ✅ Full | ✅ Full |
| Call traces | ❌ No | ✅ Yes | ❌ No |
| Archive state queries | ✅ Yes | ✅ Yes | ❌ No |
| Transaction submission | ✅ Yes | ✅ Yes | ✅ Yes |
### Endpoint Types
#### Public RPC Endpoint
- **Internal URL**: `http://192.168.11.221:8545` (VMID 2201 - direct connection, recommended for internal services)
- **Public URL**: `https://rpc-http-pub.d-bis.org` (via proxy - for external access)
- **Authentication**: Optional API keys for rate limiting
- **Rate Limits**:
- Unauthenticated: 10 req/s
- Authenticated: 100 req/s (tiered)
- **CORS**: Enabled for web applications
#### Private RPC Endpoint
- **URL**: `https://rpc-http-prv.d-bis.org`
- **Authentication**: Required (authorized access)
- **Rate Limits**: Higher limits for authenticated users
- **CORS**: Enabled for authorized domains
#### Internal RPC Endpoint
- **URL**: `http://192.168.11.211:8545` (internal network - VMID 2101)
- **Authentication**: Network-based (firewall rules)
- **Rate Limits**: Higher limits for internal services
- **CORS**: Disabled
### WebSocket Endpoint
**Configuration**:
- **Public WS (Internal)**: `ws://192.168.11.221:8546` (VMID 2201 - direct connection, recommended for internal services)
- **Public WS (Public)**: `wss://rpc-ws-pub.d-bis.org` (via proxy - for external access)
- **Private WS (Internal)**: `ws://192.168.11.211:8546` (VMID 2101 - direct connection)
- **Private WS (Public)**: `wss://rpc-ws-prv.d-bis.org` (via proxy - for external access)
- **Subscriptions**: New blocks, pending transactions, logs
- **Connection Limits**: 100 concurrent connections per IP
- **Heartbeat**: 30-second ping/pong
**Subscription Types**:
- `newHeads`: New block headers
- `logs`: Event logs matching filter
- `pendingTransactions`: Pending transactions
- `syncing`: Sync status updates
## Rate Limiting and Access Control
### Rate Limiting Strategy
**Tiers**:
1. **Anonymous**: 10 requests/second, 100 requests/minute
2. **API Key (Free)**: 100 requests/second, 1000 requests/minute
3. **API Key (Pro)**: 500 requests/second, 5000 requests/minute
4. **Internal Services**: 1000 requests/second, unlimited
**Implementation**:
- Token bucket algorithm
- Per-IP and per-API-key limits
- Separate limits for different endpoint categories:
- Simple queries (eth_blockNumber, eth_getBalance): Higher limits
- Complex queries (trace_block, eth_call with large state): Lower limits
- Write operations (eth_sendTransaction): Strict limits
### Access Control
**API Key Management**:
- Key generation with secure random tokens
- Key metadata: name, tier, creation date, last used
- Key rotation support
- Revocation capability
**IP Whitelisting**:
- Support for IP whitelisting per API key
- CIDR notation support
- Geographic restrictions (optional)
**Rate Limit Headers**:
```
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1640995200
```
## Failover Strategies
### Automatic Failover
**Detection**:
- Health checks every 10 seconds
- Consecutive failures threshold: 3
- Recovery threshold: 2 successful checks
**Failover Actions**:
1. Mark node as unhealthy
2. Route traffic to healthy nodes
3. Continue health checking for recovery
4. Log failover events
### Manual Failover
**Scenarios**:
- Planned maintenance
- Node performance degradation
- Security incidents
**Procedure**:
1. Drain connections (wait for active requests)
2. Remove from load balancer pool
3. Perform maintenance
4. Verify health
5. Re-add to pool
## Performance Requirements
### Latency Targets
| Operation Type | p50 Target | p95 Target | p99 Target |
|----------------|------------|------------|------------|
| Simple queries (blockNumber, balance) | < 50ms | < 100ms | < 200ms |
| Block queries | < 100ms | < 200ms | < 500ms |
| Transaction queries | < 150ms | < 300ms | < 1000ms |
| Trace queries | < 500ms | < 2000ms | < 5000ms |
| Historical state queries | < 1000ms | < 5000ms | < 10000ms |
### Throughput Targets
- **Read operations**: 1000 req/s per node
- **Write operations**: 100 req/s per node
- **WebSocket subscriptions**: 1000 concurrent per node
### Availability Targets
- **Uptime**: 99.9% (8.76 hours downtime/year)
- **Failover time**: < 30 seconds
- **Data consistency**: Zero data loss
## Security Considerations
### Network Security
- **TLS/SSL**: Required for public endpoints (TLS 1.2+)
- **Firewall Rules**: Restrict access to internal endpoints
- **DDoS Protection**: WAF/CDN integration for public endpoints
- **IP Filtering**: Support for IP allowlists/blocklists
### Authentication
- **API Keys**: HMAC-based authentication
- **Key Storage**: Encrypted storage with KMS
- **Key Rotation**: Support for periodic rotation
### Audit Logging
- All RPC requests logged (with PII sanitization)
- Log retention: 90 days
- Log fields: timestamp, IP, API key (hash), endpoint, response code, latency
## Integration Points
### Indexer Service Integration
- Primary connection to archive nodes
- Batch requests for efficiency
- Connection pooling
- Retry logic with exponential backoff
### API Gateway Integration
- Proxy requests to RPC pool
- Add authentication/authorization
- Apply rate limiting
- Cache common queries
### Mempool Service Integration
- WebSocket subscription to pending transactions
- Direct connection for transaction submission
- Priority queuing for indexer submissions
## Implementation Guidelines
### Node Deployment
**Infrastructure**:
- Use container orchestration (Kubernetes) for scalability
- Separate deployments for archive and tracing nodes
- Horizontal scaling based on load
**Configuration Management**:
- Use configuration files (TOML/YAML)
- Environment-specific overrides
- Secrets management via KMS/Vault
### Monitoring Integration
**Metrics to Export**:
- Prometheus metrics for node health
- Custom metrics for RPC performance
- Integration with Grafana dashboards
**Alerting Rules**:
- Node down alert
- High latency alert
- High error rate alert
- Low peer count alert
## Testing Strategy
### Unit Tests
- RPC endpoint implementations
- Rate limiting logic
- Health check logic
### Integration Tests
- Load balancer failover
- Rate limiting enforcement
- Authentication/authorization
### Load Tests
- Simulate production load
- Test failover scenarios
- Validate performance targets
## Migration from Existing Setup
### Current Setup (Blockscout Integration)
- **Public RPC (Internal)**: `http://192.168.11.221:8545` (VMID 2201 - recommended for internal services)
- **Public RPC (Public)**: `https://rpc-http-pub.d-bis.org` (via proxy - for external access)
- **Private RPC (Internal)**: `http://192.168.11.211:8545` (VMID 2101)
- **Private RPC (Public)**: `https://rpc-http-prv.d-bis.org` (via proxy)
- **Deprecated**: `https://rpc-core.d-bis.org` (no longer public, internal only)
- **Internal RPC**: `http://192.168.11.250:8545`
- **WebSocket**: `ws://192.168.11.250:8546`
### Migration Steps
1. Deploy new load balancer layer
2. Configure health checks
3. Gradually migrate traffic
4. Monitor performance
5. Decommission old direct connections
## References
- Existing Blockscout deployment: `docs/BLOCKSCOUT_COMPLETE_SUMMARY.md`
- Network architecture: See `network-topology.md`
- API Gateway: See `../api/api-gateway.md`

275
docs/specs/mempool/fee-oracle.md Normal file
View File

@@ -0,0 +1,275 @@
# Fee Oracle Specification
## Overview
This document specifies the fee estimation service that calculates and predicts transaction fees based on historical data and current mempool conditions.
## Architecture
```mermaid
flowchart TB
subgraph Data[Data Sources]
Blocks[Recent Blocks]
Mempool[Current Mempool]
History[Historical Data]
end
subgraph Analysis[Analysis]
Percentile[Percentile Calculator]
Prediction[Prediction Model]
Adjust[Market Adjuster]
end
subgraph Output[Output]
Estimates[Fee Estimates]
API[API Endpoint]
Cache[Cache]
end
Blocks --> Percentile
Mempool --> Percentile
History --> Prediction
Percentile --> Adjust
Prediction --> Adjust
Adjust --> Estimates
Estimates --> API
Estimates --> Cache
```
## Fee Calculation Methods
### Method 1: Percentile-Based (Primary)
**Algorithm**:
1. Collect gas prices from recent N blocks (e.g., 100 blocks)
2. Calculate percentiles: 25th, 50th, 75th, 95th
3. Use percentiles as fee estimates
**Percentile Mapping**:
- Slow: 25th percentile
- Standard: 50th percentile (median)
- Fast: 75th percentile
- Urgent: 95th percentile
**Update Frequency**: Every new block
### Method 2: Mempool-Based Adjustment
**Algorithm**:
1. Start with percentile-based estimates
2. Analyze current mempool:
- Average gas price of pending transactions
- Pending transaction count
- Mempool pressure (pending / recent block capacity)
3. Adjust estimates based on mempool conditions
**Adjustment Factors**:
- High mempool pressure → Increase estimates
- Low mempool pressure → Decrease estimates
- Recent confirmed gas prices → Weight recent data more
### Method 3: Time-Based Prediction
**Algorithm**:
- Predict fees for different confirmation times
- Example: "To be confirmed in 1 block, use X gas price"
- Use historical data to train model
**Prediction Targets**:
- 1 block (next block)
- 3 blocks (within 3 blocks)
- 10 blocks (within 10 blocks)
## Fee Estimate Structure
### Standard Estimates
```json
{
"chain_id": 138,
"block_number": 12345,
"estimates": {
"slow": {
"gas_price": "20000000000",
"max_fee_per_gas": "20000000000",
"max_priority_fee_per_gas": "1000000000",
"confidence": 0.95,
"estimated_confirmation_time": "5 minutes"
},
"standard": {
"gas_price": "30000000000",
"max_fee_per_gas": "30000000000",
"max_priority_fee_per_gas": "2000000000",
"confidence": 0.90,
"estimated_confirmation_time": "2 minutes"
},
"fast": {
"gas_price": "50000000000",
"max_fee_per_gas": "50000000000",
"max_priority_fee_per_gas": "3000000000",
"confidence": 0.85,
"estimated_confirmation_time": "30 seconds"
},
"urgent": {
"gas_price": "100000000000",
"max_fee_per_gas": "100000000000",
"max_priority_fee_per_gas": "5000000000",
"confidence": 0.80,
"estimated_confirmation_time": "next block"
}
},
"timestamp": "2024-01-01T00:00:00Z"
}
```
### EIP-1559 Support
**For EIP-1559 Chains**:
- Provide `max_fee_per_gas` and `max_priority_fee_per_gas`
- Calculate base fee prediction
- Estimate priority fee needed for desired speed
## Historical Fee Analysis
### Data Collection
**Sources**:
- Recent blocks (last 100-1000 blocks)
- Historical fee data (time-series database)
**Metrics Tracked**:
- Min, max, average gas price per block
- Percentiles (25th, 50th, 75th, 95th, 99th)
- Gas price distribution
- Confirmation time by gas price
### Analysis Windows
**Short-term** (Last 10 blocks):
- Most recent trends
- Quick response to market changes
**Medium-term** (Last 100 blocks):
- Stable baseline
- Primary estimate source
**Long-term** (Last 1000 blocks):
- Historical patterns
- Anomaly detection
## Prediction Models
### Simple Moving Average
**Algorithm**: Average of recent N blocks
**Pros**: Simple, fast
**Cons**: Slow to adapt to changes
### Weighted Moving Average
**Algorithm**: Weight recent blocks more heavily
**Pros**: Faster adaptation
**Cons**: More volatile
### Machine Learning Models (Future)
**Potential Models**:
- Linear regression
- Time series forecasting (ARIMA, LSTM)
- Classification (predict confirmation time)
**Features**:
- Historical gas prices
- Mempool metrics
- Block time
- Network activity
## API Endpoint
### Get Fee Estimates
`GET /api/v1/mempool/{chain_id}/fees`
**Query Parameters**:
- `block_target` (integer): Target confirmation block (optional)
**Response**: Fee estimates object (see structure above)
### Get Fee History
`GET /api/v1/mempool/{chain_id}/fees/history`
**Query Parameters**:
- `period` (string): Time period (1h, 24h, 7d, 30d)
- `interval` (string): Aggregation interval (1m, 5m, 1h)
**Response**: Historical fee data with timestamps
## Caching Strategy
### Cache Duration
- **Fee Estimates**: 10 seconds (update every block)
- **Fee History**: 5 minutes (less frequently updated)
### Cache Invalidation
- Invalidate on new block
- Invalidate on significant mempool changes
## Performance Considerations
### Calculation Performance
**Target**: Calculate estimates in < 100ms
**Optimization**:
- Pre-calculate percentiles
- Cache historical data
- Use efficient algorithms
### Update Frequency
**Block-based**: Update on every new block
**Time-based**: Update every 10 seconds (fallback)
## Accuracy Metrics
### Tracking Accuracy
**Metrics**:
- Predicted vs actual confirmation time
- Predicted vs actual gas price used
- Estimate accuracy by tier (slow/standard/fast/urgent)
### Confidence Scores
**Calculation**:
- Based on historical accuracy
- Variance in recent data
- Mempool stability
**Display**: Include confidence score in estimates
## Monitoring
### Metrics
- Fee estimate accuracy
- Update latency
- Calculation time
- Cache hit rate
### Alerts
- High prediction error (> 50% off)
- Calculation timeout
- Cache miss rate spike
## References
- Mempool Service: See `mempool-service.md`
- Time-Series Schema: See `../database/timeseries-schema.md`

251
docs/specs/mempool/mempool-service.md Normal file
View File

@@ -0,0 +1,251 @@
# Mempool Service Architecture Specification
## Overview
This document specifies the mempool service that tracks pending transactions, monitors transaction propagation, and provides real-time mempool insights.
## Architecture
```mermaid
flowchart TB
subgraph RPC[RPC Node]
WS[WebSocket<br/>New Pending Tx]
Poll[Poll<br/>Pending Tx]
end
subgraph Ingest[Ingestion]
Listener[Transaction Listener]
Validator[Transaction Validator]
Queue[Message Queue]
end
subgraph Process[Processing]
Tracker[Propagation Tracker]
RBF[RBF Detector]
Fee[Fee Calculator]
Storage[Storage]
end
subgraph Output[Output]
API[API Endpoints]
WS_Out[WebSocket<br/>Subscriptions]
Metrics[Metrics]
end
WS --> Listener
Poll --> Listener
Listener --> Validator
Validator --> Queue
Queue --> Tracker
Queue --> RBF
Queue --> Fee
Tracker --> Storage
RBF --> Storage
Fee --> Storage
Storage --> API
Storage --> WS_Out
Storage --> Metrics
```
## Transaction Ingestion
### Sources
**1. WebSocket Subscription**:
- Subscribe to `pendingTransactions` via `eth_subscribe`
- Real-time updates as transactions enter mempool
**2. Polling**:
- Poll `eth_pendingTransactions` every 1-2 seconds
- Fallback if WebSocket unavailable
**3. Transaction Submission**:
- Track transactions submitted via our API
- Link user submissions to mempool status
### Transaction Validation
**Validation Checks**:
- Valid transaction format
- Valid signature
- Sufficient nonce (account state check)
- Sufficient balance (for value transfers)
- Valid gas price (above minimum)
**Invalid Transactions**: Log but don't track (will be rejected by network)
## Transaction Tracking
### Transaction State
**States**:
- `pending`: In mempool
- `confirmed`: Included in block
- `dropped`: Removed from mempool (replaced or expired)
- `replaced`: Replaced by higher gas price transaction (RBF)
### Propagation Tracking
**Metrics Tracked**:
- First seen timestamp
- Last seen timestamp
- Propagation time (first seen → confirmed)
- Propagation path (which nodes saw it first)
**Method**:
- Track when transaction first appears
- Monitor mempool across multiple nodes
- Calculate propagation statistics
### RBF (Replace-by-Fee) Detection
**Detection**:
- Monitor transactions with same nonce
- Detect higher gas price replacements
- Link old transaction to new transaction
**Data Stored**:
- Original transaction hash
- Replacement transaction hash
- Gas price increase
- Replacement timestamp
### Bundle/MEV Visibility
**Detection** (where supported):
- Identify transaction bundles
- Detect MEV patterns
- Track front-running/back-running
**Limitations**: Depends on chain/node capabilities
### Private Transaction Markers
**Detection**:
- Identify private transaction services (Flashbots, etc.)
- Mark transactions as private
- Track private vs public mempool
## Storage Schema
See `../database/timeseries-schema.md` for detailed schema.
**Key Fields**:
- Transaction hash
- From/to addresses
- Value, gas price, gas limit
- Nonce
- First seen timestamp
- Status
- Confirmed block number (when confirmed)
- Confirmed timestamp
## Fee Estimation
### Fee Calculation
**Methods**:
1. **Historical Analysis**: Analyze recent block fees
2. **Percentile Method**: Calculate percentiles of recent transactions
3. **Market-Based**: Track current mempool competition
**Fee Estimates**:
- Slow: 25th percentile
- Standard: 50th percentile (median)
- Fast: 75th percentile
- Urgent: 95th percentile
**Update Frequency**: Every block (real-time)
## API Endpoints
### Get Pending Transactions
`GET /api/v1/mempool/{chain_id}/transactions`
**Query Parameters**:
- `from_address`: Filter by sender
- `to_address`: Filter by recipient
- `min_value`: Minimum value
- `min_gas_price`: Minimum gas price
- `limit`: Max results (default: 100)
**Response**: Array of pending transactions
### Get Transaction Status
`GET /api/v1/mempool/{chain_id}/transactions/{hash}`
**Response**: Transaction status and propagation info
### Get Fee Estimates
`GET /api/v1/mempool/{chain_id}/fees`
**Response**:
```json
{
"slow": "20000000000",
"standard": "30000000000",
"fast": "50000000000",
"urgent": "100000000000"
}
```
## WebSocket Subscriptions
See `../api/websocket-api.md` for WebSocket API details.
**Channels**:
- `pending_transactions`: New pending transactions
- `transaction_status`: Status updates for specific transactions
- `fee_updates`: Fee estimate updates
## Data Retention
**Raw Data**: 7 days (detailed transaction data)
**Aggregated Data**: 30 days (fee statistics, propagation metrics)
**Archived Data**: Move to data lake after retention period
## Performance Considerations
### Throughput
**Target**: Process 1000 transactions/second
**Scalability**: Horizontal scaling with message queue
### Latency
**Target**:
- Ingestion latency: < 1 second
- API response time: < 100ms (p95)
### Storage Optimization
**Strategy**:
- Time-series database for efficient queries
- Partition by time (daily partitions)
- Automatic cleanup of old data
## Monitoring
### Metrics
- Pending transaction count
- Transaction ingestion rate
- Confirmation rate
- Average propagation time
- Fee estimate accuracy
### Alerts
- High pending transaction count (> 10,000)
- Low confirmation rate (< 50% within 5 minutes)
- Fee estimate errors
## References
- Time-Series Schema: See `../database/timeseries-schema.md`
- WebSocket API: See `../api/websocket-api.md`
- Fee Oracle: See `fee-oracle.md`

267
docs/specs/mempool/realtime-events.md Normal file
View File

@@ -0,0 +1,267 @@
# Real-time Event System Specification
## Overview
This document specifies the event bus architecture for real-time event distribution across the explorer platform services.
## Architecture
**Technology**: Kafka or RabbitMQ
**Recommendation**: Kafka for high throughput, RabbitMQ for simpler setup
## Event Bus Architecture
```mermaid
flowchart LR
Producers[Event Producers]
Bus[Event Bus<br/>Kafka/RabbitMQ]
Consumers[Event Consumers]
Producers --> Bus
Bus --> Consumers
```
## Event Types
### Block Events
**Event**: `block.new`
**Payload**:
```json
{
"chain_id": 138,
"block_number": 12345,
"hash": "0x...",
"timestamp": "2024-01-01T00:00:00Z",
"transaction_count": 100
}
```
**Producers**: Indexer
**Consumers**: API Gateway (cache invalidation), WebSocket service, Analytics
### Transaction Events
**Event**: `transaction.confirmed`
**Payload**:
```json
{
"chain_id": 138,
"hash": "0x...",
"block_number": 12345,
"from_address": "0x...",
"to_address": "0x...",
"status": "success"
}
```
**Producers**: Indexer, Mempool service
**Consumers**: WebSocket service, Search indexer, Analytics
### Mempool Events
**Event**: `transaction.pending`
**Payload**: Transaction data
**Producers**: Mempool service
**Consumers**: WebSocket service, Fee oracle
### Token Transfer Events
**Event**: `token.transfer`
**Payload**:
```json
{
"chain_id": 138,
"token_address": "0x...",
"from_address": "0x...",
"to_address": "0x...",
"amount": "1000000000000000000",
"transaction_hash": "0x..."
}
```
**Producers**: Indexer
**Consumers**: Token balance updater, Analytics, Notifications
### Contract Verification Events
**Event**: `contract.verified`
**Payload**:
```json
{
"chain_id": 138,
"address": "0x...",
"verification_status": "verified",
"verified_at": "2024-01-01T00:00:00Z"
}
```
**Producers**: Verification service
**Consumers**: Search indexer, Cache invalidation
## Event Schema
### Standard Event Format
```json
{
"event_type": "block.new",
"event_id": "uuid",
"timestamp": "2024-01-01T00:00:00Z",
"chain_id": 138,
"payload": { ... },
"metadata": {
"producer": "indexer",
"version": "1.0"
}
}
```
## Topic/Queue Structure
### Kafka Topics
**Naming**: `{event_type}.{chain_id}` (e.g., `block.new.138`)
**Topics**:
- `block.new.{chain_id}`
- `transaction.confirmed.{chain_id}`
- `transaction.pending.{chain_id}`
- `token.transfer.{chain_id}`
- `contract.verified.{chain_id}`
**Partitioning**: By chain_id (all events for chain in same partition)
### RabbitMQ Exchanges
**Exchange Type**: Topic exchange
**Exchange Name**: `explorer.events`
**Routing Keys**: `{event_type}.{chain_id}` (e.g., `block.new.138`)
## Consumer Groups and Partitioning
### Consumer Groups
**Purpose**: Enable parallel processing and load balancing
**Groups**:
- `websocket-service`: WebSocket real-time updates
- `search-indexer`: Search index updates
- `analytics`: Analytics aggregation
- `cache-invalidation`: Cache invalidation
### Partitioning Strategy
**Kafka**:
- Partition by chain_id
- Same chain_id → same partition (maintains ordering)
**RabbitMQ**:
- Use consistent hash exchange for partitioning
- Partition by chain_id
## Delivery Guarantees
### At-Least-Once Delivery
**Default**: At-least-once delivery (events may be delivered multiple times)
**Handling**:
- Idempotent consumers
- Deduplication by event_id
- Track processed events
### Exactly-Once Delivery
**Use Case**: Critical financial events (banking layer)
**Implementation**:
- Kafka exactly-once semantics (if using Kafka)
- Idempotent consumers with deduplication
- Transactional processing
## Backpressure Handling
### Strategy
**Flow Control**:
- Consumer lag monitoring
- Slow consumer detection
- Automatic scaling
**Handling Slow Consumers**:
1. Scale consumers horizontally
2. Increase consumer resources
3. Alert on persistent lag
### Lag Monitoring
**Metrics**:
- Consumer lag per topic
- Processing rate
- Consumer health
**Alerts**: Lag > 1000 events or > 5 minutes
## Event Ordering
### Ordering Guarantees
**Within Partition**: Events processed in order
**Across Partitions**: No ordering guarantee
**Use Cases Requiring Ordering**:
- Block events (must process in order)
- Transaction events per address (maintain order)
**Implementation**: Use same partition for related events
## Error Handling
### Dead Letter Queue
**Purpose**: Store failed events for retry or investigation
**Strategy**:
- Retry failed events (exponential backoff)
- Move to DLQ after max retries
- Alert on DLQ events
### Retry Strategy
**Configuration**:
- Max retries: 3
- Backoff: Exponential (1s, 2s, 4s)
- Dead letter after max retries
## Monitoring
### Metrics
- Event production rate
- Event consumption rate
- Consumer lag
- Error rate
- Dead letter queue size
### Dashboards
- Event throughput per topic
- Consumer lag per consumer group
- Error rates
- System health
## References
- Mempool Service: See `mempool-service.md`
- WebSocket API: See `../api/websocket-api.md`

293
docs/specs/multichain/chain-adapter-interface.md Normal file
View File

@@ -0,0 +1,293 @@
# Chain Adapter Interface Specification
## Overview
This document specifies the abstract chain adapter interface that enables multi-chain support by abstracting chain-specific differences behind a unified interface.
## Adapter Interface
### Core Interface
```typescript
interface ChainAdapter {
// Chain identification
getChainId(): number;
getChainName(): string;
// RPC capabilities
getBlock(blockNumber: number | string): Promise<Block>;
getBlockByHash(hash: string): Promise<Block>;
getTransaction(hash: string): Promise<Transaction>;
getTransactionReceipt(hash: string): Promise<Receipt>;
getBalance(address: string, blockNumber?: string): Promise<string>;
call(transaction: CallRequest, blockNumber?: string): Promise<string>;
// Advanced capabilities
traceTransaction(hash: string): Promise<Trace[]>;
traceBlock(blockNumber: number): Promise<Trace[]>;
getCode(address: string, blockNumber?: string): Promise<string>;
// Subscription capabilities
subscribeNewBlocks(callback: (block: Block) => void): Subscription;
subscribePendingTransactions(callback: (tx: Transaction) => void): Subscription;
// Capability queries
supportsTracing(): boolean;
supportsArchive(): boolean;
supportsDebug(): boolean;
}
```
## RPC Capabilities Abstraction
### Archive Mode
**Interface**:
```typescript
interface ArchiveCapability {
getHistoricalBalance(address: string, blockNumber: number): Promise<string>;
getHistoricalCode(address: string, blockNumber: number): Promise<string>;
getHistoricalStorage(address: string, slot: string, blockNumber: number): Promise<string>;
}
```
**Implementation**:
- EVM chains: Use `eth_getBalance` with block number
- Non-EVM: Chain-specific historical queries
### Tracing Capabilities
**Interface**:
```typescript
interface TracingCapability {
traceCall(call: CallRequest, traceType: string[]): Promise<TraceResult>;
traceTransaction(hash: string, traceType: string[]): Promise<Trace[]>;
traceBlock(blockNumber: number, traceType: string[]): Promise<Trace[][]>;
}
```
**Trace Types**:
- `call`: Call traces
- `trace`: Full trace information
- `vmTrace`: VM execution trace
- `stateDiff`: State differences
**Implementation**:
- EVM chains: Use `debug_trace*` methods
- Non-EVM: Chain-specific trace methods
### Debug Capabilities
**Interface**:
```typescript
interface DebugCapability {
debugTraceTransaction(hash: string, options: TraceOptions): Promise<TraceResult>;
debugTraceBlock(blockNumber: number, options: TraceOptions): Promise<TraceResult[]>;
debugTraceCall(call: CallRequest, blockNumber: number, options: TraceOptions): Promise<TraceResult>;
}
```
**Use Cases**: Advanced debugging, detailed transaction analysis
## Token Standards Abstraction
### Token Interface
```typescript
interface TokenStandard {
// Common methods
getBalance(address: string, tokenAddress: string): Promise<string>;
getTotalSupply(tokenAddress: string): Promise<string>;
getDecimals(tokenAddress: string): Promise<number>;
getName(tokenAddress: string): Promise<string>;
getSymbol(tokenAddress: string): Promise<string>;
// Standard-specific
getTokenId(owner: string, index: number): Promise<string>; // ERC-721/1155
getTokenURI(tokenAddress: string, tokenId: string): Promise<string>; // ERC-721/1155
}
```
### Standard Detection
**ERC-20**:
- Detect `balanceOf`, `totalSupply`, `transfer` functions
- Standard interface ID: `0x36372b07`
**ERC-721**:
- Detect `balanceOf`, `ownerOf`, `tokenURI` functions
- Standard interface ID: `0x80ac58cd`
**ERC-1155**:
- Detect `balanceOf`, `balanceOfBatch`, `uri` functions
- Standard interface ID: `0xd9b67a26`
**Non-EVM Standards**:
- Chain-specific token standards
- Custom detection logic
## Gas Model Abstraction
### Gas Interface
```typescript
interface GasModel {
// Gas estimation
estimateGas(transaction: TransactionRequest): Promise<number>;
// Fee calculation
getGasPrice(): Promise<string>;
getFeeData(): Promise<FeeData>; // EIP-1559
// Gas limit
getBlockGasLimit(): Promise<number>;
getBlockGasUsed(): Promise<number>;
}
```
### EIP-1559 Support
**Interface**:
```typescript
interface EIP1559GasModel extends GasModel {
getBaseFee(): Promise<string>;
getMaxFeePerGas(): Promise<string>;
getMaxPriorityFeePerGas(): Promise<string>;
}
```
**Detection**: Check for `baseFeePerGas` in block header
### Legacy Gas Model
**Interface**: Simple gas price model (pre-EIP-1559)
## Finality Model Abstraction
### Finality Interface
```typescript
interface FinalityModel {
getFinalityDepth(): number; // Blocks until final
isFinal(blockNumber: number): Promise<boolean>;
getFinalityTime(): number; // Estimated time to finality (seconds)
}
```
### Finality Types
**Immediate Finality** (BFT chains):
- Finality depth: 1
- Examples: Cosmos, Polkadot (with finality gadgets)
**Probabilistic Finality** (PoW):
- Finality depth: 12-100 blocks
- Examples: Ethereum (pre-merge), Bitcoin
**Economic Finality** (PoS):
- Finality depth: 1-2 epochs
- Examples: Ethereum (post-merge), Cardano
## Implementation Examples
### EVM Chain Adapter
```typescript
class EVMChainAdapter implements ChainAdapter {
constructor(private rpcUrl: string, private chainId: number) {}
async getBlock(blockNumber: number | string): Promise<Block> {
const response = await this.rpcCall('eth_getBlockByNumber', [
blockNumber === 'latest' ? 'latest' : `0x${blockNumber.toString(16)}`,
true
]);
return this.transformBlock(response);
}
supportsTracing(): boolean {
return true; // Most EVM chains support tracing
}
// ... other methods
}
```
### Non-EVM Chain Adapter
```typescript
class NonEVMChainAdapter implements ChainAdapter {
// Chain-specific implementation
// Map chain-specific RPC calls to standard interface
}
```
## Adapter Registry
### Registry Interface
```typescript
interface ChainAdapterRegistry {
register(chainId: number, adapter: ChainAdapter): void;
get(chainId: number): ChainAdapter;
list(): ChainAdapter[];
supports(chainId: number): boolean;
}
```
### Configuration
```yaml
chains:
- chain_id: 138
name: "ChainID 138"
adapter: "evm"
rpc_url: "http://192.168.11.221:8545" # Internal IP for direct connection
supports_tracing: true
supports_archive: true
- chain_id: 1
name: "Ethereum Mainnet"
adapter: "evm"
rpc_url: "https://eth-mainnet.alchemyapi.io/v2/..."
supports_tracing: true
supports_archive: true
```
## Error Handling
### Chain-Specific Errors
**Handling**:
- Map chain-specific errors to standard error codes
- Provide error context (chain_id, method called)
- Log chain-specific error details
### Capability Errors
**Handling**:
- Check capability before calling method
- Return appropriate error if capability not supported
- Provide alternative methods when possible
## Testing
### Adapter Tests
**Test Cases**:
- Standard interface methods
- Error handling
- Capability detection
- Data transformation
### Integration Tests
**Test Cases**:
- Multi-chain queries
- Adapter switching
- Error propagation
## References
- Multi-chain Indexing: See `multichain-indexing.md`
- Database Schema: See `../database/postgres-schema.md`

154
docs/specs/multichain/entity-graph.md Normal file
View File

@@ -0,0 +1,154 @@
# Cross-Chain Entity Graph
## Overview
This document specifies the cross-chain entity graph for tracking relationships between addresses, contracts, and protocols across multiple chains.
## Graph Structure
See `../database/graph-schema.md` for detailed Neo4j schema.
### Node Types
- **Address**: Blockchain addresses (can exist on multiple chains)
- **Contract**: Smart contracts (chain-specific instances)
- **Token**: Token contracts
- **Protocol**: DeFi protocols (cross-chain presence)
### Relationship Types
- **TRANSFERRED_TO**: Token transfers
- **CALLED**: Contract calls
- **OWNS**: Token ownership
- **CLUSTERED_WITH**: Address clustering (same entity)
- **CCIP_MESSAGE_LINK**: Cross-chain message links
## Address Clustering
### Clustering Heuristics
**1. Multi-Signature Wallets**:
- Addresses controlled by same set of signers
- High confidence
**2. Transaction Patterns**:
- Addresses that frequently transact together
- Similar transaction timing
- Medium confidence
**3. Funding Patterns**:
- Addresses funded from same source
- Sequential funding
- Medium confidence
**4. Contract Deployment**:
- Addresses deploying contracts with similar code
- Same deployer patterns
- Low-medium confidence
### Confidence Scores
**High (0.9+)**: Multi-sig, verified relationships
**Medium (0.7-0.9)**: Pattern-based clustering
**Low (0.5-0.7)**: Weak patterns, needs verification
### Implementation
**Algorithm**:
1. Analyze transaction patterns
2. Calculate similarity scores
3. Cluster addresses above threshold
4. Store relationships in graph database
## Contract/Protocol Tagging
### Protocol Identification
**Methods**:
1. **Known Address Lists**: Maintain list of known protocol addresses
2. **Code Similarity**: Compare contract code
3. **Interaction Patterns**: Analyze usage patterns
4. **Labels**: User-submitted labels
### Protocol Categories
- **DEX**: Decentralized exchanges
- **Lending**: Lending protocols
- **Bridge**: Cross-chain bridges
- **NFT Marketplace**: NFT trading platforms
- **Governance**: DAOs and governance protocols
### Tagging System
**Tags**:
- Category tags (DEX, Lending, etc.)
- Protocol tags (Uniswap, Aave, etc.)
- Risk tags (verified, audited, etc.)
## Cross-Chain Entity Resolution
### Address Resolution
**Challenge**: Same entity may use different addresses on different chains
**Solution**:
- Use clustering to link addresses
- Use CCIP message links
- Use bridge transaction patterns
### Entity Unification
**Process**:
1. Identify entity across chains
2. Create unified entity node
3. Link chain-specific addresses to unified entity
4. Aggregate metrics across chains
## Graph Query Patterns
### Find Entity Across Chains
```cypher
MATCH (e:Entity)-[:HAS_ADDRESS]->(a:Address)
WHERE e.id = "entity_123"
RETURN a.address, a.chainId;
```
### Find Cross-Chain Transactions
```cypher
MATCH (a1:Address {address: "0x...", chainId: 138})
-[:CCIP_MESSAGE_LINK]->
(a2:Address {chainId: 1})
RETURN a2.address, a2.chainId;
```
### Find Protocol Users Across Chains
```cypher
MATCH (a:Address)-[r:INTERACTS_WITH]->(p:Protocol {name: "Uniswap"})
RETURN a.address, a.chainId, count(r) as interactions
ORDER BY interactions DESC;
```
## Data Ingestion
### Cross-Chain Data Sources
1. **Blockchain Data**: Indexed transactions and transfers
2. **CCIP Messages**: Cross-chain message links
3. **Bridge Transactions**: Bridge usage patterns
4. **User Labels**: User-submitted entity relationships
### Update Frequency
**Real-time**: New transactions and transfers
**Batch**: Clustering analysis (daily)
**On-demand**: User-requested clustering
## References
- Graph Database Schema: See `../database/graph-schema.md`
- CCIP Integration: See `../ccip/ccip-tracking.md`
- Multi-chain Indexing: See `multichain-indexing.md`

179
docs/specs/multichain/multichain-indexing.md Normal file
View File

@@ -0,0 +1,179 @@
# Multi-Chain Indexing Strategy
## Overview
This document specifies the strategy for indexing multiple blockchain networks with shared schema and per-chain workers.
## Architecture
```mermaid
flowchart TB
subgraph Chains[Blockchain Networks]
C1[Chain 138]
C2[Chain 1]
C3[Chain 137]
end
subgraph Workers[Indexer Workers]
W1[Worker: Chain 138]
W2[Worker: Chain 1]
W3[Worker: Chain 137]
end
subgraph DB[(Shared Database)]
PG[(PostgreSQL<br/>chain_id partitioned)]
ES[(Elasticsearch<br/>per-chain indices)]
end
C1 --> W1
C2 --> W2
C3 --> W3
W1 --> PG
W2 --> PG
W3 --> PG
W1 --> ES
W2 --> ES
W3 --> ES
```
## Per-Chain Indexer Workers
### Worker Architecture
**Design**: One indexer worker process per chain
**Isolation**:
- Separate processes/containers per chain
- Independent scaling per chain
- Chain-specific configuration
**Shared Components**:
- Database connection pool
- Message queue consumers
- Common indexing logic
### Worker Configuration
```yaml
workers:
- chain_id: 138
name: "indexer-chain-138"
rpc_url: "http://192.168.11.221:8545" # Internal IP for direct connection
adapter: "evm"
enabled: true
- chain_id: 1
name: "indexer-chain-1"
rpc_url: "https://eth-mainnet.alchemyapi.io/..."
adapter: "evm"
enabled: true
```
### Worker Scaling
**Horizontal Scaling**:
- Multiple workers per chain for high-throughput chains
- Partition blocks across workers (by block number range)
- Coordinate via database locks or message queue
**Scaling Strategy**:
- Scale workers based on chain activity
- Monitor worker lag per chain
- Auto-scale based on metrics
## Shared Schema with Chain ID Partitioning
### Database Partitioning
**Strategy**: Partition all tables by `chain_id`
**Benefits**:
- Efficient queries (partition pruning)
- Independent maintenance per chain
- Easy data isolation
**Implementation**: See `../database/postgres-schema.md`
### Index Strategy
**Global Indexes**: Across all chains for unified queries
**Chain-Specific Indexes**: Within partitions for chain-specific queries
## Cross-Chain Data Consistency
### Consistency Model
**Per-Chain Consistency**: Strong consistency within each chain
**Cross-Chain Consistency**: Eventual consistency (independent chains)
### Transaction Ordering
**Within Chain**: Maintain strict block/transaction ordering
**Across Chains**: No ordering guarantees (independent chains)
## Worker Coordination
### Block Processing Coordination
**Strategy**: Use database locks or message queue for coordination
**Approach 1: Database Locks**
- Acquire lock before processing block
- Release lock after completion
- Prevent duplicate processing
**Approach 2: Message Queue**
- Single consumer per chain partition
- Queue ensures ordering
- Automatic retry on failure
### Checkpointing
**Per-Chain Checkpoints**:
- Track last processed block per chain
- Stored in database
- Enable recovery and resume
## Performance Optimization
### Chain-Specific Optimizations
**High-Activity Chains**:
- More workers
- Larger batch sizes
- Optimized processing
**Low-Activity Chains**:
- Fewer workers
- Longer polling intervals
- Resource-efficient processing
### Resource Allocation
**CPU/Memory**: Allocate based on chain activity
**Database Connections**: Pool per chain
**Network Bandwidth**: Prioritize high-activity chains
## Monitoring
### Per-Chain Metrics
- Block lag (current block - last indexed)
- Processing rate (blocks/minute)
- Error rate
- Worker health
### Cross-Chain Metrics
- Total chains indexed
- Total blocks indexed across all chains
- System-wide throughput
## References
- Chain Adapter Interface: See `chain-adapter-interface.md`
- Database Schema: See `../database/postgres-schema.md`
- Indexer Architecture: See `../indexing/indexer-architecture.md`

192
docs/specs/multichain/unified-search.md Normal file
View File

@@ -0,0 +1,192 @@
# Unified Search Architecture
## Overview
This document specifies the unified search architecture that enables searching across multiple chains and entity types with relevance ranking.
## Architecture
```mermaid
flowchart TB
Query[Search Query]
Router[Query Router]
subgraph Search[Search Services]
ES1[Elasticsearch<br/>Chain 138]
ES2[Elasticsearch<br/>Chain 1]
ES3[Elasticsearch<br/>Chain 137]
end
Agg[Aggregator]
Rank[Relevance Ranking]
Results[Unified Results]
Query --> Router
Router --> ES1
Router --> ES2
Router --> ES3
ES1 --> Agg
ES2 --> Agg
ES3 --> Agg
Agg --> Rank
Rank --> Results
```
## Search Algorithm
### Query Processing
**Steps**:
1. Parse query (extract terms, filters)
2. Determine chain scope (all chains or specific chain)
3. Route to appropriate search indices
4. Execute searches in parallel
5. Aggregate results
6. Rank by relevance
7. Return unified results
### Query Types
**Exact Match** (Hash, Address):
- Direct lookup in specific chain
- Return single result if found
**Full-Text Search** (Name, Symbol, Label):
- Search across all chains
- Rank by relevance
- Return top N results per chain
**Fuzzy Search** (Typos, Partial matches):
- Use fuzzy matching
- Rank by similarity
- Include suggestions
## Ranking and Relevance Scoring
### Relevance Factors
**1. Exact Match Score**:
- Exact match: 100%
- Prefix match: 80%
- Fuzzy match: 60%
**2. Chain Relevance**:
- User's preferred chain: +20%
- Popular chains: +10%
**3. Entity Type Relevance**:
- Addresses: Highest (most specific)
- Transactions: High
- Blocks: Medium
- Tokens: Medium
- Contracts: Lower (unless verified)
**4. Popularity Score**:
- Transaction count
- Token holder count
- Contract usage
### Scoring Formula
```
score = (exact_match_score * 0.5) +
(chain_relevance * 0.2) +
(entity_type_relevance * 0.2) +
(popularity_score * 0.1)
```
## Result Aggregation
### Aggregation Strategy
**Per-Chain Results**:
- Limit results per chain (e.g., top 10)
- Combine across chains
- Remove duplicates (same address on multiple chains)
### Result Format
```json
{
"query": "0x123...",
"total_results": 5,
"results": [
{
"type": "address",
"chain_id": 138,
"address": "0x123...",
"label": "My Wallet",
"score": 0.95
},
{
"type": "transaction",
"chain_id": 138,
"hash": "0x123...",
"score": 0.80
}
],
"chains_searched": [138, 1, 137]
}
```
## Performance Optimization
### Caching
**Cache Strategy**:
- Cache popular queries (top 1000)
- Cache duration: 1 minute
- Invalidate on data updates
### Parallel Search
**Strategy**: Execute searches across chains in parallel
**Benefits**:
- Faster response time
- Better resource utilization
### Result Limiting
**Per-Chain Limit**: Top 10-20 results per chain
**Total Limit**: Top 50-100 results total
## Search Indexes
### Per-Chain Indices
**Index Names**: `{entity_type}-{chain_id}` (e.g., `addresses-138`)
**Benefits**:
- Independent scaling per chain
- Chain-specific optimizations
- Easy chain addition/removal
### Global Index (Optional)
**Use Case**: Quick lookup across all chains
**Implementation**:
- Separate index with chain_id field
- Less detailed than per-chain indices
- Faster for simple queries
## API Endpoint
### Unified Search
`GET /api/v1/search`
**Query Parameters**:
- `q` (string, required): Search query
- `chain_id` (integer, optional): Filter by chain
- `type` (string, optional): Filter by type (address, transaction, block, token, contract)
- `limit` (integer, default: 50): Max results
**Response**: Unified search results (see format above)
## References
- Search Index Schema: See `../database/search-index-schema.md`
- Multi-chain Indexing: See `multichain-indexing.md`

68
docs/specs/observability/dashboards.md Normal file
View File

@@ -0,0 +1,68 @@
# Dashboards Specification
## Overview
Observability dashboards for monitoring platform health and performance.
## Indexer Lag Dashboard
### Metrics
- Current block vs indexed block (per chain)
- Time lag (minutes behind)
- Processing rate (blocks/minute)
- Historical lag trends
### Visualizations
- Lag over time (line chart)
- Current lag by chain (bar chart)
- Alert status (indicator)
## CCIP Message Lifecycle Dashboard
### Metrics
- Messages by status
- Success rate
- Average execution time
- Failure reasons
### Visualizations
- Message flow diagram
- Status distribution (pie chart)
- Latency over time
- Chain pair statistics
## Transaction Funnel Analytics
### Funnel Stages
1. Quote requested
2. User approved
3. Transaction signed
4. Transaction broadcast
5. Transaction confirmed
### Metrics
- Conversion rate at each stage
- Drop-off reasons
- Time at each stage
## System Health Dashboard
### Components
- Service health status
- Error rates
- Request rates
- Resource usage
- Database health
## References
- Metrics & Monitoring: See `metrics-monitoring.md`
- Logging: See `logging.md`

75
docs/specs/observability/logging.md Normal file
View File

@@ -0,0 +1,75 @@
# Logging Architecture Specification
## Overview
Centralized logging architecture for the explorer platform.
## Log Aggregation Strategy
**Solution**: ELK Stack (Elasticsearch, Logstash, Kibana) or Loki + Grafana
**Flow**:
1. Services emit logs
2. Log collectors aggregate logs
3. Logs stored in central store
4. Dashboards and queries via UI
## Log Levels and Categorization
### Log Levels
- **DEBUG**: Detailed debugging information
- **INFO**: General informational messages
- **WARN**: Warning messages
- **ERROR**: Error messages
- **FATAL**: Critical errors
### Categories
**Application Logs**: Business logic, API requests
**Access Logs**: HTTP requests, authentication
**System Logs**: Infrastructure, system events
**Audit Logs**: Security events, compliance
## Structured Logging Format
### Log Format
```json
{
"timestamp": "2024-01-01T00:00:00Z",
"level": "INFO",
"service": "explorer-api",
"message": "Request processed",
"request_id": "uuid",
"user_id": "uuid",
"chain_id": 138,
"method": "GET",
"path": "/api/v1/blocks",
"status_code": 200,
"duration_ms": 45,
"metadata": {}
}
```
## Log Retention Policies
**Development**: 7 days
**Staging**: 30 days
**Production**: 90 days (hot), 1 year (cold archive)
## PII Sanitization in Logs
**Strategy**: Remove PII before logging
**Fields to Sanitize**:
- Email addresses
- Personal names
- Addresses
- API keys (partial masking)
**Implementation**: Log sanitization middleware
## References
- Metrics & Monitoring: See `metrics-monitoring.md`

104
docs/specs/observability/metrics-monitoring.md Normal file
View File

@@ -0,0 +1,104 @@
# Metrics & Monitoring Specification
## Overview
Metrics collection and monitoring for the explorer platform.
## Metrics Catalog
### API Metrics
- Request rate (requests/second)
- Response time (p50, p95, p99)
- Error rate (by status code)
- Endpoint usage
### Indexer Metrics
- Blocks processed per minute
- Transactions processed per minute
- Block lag (current block - last indexed)
- Error rate
- Processing time
### Database Metrics
- Query performance
- Connection pool usage
- Slow queries
- Replication lag
### Infrastructure Metrics
- CPU usage
- Memory usage
- Disk I/O
- Network I/O
## Dashboard Specifications
### Key Dashboards
**1. System Health**:
- Overall system status
- Service health
- Error rates
- Resource usage
**2. API Performance**:
- Request rates
- Latency percentiles
- Error rates
- Top endpoints
**3. Indexer Performance**:
- Block processing rate
- Indexer lag
- Error rates
- Chain status
## Alerting Rules
### Alert Conditions
**Critical**:
- Service down
- Error rate > 5%
- Indexer lag > 100 blocks
- Database connection failures
**Warning**:
- Error rate > 1%
- Indexer lag > 10 blocks
- High latency (p95 > 1s)
- High resource usage (> 80%)
### Alert Channels
- Email
- Slack
- PagerDuty (for critical)
## SLO Definitions
### API SLOs
- **Availability**: 99.9% uptime
- **Latency**: p95 < 500ms
- **Error Rate**: < 0.1%
### Indexer SLOs
- **Lag**: < 10 blocks behind chain head
- **Processing Time**: p95 < 5 seconds per block
### WebSocket SLOs
- **Delivery**: 99.9% message delivery
- **Latency**: < 100ms message delivery
## References
- Logging: See `logging.md`
- Tracing: See `tracing.md`

66
docs/specs/observability/tracing.md Normal file
View File

@@ -0,0 +1,66 @@
# Distributed Tracing Specification
## Overview
Distributed tracing for request tracking across services.
## Distributed Tracing Strategy
**Solution**: OpenTelemetry or Jaeger
**Implementation**:
- Instrument services with tracing
- Propagate trace context
- Collect and store traces
- Visualize in UI
## Trace Sampling
### Sampling Strategy
**Head-Based Sampling**:
- Sample rate: 1% of requests
- Always sample errors
- Always sample slow requests (> 1s)
**Tail-Based Sampling** (optional):
- Sample based on trace characteristics
- More efficient storage
## Trace Correlation Across Services
### Trace Context Propagation
**Headers**:
- `traceparent` (W3C Trace Context)
- `tracestate` (W3C Trace Context)
**Propagation**: HTTP headers, message queue metadata
### Trace Structure
```
Trace (request)
├── Span (API Gateway)
│ ├── Span (Explorer API)
│ │ ├── Span (Database Query)
│ │ └── Span (Cache Lookup)
│ └── Span (Search Service)
└── Span (Response)
```
## Performance Analysis Workflows
### Analysis Steps
1. Identify slow requests
2. Trace request path
3. Identify bottlenecks
4. Optimize slow components
5. Verify improvements
## References
- Logging: See `logging.md`
- Metrics: See `metrics-monitoring.md`

100
docs/specs/security/auth-spec.md Normal file
View File

@@ -0,0 +1,100 @@
# Authentication & Authorization Specification
## Overview
Authentication and authorization system for user access and API access.
## User Authentication Flows
### Authentication Methods
**1. Email/Password**:
- Registration with email
- Password hashing (bcrypt/argon2)
- Email verification
**2. OAuth**:
- Google, GitHub, etc.
- OAuth 2.0 flow
- Token-based authentication
**3. Wallet Authentication**:
- Signature-based authentication
- Prove ownership of address
### Session Management
**Storage**: HTTP-only cookies or JWT tokens
**Expiration**: Configurable (default: 24 hours)
**Refresh**: Refresh tokens for extended sessions
## API Key Management
### Key Generation
**Format**: Secure random tokens
**Storage**: Hashed (not plaintext)
**Metadata**: Name, tier, permissions, expiration
### Key Usage
**Authentication**: Via `X-API-Key` header
**Rate Limiting**: Based on key tier
**Revocation**: Support key revocation
## RBAC (Role-Based Access Control)
### Roles
**Public**: Unauthenticated users
**User**: Authenticated users
**Pro**: Paid tier users
**Admin**: Platform administrators
**Compliance**: Compliance officers
### Permissions
**Read**: View data
**Write**: Create/update data
**Admin**: Full access
**Compliance**: Compliance-specific access
## OAuth Integration
### OAuth Providers
- Google
- GitHub
- Others as needed
### OAuth Flow
1. User initiates OAuth login
2. Redirect to provider
3. User authorizes
4. Callback with code
5. Exchange code for tokens
6. Create/login user account
7. Establish session
## Session Management
### Session Storage
**Options**:
- Server-side sessions (Redis)
- JWT tokens (stateless)
**Recommendation**: Server-side sessions for better security
### Session Security
- Secure cookies (HTTPS only)
- HttpOnly flag
- SameSite attribute
- CSRF protection
## References
- Security Architecture: See `security-architecture.md`

62
docs/specs/security/ddos-protection.md Normal file
View File

@@ -0,0 +1,62 @@
# DDoS Protection Specification
## Overview
DDoS protection via WAF, CDN, and rate limiting.
## WAF Rules
### Rule Categories
**1. IP Reputation**:
- Block known malicious IPs
- Rate limit suspicious IPs
**2. Request Patterns**:
- Detect bot patterns
- Block automated attacks
**3. Geographic**:
- Optional geographic restrictions
- Block high-risk regions
## CDN Integration
**Provider**: Cloudflare
**Benefits**:
- DDoS mitigation
- Geographic distribution
- Caching
## Rate Limiting Strategies
### Rate Limit Levels
**Per IP**: 100 requests/minute
**Per API Key**: Based on tier
**Per Endpoint**: Varies by endpoint complexity
### Implementation
**Method**: Token bucket or sliding window
**Storage**: Redis for distributed rate limiting
**Headers**: Rate limit headers in responses
## IP Blocking and Whitelisting
### Blocking
**Automatic**: Block IPs exceeding rate limits
**Manual**: Admin can block specific IPs
**Temporary**: Auto-unblock after cooldown period
### Whitelisting
**Use Case**: Known good IPs (partners, internal)
**Implementation**: Bypass rate limits for whitelisted IPs
## References
- API Gateway: See `../api/api-gateway.md`
- Security Architecture: See `security-architecture.md`

63
docs/specs/security/privacy-controls.md Normal file
View File

@@ -0,0 +1,63 @@
# Privacy Controls Specification
## Overview
Privacy controls for PII protection and data residency.
## PII Separation Strategy
### Data Segregation
**Public Data**: Blockchain data (no PII)
**Private Data**: User accounts, KYC data (PII)
**Storage**: Separate databases/partitions
### Access Control
**Public Data**: Open access
**Private Data**: Strict access control, encryption
## Tokenization/Encryption
### Identity Artifacts
**Encryption**: Encrypt sensitive identity documents
**Storage**: Encrypted at rest
**Access**: Decrypt only when needed, audit logged
### Tokenization
**Use Case**: Reference PII without exposing it
**Implementation**: Store tokens, map to PII securely
## Regional Data Residency Controls
### Data Residency
**Requirement**: Store data in specific regions per regulations
**Implementation**: Regional databases/partitions
**Routing**: Route user data to appropriate region
### Compliance
- GDPR (EU)
- CCPA (California)
- Others as needed
## Data Retention Policies
### Retention Periods
**User Data**: Per regulatory requirements
**Transaction Data**: Per regulatory requirements
**Logs**: Per security requirements
### Deletion
**Right to Deletion**: Support user data deletion requests
**Process**: Secure deletion, audit logged
## References
- Security Architecture: See `security-architecture.md`

69
docs/specs/security/security-architecture.md Normal file
View File

@@ -0,0 +1,69 @@
# Security Architecture Specification
## Overview
Security architecture including threat model, security boundaries, and security controls.
## Threat Model
### Threat Categories
**1. External Threats**:
- DDoS attacks
- SQL injection
- XSS attacks
- API abuse
**2. Internal Threats**:
- Privileged user abuse
- Data breaches
- Insider threats
**3. Infrastructure Threats**:
- Node compromise
- Database breaches
- Network attacks
## Security Boundaries
**DMZ**: Public-facing services
**Internal Network**: Backend services
**Data Layer**: Database and storage (isolated)
**Blockchain Network**: Node network (isolated)
## KMS/HSM Integration
### Key Management
**Storage**: Hardware Security Module (HSM)
**Usage**: API keys, signing keys, encryption keys
**Access**: Role-based, audit logged
## Secrets Management
### Secret Storage
**Solution**: Vault, AWS Secrets Manager, or similar
**Encryption**: At rest and in transit
**Rotation**: Regular key rotation
**Access**: Least privilege principle
## Signed Builds and SBOM
### Build Signing
**Process**: Sign all builds
**Verification**: Verify signatures before deployment
**Tooling**: Code signing certificates
### SBOM (Software Bill of Materials)
**Generation**: Generate SBOM for all dependencies
**Storage**: Store SBOM for audit
**Vulnerability Scanning**: Scan SBOM for known vulnerabilities
## References
- Authentication: See `auth-spec.md`
- Privacy: See `privacy-controls.md`

100
docs/specs/vtm/conversation-state.md Normal file
View File

@@ -0,0 +1,100 @@
# Conversation State Management Specification
## Overview
Conversation state management for multi-turn VTM conversations.
## State Schema
```json
{
"session_id": "uuid",
"user_id": "uuid",
"context": {
"current_workflow": "bridge_action",
"workflow_state": {...},
"entities": {
"amount": "100",
"source_chain": "138",
"destination_chain": "1"
}
},
"conversation_history": [
{
"turn": 1,
"user": "I want to bridge tokens",
"assistant": "Which chains would you like to bridge between?",
"timestamp": "2024-01-01T00:00:00Z"
}
],
"intents": ["bridge_initiation"],
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:05:00Z"
}
```
## Context Preservation
### Context Types
**Workflow Context**: Current workflow state
**Entity Context**: Extracted entities (amounts, addresses, etc.)
**User Context**: User preferences, history
**Session Context**: Session metadata
### Context Updates
**Triggers**:
- User input
- Workflow progression
- Entity extraction
- External events
## Multi-Turn Conversation Handling
### Turn Management
**Track**:
- Turn number
- User input
- Assistant response
- Extracted entities
- Intent changes
### Entity Extraction
**Entities**:
- Amounts
- Addresses
- Chain names/IDs
- Token symbols
- Dates/times
### Disambiguation
**Handling Ambiguity**:
- Ask clarifying questions
- Suggest options
- Use context to infer
## Session Timeout and Recovery
### Timeout
**Duration**: 30 minutes of inactivity
**Action**: Save state, end session
**Recovery**: Resume from saved state if user returns
### Recovery
**Process**:
1. User returns within timeout window
2. Restore conversation state
3. Continue from last turn
4. Prompt user to continue
## References
- Teller Orchestrator: See `teller-orchestrator.md`
- Workflows: See `teller-workflows.md`

87
docs/specs/vtm/soul-machines-integration.md Normal file
View File

@@ -0,0 +1,87 @@
# Soul Machines Integration Specification
## Overview
Integration with Soul Machines digital humans for Virtual Teller Machine (VTM) experience.
## Digital Human UI Embedding
### Web Integration
**Method**: Embed Soul Machines SDK/widget
**Container**: React component or iframe
**Positioning**: Modal overlay or side panel
### Mobile Integration
**Method**: Native SDK integration
**UI**: Full-screen or overlay
### XR Integration
**Method**: 3D digital human in XR environment
**Platform**: WebXR or native XR app
## API Integration Points
### Session Management
**Endpoints**:
- Create session
- End session
- Get session status
### Conversation Management
**Events**:
- User message
- Digital human response
- Intent detection
- Workflow triggers
## Session Management
### Session Lifecycle
1. User initiates VTM session
2. Create Soul Machines session
3. Authenticate user (if required)
4. Load user context
5. Start conversation
6. End session (user or timeout)
### Session Data
**Stored**:
- Session ID
- User ID
- Conversation history
- Workflow state
- Timestamps
## Conversation State Persistence
### State Storage
**Storage**: Database or session store
**Retention**: Per session, archive after completion
### State Schema
```json
{
"session_id": "uuid",
"user_id": "uuid",
"conversation_history": [...],
"workflow_state": {...},
"intents": [...],
"created_at": "timestamp",
"updated_at": "timestamp"
}
```
## References
- Teller Orchestrator: See `teller-orchestrator.md`
- Workflows: See `teller-workflows.md`

113
docs/specs/vtm/teller-orchestrator.md Normal file
View File

@@ -0,0 +1,113 @@
# Teller Orchestrator Specification
## Overview
Orchestrator that connects Soul Machines digital humans with backend services and workflows.
## Workflow Engine Architecture
```mermaid
flowchart TB
DH[Digital Human UI]
NLU[Intent/Policy Layer]
WF[Workflow Engine]
BANK[Banking API]
ACT[Action Orchestrator]
EXP[Explorer Services]
HUM[Human Agent Console]
DH --> NLU
NLU --> WF
WF --> BANK
WF --> ACT
WF --> EXP
WF --> HUM
```
## Intent Recognition and Routing
### Intent Types
**Banking Intents**:
- Account opening
- Balance inquiry
- Transaction explanation
- Limit increase request
**Explorer Intents**:
- Transaction lookup
- Address information
- Token information
**Action Intents**:
- Bridge initiation
- Swap assistance
- Wallet setup
### Intent Recognition
**Method**: NLU from Soul Machines + custom intent classification
**Routing**: Route to appropriate workflow handler
## Policy Enforcement Layer
### Policies
**Role-Based Access**:
- Customer permissions
- Feature access
- Limit enforcement
**Compliance Policies**:
- KYC requirements
- Transaction limits
- Regulatory restrictions
### Enforcement
**Checks**:
- Verify user permissions
- Check compliance status
- Validate limits
- Enforce policies
## Action Authorization
### Authorization Flow
1. User requests action via VTM
2. Teller orchestrator validates request
3. Check user permissions
4. Check compliance status
5. Request explicit confirmation
6. Execute action if authorized
### Authorization Levels
**Level 1**: Information queries (no authorization needed)
**Level 2**: Read-only actions (view account)
**Level 3**: Financial actions (require explicit confirmation)
## Human Escalation Workflow
### Escalation Triggers
**Conditions**:
- Complex issues beyond VTM capability
- User request for human agent
- Compliance concerns
- Technical errors
### Escalation Process
1. Identify escalation need
2. Create support ticket
3. Notify human agent
4. Transfer conversation context
5. Human agent takes over
## References
- Soul Machines Integration: See `soul-machines-integration.md`
- Workflows: See `teller-workflows.md`

98
docs/specs/vtm/teller-workflows.md Normal file
View File

@@ -0,0 +1,98 @@
# Teller Workflows Specification
## Overview
Workflow definitions for common VTM teller interactions.
## Workflow: Account Opening
**State Machine**:
```
[Start] → [Identity Verification] → [KYC Check] → [Account Creation] → [Wallet Setup] → [Complete]
↓ ↓
[Failed] [Manual Review]
```
**Steps**:
1. Collect user information
2. Initiate identity verification
3. Run KYC checks
4. Create account if approved
5. Set up wallet
6. Complete onboarding
## Workflow: Bridge Action
**State Machine**:
```
[Start] → [Select Chains] → [Enter Amount] → [Risk Check] → [Confirmation] → [Execute] → [Monitor] → [Complete]
↓
[Failed]
```
**Steps**:
1. User requests bridge
2. Select source and destination chains
3. Enter amount
4. Get quote and show details
5. Risk assessment
6. User confirmation
7. Execute bridge
8. Monitor status
9. Confirm completion
## Workflow: Transaction Explanation
**Steps**:
1. User provides transaction hash
2. Fetch transaction details
3. Explain transaction components
4. Answer user questions
5. Provide additional context if needed
## User Consent Checkpoints
### Checkpoint Types
**Implicit Consent**: Information queries
**Explicit Consent**: Financial actions
**Strong Consent**: High-value or high-risk actions
### Consent Recording
**Record**:
- Action requested
- User acknowledgment
- Timestamp
- Method (voice, text, button)
## Audit Logging Requirements
### Logged Events
**All Events**:
- Workflow start/completion
- User inputs
- System actions
- Authorizations
- Escalations
- Errors
### Log Format
```json
{
"session_id": "uuid",
"user_id": "uuid",
"workflow": "bridge_action",
"event": "user_confirmation",
"data": {...},
"timestamp": "2024-01-01T00:00:00Z"
}
```
## References
- Teller Orchestrator: See `teller-orchestrator.md`
- Conversation State: See `conversation-state.md`

44
docs/specs/xr/xr-architecture.md Normal file
View File

@@ -0,0 +1,44 @@
# XR Architecture Specification
## Overview
XR (Extended Reality) architecture for immersive blockchain exploration experiences.
## WebXR Integration Strategy
### Primary Approach
**Technology**: WebXR API
**Benefits**: Browser-based, no app installation
**Limitations**: Device compatibility
### Fallback
**Unity/Unreal**: Native apps for high-fidelity experiences
**Use Case**: Advanced XR features, better performance
## Shared Backend API Usage
**Strategy**: XR clients use same APIs as web/mobile
**Benefits**: Consistency, reuse, easier maintenance
## Scene Management
### Scene Types
- Block graph space
- Transaction flow visualization
- Cross-chain message tunnels
- Virtual bank branch
### Scene Transitions
- Smooth transitions between scenes
- Loading states
- Scene persistence
## References
- XR Scenes: See `xr-scenes.md`
- XR Performance: See `xr-performance.md`

48
docs/specs/xr/xr-performance.md Normal file
View File

@@ -0,0 +1,48 @@
# XR Performance Specification
## Overview
Performance requirements and optimization strategies for XR experiences.
## Frame Rate Targets
**Target**: 90 FPS (VR) or 60 FPS (AR)
**Minimum**: 72 FPS (VR) or 30 FPS (AR)
**Measurement**: Consistent frame timing
## LOD (Level of Detail) Strategies
### Strategy
**Distance-Based LOD**:
- High detail for nearby objects
- Reduced detail for distant objects
- Cull objects beyond view distance
**Performance-Based LOD**:
- Adjust detail based on frame rate
- Dynamic quality adjustment
## Asset Optimization
### Optimization Techniques
- Texture compression
- Geometry simplification
- Animation optimization
- Occlusion culling
## Network Data Streaming
### Strategy
- Load critical data first
- Stream additional data progressively
- Cache frequently accessed data
- Preload based on user movement
## References
- XR Architecture: See `xr-architecture.md`
- XR Scenes: See `xr-scenes.md`

60
docs/specs/xr/xr-scenes.md Normal file
View File

@@ -0,0 +1,60 @@
# XR Scenes Specification
## Overview
XR scene designs for blockchain visualization and VTM interactions.
## Block/Transaction Graph Space
### Design
**Concept**: 3D graph of blocks and transactions
**Navigation**: Fly through, zoom, select
**Interactions**: Select block/transaction to view details
### Elements
- Blocks as nodes
- Transactions as edges
- Time axis (depth)
- Network activity visualization
## Cross-Chain Message Tunnels
### Design
**Concept**: Visualize CCIP messages as tunnels between chains
**Navigation**: Follow message path
**Interactions**: View message details at each hop
## Virtual Bank Branch
### Design
**Concept**: 3D bank branch environment
**Purpose**: VTM interactions in immersive setting
**Elements**: Teller desk, digital human, waiting area
## Navigation Patterns
### Teleport Navigation
**Method**: Point and teleport
**Benefits**: Reduces motion sickness
**Use Case**: Large scene navigation
### Smooth Movement
**Method**: Continuous movement (optional)
**Settings**: Enable/disable based on user preference
## Accessibility Fallback to 2D
**Requirement**: Always provide 2D fallback
**Trigger**: User preference or device capability
**Implementation**: Same data, 2D UI
## References
- XR Architecture: See `xr-architecture.md`