Add full monorepo: virtual-banker, backend, frontend, docs, scripts, deployment

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/specs/mempool/fee-oracle.md

@@ -0,0 +1,275 @@
+# Fee Oracle Specification
+
+## Overview
+
+This document specifies the fee estimation service that calculates and predicts transaction fees based on historical data and current mempool conditions.
+
+## Architecture
+
+```mermaid
+flowchart TB
+    subgraph Data[Data Sources]
+        Blocks[Recent Blocks]
+        Mempool[Current Mempool]
+        History[Historical Data]
+    end
+    
+    subgraph Analysis[Analysis]
+        Percentile[Percentile Calculator]
+        Prediction[Prediction Model]
+        Adjust[Market Adjuster]
+    end
+    
+    subgraph Output[Output]
+        Estimates[Fee Estimates]
+        API[API Endpoint]
+        Cache[Cache]
+    end
+    
+    Blocks --> Percentile
+    Mempool --> Percentile
+    History --> Prediction
+    Percentile --> Adjust
+    Prediction --> Adjust
+    Adjust --> Estimates
+    Estimates --> API
+    Estimates --> Cache
+```
+
+## Fee Calculation Methods
+
+### Method 1: Percentile-Based (Primary)
+
+**Algorithm**:
+1. Collect gas prices from recent N blocks (e.g., 100 blocks)
+2. Calculate percentiles: 25th, 50th, 75th, 95th
+3. Use percentiles as fee estimates
+
+**Percentile Mapping**:
+- Slow: 25th percentile
+- Standard: 50th percentile (median)
+- Fast: 75th percentile
+- Urgent: 95th percentile
+
+**Update Frequency**: Every new block
+
+### Method 2: Mempool-Based Adjustment
+
+**Algorithm**:
+1. Start with percentile-based estimates
+2. Analyze current mempool:
+   - Average gas price of pending transactions
+   - Pending transaction count
+   - Mempool pressure (pending / recent block capacity)
+3. Adjust estimates based on mempool conditions
+
+**Adjustment Factors**:
+- High mempool pressure → Increase estimates
+- Low mempool pressure → Decrease estimates
+- Recent confirmed gas prices → Weight recent data more
+
+### Method 3: Time-Based Prediction
+
+**Algorithm**:
+- Predict fees for different confirmation times
+- Example: "To be confirmed in 1 block, use X gas price"
+- Use historical data to train model
+
+**Prediction Targets**:
+- 1 block (next block)
+- 3 blocks (within 3 blocks)
+- 10 blocks (within 10 blocks)
+
+## Fee Estimate Structure
+
+### Standard Estimates
+
+```json
+{
+  "chain_id": 138,
+  "block_number": 12345,
+  "estimates": {
+    "slow": {
+      "gas_price": "20000000000",
+      "max_fee_per_gas": "20000000000",
+      "max_priority_fee_per_gas": "1000000000",
+      "confidence": 0.95,
+      "estimated_confirmation_time": "5 minutes"
+    },
+    "standard": {
+      "gas_price": "30000000000",
+      "max_fee_per_gas": "30000000000",
+      "max_priority_fee_per_gas": "2000000000",
+      "confidence": 0.90,
+      "estimated_confirmation_time": "2 minutes"
+    },
+    "fast": {
+      "gas_price": "50000000000",
+      "max_fee_per_gas": "50000000000",
+      "max_priority_fee_per_gas": "3000000000",
+      "confidence": 0.85,
+      "estimated_confirmation_time": "30 seconds"
+    },
+    "urgent": {
+      "gas_price": "100000000000",
+      "max_fee_per_gas": "100000000000",
+      "max_priority_fee_per_gas": "5000000000",
+      "confidence": 0.80,
+      "estimated_confirmation_time": "next block"
+    }
+  },
+  "timestamp": "2024-01-01T00:00:00Z"
+}
+```
+
+### EIP-1559 Support
+
+**For EIP-1559 Chains**:
+- Provide `max_fee_per_gas` and `max_priority_fee_per_gas`
+- Calculate base fee prediction
+- Estimate priority fee needed for desired speed
+
+## Historical Fee Analysis
+
+### Data Collection
+
+**Sources**:
+- Recent blocks (last 100-1000 blocks)
+- Historical fee data (time-series database)
+
+**Metrics Tracked**:
+- Min, max, average gas price per block
+- Percentiles (25th, 50th, 75th, 95th, 99th)
+- Gas price distribution
+- Confirmation time by gas price
+
+### Analysis Windows
+
+**Short-term** (Last 10 blocks):
+- Most recent trends
+- Quick response to market changes
+
+**Medium-term** (Last 100 blocks):
+- Stable baseline
+- Primary estimate source
+
+**Long-term** (Last 1000 blocks):
+- Historical patterns
+- Anomaly detection
+
+## Prediction Models
+
+### Simple Moving Average
+
+**Algorithm**: Average of recent N blocks
+
+**Pros**: Simple, fast
+**Cons**: Slow to adapt to changes
+
+### Weighted Moving Average
+
+**Algorithm**: Weight recent blocks more heavily
+
+**Pros**: Faster adaptation
+**Cons**: More volatile
+
+### Machine Learning Models (Future)
+
+**Potential Models**:
+- Linear regression
+- Time series forecasting (ARIMA, LSTM)
+- Classification (predict confirmation time)
+
+**Features**:
+- Historical gas prices
+- Mempool metrics
+- Block time
+- Network activity
+
+## API Endpoint
+
+### Get Fee Estimates
+
+`GET /api/v1/mempool/{chain_id}/fees`
+
+**Query Parameters**:
+- `block_target` (integer): Target confirmation block (optional)
+
+**Response**: Fee estimates object (see structure above)
+
+### Get Fee History
+
+`GET /api/v1/mempool/{chain_id}/fees/history`
+
+**Query Parameters**:
+- `period` (string): Time period (1h, 24h, 7d, 30d)
+- `interval` (string): Aggregation interval (1m, 5m, 1h)
+
+**Response**: Historical fee data with timestamps
+
+## Caching Strategy
+
+### Cache Duration
+
+- **Fee Estimates**: 10 seconds (update every block)
+- **Fee History**: 5 minutes (less frequently updated)
+
+### Cache Invalidation
+
+- Invalidate on new block
+- Invalidate on significant mempool changes
+
+## Performance Considerations
+
+### Calculation Performance
+
+**Target**: Calculate estimates in < 100ms
+
+**Optimization**:
+- Pre-calculate percentiles
+- Cache historical data
+- Use efficient algorithms
+
+### Update Frequency
+
+**Block-based**: Update on every new block
+**Time-based**: Update every 10 seconds (fallback)
+
+## Accuracy Metrics
+
+### Tracking Accuracy
+
+**Metrics**:
+- Predicted vs actual confirmation time
+- Predicted vs actual gas price used
+- Estimate accuracy by tier (slow/standard/fast/urgent)
+
+### Confidence Scores
+
+**Calculation**:
+- Based on historical accuracy
+- Variance in recent data
+- Mempool stability
+
+**Display**: Include confidence score in estimates
+
+## Monitoring
+
+### Metrics
+
+- Fee estimate accuracy
+- Update latency
+- Calculation time
+- Cache hit rate
+
+### Alerts
+
+- High prediction error (> 50% off)
+- Calculation timeout
+- Cache miss rate spike
+
+## References
+
+- Mempool Service: See `mempool-service.md`
+- Time-Series Schema: See `../database/timeseries-schema.md`
+

docs/specs/mempool/mempool-service.md

@@ -0,0 +1,251 @@
+# Mempool Service Architecture Specification
+
+## Overview
+
+This document specifies the mempool service that tracks pending transactions, monitors transaction propagation, and provides real-time mempool insights.
+
+## Architecture
+
+```mermaid
+flowchart TB
+    subgraph RPC[RPC Node]
+        WS[WebSocket<br/>New Pending Tx]
+        Poll[Poll<br/>Pending Tx]
+    end
+    
+    subgraph Ingest[Ingestion]
+        Listener[Transaction Listener]
+        Validator[Transaction Validator]
+        Queue[Message Queue]
+    end
+    
+    subgraph Process[Processing]
+        Tracker[Propagation Tracker]
+        RBF[RBF Detector]
+        Fee[Fee Calculator]
+        Storage[Storage]
+    end
+    
+    subgraph Output[Output]
+        API[API Endpoints]
+        WS_Out[WebSocket<br/>Subscriptions]
+        Metrics[Metrics]
+    end
+    
+    WS --> Listener
+    Poll --> Listener
+    Listener --> Validator
+    Validator --> Queue
+    Queue --> Tracker
+    Queue --> RBF
+    Queue --> Fee
+    Tracker --> Storage
+    RBF --> Storage
+    Fee --> Storage
+    Storage --> API
+    Storage --> WS_Out
+    Storage --> Metrics
+```
+
+## Transaction Ingestion
+
+### Sources
+
+**1. WebSocket Subscription**:
+- Subscribe to `pendingTransactions` via `eth_subscribe`
+- Real-time updates as transactions enter mempool
+
+**2. Polling**:
+- Poll `eth_pendingTransactions` every 1-2 seconds
+- Fallback if WebSocket unavailable
+
+**3. Transaction Submission**:
+- Track transactions submitted via our API
+- Link user submissions to mempool status
+
+### Transaction Validation
+
+**Validation Checks**:
+- Valid transaction format
+- Valid signature
+- Sufficient nonce (account state check)
+- Sufficient balance (for value transfers)
+- Valid gas price (above minimum)
+
+**Invalid Transactions**: Log but don't track (will be rejected by network)
+
+## Transaction Tracking
+
+### Transaction State
+
+**States**:
+- `pending`: In mempool
+- `confirmed`: Included in block
+- `dropped`: Removed from mempool (replaced or expired)
+- `replaced`: Replaced by higher gas price transaction (RBF)
+
+### Propagation Tracking
+
+**Metrics Tracked**:
+- First seen timestamp
+- Last seen timestamp
+- Propagation time (first seen → confirmed)
+- Propagation path (which nodes saw it first)
+
+**Method**:
+- Track when transaction first appears
+- Monitor mempool across multiple nodes
+- Calculate propagation statistics
+
+### RBF (Replace-by-Fee) Detection
+
+**Detection**:
+- Monitor transactions with same nonce
+- Detect higher gas price replacements
+- Link old transaction to new transaction
+
+**Data Stored**:
+- Original transaction hash
+- Replacement transaction hash
+- Gas price increase
+- Replacement timestamp
+
+### Bundle/MEV Visibility
+
+**Detection** (where supported):
+- Identify transaction bundles
+- Detect MEV patterns
+- Track front-running/back-running
+
+**Limitations**: Depends on chain/node capabilities
+
+### Private Transaction Markers
+
+**Detection**:
+- Identify private transaction services (Flashbots, etc.)
+- Mark transactions as private
+- Track private vs public mempool
+
+## Storage Schema
+
+See `../database/timeseries-schema.md` for detailed schema.
+
+**Key Fields**:
+- Transaction hash
+- From/to addresses
+- Value, gas price, gas limit
+- Nonce
+- First seen timestamp
+- Status
+- Confirmed block number (when confirmed)
+- Confirmed timestamp
+
+## Fee Estimation
+
+### Fee Calculation
+
+**Methods**:
+1. **Historical Analysis**: Analyze recent block fees
+2. **Percentile Method**: Calculate percentiles of recent transactions
+3. **Market-Based**: Track current mempool competition
+
+**Fee Estimates**:
+- Slow: 25th percentile
+- Standard: 50th percentile (median)
+- Fast: 75th percentile
+- Urgent: 95th percentile
+
+**Update Frequency**: Every block (real-time)
+
+## API Endpoints
+
+### Get Pending Transactions
+
+`GET /api/v1/mempool/{chain_id}/transactions`
+
+**Query Parameters**:
+- `from_address`: Filter by sender
+- `to_address`: Filter by recipient
+- `min_value`: Minimum value
+- `min_gas_price`: Minimum gas price
+- `limit`: Max results (default: 100)
+
+**Response**: Array of pending transactions
+
+### Get Transaction Status
+
+`GET /api/v1/mempool/{chain_id}/transactions/{hash}`
+
+**Response**: Transaction status and propagation info
+
+### Get Fee Estimates
+
+`GET /api/v1/mempool/{chain_id}/fees`
+
+**Response**:
+```json
+{
+  "slow": "20000000000",
+  "standard": "30000000000",
+  "fast": "50000000000",
+  "urgent": "100000000000"
+}
+```
+
+## WebSocket Subscriptions
+
+See `../api/websocket-api.md` for WebSocket API details.
+
+**Channels**:
+- `pending_transactions`: New pending transactions
+- `transaction_status`: Status updates for specific transactions
+- `fee_updates`: Fee estimate updates
+
+## Data Retention
+
+**Raw Data**: 7 days (detailed transaction data)
+**Aggregated Data**: 30 days (fee statistics, propagation metrics)
+**Archived Data**: Move to data lake after retention period
+
+## Performance Considerations
+
+### Throughput
+
+**Target**: Process 1000 transactions/second
+**Scalability**: Horizontal scaling with message queue
+
+### Latency
+
+**Target**: 
+- Ingestion latency: < 1 second
+- API response time: < 100ms (p95)
+
+### Storage Optimization
+
+**Strategy**:
+- Time-series database for efficient queries
+- Partition by time (daily partitions)
+- Automatic cleanup of old data
+
+## Monitoring
+
+### Metrics
+
+- Pending transaction count
+- Transaction ingestion rate
+- Confirmation rate
+- Average propagation time
+- Fee estimate accuracy
+
+### Alerts
+
+- High pending transaction count (> 10,000)
+- Low confirmation rate (< 50% within 5 minutes)
+- Fee estimate errors
+
+## References
+
+- Time-Series Schema: See `../database/timeseries-schema.md`
+- WebSocket API: See `../api/websocket-api.md`
+- Fee Oracle: See `fee-oracle.md`
+

docs/specs/mempool/realtime-events.md

@@ -0,0 +1,267 @@
+# Real-time Event System Specification
+
+## Overview
+
+This document specifies the event bus architecture for real-time event distribution across the explorer platform services.
+
+## Architecture
+
+**Technology**: Kafka or RabbitMQ
+
+**Recommendation**: Kafka for high throughput, RabbitMQ for simpler setup
+
+## Event Bus Architecture
+
+```mermaid
+flowchart LR
+    Producers[Event Producers]
+    Bus[Event Bus<br/>Kafka/RabbitMQ]
+    Consumers[Event Consumers]
+    
+    Producers --> Bus
+    Bus --> Consumers
+```
+
+## Event Types
+
+### Block Events
+
+**Event**: `block.new`
+
+**Payload**:
+```json
+{
+  "chain_id": 138,
+  "block_number": 12345,
+  "hash": "0x...",
+  "timestamp": "2024-01-01T00:00:00Z",
+  "transaction_count": 100
+}
+```
+
+**Producers**: Indexer
+**Consumers**: API Gateway (cache invalidation), WebSocket service, Analytics
+
+### Transaction Events
+
+**Event**: `transaction.confirmed`
+
+**Payload**:
+```json
+{
+  "chain_id": 138,
+  "hash": "0x...",
+  "block_number": 12345,
+  "from_address": "0x...",
+  "to_address": "0x...",
+  "status": "success"
+}
+```
+
+**Producers**: Indexer, Mempool service
+**Consumers**: WebSocket service, Search indexer, Analytics
+
+### Mempool Events
+
+**Event**: `transaction.pending`
+
+**Payload**: Transaction data
+
+**Producers**: Mempool service
+**Consumers**: WebSocket service, Fee oracle
+
+### Token Transfer Events
+
+**Event**: `token.transfer`
+
+**Payload**:
+```json
+{
+  "chain_id": 138,
+  "token_address": "0x...",
+  "from_address": "0x...",
+  "to_address": "0x...",
+  "amount": "1000000000000000000",
+  "transaction_hash": "0x..."
+}
+```
+
+**Producers**: Indexer
+**Consumers**: Token balance updater, Analytics, Notifications
+
+### Contract Verification Events
+
+**Event**: `contract.verified`
+
+**Payload**:
+```json
+{
+  "chain_id": 138,
+  "address": "0x...",
+  "verification_status": "verified",
+  "verified_at": "2024-01-01T00:00:00Z"
+}
+```
+
+**Producers**: Verification service
+**Consumers**: Search indexer, Cache invalidation
+
+## Event Schema
+
+### Standard Event Format
+
+```json
+{
+  "event_type": "block.new",
+  "event_id": "uuid",
+  "timestamp": "2024-01-01T00:00:00Z",
+  "chain_id": 138,
+  "payload": { ... },
+  "metadata": {
+    "producer": "indexer",
+    "version": "1.0"
+  }
+}
+```
+
+## Topic/Queue Structure
+
+### Kafka Topics
+
+**Naming**: `{event_type}.{chain_id}` (e.g., `block.new.138`)
+
+**Topics**:
+- `block.new.{chain_id}`
+- `transaction.confirmed.{chain_id}`
+- `transaction.pending.{chain_id}`
+- `token.transfer.{chain_id}`
+- `contract.verified.{chain_id}`
+
+**Partitioning**: By chain_id (all events for chain in same partition)
+
+### RabbitMQ Exchanges
+
+**Exchange Type**: Topic exchange
+
+**Exchange Name**: `explorer.events`
+
+**Routing Keys**: `{event_type}.{chain_id}` (e.g., `block.new.138`)
+
+## Consumer Groups and Partitioning
+
+### Consumer Groups
+
+**Purpose**: Enable parallel processing and load balancing
+
+**Groups**:
+- `websocket-service`: WebSocket real-time updates
+- `search-indexer`: Search index updates
+- `analytics`: Analytics aggregation
+- `cache-invalidation`: Cache invalidation
+
+### Partitioning Strategy
+
+**Kafka**:
+- Partition by chain_id
+- Same chain_id → same partition (maintains ordering)
+
+**RabbitMQ**:
+- Use consistent hash exchange for partitioning
+- Partition by chain_id
+
+## Delivery Guarantees
+
+### At-Least-Once Delivery
+
+**Default**: At-least-once delivery (events may be delivered multiple times)
+
+**Handling**:
+- Idempotent consumers
+- Deduplication by event_id
+- Track processed events
+
+### Exactly-Once Delivery
+
+**Use Case**: Critical financial events (banking layer)
+
+**Implementation**:
+- Kafka exactly-once semantics (if using Kafka)
+- Idempotent consumers with deduplication
+- Transactional processing
+
+## Backpressure Handling
+
+### Strategy
+
+**Flow Control**:
+- Consumer lag monitoring
+- Slow consumer detection
+- Automatic scaling
+
+**Handling Slow Consumers**:
+1. Scale consumers horizontally
+2. Increase consumer resources
+3. Alert on persistent lag
+
+### Lag Monitoring
+
+**Metrics**:
+- Consumer lag per topic
+- Processing rate
+- Consumer health
+
+**Alerts**: Lag > 1000 events or > 5 minutes
+
+## Event Ordering
+
+### Ordering Guarantees
+
+**Within Partition**: Events processed in order
+**Across Partitions**: No ordering guarantee
+
+**Use Cases Requiring Ordering**:
+- Block events (must process in order)
+- Transaction events per address (maintain order)
+
+**Implementation**: Use same partition for related events
+
+## Error Handling
+
+### Dead Letter Queue
+
+**Purpose**: Store failed events for retry or investigation
+
+**Strategy**:
+- Retry failed events (exponential backoff)
+- Move to DLQ after max retries
+- Alert on DLQ events
+
+### Retry Strategy
+
+**Configuration**:
+- Max retries: 3
+- Backoff: Exponential (1s, 2s, 4s)
+- Dead letter after max retries
+
+## Monitoring
+
+### Metrics
+
+- Event production rate
+- Event consumption rate
+- Consumer lag
+- Error rate
+- Dead letter queue size
+
+### Dashboards
+
+- Event throughput per topic
+- Consumer lag per consumer group
+- Error rates
+- System health
+
+## References
+
+- Mempool Service: See `mempool-service.md`
+- WebSocket API: See `../api/websocket-api.md`
+