d-bis/explorer-monorepo
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add full monorepo: virtual-banker, backend, frontend, docs, scripts, deployment

Browse Source 
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
2026-02-10 11:32:49 -08:00
parent aafcd913c2
commit 88bc76da91
815 changed files with 125522 additions and 264 deletions
Download Patch File Download Diff File Expand all files Collapse all files

402
docs/specs/infrastructure/network-topology.md Normal file
View File

@@ -0,0 +1,402 @@
# Network Topology Specification
## Overview
This document specifies the network architecture for the ChainID 138 Explorer+ platform, including internal network structure, external access patterns, security boundaries, and CDN/WAF integration.
## Network Architecture
```mermaid
flowchart TB
subgraph Internet[Internet]
Users[End Users]
Bots[Bots/Crawlers]
end
subgraph CDN[CDN Layer]
Cloudflare[Cloudflare CDN/WAF]
end
subgraph Edge[Edge Layer]
WAF[Web Application Firewall]
LB[Load Balancer]
end
subgraph DMZ[DMZ Network<br/>192.168.11.0/24]
API_GW[API Gateway]
Web_Server[Web Server]
end
subgraph Internal[Internal Network<br/>10.0.0.0/16]
Indexer[Indexer Service]
Explorer_API[Explorer API]
Mempool[Mempool Service]
Search[Search Service]
end
subgraph Blockchain[Blockchain Network<br/>192.168.11.250:8545]
Node1[ChainID 138 Node 1]
Node2[ChainID 138 Node 2]
end
subgraph Data[Data Layer<br/>Private Network]
DB[(PostgreSQL)]
ES[(Elasticsearch)]
Redis[(Redis Cache)]
end
Users --> Cloudflare
Bots --> Cloudflare
Cloudflare --> WAF
WAF --> LB
LB --> API_GW
LB --> Web_Server
API_GW --> Explorer_API
Web_Server --> Explorer_API
Explorer_API --> Indexer
Explorer_API --> Mempool
Explorer_API --> Search
Explorer_API --> Redis
Indexer --> Node1
Indexer --> Node2
Mempool --> Node1
Explorer_API --> DB
Indexer --> DB
Search --> ES
```
## Network Segments
### External Network (Internet)
**Description**: Public internet access points for end users and bots.
**Access Patterns**:
- HTTPS requests to `https://explorer.d-bis.org`
- API requests to `https://api.explorer.d-bis.org`
- WebSocket connections for real-time updates
**Security Requirements**:
- All traffic must pass through CDN/WAF
- DDoS protection enabled
- Rate limiting at edge
- SSL/TLS termination
### CDN Layer (Cloudflare)
**Description**: Content delivery network and web application firewall.
**Configuration**:
- **CDN**: Static asset caching, global distribution
- **WAF**: DDoS protection, bot mitigation, rate limiting
- **SSL/TLS**: End-to-end encryption, certificate management
- **DNS**: DNS resolution and routing
**Rules**:
- Cache static assets (HTML, CSS, JS, images) for 1 hour
- Cache API responses for 1 minute (configurable)
- Block known malicious IPs
- Rate limit: 100 requests/minute per IP
- Challenge suspicious traffic (CAPTCHA, JavaScript challenge)
**Integration Points**:
- Origin: `192.168.11.140:443` (Blockscout/Web server)
- Tunnel: Cloudflare Zero Trust tunnel for internal services
### Edge Layer (DMZ - 192.168.11.0/24)
**Description**: Demilitarized zone containing publicly accessible services.
**Subnet**: `192.168.11.0/24`
**Components**:
- **API Gateway** (192.168.11.141)
- Reverse proxy for API requests
- Authentication/authorization
- Rate limiting per API key
- Request routing
- **Web Server** (192.168.11.140)
- Serves frontend application
- Static file hosting
- Server-side rendering (if applicable)
**Security Rules**:
- No direct database access
- Communication with internal network via firewall rules only
- All outbound connections logged
- Intrusion detection system (IDS) monitoring
### Internal Network (10.0.0.0/16)
**Description**: Private network for backend services.
**Subnet**: `10.0.0.0/16`
**Service Allocation**:
- **10.0.1.0/24**: Explorer services (API, indexer, mempool)
- **10.0.2.0/24**: Search and analytics services
- **10.0.3.0/24**: Banking and compliance services
- **10.0.4.0/24**: VTM and conversational services
**Components**:
- Indexer Service (10.0.1.10)
- Explorer API (10.0.1.20)
- Mempool Service (10.0.1.30)
- Search Service (10.0.2.10)
- Analytics Service (10.0.2.20)
**Security Rules**:
- No internet access (outbound firewall)
- Communication with DMZ only via specific ports
- Internal service-to-service communication allowed
- All traffic encrypted (mTLS recommended)
### Blockchain Network (192.168.11.250:8545)
**Description**: Network segment containing blockchain nodes.
**Nodes**:
- **Primary Node**: `192.168.11.250:8545` (RPC)
- **Primary Node WS**: `192.168.11.250:8546` (WebSocket)
- **Backup Nodes**: Additional nodes for redundancy
**Access Rules**:
- Indexer service: Full access (archive, trace)
- Mempool service: Read access (pending tx), write access (submit tx)
- External services: Rate-limited public access via RPC endpoint
- Internal services: Direct access without rate limits
### Data Layer (Private Network)
**Description**: Network segment for data storage systems.
**Subnet**: `10.0.10.0/24` (isolated)
**Components**:
- **PostgreSQL** (10.0.10.10:5432)
- Primary database
- Read replicas for scaling
- **Elasticsearch** (10.0.10.20:9200)
- Search index
- Log aggregation
- **Redis** (10.0.10.30:6379)
- Cache layer
- Session storage
- Rate limiting counters
- **TimescaleDB** (10.0.10.40:5432)
- Time-series data
- Mempool metrics
- **Neo4j** (10.0.10.50:7687)
- Graph database
- Cross-chain entity graph
**Security Rules**:
- No internet access
- Access only from internal services
- Network encryption (TLS)
- Database authentication required
- Backup network isolation
## External Access Patterns
### Public Web Access
**Flow**:
1. User → Cloudflare CDN (DNS resolution)
2. Cloudflare → WAF (security checks)
3. WAF → Load Balancer (traffic distribution)
4. Load Balancer → Web Server (content delivery)
**Ports**:
- HTTP: 80 (redirects to HTTPS)
- HTTPS: 443
### Public API Access
**Flow**:
1. Client → Cloudflare CDN
2. Cloudflare → WAF (rate limiting, bot detection)
3. WAF → API Gateway (authentication, routing)
4. API Gateway → Explorer API (request processing)
**Endpoints**:
- `https://api.explorer.d-bis.org/v1/*` (REST API)
- `https://api.explorer.d-bis.org/graphql` (GraphQL API)
- `wss://api.explorer.d-bis.org/ws` (WebSocket API)
### Internal Service Access
**Flow**:
- Direct communication within internal network
- Service discovery via DNS or service mesh
- Load balancing via service mesh or internal LB
## Security Boundaries
### Firewall Rules
**DMZ → Internal**:
- Allow: HTTP/HTTPS to API services (80, 443)
- Allow: Specific service ports (indexer, mempool)
- Deny: Database ports (5432, 9200, 6379)
- Deny: All other ports
**Internal → Data Layer**:
- Allow: Database connections from specific services
- Allow: Specific ports only (5432, 9200, 6379, 7687)
- Deny: All other access
**Internal → Blockchain Network**:
- Allow: RPC connections (8545, 8546)
- Deny: All other ports
**Internet → DMZ**:
- Allow: HTTP (80) → HTTPS redirect
- Allow: HTTPS (443) → Web/API servers
- Deny: All other ports
**Internal → Internet**:
- Deny: All outbound (except specific whitelisted services)
- Allow: Package updates (via proxy)
- Allow: External API calls (via gateway)
### Network Segmentation Benefits
1. **Isolation**: Compromise in one segment doesn't affect others
2. **Access Control**: Granular firewall rules
3. **Monitoring**: Network traffic analysis per segment
4. **Compliance**: Easier to demonstrate security controls
## CDN/WAF Integration
### Cloudflare Configuration
**DNS Records**:
- `explorer.d-bis.org` → Cloudflare proxy (orange cloud)
- `api.explorer.d-bis.org` → Cloudflare proxy
- Tunnel target: `192.168.11.140:80` (Web) or `192.168.11.141:443` (API)
**WAF Rules**:
1. **DDoS Protection**: Automatic mitigation
2. **Rate Limiting**:
- 100 requests/minute per IP (general)
- 10 requests/second per IP (API)
- 1000 requests/minute per API key
3. **Bot Management**: Challenge suspicious bots
4. **Geographic Blocking**: Optional country restrictions
5. **IP Access Rules**: Whitelist/blacklist specific IPs
**Cache Rules**:
- Static assets: Cache for 1 hour
- HTML pages: Cache for 5 minutes
- API responses: No cache (dynamic data)
- GraphQL: No cache
### Cloudflare Tunnel (Zero Trust)
**Purpose**: Secure access to internal services without exposing ports.
**Configuration**:
- Tunnel endpoint: Internal load balancer
- Routes:
- `explorer.d-bis.org/*` → `http://192.168.11.140:80`
- `api.explorer.d-bis.org/*` → `http://192.168.11.141:443`
**Benefits**:
- No open ports to internet
- Encrypted connections
- Access control via Zero Trust policies
- Automatic failover
## Monitoring and Observability
### Network Monitoring
**Metrics to Track**:
- Bandwidth utilization per segment
- Connection counts
- Latency between segments
- Packet loss
- Firewall rule hit counts
**Tools**:
- Netflow/sFlow analysis
- Network packet capture (tcpdump, Wireshark)
- Network performance monitoring (PRTG, Zabbix)
### Security Monitoring
**Events to Monitor**:
- Firewall rule violations
- Unusual traffic patterns
- DDoS attack attempts
- Port scan attempts
- Failed authentication attempts
**Tools**:
- Intrusion Detection System (IDS)
- Security Information and Event Management (SIEM)
- Cloudflare security logs
- Firewall logs aggregation
## Implementation Guidelines
### Network Configuration
**IP Address Allocation**:
- Use CIDR notation for subnets
- Reserve IP ranges for future expansion
- Document all IP assignments
**DNS Configuration**:
- Internal DNS for service discovery
- External DNS via Cloudflare
- Service names: `service-name.internal.explorer`
**Load Balancing**:
- Use HAProxy or NGINX for internal LB
- Health checks for backend services
- Session affinity where needed
### Security Hardening
1. **Disable unused services**: Close unnecessary ports
2. **Use strong encryption**: TLS 1.2+ for all connections
3. **Implement network segmentation**: Isolate sensitive systems
4. **Regular security audits**: Review firewall rules
5. **Monitor network traffic**: Detect anomalies
## Migration from Current Setup
### Current Network Configuration
- **Blockscout Container**: `192.168.11.140:4000`
- **RPC Endpoint**: `192.168.11.250:8545`
- **Cloudflare Tunnel**: Configured for `explorer.d-bis.org`
### Migration Steps
1. Deploy new network segments
2. Configure firewall rules
3. Deploy services to appropriate segments
4. Update DNS and routing
5. Test connectivity
6. Monitor for issues
7. Gradually migrate traffic
## References
- Node & RPC Architecture: See `node-rpc-architecture.md`
- API Gateway: See `../api/api-gateway.md`
- Security Architecture: See `../security/security-architecture.md`
- Existing Blockscout setup: `../../docs/BLOCKSCOUT_COMPLETE_SUMMARY.md`

387
docs/specs/infrastructure/node-rpc-architecture.md Normal file
View File

@@ -0,0 +1,387 @@
# Node & RPC Architecture Specification
## Overview
This document specifies the architecture for ChainID 138 blockchain nodes and RPC endpoints that power the explorer platform. The architecture ensures high availability, performance, and reliability for block data retrieval and transaction submission.
## Architecture
```mermaid
flowchart TB
subgraph Nodes[ChainID 138 Nodes]
AN[Archive Node<br/>Full History]
TN[Tracing Node<br/>Call Traces]
VN[Validator Node<br/>Block Production]
end
subgraph LoadBalancer[Load Balancer Layer]
LB[NGINX/HAProxy<br/>SSL Termination]
end
subgraph RPCPool[RPC Endpoint Pool]
RPC1[RPC Endpoint 1<br/>Archive + Trace]
RPC2[RPC Endpoint 2<br/>Archive + Trace]
RPC3[RPC Endpoint 3<br/>Archive Only]
end
subgraph Services[Explorer Services]
Indexer[Indexer Service]
API[API Gateway]
Mempool[Mempool Service]
end
AN --> RPC1
TN --> RPC1
AN --> RPC2
TN --> RPC2
AN --> RPC3
LB --> RPC1
LB --> RPC2
LB --> RPC3
Indexer --> LB
API --> LB
Mempool --> LB
```
## Node Setup Requirements
### Archive Node Configuration
**Purpose**: Store complete blockchain history for historical queries and data indexing.
**Requirements**:
- **Archive mode**: Full historical state access (`--sync-mode=full --gcmode=archive`)
- **Storage**: High-capacity storage with SSD for recent blocks, HDD for cold storage
- **Memory**: Minimum 32GB RAM for efficient state access
- **Database**: Fast state database (LevelDB, RocksDB)
- **Network**: High-bandwidth connection for syncing
**Configuration Example** (Hyperledger Besu):
```toml
data-path="./data"
genesis-file="./genesis.json"
network-id=138
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","WEB3","ADMIN","DEBUG","TRACE","EEA","PRIV"]
rpc-http-cors-origins=["*"]
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
rpc-ws-api=["ETH","NET","WEB3"]
sync-mode="FAST"
data-storage-format="BONSAI"
```
### Tracing Node Configuration
**Purpose**: Provide call trace capabilities for transaction debugging and internal transaction tracking.
**Requirements**:
- Archive mode enabled for historical trace access
- Debug and trace API endpoints enabled
- Additional CPU for trace computation
- Trace storage strategy (on-demand vs cached)
**Trace Types Supported**:
- `trace_call`: Single call trace
- `trace_block`: All traces in a block
- `trace_transaction`: All traces in a transaction
- `trace_replayBlockTransactions`: Replay with state
- `trace_replayTransaction`: Replay single transaction
### Node Health Monitoring
**Health Check Endpoints**:
- `eth_blockNumber`: Returns latest block number
- `eth_syncing`: Returns sync status
- `net_peerCount`: Returns connected peer count
**Health Check Criteria**:
- Latest block is within 5 blocks of network head
- Peer count > 5 (configurable)
- Response time < 500ms for simple queries
- No critical errors in logs
**Monitoring Metrics**:
- Block height lag vs network
- Peer connection count
- RPC request latency (p50, p95, p99)
- Error rate by endpoint
- CPU and memory usage
- Disk I/O and storage usage
## RPC Endpoint Architecture
### Load Balancing Strategy
**Distribution Method**: Round-robin with health-aware routing
**Failover Strategy**:
- Primary: Archive + Trace capable nodes
- Secondary: Archive-only nodes (fallback for non-trace requests)
- Automatic failover on health check failure
- Circuit breaker pattern to prevent cascading failures
**Session Affinity**: Not required for stateless RPC requests
### RPC Capabilities Matrix
| Capability | Archive Node | Tracing Node | Light Node |
|------------|--------------|--------------|------------|
| Historical blocks | ✅ Full | ✅ Full | ❌ Limited |
| Current state | ✅ Full | ✅ Full | ✅ Full |
| Call traces | ❌ No | ✅ Yes | ❌ No |
| Archive state queries | ✅ Yes | ✅ Yes | ❌ No |
| Transaction submission | ✅ Yes | ✅ Yes | ✅ Yes |
### Endpoint Types
#### Public RPC Endpoint
- **Internal URL**: `http://192.168.11.221:8545` (VMID 2201 - direct connection, recommended for internal services)
- **Public URL**: `https://rpc-http-pub.d-bis.org` (via proxy - for external access)
- **Authentication**: Optional API keys for rate limiting
- **Rate Limits**:
- Unauthenticated: 10 req/s
- Authenticated: 100 req/s (tiered)
- **CORS**: Enabled for web applications
#### Private RPC Endpoint
- **URL**: `https://rpc-http-prv.d-bis.org`
- **Authentication**: Required (authorized access)
- **Rate Limits**: Higher limits for authenticated users
- **CORS**: Enabled for authorized domains
#### Internal RPC Endpoint
- **URL**: `http://192.168.11.211:8545` (internal network - VMID 2101)
- **Authentication**: Network-based (firewall rules)
- **Rate Limits**: Higher limits for internal services
- **CORS**: Disabled
### WebSocket Endpoint
**Configuration**:
- **Public WS (Internal)**: `ws://192.168.11.221:8546` (VMID 2201 - direct connection, recommended for internal services)
- **Public WS (Public)**: `wss://rpc-ws-pub.d-bis.org` (via proxy - for external access)
- **Private WS (Internal)**: `ws://192.168.11.211:8546` (VMID 2101 - direct connection)
- **Private WS (Public)**: `wss://rpc-ws-prv.d-bis.org` (via proxy - for external access)
- **Subscriptions**: New blocks, pending transactions, logs
- **Connection Limits**: 100 concurrent connections per IP
- **Heartbeat**: 30-second ping/pong
**Subscription Types**:
- `newHeads`: New block headers
- `logs`: Event logs matching filter
- `pendingTransactions`: Pending transactions
- `syncing`: Sync status updates
## Rate Limiting and Access Control
### Rate Limiting Strategy
**Tiers**:
1. **Anonymous**: 10 requests/second, 100 requests/minute
2. **API Key (Free)**: 100 requests/second, 1000 requests/minute
3. **API Key (Pro)**: 500 requests/second, 5000 requests/minute
4. **Internal Services**: 1000 requests/second, unlimited
**Implementation**:
- Token bucket algorithm
- Per-IP and per-API-key limits
- Separate limits for different endpoint categories:
- Simple queries (eth_blockNumber, eth_getBalance): Higher limits
- Complex queries (trace_block, eth_call with large state): Lower limits
- Write operations (eth_sendTransaction): Strict limits
### Access Control
**API Key Management**:
- Key generation with secure random tokens
- Key metadata: name, tier, creation date, last used
- Key rotation support
- Revocation capability
**IP Whitelisting**:
- Support for IP whitelisting per API key
- CIDR notation support
- Geographic restrictions (optional)
**Rate Limit Headers**:
```
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1640995200
```
## Failover Strategies
### Automatic Failover
**Detection**:
- Health checks every 10 seconds
- Consecutive failures threshold: 3
- Recovery threshold: 2 successful checks
**Failover Actions**:
1. Mark node as unhealthy
2. Route traffic to healthy nodes
3. Continue health checking for recovery
4. Log failover events
### Manual Failover
**Scenarios**:
- Planned maintenance
- Node performance degradation
- Security incidents
**Procedure**:
1. Drain connections (wait for active requests)
2. Remove from load balancer pool
3. Perform maintenance
4. Verify health
5. Re-add to pool
## Performance Requirements
### Latency Targets
| Operation Type | p50 Target | p95 Target | p99 Target |
|----------------|------------|------------|------------|
| Simple queries (blockNumber, balance) | < 50ms | < 100ms | < 200ms |
| Block queries | < 100ms | < 200ms | < 500ms |
| Transaction queries | < 150ms | < 300ms | < 1000ms |
| Trace queries | < 500ms | < 2000ms | < 5000ms |
| Historical state queries | < 1000ms | < 5000ms | < 10000ms |
### Throughput Targets
- **Read operations**: 1000 req/s per node
- **Write operations**: 100 req/s per node
- **WebSocket subscriptions**: 1000 concurrent per node
### Availability Targets
- **Uptime**: 99.9% (8.76 hours downtime/year)
- **Failover time**: < 30 seconds
- **Data consistency**: Zero data loss
## Security Considerations
### Network Security
- **TLS/SSL**: Required for public endpoints (TLS 1.2+)
- **Firewall Rules**: Restrict access to internal endpoints
- **DDoS Protection**: WAF/CDN integration for public endpoints
- **IP Filtering**: Support for IP allowlists/blocklists
### Authentication
- **API Keys**: HMAC-based authentication
- **Key Storage**: Encrypted storage with KMS
- **Key Rotation**: Support for periodic rotation
### Audit Logging
- All RPC requests logged (with PII sanitization)
- Log retention: 90 days
- Log fields: timestamp, IP, API key (hash), endpoint, response code, latency
## Integration Points
### Indexer Service Integration
- Primary connection to archive nodes
- Batch requests for efficiency
- Connection pooling
- Retry logic with exponential backoff
### API Gateway Integration
- Proxy requests to RPC pool
- Add authentication/authorization
- Apply rate limiting
- Cache common queries
### Mempool Service Integration
- WebSocket subscription to pending transactions
- Direct connection for transaction submission
- Priority queuing for indexer submissions
## Implementation Guidelines
### Node Deployment
**Infrastructure**:
- Use container orchestration (Kubernetes) for scalability
- Separate deployments for archive and tracing nodes
- Horizontal scaling based on load
**Configuration Management**:
- Use configuration files (TOML/YAML)
- Environment-specific overrides
- Secrets management via KMS/Vault
### Monitoring Integration
**Metrics to Export**:
- Prometheus metrics for node health
- Custom metrics for RPC performance
- Integration with Grafana dashboards
**Alerting Rules**:
- Node down alert
- High latency alert
- High error rate alert
- Low peer count alert
## Testing Strategy
### Unit Tests
- RPC endpoint implementations
- Rate limiting logic
- Health check logic
### Integration Tests
- Load balancer failover
- Rate limiting enforcement
- Authentication/authorization
### Load Tests
- Simulate production load
- Test failover scenarios
- Validate performance targets
## Migration from Existing Setup
### Current Setup (Blockscout Integration)
- **Public RPC (Internal)**: `http://192.168.11.221:8545` (VMID 2201 - recommended for internal services)
- **Public RPC (Public)**: `https://rpc-http-pub.d-bis.org` (via proxy - for external access)
- **Private RPC (Internal)**: `http://192.168.11.211:8545` (VMID 2101)
- **Private RPC (Public)**: `https://rpc-http-prv.d-bis.org` (via proxy)
- **Deprecated**: `https://rpc-core.d-bis.org` (no longer public, internal only)
- **Internal RPC**: `http://192.168.11.250:8545`
- **WebSocket**: `ws://192.168.11.250:8546`
### Migration Steps
1. Deploy new load balancer layer
2. Configure health checks
3. Gradually migrate traffic
4. Monitor performance
5. Decommission old direct connections
## References
- Existing Blockscout deployment: `docs/BLOCKSCOUT_COMPLETE_SUMMARY.md`
- Network architecture: See `network-topology.md`
- API Gateway: See `../api/api-gateway.md`