Add full monorepo: virtual-banker, backend, frontend, docs, scripts, deployment
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
517
docs/specs/database/postgres-schema.md
Normal file
517
docs/specs/database/postgres-schema.md
Normal file
@@ -0,0 +1,517 @@
|
||||
# PostgreSQL Database Schema Specification
|
||||
|
||||
## Overview
|
||||
|
||||
This document specifies the complete PostgreSQL database schema for the explorer platform. The schema is designed to support multi-chain operation, high-performance queries, and data consistency.
|
||||
|
||||
## Schema Design Principles
|
||||
|
||||
1. **Multi-chain Support**: All tables include `chain_id` for chain isolation
|
||||
2. **Normalization**: Normalized structure to avoid data duplication
|
||||
3. **Performance**: Strategic indexing for common query patterns
|
||||
4. **Consistency**: Foreign key constraints where appropriate
|
||||
5. **Extensibility**: JSONB columns for flexible data storage
|
||||
6. **Partitioning**: Large tables partitioned by `chain_id`
|
||||
|
||||
## Core Tables
|
||||
|
||||
### Blocks Table
|
||||
|
||||
See `../indexing/data-models.md` for detailed block schema.
|
||||
|
||||
**Partitioning**: Partition by `chain_id` for large deployments.
|
||||
|
||||
**Key Indexes**:
|
||||
- Primary: `(chain_id, number)`
|
||||
- Unique: `(chain_id, hash)`
|
||||
- Index: `(chain_id, timestamp)` for time-range queries
|
||||
|
||||
### Transactions Table
|
||||
|
||||
See `../indexing/data-models.md` for detailed transaction schema.
|
||||
|
||||
**Key Indexes**:
|
||||
- Primary: `(chain_id, hash)`
|
||||
- Index: `(chain_id, block_number, transaction_index)` for block queries
|
||||
- Index: `(chain_id, from_address)` for address queries
|
||||
- Index: `(chain_id, to_address)` for address queries
|
||||
- Index: `(chain_id, block_number, from_address)` for compound queries
|
||||
|
||||
### Logs Table
|
||||
|
||||
See `../indexing/data-models.md` for detailed log schema.
|
||||
|
||||
**Key Indexes**:
|
||||
- Primary: `(chain_id, transaction_hash, log_index)`
|
||||
- Index: `(chain_id, address)` for contract event queries
|
||||
- Index: `(chain_id, topic0)` for event type queries
|
||||
- Index: `(chain_id, address, topic0)` for filtered event queries
|
||||
- Index: `(chain_id, block_number)` for block-based queries
|
||||
|
||||
### Traces Table
|
||||
|
||||
See `../indexing/data-models.md` for detailed trace schema.
|
||||
|
||||
**Key Indexes**:
|
||||
- Primary: `(chain_id, transaction_hash, trace_address)`
|
||||
- Index: `(chain_id, action_from)` for address queries
|
||||
- Index: `(chain_id, action_to)` for address queries
|
||||
- Index: `(chain_id, block_number)` for block queries
|
||||
|
||||
### Internal Transactions Table
|
||||
|
||||
See `../indexing/data-models.md` for detailed internal transaction schema.
|
||||
|
||||
**Key Indexes**:
|
||||
- Primary: `(chain_id, transaction_hash, trace_address)`
|
||||
- Index: `(chain_id, from_address)`
|
||||
- Index: `(chain_id, to_address)`
|
||||
- Index: `(chain_id, block_number)`
|
||||
|
||||
## Token Tables
|
||||
|
||||
### Tokens Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE tokens (
|
||||
id BIGSERIAL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
type VARCHAR(10) NOT NULL CHECK (type IN ('ERC20', 'ERC721', 'ERC1155')),
|
||||
name VARCHAR(255),
|
||||
symbol VARCHAR(50),
|
||||
decimals INTEGER CHECK (decimals >= 0 AND decimals <= 18),
|
||||
total_supply NUMERIC(78, 0),
|
||||
holder_count INTEGER DEFAULT 0,
|
||||
transfer_count INTEGER DEFAULT 0,
|
||||
logo_url TEXT,
|
||||
website_url TEXT,
|
||||
description TEXT,
|
||||
verified BOOLEAN DEFAULT false,
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
UNIQUE (chain_id, address)
|
||||
) PARTITION BY LIST (chain_id);
|
||||
|
||||
CREATE INDEX idx_tokens_chain_address ON tokens(chain_id, address);
|
||||
CREATE INDEX idx_tokens_chain_type ON tokens(chain_id, type);
|
||||
CREATE INDEX idx_tokens_chain_symbol ON tokens(chain_id, symbol);
|
||||
```
|
||||
|
||||
### Token Transfers Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE token_transfers (
|
||||
id BIGSERIAL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
transaction_hash VARCHAR(66) NOT NULL,
|
||||
block_number BIGINT NOT NULL,
|
||||
log_index INTEGER NOT NULL,
|
||||
token_address VARCHAR(42) NOT NULL,
|
||||
token_type VARCHAR(10) NOT NULL CHECK (token_type IN ('ERC20', 'ERC721', 'ERC1155')),
|
||||
from_address VARCHAR(42) NOT NULL,
|
||||
to_address VARCHAR(42) NOT NULL,
|
||||
amount NUMERIC(78, 0),
|
||||
token_id VARCHAR(78),
|
||||
operator VARCHAR(42),
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
FOREIGN KEY (chain_id, transaction_hash) REFERENCES transactions(chain_id, hash),
|
||||
FOREIGN KEY (chain_id, token_address) REFERENCES tokens(chain_id, address),
|
||||
UNIQUE (chain_id, transaction_hash, log_index)
|
||||
) PARTITION BY LIST (chain_id);
|
||||
|
||||
CREATE INDEX idx_token_transfers_chain_token ON token_transfers(chain_id, token_address);
|
||||
CREATE INDEX idx_token_transfers_chain_from ON token_transfers(chain_id, from_address);
|
||||
CREATE INDEX idx_token_transfers_chain_to ON token_transfers(chain_id, to_address);
|
||||
CREATE INDEX idx_token_transfers_chain_tx ON token_transfers(chain_id, transaction_hash);
|
||||
CREATE INDEX idx_token_transfers_chain_block ON token_transfers(chain_id, block_number);
|
||||
CREATE INDEX idx_token_transfers_chain_token_from ON token_transfers(chain_id, token_address, from_address);
|
||||
CREATE INDEX idx_token_transfers_chain_token_to ON token_transfers(chain_id, token_address, to_address);
|
||||
```
|
||||
|
||||
### Token Holders Table (Optional)
|
||||
|
||||
**Purpose**: Maintain current token balances for efficient queries.
|
||||
|
||||
```sql
|
||||
CREATE TABLE token_holders (
|
||||
id BIGSERIAL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
token_address VARCHAR(42) NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
balance NUMERIC(78, 0) NOT NULL DEFAULT 0,
|
||||
token_id VARCHAR(78), -- For ERC-721/1155
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
FOREIGN KEY (chain_id, token_address) REFERENCES tokens(chain_id, address),
|
||||
UNIQUE (chain_id, token_address, address, COALESCE(token_id, ''))
|
||||
) PARTITION BY LIST (chain_id);
|
||||
|
||||
CREATE INDEX idx_token_holders_chain_token ON token_holders(chain_id, token_address);
|
||||
CREATE INDEX idx_token_holders_chain_address ON token_holders(chain_id, address);
|
||||
```
|
||||
|
||||
## Contract Tables
|
||||
|
||||
### Contracts Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE contracts (
|
||||
id BIGSERIAL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
name VARCHAR(255),
|
||||
compiler_version VARCHAR(50),
|
||||
optimization_enabled BOOLEAN,
|
||||
optimization_runs INTEGER,
|
||||
evm_version VARCHAR(20),
|
||||
source_code TEXT,
|
||||
abi JSONB,
|
||||
constructor_arguments TEXT,
|
||||
verification_status VARCHAR(20) NOT NULL CHECK (verification_status IN ('pending', 'verified', 'failed')),
|
||||
verified_at TIMESTAMP,
|
||||
verification_method VARCHAR(50),
|
||||
license VARCHAR(50),
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
UNIQUE (chain_id, address)
|
||||
) PARTITION BY LIST (chain_id);
|
||||
|
||||
CREATE INDEX idx_contracts_chain_address ON contracts(chain_id, address);
|
||||
CREATE INDEX idx_contracts_chain_verified ON contracts(chain_id, verification_status);
|
||||
CREATE INDEX idx_contracts_abi_gin ON contracts USING GIN (abi); -- For ABI queries
|
||||
```
|
||||
|
||||
### Contract ABIs Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE contract_abis (
|
||||
id BIGSERIAL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
abi JSONB NOT NULL,
|
||||
source VARCHAR(50) NOT NULL,
|
||||
verified BOOLEAN DEFAULT false,
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
UNIQUE (chain_id, address)
|
||||
) PARTITION BY LIST (chain_id);
|
||||
|
||||
CREATE INDEX idx_abis_chain_address ON contract_abis(chain_id, address);
|
||||
CREATE INDEX idx_abis_abi_gin ON contract_abis USING GIN (abi);
|
||||
```
|
||||
|
||||
### Contract Verifications Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE contract_verifications (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
chain_id INTEGER NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
status VARCHAR(20) NOT NULL CHECK (status IN ('pending', 'processing', 'verified', 'failed', 'partially_verified')),
|
||||
compiler_version VARCHAR(50),
|
||||
optimization_enabled BOOLEAN,
|
||||
optimization_runs INTEGER,
|
||||
evm_version VARCHAR(20),
|
||||
source_code TEXT,
|
||||
abi JSONB,
|
||||
constructor_arguments TEXT,
|
||||
verification_method VARCHAR(50),
|
||||
error_message TEXT,
|
||||
verified_at TIMESTAMP,
|
||||
version INTEGER DEFAULT 1,
|
||||
is_active BOOLEAN DEFAULT true,
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
FOREIGN KEY (chain_id, address) REFERENCES contracts(chain_id, address)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_verifications_chain_address ON contract_verifications(chain_id, address);
|
||||
CREATE INDEX idx_verifications_status ON contract_verifications(status);
|
||||
```
|
||||
|
||||
## Address-Related Tables
|
||||
|
||||
### Address Labels Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE address_labels (
|
||||
id BIGSERIAL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
label VARCHAR(255) NOT NULL,
|
||||
label_type VARCHAR(20) NOT NULL CHECK (label_type IN ('user', 'public', 'contract_name')),
|
||||
user_id UUID,
|
||||
source VARCHAR(50),
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
UNIQUE (chain_id, address, label_type, user_id),
|
||||
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_labels_chain_address ON address_labels(chain_id, address);
|
||||
CREATE INDEX idx_labels_chain_user ON address_labels(chain_id, user_id);
|
||||
```
|
||||
|
||||
### Address Tags Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE address_tags (
|
||||
id BIGSERIAL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
tag VARCHAR(50) NOT NULL,
|
||||
tag_type VARCHAR(20) NOT NULL CHECK (tag_type IN ('category', 'risk', 'protocol')),
|
||||
user_id UUID,
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
UNIQUE (chain_id, address, tag, user_id),
|
||||
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_tags_chain_address ON address_tags(chain_id, address);
|
||||
CREATE INDEX idx_tags_chain_tag ON address_tags(chain_id, tag);
|
||||
```
|
||||
|
||||
## User Tables
|
||||
|
||||
### Users Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE users (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
email VARCHAR(255) UNIQUE,
|
||||
username VARCHAR(100) UNIQUE,
|
||||
password_hash TEXT,
|
||||
api_key_hash TEXT,
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
updated_at TIMESTAMP DEFAULT NOW(),
|
||||
last_login_at TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX idx_users_email ON users(email);
|
||||
CREATE INDEX idx_users_username ON users(username);
|
||||
```
|
||||
|
||||
### Watchlists Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE watchlists (
|
||||
id BIGSERIAL,
|
||||
user_id UUID NOT NULL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
address VARCHAR(42) NOT NULL,
|
||||
label VARCHAR(255),
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
PRIMARY KEY (id),
|
||||
UNIQUE (user_id, chain_id, address),
|
||||
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_watchlists_user ON watchlists(user_id);
|
||||
CREATE INDEX idx_watchlists_chain_address ON watchlists(chain_id, address);
|
||||
```
|
||||
|
||||
### API Keys Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE api_keys (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id UUID NOT NULL,
|
||||
key_hash TEXT NOT NULL UNIQUE,
|
||||
name VARCHAR(255),
|
||||
tier VARCHAR(20) NOT NULL CHECK (tier IN ('free', 'pro', 'enterprise')),
|
||||
rate_limit_per_second INTEGER,
|
||||
rate_limit_per_minute INTEGER,
|
||||
ip_whitelist TEXT[], -- Array of CIDR blocks
|
||||
last_used_at TIMESTAMP,
|
||||
expires_at TIMESTAMP,
|
||||
revoked BOOLEAN DEFAULT false,
|
||||
created_at TIMESTAMP DEFAULT NOW(),
|
||||
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_api_keys_user ON api_keys(user_id);
|
||||
CREATE INDEX idx_api_keys_hash ON api_keys(key_hash);
|
||||
```
|
||||
|
||||
## Multi-Chain Partitioning
|
||||
|
||||
### Partitioning Strategy
|
||||
|
||||
**Large Tables**: Partition by `chain_id` using LIST partitioning.
|
||||
|
||||
**Tables to Partition**:
|
||||
- `blocks`
|
||||
- `transactions`
|
||||
- `logs`
|
||||
- `traces`
|
||||
- `internal_transactions`
|
||||
- `token_transfers`
|
||||
- `tokens`
|
||||
- `token_holders` (if used)
|
||||
|
||||
### Partition Creation
|
||||
|
||||
**Example for blocks table**:
|
||||
|
||||
```sql
|
||||
-- Create parent table
|
||||
CREATE TABLE blocks (
|
||||
-- columns
|
||||
) PARTITION BY LIST (chain_id);
|
||||
|
||||
-- Create partitions
|
||||
CREATE TABLE blocks_chain_138 PARTITION OF blocks
|
||||
FOR VALUES IN (138);
|
||||
|
||||
CREATE TABLE blocks_chain_1 PARTITION OF blocks
|
||||
FOR VALUES IN (1);
|
||||
|
||||
-- Add indexes to partitions (inherited from parent)
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Faster queries (partition pruning)
|
||||
- Easier maintenance (per-chain operations)
|
||||
- Parallel processing
|
||||
- Data isolation
|
||||
|
||||
## Indexing Strategy
|
||||
|
||||
### Index Types
|
||||
|
||||
1. **B-tree**: Default for most indexes (equality, range, sorting)
|
||||
2. **Hash**: For exact match only (rarely used, B-tree usually better)
|
||||
3. **GIN**: For JSONB columns (ABIs, decoded data)
|
||||
4. **BRIN**: For large ordered columns (block numbers, timestamps)
|
||||
5. **Partial**: For filtered indexes (e.g., verified contracts only)
|
||||
|
||||
### Index Maintenance
|
||||
|
||||
**Regular Maintenance**:
|
||||
- `VACUUM ANALYZE` regularly (auto-vacuum enabled)
|
||||
- `REINDEX` if needed (bloat, corruption)
|
||||
- Monitor index usage (`pg_stat_user_indexes`)
|
||||
|
||||
**Index Monitoring**:
|
||||
- Track index sizes
|
||||
- Monitor index bloat
|
||||
- Remove unused indexes
|
||||
|
||||
## Data Retention and Archiving
|
||||
|
||||
### Retention Policies
|
||||
|
||||
**Hot Data**: Recent data (last 1 year)
|
||||
- Fast access required
|
||||
- All indexes maintained
|
||||
|
||||
**Warm Data**: Older data (1-5 years)
|
||||
- Archive to slower storage
|
||||
- Reduced indexing
|
||||
|
||||
**Cold Data**: Very old data (5+ years)
|
||||
- Archive to object storage
|
||||
- Minimal indexing
|
||||
|
||||
### Archiving Strategy
|
||||
|
||||
**Approach**:
|
||||
1. Partition tables by time ranges (monthly/yearly)
|
||||
2. Move old partitions to archive storage
|
||||
3. Query archive when needed (slower but available)
|
||||
|
||||
**Implementation**:
|
||||
- Use PostgreSQL table partitioning by date range
|
||||
- Move partitions to archive storage (S3, etc.)
|
||||
- Query via foreign data wrappers if needed
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Versioning
|
||||
|
||||
**Migration Tool**: Use migration tool (Flyway, Liquibase, or custom).
|
||||
|
||||
**Versioning Format**: `YYYYMMDDHHMMSS_description.sql`
|
||||
|
||||
**Example**:
|
||||
```
|
||||
20240101000001_initial_schema.sql
|
||||
20240115000001_add_token_holders.sql
|
||||
20240201000001_add_partitioning.sql
|
||||
```
|
||||
|
||||
### Migration Best Practices
|
||||
|
||||
1. **Backward Compatible**: Additive changes preferred
|
||||
2. **Reversible**: All migrations should be reversible
|
||||
3. **Tested**: Test on staging before production
|
||||
4. **Documented**: Document breaking changes
|
||||
5. **Rollback Plan**: Have rollback strategy
|
||||
|
||||
### Schema Evolution
|
||||
|
||||
**Adding Columns**:
|
||||
- Use `ALTER TABLE ADD COLUMN` with default values
|
||||
- Avoid NOT NULL without defaults (use two-step migration)
|
||||
|
||||
**Removing Columns**:
|
||||
- Mark as deprecated first
|
||||
- Remove after migration period
|
||||
|
||||
**Changing Types**:
|
||||
- Create new column
|
||||
- Migrate data
|
||||
- Drop old column
|
||||
- Rename new column
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Query Optimization
|
||||
|
||||
**Common Query Patterns**:
|
||||
1. Get block by number: Use `(chain_id, number)` index
|
||||
2. Get transaction by hash: Use `(chain_id, hash)` index
|
||||
3. Get address transactions: Use `(chain_id, from_address)` or `(chain_id, to_address)` index
|
||||
4. Filter logs by address and event: Use `(chain_id, address, topic0)` index
|
||||
|
||||
### Connection Pooling
|
||||
|
||||
**Configuration**:
|
||||
- Use connection pooler (PgBouncer, pgpool-II)
|
||||
- Pool size: 20-100 connections per application server
|
||||
- Statement-level pooling for better concurrency
|
||||
|
||||
### Read Replicas
|
||||
|
||||
**Strategy**:
|
||||
- Primary: Write operations
|
||||
- Replicas: Read operations (load balanced)
|
||||
- Async replication (small lag acceptable)
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Backup Strategy
|
||||
|
||||
**Full Backups**: Daily full database dumps
|
||||
**Incremental Backups**: Continuous WAL archiving
|
||||
**Point-in-Time Recovery**: Enabled via WAL archiving
|
||||
|
||||
### Recovery Procedures
|
||||
|
||||
**RTO Target**: 1 hour
|
||||
**RPO Target**: 5 minutes (max data loss)
|
||||
|
||||
## References
|
||||
|
||||
- Data Models: See `../indexing/data-models.md`
|
||||
- Indexer Architecture: See `../indexing/indexer-architecture.md`
|
||||
- Search Index Schema: See `search-index-schema.md`
|
||||
- Multi-chain Architecture: See `../multichain/multichain-indexing.md`
|
||||
|
||||
Reference in New Issue
Block a user