explorer-monorepo/docs/specs/indexing/reorg-handling.md

Reorg Handling Specification

Overview

This document specifies how the indexer handles blockchain reorganizations (reorgs), where the canonical chain changes and previously indexed blocks become invalid. Reorg handling ensures data consistency and maintains accurate blockchain state.

Reorg Detection

Detection Methods

1. Block Hash Mismatch

• Compare stored block hash with RPC node block hash
• If mismatch detected, reorg has occurred

2. Parent Hash Validation

• Verify each block's parent_hash matches previous block's hash
• Chain break indicates reorg point

3. Block Height Comparison

• Monitor chain head block number
• Sudden decrease indicates potential reorg

4. Chain Head Monitoring

• Poll eth_blockNumber periodically
• Compare with last indexed block number
• Detect rollback scenarios

Detection Strategy

Real-time Monitoring:

• Check block hash after each new block ingestion
• Compare with RPC node's block hash for same block number
• Immediate detection for recent blocks

Periodic Validation:

• Validate last N blocks (e.g., 100 blocks) every M seconds
• Detect deep reorgs that may have been missed
• Background validation job

Checkpoint Validation:

• Validate checkpoint blocks (every 1000 blocks)
• Ensure checkpoint blocks still exist in canonical chain
• Detect deep reorgs quickly

Reorg Handling Flow

flowchart TB
    Detect[Detect Reorg]
    Identify[Identify Reorg Point<br/>Find Common Ancestor]
    Mark[Mark Blocks as Orphaned]
    Delete[Delete Orphaned Data]
    Reindex[Re-index New Chain]
    Verify[Verify Data Consistency]
    
    Detect --> Identify
    Identify --> Mark
    Mark --> Delete
    Delete --> Reindex
    Reindex --> Verify
    Verify --> Done[Complete]
    
    Verify -->|Inconsistency| Delete

Step 1: Identify Reorg Point

Goal: Find the common ancestor block (last block that's still valid).

Algorithm:

1. Start from current chain head
2. Compare block hash with stored hash at each block number
3. When hash matches, that's the common ancestor
4. All blocks after common ancestor need reorg handling

Optimization:

• Binary search to find reorg point quickly
• Start from recent blocks and work backwards
• Cache last N block hashes for faster comparison

Step 2: Mark Blocks as Orphaned

Strategy: Mark blocks as orphaned before deletion (for audit trail).

Database Update:

UPDATE blocks 
SET orphaned = true, orphaned_at = NOW()
WHERE chain_id = ? 
  AND block_number > ?
  AND orphaned = false;

Benefits:

• Audit trail of reorgs
• Ability to query orphaned blocks
• Easier debugging

Step 3: Delete Orphaned Data

Cascade Deletion Order:

1. Token transfers (depends on transactions)
2. Logs (depends on transactions)
3. Traces (depends on transactions)
4. Internal transactions (depends on transactions)
5. Transactions (depends on blocks)
6. Blocks (orphaned blocks)

Implementation:

• Use database transactions for atomicity
• Cascade deletes via foreign keys (if enabled)
• Or explicit deletion in correct order

Performance Considerations:

• Batch deletions for large reorgs
• Index on block_number for efficient deletion
• Consider soft deletes (mark as deleted) vs hard deletes

Step 4: Re-index New Chain

Process:

1. Fetch new blocks from RPC node (from reorg point onward)
2. Process blocks through normal indexing pipeline
3. Verify each block before marking as indexed

Optimization:

• Parallel processing for multiple blocks (if safe)
• Use existing indexer workers
• Priority queue: reorg blocks before new blocks

Step 5: Verify Data Consistency

Validation Checks:

• Block hashes match RPC node
• Transaction counts match
• Parent hashes form valid chain
• No gaps in block numbers

Metrics:

• Reorg depth (number of blocks affected)
• Reorg duration (time to handle)
• Data consistency verification

Data Consistency Guarantees

Transaction Isolation

Strategy: Use database transactions to ensure atomic reorg handling.

Implementation:

BEGIN;
-- Mark blocks as orphaned
-- Delete orphaned data
-- Insert new blocks
-- Verify consistency
COMMIT;

Rollback: If any step fails, rollback entire operation.

Idempotency

Requirement: Reorg handling must be idempotent (safe to retry).

Mechanisms:

• Check block hash before processing
• Skip blocks already correctly indexed
• Use unique constraints to prevent duplicates

Finality Considerations

Reorg Depth Limits:

• Only handle reorgs within finality window
• For PoW: Typically 12-100 blocks deep
• For PoS: Typically 1-2 epochs (32-64 blocks)
• For BFT: Typically immediate finality

Configuration:

• Configurable reorg depth limit per chain
• Skip reorgs beyond finality window (log for investigation)

Performance Optimization

Handling Frequent Reorgs

Problem: Some chains have frequent small reorgs (1-2 blocks).

Solution:

• Batch reorg handling (wait for stability)
• Detect reorgs but delay handling for short period
• Only handle if reorg persists

Configuration:

• Reorg confirmation period: 30 seconds
• Maximum reorg depth to handle immediately: 5 blocks
• Deeper reorgs: Manual investigation

Handling Deep Reorgs

Problem: Deep reorgs require deleting and re-indexing many blocks.

Optimization Strategies:

1. Parallel Processing: Process new chain blocks in parallel
2. Batch Operations: Batch database operations
3. Incremental Updates: Only update changed data
4. Selective Deletion: Only delete affected data (not entire blocks if possible)

Index Maintenance

During Reorg:

• Pause index updates temporarily
• Resume after reorg handling complete
• Rebuild affected indexes if necessary

Monitoring and Alerting

Metrics

Reorg Metrics:

• Reorg count (per chain, per time period)
• Reorg depth distribution
• Reorg handling duration (p50, p95, p99)
• Data consistency check results

Alerting Rules:

• Reorg depth > 10 blocks: Warning (investigate)
• Reorg depth > 100 blocks: Critical (potential chain issue)
• Reorg handling duration > 5 minutes: Warning
• Data consistency check failure: Critical

Logging

Log Events:

• Reorg detection (block number, depth)
• Reorg point identification (common ancestor)
• Blocks orphaned (count, block numbers)
• Re-indexing progress
• Data consistency verification results

Log Levels:

• INFO: Normal reorgs (< 5 blocks)
• WARN: Unusual reorgs (5-10 blocks)
• ERROR: Deep reorgs (> 10 blocks) or failures

Edge Cases

Multiple Reorgs in Quick Succession

Scenario: Chain reorgs, then reorgs again before first reorg is handled.

Handling:

• Cancel in-progress reorg handling
• Start new reorg handling from latest state
• Ensure idempotency

Reorg During Backfill

Scenario: Historical block indexing encounters a reorg.

Handling:

• Pause backfill
• Handle reorg
• Resume backfill from reorg point

Concurrent Reorg Handling

Prevention:

• Use database locks to prevent concurrent reorg handling
• Single reorg handler per chain
• Queue reorg events if handler is busy

Recovery Procedures

Manual Reorg Handling

When to Use:

• Automatic handling fails
• Deep reorgs beyond normal limits
• Data corruption detected

Procedure:

1. Identify reorg point manually
2. Verify with RPC node
3. Mark blocks as orphaned
4. Delete orphaned data
5. Trigger re-indexing
6. Verify data consistency

Data Recovery

Backup Strategy:

• Regular database backups
• Point-in-time recovery capability
• Ability to restore to pre-reorg state

Recovery Steps:

1. Restore database to point before reorg
2. Re-run indexer from that point
3. Let automatic reorg handling process naturally

Testing Strategy

Unit Tests

• Reorg detection logic
• Common ancestor identification
• Orphan marking
• Data deletion logic

Integration Tests

• Simulate reorgs (testnet with known reorgs)
• Verify data consistency after reorg
• Test concurrent reorg handling
• Test deep reorgs

Load Tests

• Simulate frequent reorgs
• Measure performance impact
• Test reorg handling during high load

Configuration

Reorg Handling Configuration

reorg:
  detection:
    check_interval_seconds: 10
    validation_depth: 100
    checkpoint_interval: 1000
  
  handling:
    max_depth: 100
    confirmation_period_seconds: 30
    batch_size: 1000
    parallel_processing: true
  
  finality:
    chain_138:
      type: "poa" # or "pos", "pow", "bft"
      depth_limit: 50
      finality_blocks: 12

References

• Indexer Architecture: See indexer-architecture.md
• Data Models: See data-models.md
• Database Schema: See ../database/postgres-schema.md