proxmox/docs/06-besu/TRANSACTION_PERSISTENCE_INVESTIGATION.md

Transaction Persistence Investigation

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2025-01-20
Status: 🔍 Investigation In Progress
Issue: Transactions persist in blockchain state despite mempool flush

---

Problem Statement

After flushing mempools (restarting all RPC nodes and validators) and clearing transaction pool databases, transactions still persist, causing:

• "Known transaction" errors
• "Replacement transaction underpriced" errors
• Nonce stuck at 13104

---

Investigation Steps

Step 1: Transaction History Check

Purpose: Determine if transactions were actually mined (even if failed)

Method: Search last 50-200 blocks for transactions from deployer account

Expected Results:

• If transaction found in block → Transaction was mined (success or failure)
• If transaction not found → Transaction is in mempool/pending state

Step 2: Transaction Receipt Check

Purpose: Check if transaction with nonce 13104 exists in blockchain

Method: Search blocks for transactions with specific nonce

Expected Results:

• Receipt found → Transaction was included in block
• Receipt not found → Transaction never mined

Step 3: Transaction Pool Status

Purpose: Check current state of transaction pool

Methods:

• eth_pendingTransactions - Standard Ethereum RPC
• txpool_besuPendingTransactions - Besu-specific RPC
• txpool_content - Detailed pool content

Expected Results:

• Empty pool → Transactions not in mempool
• Non-empty pool → Transactions still pending

Step 4: Besu Logs Analysis

Purpose: Find transaction-related errors in Besu logs

Locations:

• /var/log/besu/
• /data/besu/logs/
• /opt/besu/logs/
• Systemd journal: journalctl -u besu-rpc.service

What to Look For:

• Transaction rejection reasons
• Nonce validation errors
• Gas price issues
• Transaction pool errors

Step 5: Account Transaction Count

Purpose: Compare pending vs latest transaction count

Method:

• eth_getTransactionCount(address, "latest") - Confirmed transactions
• eth_getTransactionCount(address, "pending") - Includes pending

Expected Results:

• Same count → No pending transactions
• Different count → Pending transactions exist

Step 6: Recent Block Search

Purpose: Find any transactions from deployer in recent blocks

Method: Search last 200 blocks for deployer transactions

Expected Results:

• Transactions found → Account is active, check nonces
• No transactions → Account may be stuck

Step 7: Besu Database Check

Purpose: Find transaction-related database files

Locations:

• /data/besu/ - Main data directory
• Transaction pool databases
• Block database files

What to Look For:

• Transaction pool database files
• Persistent transaction storage
• Database files that might contain transactions

Step 8: Broadcast Cache Analysis

Purpose: Check forge broadcast cache for transaction hashes

Method: Check broadcast/ directory for saved transaction hashes

Expected Results:

• Transaction hashes found → Check if they exist on-chain
• No hashes → No recent deployment attempts

---

Possible Root Causes

1. Transaction Already Mined (Failed)

Scenario: Transaction was included in a block but failed execution

Evidence:

• Transaction receipt exists with status: 0x0 (failed)
• Nonce was consumed by failed transaction
• Next transaction must use nonce + 1

Solution: Use next nonce (13105) for new transactions

2. Transaction in Validator Mempool

Scenario: Transaction is in a validator's mempool, not RPC node

Evidence:

• RPC node mempool appears empty
• Validators have separate mempools
• Transaction not visible via RPC

Solution: Restart validators (already done) or wait for expiry

3. Transaction in Blockchain State

Scenario: Transaction is part of blockchain state, not just mempool

Evidence:

• Transaction exists in block
• Nonce is part of blockchain state
• Cannot be cleared without database manipulation

Solution:

• If transaction failed → Use next nonce
• If transaction succeeded → Contract already deployed

4. Besu Transaction Pool Persistence

Scenario: Besu persists transactions in database, not just memory

Evidence:

• Transactions persist after restart
• Database files contain transaction data
• Clearing mempool doesn't clear database

Solution: Clear transaction pool database files (already attempted)

5. Nonce Mismatch

Scenario: Local nonce calculation doesn't match blockchain state

Evidence:

• eth_getTransactionCount returns different value
• Pending vs latest count differs
• Nonce calculation error

Solution: Use eth_getTransactionCount(address, "pending") for accurate nonce

6. Transaction in Multiple Nodes

Scenario: Transaction exists in multiple node mempools

Evidence:

• Transaction visible on some nodes but not others
• Network sync issues
• Transaction propagation delay

Solution: Wait for network sync or clear all nodes

---

Diagnostic Commands

Check Transaction History

# Search last 50 blocks for deployer transactions
cast block <block_number> --rpc-url $RPC | grep $DEPLOYER

# Check transaction receipt
cast receipt <tx_hash> --rpc-url $RPC

Check Transaction Pool

# Standard pending transactions
cast rpc eth_pendingTransactions --rpc-url $RPC

# Besu-specific pending transactions
cast rpc txpool_besuPendingTransactions --rpc-url $RPC

# Transaction pool content
cast rpc txpool_content --rpc-url $RPC

Check Account Nonce

# Latest (confirmed) nonce
cast nonce $DEPLOYER --rpc-url $RPC

# Pending nonce (includes pending transactions)
cast rpc eth_getTransactionCount $DEPLOYER pending --rpc-url $RPC

Check Besu Logs

# Systemd journal
journalctl -u besu-rpc.service --since "1 hour ago" | grep -i transaction

# Log files
grep -i "transaction\|nonce\|13104" /var/log/besu/*.log

Check Database Files

# Find transaction-related files
find /data/besu -name "*transaction*" -o -name "*pool*"

# Check database directories
ls -la /data/besu/

---

Expected Findings

Based on the investigation, we should determine:

1. Is transaction in blockchain?

   • Yes → Transaction was mined (check status)
   • No → Transaction is pending or in mempool

2. Is transaction in mempool?

   • Yes → Clear mempool or wait
   • No → Transaction may be in blockchain state

3. Is nonce correct?

   • Yes → Use next nonce
   • No → Fix nonce calculation

4. Are there pending transactions?

   • Yes → Wait or clear
   • No → Issue may be elsewhere

---

Next Steps Based on Findings

If Transaction Was Mined (Failed)

• Use nonce 13105 for next transaction
• Check why transaction failed
• Retry with corrected parameters

If Transaction Is Pending

• Wait for transaction to be mined or expire
• Clear transaction pool database
• Use higher gas price to replace

If Transaction Is in Blockchain State

• Check transaction receipt
• Verify contract deployment status
• Use next nonce if transaction failed

If Nonce Mismatch

• Use eth_getTransactionCount(address, "pending")
• Update nonce calculation
• Retry with correct nonce

---

Status: Investigation in progress
Next: Execute diagnostic commands and analyze results