proxmox/docs/06-besu/NETWORK_RECOVERED_BUT_TX_ISSUE.md

Network Recovered - Transaction Propagation Issue

Historical note: This incident note records a specific recovery window and the transaction issues observed at that time. Preserve it as troubleshooting history, not as the current network status.

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-24 01:35 PST
Status: ⚠️ PARTIAL SUCCESS - Blocks Producing, Transactions Not Confirming

---

✅ What Was Fixed

1. Found All Validators

Problem: Thought we only had 2 validators
Reality: All 5 validators exist and are running!

VMID	IP	Host	Status
1000	192.168.11.100	r630-01	✅ Running
1001	192.168.11.101	r630-01	✅ Running
1002	192.168.11.102	r630-01	✅ Running
1003	192.168.11.103	ml110	✅ Running
1004	192.168.11.104	ml110	✅ Running

2. Enabled Mining on All Validators

Problem: miner-enabled=false on all validators
Solution: Set miner-enabled=true on all 5 validators
Result: ✅ Block production resumed!

3. Network Statistics

• Current block: 1,301,181+ (and counting)
• Block time: ~2 seconds
• Consensus: QBFT with 5 validators
• Quorum: 4/5 required, all 5 active
• Peers: 14 connected

---

⚠️ Remaining Issue: Empty Blocks

Problem

• ✅ Blocks ARE producing (network recovered)
• ❌ Blocks are EMPTY (0 transactions)
• ❌ Transactions stuck in RPC mempool
• ❌ Validators not selecting transactions

Symptoms

{
  "number": "0x13dac9",  // Block 1,301,193
  "transactions": 0,       // EMPTY!
  "gasUsed": "0x0"        // No gas used
}

Root Cause Analysis

1. RPC has transactions in mempool (stuck at nonces 13104-13115)
2. Validators have correct TX-pool config (layered, with proper capacity)
3. Transactions not propagating from RPC to validators
4. Or validators not accepting transactions from mempool

---

🔍 Diagnostic Results

Validator Configuration ✅

# TX-Pool Configuration (CORRECT)
tx-pool-max-future-by-sender=200
tx-pool-layer-max-capacity=12500000
tx-pool-max-prioritized=2000

# Mining Configuration (CORRECT)
miner-enabled=true

# Sync Configuration (CORRECT)
sync-mode="FAST"

Network Topology ✅

• Static nodes: Configured correctly
• Permissioned nodes: All 5 validators listed
• Peer connections: 14 peers connected
• P2P network: Functioning

What's NOT Working ❌

• Transaction propagation RPC → Validators
• Transaction selection by validators
• Transaction confirmation

---

🎯 Possible Causes

1. RPC Not Propagating to Validators

• RPC nodes may not be connected to validators
• Transactions stay in RPC mempool only
• Need to verify RPC peering

2. Validator Transaction Acceptance

• Validators may have additional filters
• Min gas price mismatch
• Transaction pool limits

3. Permissions/ACL Issues

• accounts-allowlist might be blocking
• Transaction validation rules
• Smart contract permissions

---

🔧 Next Steps to Fix

Option 1: Send Transaction Directly to Validator

Instead of using RPC at 192.168.11.211, send directly to a validator:

# Try sending to validator 1003 at 192.168.11.103
export PRIVATE_KEY=0x5373d11ee2cad4ed82b9208526a8c358839cbfe325919fb250f062a25153d1c8
cast send 0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2 "deposit()" \
  --value 1000000000000000 \
  --private-key $PRIVATE_KEY \
  --rpc-url http://192.168.11.103:8545 \
  --gas-limit 50000 \
  --gas-price 1000000000 \
  --nonce 13116

Option 2: Enable RPC on Validators

Check if validators have RPC enabled:

ssh root@192.168.11.10 "pct exec 1003 -- grep 'rpc-http-enabled' /etc/besu/config-validator.toml"

If rpc-http-enabled=false, enable it and send transactions directly.

Option 3: Clear RPC Transaction Database

Force clear stuck transactions from RPC:

# Stop RPC nodes
# Clear transaction pool database
# Restart RPC nodes
# Resend transaction with next nonce

---

📊 Current Status Summary

Component	Status	Details
Validators	✅ Running	All 5 active
Mining	✅ Enabled	All validators
Block Production	✅ Working	~2s block time
Network Quorum	✅ Met	5/5 validators
TX-Pool Config	✅ Correct	Layered settings
Transaction Confirmation	❌ Not working	Empty blocks
Bridge Execution	❌ Blocked	Can't confirm txs

---

💡 Key Insights

What We Learned

1. All 5 validators exist - they're split across 2 Proxmox hosts
2. Mining was disabled - this was the main blocker
3. Network can recover - blocks producing again
4. Transaction propagation - separate issue from consensus

Why Bridge Still Can't Execute

• Network is operational ✅
• Bridge contracts deployed ✅
• Validators working ✅
• But transactions can't confirm ❌

---

🚀 Immediate Next Action

Try sending transaction directly to a validator node instead of through RPC:

cd /home/intlc/projects/proxmox
export PRIVATE_KEY=0x5373d11ee2cad4ed82b9208526a8c358839cbfe325919fb250f062a25153d1c8

# Try validator 1003 which we know is running
cast send 0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2 "deposit()" \
  --value 1000000000000000 \
  --private-key $PRIVATE_KEY \
  --rpc-url http://192.168.11.103:8545 \
  --gas-limit 50000 \
  --gas-price 1000000000 \
  --nonce 13116

If validator doesn't have RPC enabled, we need to:

1. Enable RPC on one validator temporarily
2. Or fix the RPC→Validator transaction propagation

---

Status: Network operational, investigating transaction confirmation issue
Progress: 90% complete - just need transactions to confirm
Blocker: Transaction propagation RPC→Validators

Last Updated: 2026-01-24 01:35 PST