proxmox/docs/06-besu/RPC_REVIEW_COMPREHENSIVE_FINDINGS.md

Comprehensive RPC Review for Hyperledger Besu Permissioned Blockchain

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2025-01-20
Purpose: Full review of RPC configuration for Foundry deployment
Issues: Connection timeout and "Known transaction" errors

---

Executive Summary

Issues

1. Connection Timeout: RPC connection timeouts during deployment
2. "Known Transaction" Error: Transactions already exist in validator transaction pools

Current Status

• Pending Transactions: 3 (nonces 13105-13107)
• Latest Nonce: 13104 (confirmed)
• Pending Nonce: 13107 (3 transactions ahead)
• Network: ✅ Operational
• Block Production: ✅ Active

---

1. RPC Endpoint Analysis

Configuration

• URL: http://192.168.11.211:8545
• Type: Core/Admin RPC node
• Configuration File: config-rpc-core.toml
• Network: Internal (192.168.11.0/24)

Connectivity Status

✅ Ping Test: Success (0% packet loss, avg 1.5ms)
✅ Port Test: Port 8545 open and accessible
✅ RPC Response: Working (~150ms response time)
✅ Chain ID: Correct (138)

Network Health

• Status: ✅ Healthy
• Latency: Low (~150ms)
• Reliability: Good (0% packet loss)

---

2. Besu RPC Configuration

RPC APIs Enabled

rpc-http-api=["ETH","NET","WEB3","TXPOOL","QBFT","ADMIN","DEBUG","TRACE"]

✅ Status: All required APIs enabled

Transaction Pool API

• TXPOOL API: ✅ Enabled
• Note: Some txpool methods may require specific RPC call format
• Status: Available but may have implementation differences

Configuration Issues

⚠️ Missing RPC Timeout Configuration

# Current: Not explicitly set
# Default: 60 seconds
# Recommended: 120 seconds for large deployments
rpc-http-timeout=120

⚠️ Transaction Pool Not Configured

# Current: Empty section (using Besu defaults)
# Transaction Pool

# Recommended: Add explicit limits
tx-pool-max-size=8192
tx-pool-limit-by-account-percentage=0.5

---

3. Transaction Pool Analysis

Current State

• Latest Nonce: 13104 (confirmed in blockchain)
• Pending Nonce: 13107 (3 transactions ahead)
• Pending Count: 3 transactions
• Status: Transactions stuck in validator transaction pools

"Known Transaction" Error

Root Cause

1. Previous deployments created transactions at nonces 13105-13107
2. Transactions were accepted but not included in blocks
3. Transactions remain in validator pools (not just RPC node)
4. Validators reject duplicate transactions with same hash
5. Replacement requires higher gas price (which we attempted)

Why They Persist

• Besu transaction retention: Validators maintain transaction pools
• Network sync: Transactions propagate between nodes
• Pool database: Transactions stored persistently
• Not in blocks: Transactions never confirmed or rejected

Transaction Pool Clearing Challenges

• ✅ Can restart RPC nodes (clears RPC pool)
• ⚠️ Validator pools persist (need validator restarts)
• ⚠️ Network sync re-adds transactions after restart
• ⚠️ Database persistence may retain transaction state

---

4. Connection Timeout Issues

Observed Behavior

• Error: Connection timed out (os error 110)
• Frequency: Intermittent
• Context: During cast send deployment

Possible Causes

1. RPC Timeout Too Short

• Current: Default 60 seconds
• Issue: Large contract deployments may exceed timeout
• Solution: Increase rpc-http-timeout to 120 seconds

2. Network Latency

• Observed: ~150ms response time
• Issue: Multiple RPC calls may accumulate delay
• Solution: Not critical, but monitor for degradation

3. RPC Node Load

• Issue: Node under heavy load
• Solution: Monitor node performance, add timeout

4. Transaction Processing Delay

• Issue: Besu processing large transactions slowly
• Solution: Increase timeout, optimize transaction size

---

5. Permission Configuration

Account Permissioning

• Status: ✅ Disabled (permissions-accounts-config-file-enabled=false)
• Impact: All accounts allowed (not blocking deployments)
• Recommendation: Keep disabled for deployment, enable for production

Node Permissioning

• Status: ✅ Enabled (permissions-nodes-config-file-enabled=true)
• Impact: Only allowlisted nodes can connect (normal for permissioned network)
• Recommendation: Verify deployment node is allowlisted

---

6. Besu Permissioned Network Specifics

Key Characteristics

1. QBFT Consensus: Requires validator quorum for blocks
2. Permissioned Nodes: Only allowlisted nodes participate
3. Account Permissioning: Can restrict transaction sources
4. Transaction Pool: Shared across permissioned network

Transaction Handling Differences

• Immediate propagation: Transactions propagate to all validators quickly
• Validation rules: More strict validation than public networks
• Transaction retention: Transactions may persist longer
• Pool synchronization: All validators must agree on pool state

---

7. Recommendations

Immediate Actions

1. Configure RPC Timeout

# Add to config-rpc-core.toml
rpc-http-timeout=120  # Increase from default 60s

2. Check Transaction Pool Status

# Verify transactions in pool
cast rpc eth_getTransactionCount <DEPLOYER> pending \
  --rpc-url http://192.168.11.211:8545

# Check latest confirmed
cast rpc eth_getTransactionCount <DEPLOYER> latest \
  --rpc-url http://192.168.11.211:8545

3. Clear Transaction Pools

# Option A: Restart all Besu nodes (RPC + Validators)
# Option B: Use admin API to clear pool (if available)
# Option C: Use next nonce (skip to 13108+)

4. Use Higher Gas Price for Replacements

# Use significantly higher gas price (3-5 gwei)
cast send ... --gas-price 5000000000  # 5 gwei

Configuration Changes

1. Add Transaction Pool Limits

# config-rpc-core.toml
tx-pool-max-size=8192
tx-pool-limit-by-account-percentage=0.5
tx-pool-price-bump=10  # Minimum gas price increase for replacement (%)

2. Increase RPC Timeout

# config-rpc-core.toml
rpc-http-timeout=120

3. Enable Transaction Pool Monitoring

# Already enabled
rpc-http-api=["ETH","NET","WEB3","TXPOOL","QBFT","ADMIN","DEBUG","TRACE"]

Deployment Strategy

Option 1: Clear and Redeploy (Recommended)

1. Stop all Besu nodes (RPC + Validators)
2. Clear transaction pool databases
3. Restart nodes (validators first, then RPC)
4. Wait for network stabilization
5. Redeploy with correct gas price

Option 2: Skip Nonce

1. Use nonce 13108 (skip pending transactions)
2. Deploy with explicit gas price (2 gwei)
3. Accept nonce gap (if pending transactions eventually confirm)

Option 3: Replace with Higher Gas Price

1. Use gas price 3-5 gwei
2. Replace transactions at nonces 13105-13107
3. Monitor for confirmation

---

8. Foundry Deployment Best Practices

Timeout Configuration

# foundry.toml
[rpc_endpoints]
chain138 = "http://192.168.11.211:8545"

[rpc_endpoints.chain138]
timeout = 30000  # 30 seconds

Transaction Retry Logic

# Implement retry with exponential backoff
MAX_RETRIES=3
RETRY_DELAY=5

for i in {1..$MAX_RETRIES}; do
  cast send ... --gas-price 2000000000
  if [ $? -eq 0 ]; then
    break
  fi
  sleep $RETRY_DELAY
  RETRY_DELAY=$((RETRY_DELAY * 2))
done

Transaction Status Check

# Check before retrying
TX_COUNT=$(cast rpc eth_getTransactionCount $DEPLOYER pending \
  --rpc-url http://192.168.11.211:8545)
LATEST_COUNT=$(cast rpc eth_getTransactionCount $DEPLOYER latest \
  --rpc-url http://192.168.11.211:8545)

if [ "$TX_COUNT" -gt "$LATEST_COUNT" ]; then
  echo "Pending transactions exist - clear or use higher gas price"
fi

---

9. Diagnostic Commands

Check RPC Connectivity

# Basic connectivity
cast chain-id --rpc-url http://192.168.11.211:8545

# Response time
time curl -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' \
  http://192.168.11.211:8545

Check Transaction Status

# Latest nonce (confirmed)
cast rpc eth_getTransactionCount <DEPLOYER> latest \
  --rpc-url http://192.168.11.211:8545

# Pending nonce (includes pending)
cast rpc eth_getTransactionCount <DEPLOYER> pending \
  --rpc-url http://192.168.11.211:8545

Check Block Production

# Current block
cast block-number --rpc-url http://192.168.11.211:8545

# Verify blocks advancing
watch -n 1 'cast block-number --rpc-url http://192.168.11.211:8545'

---

10. Action Items

High Priority (Immediate)

1. ✅ RPC Review Complete (THIS DOCUMENT)
2. ⏳ Increase RPC timeout in config-rpc-core.toml
3. ⏳ Clear transaction pools (restart all Besu nodes)
4. ⏳ Add transaction pool configuration to config

Medium Priority (Short-term)

1. ⏳ Implement retry logic in deployment scripts
2. ⏳ Add transaction status checking before deployments
3. ⏳ Monitor transaction pool for stuck transactions
4. ⏳ Optimize gas price calculation

Low Priority (Long-term)

1. ⏳ Transaction pool monitoring dashboard
2. ⏳ Automated pool clearing for stuck transactions
3. ⏳ Transaction deduplication improvements
4. ⏳ Deployment health checks

---

11. Conclusion

Current Status

• ✅ RPC Configuration: Properly configured
• ✅ Network Connectivity: Healthy
• ⚠️ Transaction Pool: 3 pending transactions
• ⚠️ RPC Timeout: Needs increase
• ⚠️ Transaction Pool Config: Needs explicit limits

Root Causes

1. Connection Timeout: RPC timeout too short (60s default)
2. Known Transaction: Transactions stuck in validator pools (nonces 13105-13107)

Next Steps

1. Update RPC configuration with timeout and pool settings
2. Clear transaction pools (restart all Besu nodes)
3. Implement proper retry and status checking
4. Use appropriate gas prices for replacements

---

This comprehensive review identifies all RPC-related issues and provides actionable recommendations for resolving deployment problems.