proxmox/reports/status/RPC_TRANSACTION_FAILURE_INVESTIGATION.md

RPC Transaction Failure Investigation

Date: 2026-01-27
Status: 🔍 Investigation In Progress

---

Initial Findings

✅ All RPC Nodes Operational

From the investigation, all RPC nodes are:

• ✅ Running and synchronized
• ✅ Responding to RPC calls
• ✅ Connected to peers (5 peers each)
• ✅ No recent errors in logs
• ⚠️ Thread blocking warnings (performance issue, not transaction failures)

RPC Nodes Checked

VMID	IP Address	Hostname	Status	Chain ID	Block Number
2400	192.168.11.240	thirdweb-rpc-1	✅ Running	0x8a (138)	600172
2401	192.168.11.241	thirdweb-rpc-2	✅ Running	0x8a (138)	600172
2402	192.168.11.242	thirdweb-rpc-3	✅ Running	0x8a (138)	600172
2500	192.168.11.250	besu-rpc-1	✅ Running	0x8a (138)	600172
2501	192.168.11.251	besu-rpc-2	✅ Running	0x8a (138)	600172
2502	192.168.11.252	besu-rpc-3	✅ Running	0x8a (138)	600172
2505	192.168.11.201	besu-rpc-luis-0x8a	✅ Running	0x8a (138)	600172
2506	192.168.11.202	besu-rpc-luis-0x1	✅ Running	0x8a (138)	600172
2507	192.168.11.203	besu-rpc-putu-0x8a	✅ Running	0x8a (138)	600172
2508	192.168.11.204	besu-rpc-putu-0x1	✅ Running	0x8a (138)	600172

---

Possible Transaction Failure Scenarios

1. Transactions Not Being Submitted ❓

• Symptom: Transactions never reach the network
• Possible Causes:
  • RPC endpoint not accessible
  • Network connectivity issues
  • Firewall blocking
  • Nginx/proxy configuration issues

2. Transactions Rejected Before Entering Pool ❓

• Symptom: Transactions rejected with error codes
• Possible Causes:
  • Gas price too low
  • Nonce issues
  • Account permissioning
  • Invalid transaction format
  • Insufficient balance

3. Transactions Entering Pool But Not Mined ❓

• Symptom: Transactions stuck in mempool
• Possible Causes:
  • Gas price too low
  • Network congestion
  • Validator issues
  • Transaction pool full

4. Transactions Mined But Failing (Reverting) ❓

• Symptom: Transactions mined with status 0x0
• Possible Causes:
  • Contract execution errors
  • Out of gas
  • Invalid contract code
  • State inconsistencies

---

Investigation Scripts Created

1. Comprehensive Investigation

./scripts/investigate-rpc-transaction-failures.sh

This script checks:

• Container and service status
• RPC connectivity
• Block synchronization
• Recent error logs
• Transaction pool status
• Recent block transactions (with failure detection)
• Peer connections
• Thread blocking warnings

2. Quick Status Check

# Check a specific RPC node
curl -X POST http://192.168.11.250:8545 \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

---

Next Steps

To Identify the Specific Issue:

1. Check Recent Transaction Receipts

   # Get a recent transaction hash and check its receipt
   curl -X POST http://192.168.11.250:8545 \
     -H 'Content-Type: application/json' \
     -d '{
       "jsonrpc": "2.0",
       "method": "eth_getTransactionReceipt",
       "params": ["0x<transaction_hash>"],
       "id": 1
     }'
   

2. Check Transaction Pool (if enabled)

   curl -X POST http://192.168.11.250:8545 \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","method":"txpool_status","params":[],"id":1}'
   

3. Check Recent Blocks for Failed Transactions

   # Get recent block and check transaction statuses
   ./scripts/investigate-rpc-transaction-failures.sh
   

4. Check Validator Logs (if transactions not being mined)

   # Check validator nodes (1000-1004) for transaction processing issues
   ssh root@192.168.11.10 "pct exec 1000 -- journalctl -u besu-validator --since '1 hour ago' | grep -i transaction"
   

---

Historical Issues Found

1. Contract Deployment Failures (Historical)

• Issue: All contract deployments failing with status 0x0
• Pattern: Transactions using all available gas and reverting
• Status: May still be occurring

2. Stuck Transactions in Mempool (Historical)

• Issue: Transactions stuck in mempool, blocking new transactions
• Error: "Replacement transaction underpriced"
• Status: May require Besu restart or database clearing

3. Thread Blocking Warnings (Current)

• Issue: Vertx threads being blocked for >2000ms
• Impact: Performance degradation, potential RPC delays
• Status: Ongoing, not causing transaction failures directly

---

Questions to Clarify

1. What type of transactions are failing?

   • Contract deployments?
   • Simple transfers?
   • Contract calls?
   • All transactions?

2. What error messages are you seeing?

   • Transaction rejected?
   • Transaction reverted?
   • Transaction stuck?
   • No error but transaction not appearing?

3. Are transactions being submitted successfully?

   • Do you get a transaction hash back?
   • Or are they rejected immediately?

4. Are transactions being mined but failing?

   • Check transaction receipt status (0x0 = failed, 0x1 = success)

---

SSL Certificate Error Fix (Remembered)

Error: error:0A000086:SSL routines::certificate verify failed (596)
Fix: Run pvecm updatecerts -f on Proxmox host nodes
Script: ./scripts/fix-ssl-certificate-all-hosts.sh
Status: ✅ Fixed on ml110, r630-01, r630-02

---

Last Updated: 2026-01-27
Next Action: Need clarification on specific transaction failure symptoms