proxmox/reports/status/RPC_TRANSACTION_FAILURE_INVESTIGATION.md

# RPC Transaction Failure Investigation

**Date**: 2026-01-27  
**Status**: 🔍 Investigation In Progress

---

## Initial Findings

### ✅ All RPC Nodes Operational

From the investigation, all RPC nodes are:
- ✅ Running and synchronized
- ✅ Responding to RPC calls
- ✅ Connected to peers (5 peers each)
- ✅ No recent errors in logs
- ⚠️ Thread blocking warnings (performance issue, not transaction failures)

### RPC Nodes Checked

| VMID | IP Address | Hostname | Status | Chain ID | Block Number |
|------|------------|----------|--------|----------|--------------|
| 2400 | 192.168.11.240 | thirdweb-rpc-1 | ✅ Running | 0x8a (138) | 600172 |
| 2401 | 192.168.11.241 | thirdweb-rpc-2 | ✅ Running | 0x8a (138) | 600172 |
| 2402 | 192.168.11.242 | thirdweb-rpc-3 | ✅ Running | 0x8a (138) | 600172 |
| 2500 | 192.168.11.250 | besu-rpc-1 | ✅ Running | 0x8a (138) | 600172 |
| 2501 | 192.168.11.251 | besu-rpc-2 | ✅ Running | 0x8a (138) | 600172 |
| 2502 | 192.168.11.252 | besu-rpc-3 | ✅ Running | 0x8a (138) | 600172 |
| 2505 | 192.168.11.201 | besu-rpc-luis-0x8a | ✅ Running | 0x8a (138) | 600172 |
| 2506 | 192.168.11.202 | besu-rpc-luis-0x1 | ✅ Running | 0x8a (138) | 600172 |
| 2507 | 192.168.11.203 | besu-rpc-putu-0x8a | ✅ Running | 0x8a (138) | 600172 |
| 2508 | 192.168.11.204 | besu-rpc-putu-0x1 | ✅ Running | 0x8a (138) | 600172 |

---

## Possible Transaction Failure Scenarios

### 1. Transactions Not Being Submitted ❓
- **Symptom**: Transactions never reach the network
- **Possible Causes**:
  - RPC endpoint not accessible
  - Network connectivity issues
  - Firewall blocking
  - Nginx/proxy configuration issues

### 2. Transactions Rejected Before Entering Pool ❓
- **Symptom**: Transactions rejected with error codes
- **Possible Causes**:
  - Gas price too low
  - Nonce issues
  - Account permissioning
  - Invalid transaction format
  - Insufficient balance

### 3. Transactions Entering Pool But Not Mined ❓
- **Symptom**: Transactions stuck in mempool
- **Possible Causes**:
  - Gas price too low
  - Network congestion
  - Validator issues
  - Transaction pool full

### 4. Transactions Mined But Failing (Reverting) ❓
- **Symptom**: Transactions mined with status 0x0
- **Possible Causes**:
  - Contract execution errors
  - Out of gas
  - Invalid contract code
  - State inconsistencies

---

## Investigation Scripts Created

### 1. Comprehensive Investigation
```bash
./scripts/investigate-rpc-transaction-failures.sh
```

This script checks:
- Container and service status
- RPC connectivity
- Block synchronization
- Recent error logs
- Transaction pool status
- Recent block transactions (with failure detection)
- Peer connections
- Thread blocking warnings

### 2. Quick Status Check
```bash
# Check a specific RPC node
curl -X POST http://192.168.11.250:8545 \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```

---

## Next Steps

### To Identify the Specific Issue:

1. **Check Recent Transaction Receipts**
   ```bash
   # Get a recent transaction hash and check its receipt
   curl -X POST http://192.168.11.250:8545 \
     -H 'Content-Type: application/json' \
     -d '{
       "jsonrpc": "2.0",
       "method": "eth_getTransactionReceipt",
       "params": ["0x<transaction_hash>"],
       "id": 1
     }'
   ```

2. **Check Transaction Pool** (if enabled)
   ```bash
   curl -X POST http://192.168.11.250:8545 \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","method":"txpool_status","params":[],"id":1}'
   ```

3. **Check Recent Blocks for Failed Transactions**
   ```bash
   # Get recent block and check transaction statuses
   ./scripts/investigate-rpc-transaction-failures.sh
   ```

4. **Check Validator Logs** (if transactions not being mined)
   ```bash
   # Check validator nodes (1000-1004) for transaction processing issues
   ssh root@192.168.11.10 "pct exec 1000 -- journalctl -u besu-validator --since '1 hour ago' | grep -i transaction"
   ```

---

## Historical Issues Found

### 1. Contract Deployment Failures (Historical)
- **Issue**: All contract deployments failing with status 0x0
- **Pattern**: Transactions using all available gas and reverting
- **Status**: May still be occurring

### 2. Stuck Transactions in Mempool (Historical)
- **Issue**: Transactions stuck in mempool, blocking new transactions
- **Error**: "Replacement transaction underpriced"
- **Status**: May require Besu restart or database clearing

### 3. Thread Blocking Warnings (Current)
- **Issue**: Vertx threads being blocked for >2000ms
- **Impact**: Performance degradation, potential RPC delays
- **Status**: Ongoing, not causing transaction failures directly

---

## Questions to Clarify

1. **What type of transactions are failing?**
   - Contract deployments?
   - Simple transfers?
   - Contract calls?
   - All transactions?

2. **What error messages are you seeing?**
   - Transaction rejected?
   - Transaction reverted?
   - Transaction stuck?
   - No error but transaction not appearing?

3. **Are transactions being submitted successfully?**
   - Do you get a transaction hash back?
   - Or are they rejected immediately?

4. **Are transactions being mined but failing?**
   - Check transaction receipt status (0x0 = failed, 0x1 = success)

---

## SSL Certificate Error Fix (Remembered)

**Error**: `error:0A000086:SSL routines::certificate verify failed (596)`  
**Fix**: Run `pvecm updatecerts -f` on Proxmox host nodes  
**Script**: `./scripts/fix-ssl-certificate-all-hosts.sh`  
**Status**: ✅ Fixed on ml110, r630-01, r630-02

---

**Last Updated**: 2026-01-27  
**Next Action**: Need clarification on specific transaction failure symptoms