fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
210 lines
5.6 KiB
Markdown
210 lines
5.6 KiB
Markdown
# Network Recovered - Transaction Propagation Issue
|
|
|
|
**Last Updated:** 2026-01-31
|
|
**Document Version:** 1.0
|
|
**Status:** Active Documentation
|
|
|
|
---
|
|
|
|
**Date**: 2026-01-24 01:35 PST
|
|
**Status**: ⚠️ **PARTIAL SUCCESS - Blocks Producing, Transactions Not Confirming**
|
|
|
|
---
|
|
|
|
## ✅ What Was Fixed
|
|
|
|
### 1. Found All Validators
|
|
**Problem**: Thought we only had 2 validators
|
|
**Reality**: All 5 validators exist and are running!
|
|
|
|
| VMID | IP | Host | Status |
|
|
|------|----|----|--------|
|
|
| 1000 | 192.168.11.100 | r630-01 | ✅ Running |
|
|
| 1001 | 192.168.11.101 | r630-01 | ✅ Running |
|
|
| 1002 | 192.168.11.102 | r630-01 | ✅ Running |
|
|
| 1003 | 192.168.11.103 | ml110 | ✅ Running |
|
|
| 1004 | 192.168.11.104 | ml110 | ✅ Running |
|
|
|
|
### 2. Enabled Mining on All Validators
|
|
**Problem**: `miner-enabled=false` on all validators
|
|
**Solution**: Set `miner-enabled=true` on all 5 validators
|
|
**Result**: ✅ Block production resumed!
|
|
|
|
### 3. Network Statistics
|
|
- **Current block**: 1,301,181+ (and counting)
|
|
- **Block time**: ~2 seconds
|
|
- **Consensus**: QBFT with 5 validators
|
|
- **Quorum**: 4/5 required, all 5 active
|
|
- **Peers**: 14 connected
|
|
|
|
---
|
|
|
|
## ⚠️ Remaining Issue: Empty Blocks
|
|
|
|
### Problem
|
|
- ✅ Blocks ARE producing (network recovered)
|
|
- ❌ Blocks are EMPTY (0 transactions)
|
|
- ❌ Transactions stuck in RPC mempool
|
|
- ❌ Validators not selecting transactions
|
|
|
|
### Symptoms
|
|
```json
|
|
{
|
|
"number": "0x13dac9", // Block 1,301,193
|
|
"transactions": 0, // EMPTY!
|
|
"gasUsed": "0x0" // No gas used
|
|
}
|
|
```
|
|
|
|
### Root Cause Analysis
|
|
1. **RPC has transactions** in mempool (stuck at nonces 13104-13115)
|
|
2. **Validators have correct TX-pool config** (layered, with proper capacity)
|
|
3. **Transactions not propagating** from RPC to validators
|
|
4. **Or validators not accepting** transactions from mempool
|
|
|
|
---
|
|
|
|
## 🔍 Diagnostic Results
|
|
|
|
### Validator Configuration ✅
|
|
```toml
|
|
# TX-Pool Configuration (CORRECT)
|
|
tx-pool-max-future-by-sender=200
|
|
tx-pool-layer-max-capacity=12500000
|
|
tx-pool-max-prioritized=2000
|
|
|
|
# Mining Configuration (CORRECT)
|
|
miner-enabled=true
|
|
|
|
# Sync Configuration (CORRECT)
|
|
sync-mode="FAST"
|
|
```
|
|
|
|
### Network Topology ✅
|
|
- **Static nodes**: Configured correctly
|
|
- **Permissioned nodes**: All 5 validators listed
|
|
- **Peer connections**: 14 peers connected
|
|
- **P2P network**: Functioning
|
|
|
|
### What's NOT Working ❌
|
|
- Transaction propagation RPC → Validators
|
|
- Transaction selection by validators
|
|
- Transaction confirmation
|
|
|
|
---
|
|
|
|
## 🎯 Possible Causes
|
|
|
|
### 1. RPC Not Propagating to Validators
|
|
- RPC nodes may not be connected to validators
|
|
- Transactions stay in RPC mempool only
|
|
- Need to verify RPC peering
|
|
|
|
### 2. Validator Transaction Acceptance
|
|
- Validators may have additional filters
|
|
- Min gas price mismatch
|
|
- Transaction pool limits
|
|
|
|
### 3. Permissions/ACL Issues
|
|
- accounts-allowlist might be blocking
|
|
- Transaction validation rules
|
|
- Smart contract permissions
|
|
|
|
---
|
|
|
|
## 🔧 Next Steps to Fix
|
|
|
|
### Option 1: Send Transaction Directly to Validator
|
|
Instead of using RPC at 192.168.11.211, send directly to a validator:
|
|
|
|
```bash
|
|
# Try sending to validator 1003 at 192.168.11.103
|
|
export PRIVATE_KEY=0x5373d11ee2cad4ed82b9208526a8c358839cbfe325919fb250f062a25153d1c8
|
|
cast send 0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2 "deposit()" \
|
|
--value 1000000000000000 \
|
|
--private-key $PRIVATE_KEY \
|
|
--rpc-url http://192.168.11.103:8545 \
|
|
--gas-limit 50000 \
|
|
--gas-price 1000000000 \
|
|
--nonce 13116
|
|
```
|
|
|
|
### Option 2: Enable RPC on Validators
|
|
Check if validators have RPC enabled:
|
|
```bash
|
|
ssh root@192.168.11.10 "pct exec 1003 -- grep 'rpc-http-enabled' /etc/besu/config-validator.toml"
|
|
```
|
|
|
|
If `rpc-http-enabled=false`, enable it and send transactions directly.
|
|
|
|
### Option 3: Clear RPC Transaction Database
|
|
Force clear stuck transactions from RPC:
|
|
```bash
|
|
# Stop RPC nodes
|
|
# Clear transaction pool database
|
|
# Restart RPC nodes
|
|
# Resend transaction with next nonce
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Current Status Summary
|
|
|
|
| Component | Status | Details |
|
|
|-----------|--------|---------|
|
|
| **Validators** | ✅ Running | All 5 active |
|
|
| **Mining** | ✅ Enabled | All validators |
|
|
| **Block Production** | ✅ Working | ~2s block time |
|
|
| **Network Quorum** | ✅ Met | 5/5 validators |
|
|
| **TX-Pool Config** | ✅ Correct | Layered settings |
|
|
| **Transaction Confirmation** | ❌ Not working | Empty blocks |
|
|
| **Bridge Execution** | ❌ Blocked | Can't confirm txs |
|
|
|
|
---
|
|
|
|
## 💡 Key Insights
|
|
|
|
### What We Learned
|
|
1. **All 5 validators exist** - they're split across 2 Proxmox hosts
|
|
2. **Mining was disabled** - this was the main blocker
|
|
3. **Network can recover** - blocks producing again
|
|
4. **Transaction propagation** - separate issue from consensus
|
|
|
|
### Why Bridge Still Can't Execute
|
|
- Network is operational ✅
|
|
- Bridge contracts deployed ✅
|
|
- Validators working ✅
|
|
- But transactions can't confirm ❌
|
|
|
|
---
|
|
|
|
## 🚀 Immediate Next Action
|
|
|
|
**Try sending transaction directly to a validator node instead of through RPC:**
|
|
|
|
```bash
|
|
cd /home/intlc/projects/proxmox
|
|
export PRIVATE_KEY=0x5373d11ee2cad4ed82b9208526a8c358839cbfe325919fb250f062a25153d1c8
|
|
|
|
# Try validator 1003 which we know is running
|
|
cast send 0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2 "deposit()" \
|
|
--value 1000000000000000 \
|
|
--private-key $PRIVATE_KEY \
|
|
--rpc-url http://192.168.11.103:8545 \
|
|
--gas-limit 50000 \
|
|
--gas-price 1000000000 \
|
|
--nonce 13116
|
|
```
|
|
|
|
If validator doesn't have RPC enabled, we need to:
|
|
1. Enable RPC on one validator temporarily
|
|
2. Or fix the RPC→Validator transaction propagation
|
|
|
|
---
|
|
|
|
**Status**: Network operational, investigating transaction confirmation issue
|
|
**Progress**: 90% complete - just need transactions to confirm
|
|
**Blocker**: Transaction propagation RPC→Validators
|
|
|
|
**Last Updated**: 2026-01-24 01:35 PST
|