proxmox/smom-dbis-138-proxmox/docs/RESTART_BESU_NODE.md

Restart Besu Node (Clears Mempool)

Purpose: Restart a Besu node to clear the mempool and resolve stuck transactions
Script: scripts/restart-besu-node.sh

---

Overview

Restarting a Besu node clears the mempool, which removes all pending transactions. This is useful for resolving:

• Stuck transactions in mempool
• Mempool-related errors
• Transaction validation issues
• Network synchronization problems

⚠️ Important: Restarting clears the mempool, so all pending transactions will be lost and need to be resubmitted.

---

Usage

Basic Usage

# Restart by VMID (auto-detect service type)
./scripts/restart-besu-node.sh 1000

# Restart by hostname
./scripts/restart-besu-node.sh validator-1

# Specify service type explicitly
./scripts/restart-besu-node.sh 2500 rpc

Service Types

• validator - Besu validator node
• sentry - Besu sentry node
• rpc - Besu RPC node
• auto - Auto-detect (default)

---

What Happens When You Restart

1. Service Stops

   • Besu process terminates
   • Mempool is cleared (all pending transactions removed)
   • Network connections close

2. Service Starts

   • Besu process restarts
   • Node resyncs with network
   • Peer connections rebuild
   • Mempool is empty (ready for new transactions)

3. Recovery Time

   • Service typically starts within 10-30 seconds
   • Network resync may take 1-5 minutes depending on network state
   • Node will catch up to latest block height

---

When to Use

✅ Use Restart When:

• Transactions are stuck in mempool
• Getting "replacement transaction underpriced" errors
• Mempool appears blocked
• Node shows mempool-related errors in logs
• Need to clear transaction state

❌ Don't Use Restart For:

• Normal operation (only when troubleshooting)
• Frequent issues (investigate root cause instead)
• Production during high-traffic periods (if avoidable)
• Multiple nodes simultaneously (restart one at a time)

---

Examples

Example 1: Restart Validator Node

# Restart validator node (VMID 1000)
./scripts/restart-besu-node.sh 1000

# Output:
# [INFO] VMID: 1000
# [INFO] Hostname: validator-1
# [INFO] Detected service type: validator
# [INFO] Service: besu-validator
# [WARN] ⚠️  Restarting Besu node will clear the mempool
# Continue with restart? (y/N): y
# [✓] Service restart command executed
# [✓] Service is active
# [✓] Besu node restart completed!

Example 2: Restart RPC Node

# Restart RPC node (VMID 2500)
./scripts/restart-besu-node.sh 2500 rpc

# The script will restart besu-rpc.service

Example 3: Restart by Hostname

# Restart using hostname
./scripts/restart-besu-node.sh sentry-2

# Script finds VMID automatically

---

Manual Restart (Alternative)

If you prefer to restart manually:

# For validator node
pct exec 1000 -- systemctl restart besu-validator.service

# For sentry node
pct exec 1500 -- systemctl restart besu-sentry.service

# For RPC node
pct exec 2500 -- systemctl restart besu-rpc.service

---

Monitoring After Restart

Check Service Status

# Check if service is active
pct exec <VMID> -- systemctl status besu-validator.service

# Follow logs
pct exec <VMID> -- journalctl -u besu-validator.service -f

Verify Node Recovery

1. Check service is running

   pct exec <VMID> -- systemctl is-active besu-validator.service
   # Should return: active
   

2. Check node is syncing

   # Via RPC
   curl -X POST http://localhost:8545 \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'
   
   # Should return: {"jsonrpc":"2.0","id":1,"result":false}
   # (false means fully synced)
   

3. Check block height

   curl -X POST http://localhost:8545 \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
   

---

Troubleshooting

Service Won't Start

If the service fails to start after restart:

1. Check logs

   pct exec <VMID> -- journalctl -u besu-validator.service -n 50
   

2. Check configuration

   # Verify config file exists
   pct exec <VMID> -- ls -la /etc/besu/config-*.toml
   

3. Check permissions

   pct exec <VMID> -- ls -la /opt/besu
   pct exec <VMID> -- ls -la /var/lib/besu
   

Node Not Syncing

If the node doesn't sync after restart:

1. Check peer connections

   # Via admin RPC (if enabled)
   curl -X POST http://localhost:8545 \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}'
   

2. Check static nodes

   pct exec <VMID> -- cat /etc/besu/static-nodes.json
   

3. Wait longer

   • Network resync can take several minutes
   • Validator nodes may need to catch up to latest block

---

Impact on Network

Single Node Restart

• Minimal impact: Other nodes continue operating
• Validation: If restarting a validator, other validators continue block production
• RPC: If restarting RPC node, use other RPC nodes during downtime

Multiple Node Restart

• ⚠️ Warning: Avoid restarting multiple nodes simultaneously
• Best practice: Restart one node at a time, wait for recovery
• Validator quorum: Ensure enough validators remain running for consensus

---

Related Scripts

• scripts/check-besu-transaction-pool.sh - Check mempool status
• scripts/check-mempool-status.sh - Monitor mempool and block production
• scripts/network/bootstrap-network.sh - Restart all nodes in order

---

References

• Besu Documentation
• Transaction Pool Documentation
• Mempool Issue Resolution

---

Last Updated: 2025-12-22