Files
proxmox/docs/08-monitoring/BLOCK_PRODUCTION_MONITORING.md
defiQUG fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
docs: Ledger Live integration, contract deploy learnings, NEXT_STEPS updates
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands
- CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround
- CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check
- NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere
- MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates
- LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-12 15:46:57 -08:00

3.3 KiB

Block Production Monitoring

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation


Date: $(date)
Status: MONITORING FOR BLOCK PRODUCTION


Monitoring Plan

After applying the validator key fix, we need to monitor:

  1. Block Numbers - Should increment from 0
  2. QBFT Consensus Activity - Logs should show block proposal/production
  3. Peer Connections - Nodes should maintain connections
  4. Validator Key Usage - Confirm validators are using correct keys
  5. Errors/Warnings - Check for any issues preventing block production

Expected Behavior

Block Production

  • Blocks should be produced every 2 seconds (per genesis blockperiodseconds: 2)
  • Block numbers should increment: 0 → 1 → 2 → 3 ...
  • All nodes should see the same block numbers (consensus)

QBFT Consensus

  • Validators should participate in consensus
  • Logs should show block proposal/production activity
  • At least 4 out of 5 validators must be online (2/3 quorum)

Network Status

  • All validators should be connected (5 peers visible)
  • Sentries should connect to validators
  • No sync errors or connection issues

Monitoring Commands

Check Block Numbers

for vmid in 1500 1501 1502; do
  block=$(pct exec $vmid -- curl -s -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    -H 'Content-Type: application/json' http://localhost:8545 2>/dev/null | \
    grep -oP '"result":"\K[0-9a-f]+' | head -1)
  block_dec=$(printf '%d' 0x$block 2>/dev/null)
  echo "Sentry $vmid: Block $block_dec"
done

Check QBFT Activity

pct exec 1000 -- journalctl -u besu-validator.service --since '5 minutes ago' --no-pager | \
  grep -iE 'qbft|consensus|propose|producing|block.*produced|imported.*block'

Check Peer Connections

pct exec 1500 -- curl -s -X POST --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}' \
  -H 'Content-Type: application/json' http://localhost:8545 | \
  python3 -c "import json, sys; data=json.load(sys.stdin); print(f'Peers: {len(data.get(\"result\", []))}')"

Troubleshooting

If Blocks Are Not Producing

  1. Verify Validator Keys

    • Check that /data/besu/key contains validator keys (not node keys)
    • Verify addresses match genesis extraData
  2. Check Consensus Status

    • Look for QBFT messages in logs
    • Verify at least 4/5 validators are online
    • Check for consensus errors
  3. Verify Network Connectivity

    • All validators should have peer connections
    • Check that enode URLs are correct in static-nodes.json
  4. Check Genesis Configuration

    • Verify QBFT config in genesis.json
    • Confirm validator addresses in extraData match actual keys

Success Criteria

Block Production Working:

  • Block numbers increment from 0
  • Blocks produced approximately every 2 seconds
  • All nodes see same block numbers

QBFT Consensus Active:

  • Logs show block proposal/production messages
  • Validators participating in consensus
  • No consensus errors

Network Stable:

  • All validators connected
  • No connection errors
  • Enode URLs correct

Last Updated: $(date)
Next Check: Monitor block numbers and logs for production activity