proxmox/docs/archive/BESU_LOGS_SUMMARY.md

# Besu Logs Analysis Summary

**Date**: $(date)  
**Analysis**: Complete log review of all Besu services in LXC containers

---

## Executive Summary

🔴 **CRITICAL ISSUE**: All Besu services are failing due to missing configuration files.

**Status**: 
- All services in restart loops (45-54 restarts each)
- Services start → fail immediately → systemd restarts → repeat
- **Root Cause**: Configuration files not found

---

## Service Status Overview

| Category | VMID Range | Service Status | Restart Count | Config Status |
|----------|-----------|----------------|---------------|---------------|
| **Validators** | 1000-1004 | 🔴 Failing | 47-54 | ❌ Missing |
| **Sentries** | 1500-1502 | 🔴 Failing | 47-53 | ❌ Missing |
| **Sentry** | 1503 | ⚠️ Inactive | N/A | ❌ Service file missing |
| **RPC Nodes** | 2500-2502 | 🔴 Failing | 45-52 | ❌ Missing |

---

## Error Pattern (All Services)

### Common Error Message

```
Unable to read TOML configuration, file not found.
To display full help:
besu [COMMAND] --help
```

### Service Restart Loop

1. Service starts (systemd)
2. Besu process begins
3. Besu fails to read config file
4. Process exits immediately
5. Systemd restarts service (after 10 seconds)
6. Loop repeats

**Restart Counter**: Services have restarted 45-54 times, indicating this issue has persisted for an extended period.

---

## Detailed Logs by Service Type

### Validators (VMID 1000-1004)

**Sample Log (VMID 1000)**:
```
Dec 20 15:51:07 besu-validator-1 systemd[1]: Started Hyperledger Besu Validator Node.
Dec 20 15:51:10 besu-validator-1 besu-validator[2160]: Unable to read TOML configuration, file not found.
Dec 20 15:51:10 besu-validator-1 systemd[1]: besu-validator.service: Deactivated successfully.
Dec 20 15:51:21 systemd[1]: besu-validator.service: Scheduled restart job, restart counter is at 54.
```

**All Validators**: Showing identical pattern with restart counters 47-54.

### Sentries (VMID 1500-1502)

**Sample Log (VMID 1500)**:
```
Dec 20 15:51:12 besu-sentry-1 systemd[1]: Started Hyperledger Besu Sentry Node.
Dec 20 15:51:18 besu-sentry-1 besu-sentry[16206]: Unable to read TOML configuration, file not found.
Dec 20 15:51:18 besu-sentry-1 systemd[1]: besu-sentry.service: Deactivated successfully.
Dec 20 15:51:29 systemd[1]: besu-sentry.service: Scheduled restart job, restart counter is at 48.
```

**All Sentries**: Showing identical pattern with restart counters 47-53.

**Note**: VMID 1503 has no service file, so no logs to review.

### RPC Nodes (VMID 2500-2502)

**Sample Log (VMID 2500)**:
```
Dec 20 15:51:22 besu-rpc-1 systemd[1]: Started Hyperledger Besu RPC Node.
Dec 20 15:51:25 besu-rpc-1 besu-rpc[16213]: Unable to read TOML configuration, file not found.
Dec 20 15:51:25 besu-rpc-1 systemd[1]: besu-rpc.service: Deactivated successfully.
Dec 20 15:51:35 systemd[1]: besu-rpc.service: Scheduled restart job, restart counter is at 52.
```

**All RPC Nodes**: Showing identical pattern with restart counters 45-52.

---

## Root Cause Analysis

### Expected Configuration Files

| Service Type | Expected Path | Status |
|--------------|---------------|--------|
| Validators | `/etc/besu/config-validator.toml` | ❌ Missing |
| Sentries | `/etc/besu/config-sentry.toml` | ❌ Missing |
| RPC Nodes | `/etc/besu/config-rpc.toml` | ❌ Missing |

### Actual Files Found

| File | Status | Location |
|------|--------|----------|
| `config-validator.toml.template` | ✅ Exists | `/etc/besu/config-validator.toml.template` |
| `config-validator.toml` | ❌ Missing | Should be at `/etc/besu/config-validator.toml` |

### Service Configuration

Service files are correctly configured to use:
```ini
ExecStart=/opt/besu/bin/besu \
    --config-file=/etc/besu/config-validator.toml
```

**Issue**: The file `/etc/besu/config-validator.toml` does not exist.

---

## Impact Assessment

### Immediate Impact

- ❌ **No Besu nodes are running** - All services failing
- ❌ **Network not operational** - No consensus, no block production
- ❌ **Resources wasted** - Containers running but services restarting constantly
- ⚠️ **High restart counts** - Systemd has attempted 45-54 restarts per service

### Service Health

| Metric | Value | Status |
|--------|-------|--------|
| Services Running | 0 / 12 | 🔴 Critical |
| Services in Restart Loop | 11 / 12 | 🔴 Critical |
| Average Restart Count | ~50 | 🔴 Critical |
| Configuration Files Present | 0 / 11 | 🔴 Critical |

---

## Required Actions

### Immediate (Priority 1)

1. **Copy Configuration Files**
   - Option A: Copy template to config file (quick fix)
   - Option B: Copy from source project (recommended)

2. **Verify File Permissions**
   - Ensure files owned by `besu:besu`
   - Ensure readable permissions

3. **Restart Services**
   - Restart services after copying config files
   - Monitor logs to verify successful startup

### High Priority (Priority 2)

4. **Verify Additional Required Files**
   - `/etc/besu/genesis.json`
   - `/etc/besu/static-nodes.json`
   - `/etc/besu/permissions-nodes.toml`

5. **Check Validator Keys** (for validators only)
   - Verify keys exist in `/keys/validators/`

### Medium Priority (Priority 3)

6. **Fix VMID 1503 Service File**
   - Create service file if missing
   - Or verify if this container was intentionally excluded

7. **Monitor Services**
   - Watch logs after restart
   - Verify services stay running
   - Check for additional errors

---

## Quick Fix Command

### Copy Template to Config (Quick Solution)

```bash
# Validators
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- cp /etc/besu/config-validator.toml.template /etc/besu/config-validator.toml
    pct exec $vmid -- chown besu:besu /etc/besu/config-validator.toml
    pct exec $vmid -- systemctl restart besu-validator.service
done

# Sentries (if templates exist)
for vmid in 1500 1501 1502; do
    pct exec $vmid -- cp /etc/besu/config-sentry.toml.template /etc/besu/config-sentry.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-sentry.toml 2>/dev/null
    pct exec $vmid -- systemctl restart besu-sentry.service
done

# RPC Nodes (if templates exist)
for vmid in 2500 2501 2502; do
    pct exec $vmid -- cp /etc/besu/config-rpc.toml.template /etc/besu/config-rpc.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-rpc.toml 2>/dev/null
    pct exec $vmid -- systemctl restart besu-rpc.service
done
```

**Note**: This uses template configurations which may need customization for production use.

---

## Verification Commands

After applying fixes:

```bash
# Check if config files exist
for vmid in 1000 1001 1002 1003 1004 1500 1501 1502 2500 2501 2502; do
    if [[ $vmid -lt 1500 ]]; then
        config_file="config-validator.toml"
        service="besu-validator"
    elif [[ $vmid -lt 2500 ]]; then
        config_file="config-sentry.toml"
        service="besu-sentry"
    else
        config_file="config-rpc.toml"
        service="besu-rpc"
    fi
    
    echo "=== VMID $vmid ==="
    pct exec $vmid -- ls -la /etc/besu/$config_file 2>/dev/null && echo "✅ Config exists" || echo "❌ Config missing"
    pct exec $vmid -- systemctl is-active $service.service 2>/dev/null && echo "✅ Service active" || echo "❌ Service inactive"
    echo ""
done
```

---

## Related Documentation

- [Configuration Issue Details](BESU_CONFIGURATION_ISSUE.md)
- [Current Deployment Status](CURRENT_DEPLOYMENT_STATUS.md)
- [Files Copy Checklist](FILES_COPY_CHECKLIST.md)

---

**Analysis Completed**: $(date)  
**Status**: 🔴 **CRITICAL - IMMEDIATE ACTION REQUIRED**