# Critical Issue: Missing Besu Configuration Files

**Date**: $(date)  
**Severity**: 🔴 **CRITICAL**  
**Impact**: All Besu services failing in restart loop

---

## Issue Summary

All Besu services across all LXC containers are **failing** with the error:

```
Unable to read TOML configuration, file not found.
```

**Services Affected**:
- ✅ Validators (1000-1004): All failing
- ✅ Sentries (1500-1502): All failing  
- ✅ RPC Nodes (2500-2502): All failing
- ⚠️ Sentry 1503: Service file missing

---

## Root Cause

The systemd services are configured to use:
- **Expected Path**: `/etc/besu/config-validator.toml` (validators)
- **Expected Path**: `/etc/besu/config-sentry.toml` (sentries)
- **Expected Path**: `/etc/besu/config-rpc.toml` (RPC nodes)

**Actual Status**: Only template files exist:
- `/etc/besu/config-validator.toml.template` ✅ (exists)
- `/etc/besu/config-validator.toml` ❌ (missing)

---

## Service Status

All services are in a **restart loop**:

| Node Type | VMID Range | Restart Count | Status |
|-----------|-----------|---------------|--------|
| Validators | 1000-1004 | 47-54 restarts | 🔴 Failing |
| Sentries | 1500-1502 | 47-53 restarts | 🔴 Failing |
| RPC Nodes | 2500-2502 | 45-52 restarts | 🔴 Failing |

**Error Pattern**: Service starts → fails immediately (config file not found) → systemd restarts → repeat

---

## Verification

### What's Missing

```bash
# Service expects:
/etc/besu/config-validator.toml  ❌ NOT FOUND

# What exists:
/etc/besu/config-validator.toml.template  ✅ EXISTS
```

### Service Configuration

The systemd service files reference:
```ini
ExecStart=/opt/besu/bin/besu \
    --config-file=/etc/besu/config-validator.toml
```

---

## Solution Options

### Option 1: Copy Template to Config File (Quick Fix)

Copy the template files to the actual config files:

```bash
# For Validators
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- cp /etc/besu/config-validator.toml.template /etc/besu/config-validator.toml
    pct exec $vmid -- chown besu:besu /etc/besu/config-validator.toml
done

# For Sentries
for vmid in 1500 1501 1502 1503; do
    pct exec $vmid -- cp /etc/besu/config-sentry.toml.template /etc/besu/config-sentry.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-sentry.toml 2>/dev/null
done

# For RPC Nodes
for vmid in 2500 2501 2502; do
    pct exec $vmid -- cp /etc/besu/config-rpc.toml.template /etc/besu/config-rpc.toml 2>/dev/null || echo "Template not found for $vmid"
    pct exec $vmid -- chown besu:besu /etc/besu/config-rpc.toml 2>/dev/null
done
```

**Note**: This uses template configuration which may need customization.

### Option 2: Copy from Source Project (Recommended)

Copy actual configuration files from the source project:

```bash
# Assuming source project is at /opt/smom-dbis-138 on Proxmox host

# Validators
for vmid in 1000 1001 1002 1003 1004; do
    # Determine validator number (1-5)
    validator_num=$((vmid - 999))
    
    # Copy from source project (adjust path as needed)
    # Option A: If node-specific configs exist
    pct push $vmid /opt/smom-dbis-138/config/nodes/validator-${validator_num}/config-validator.toml \
        /etc/besu/config-validator.toml
    
    # Option B: If single template exists
    pct push $vmid /opt/smom-dbis-138/config/config-validator.toml \
        /etc/besu/config-validator.toml
    
    pct exec $vmid -- chown besu:besu /etc/besu/config-validator.toml
done

# Similar for sentries and RPC nodes
```

### Option 3: Run Configuration Deployment Script

Use the deployment scripts to properly copy and configure files:

```bash
cd /opt/smom-dbis-138-proxmox
# Check for config copy scripts
./scripts/deployment/copy-configs-to-containers.sh /opt/smom-dbis-138
```

---

## Additional Required Files

Even after fixing the main config files, ensure these files exist:

### Required for All Nodes
- ✅ `/etc/besu/genesis.json` - Network genesis block
- ✅ `/etc/besu/static-nodes.json` - Static peer list
- ✅ `/etc/besu/permissions-nodes.toml` - Node permissions

### Required for Validators
- ✅ `/keys/validators/validator-*/` - Validator signing keys

---

## Verification After Fix

After copying configuration files, verify:

```bash
# Check if config files exist
for vmid in 1000 1001 1002 1003 1004; do
    echo "VMID $vmid:"
    pct exec $vmid -- ls -la /etc/besu/config-validator.toml
done

# Restart services
for vmid in 1000 1001 1002 1003 1004; do
    pct exec $vmid -- systemctl restart besu-validator.service
    sleep 2
    pct exec $vmid -- systemctl status besu-validator.service --no-pager | head -10
done

# Check logs for errors
for vmid in 1000 1001 1002 1003 1004; do
    echo "=== VMID $vmid ==="
    pct exec $vmid -- journalctl -u besu-validator.service --since "1 minute ago" --no-pager | tail -10
done
```

---

## Current Logs Summary

All services showing identical error pattern:

```
Dec 20 15:51:XX besu-validator-X besu-validator[XXXX]: Unable to read TOML configuration, file not found.
Dec 20 15:51:XX besu-validator-X besu-validator[XXXX]: To display full help:
Dec 20 15:51:XX besu-validator-X besu-validator[XXXX]: besu [COMMAND] --help
Dec 20 15:51:XX besu-validator-X systemd[1]: besu-validator.service: Deactivated successfully.
```

**Restart Counter**: Services have restarted 45-54 times each, indicating this has been failing for an extended period.

---

## Priority Actions

1. 🔴 **URGENT**: Copy configuration files to all containers
2. 🔴 **URGENT**: Restart services after fixing config files
3. ⚠️ **HIGH**: Verify all required files (genesis.json, static-nodes.json, etc.)
4. ⚠️ **HIGH**: Check service logs after restart to ensure proper startup
5. 📋 **MEDIUM**: Verify validator keys are in place (for validators only)

---

## Related Documentation

- [Files Copy Checklist](FILES_COPY_CHECKLIST.md)
- [Path Reference](/docs/11-references/PATHS_REFERENCE.md)
- [Current Deployment Status](CURRENT_DEPLOYMENT_STATUS.md)

---

**Issue Identified**: $(date)  
**Status**: 🔴 **NEEDS IMMEDIATE ATTENTION**