proxmox/docs/04-configuration/SECRETS_KEYS_CONFIGURATION.md

# Secrets and Keys Configuration Guide

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

Complete guide for all secrets, keys, and credentials needed for deployment.

---

## 1. Proxmox API Credentials

### Configuration Location
**File**: `~/.env` (home directory)

### Required Variables
```bash
PROXMOX_HOST="192.168.11.10"
PROXMOX_PORT="8006"
PROXMOX_USER="root@pam"
PROXMOX_TOKEN_NAME="mcp-server"
PROXMOX_TOKEN_VALUE="your-actual-token-secret-value-here"
```

### How It Works
1. Scripts load variables from `~/.env` via `load_env_file()` function in `lib/common.sh`
2. Falls back to values in `config/proxmox.conf` if not in `.env`
3. `PROXMOX_TOKEN_VALUE` is preferred; `PROXMOX_TOKEN_SECRET` is supported for backwards compatibility

### Security Notes
- ✅ API tokens are preferred over passwords
- ✅ Token should never be hardcoded in scripts
- ✅ `~/.env` file should have restrictive permissions: `chmod 600 ~/.env`
- ✅ Token is loaded dynamically, not stored in repository

### Creating API Token
```bash
# On Proxmox host (via Web UI):
# 1. Go to Datacenter → Permissions → API Tokens
# 2. Click "Add"
# 3. Set Token ID: mcp-server (or custom name)
# 4. Set User: root@pam (or appropriate user)
# 5. Set Privilege Separation: enabled (recommended)
# 6. Copy the secret value immediately (cannot be retrieved later)
# 7. Add to ~/.env file as PROXMOX_TOKEN_VALUE
```

---

## 2. Besu Validator Keys

### Location
**Directory**: `/home/intlc/projects/smom-dbis-138/keys/validators/`

### Structure
```
keys/validators/
├── validator-1/
│   ├── key           # Private key (CRITICAL - keep secure!)
│   ├── key.pub       # Public key
│   └── address       # Account address
├── validator-2/
├── validator-3/
├── validator-4/
└── validator-5/
```

### Security Requirements
- ⚠️ **CRITICAL**: Private keys (`key` files) must be kept secure
- ✅ Keys are copied via `pct push` (secure transfer)
- ✅ Ownership set to `besu:besu` user in containers
- ✅ Permissions managed by deployment scripts
- ⚠️ **Never commit keys to git repositories**

### Key Mapping
- `validator-1/` → VMID 1000
- `validator-2/` → VMID 1001
- `validator-3/` → VMID 1002
- `validator-4/` → VMID 1003
- `validator-5/` → VMID 1004

### Verification
```bash
# Check keys exist
SOURCE_PROJECT="/home/intlc/projects/smom-dbis-138"
for i in 1 2 3 4 5; do
    echo "Validator $i:"
    [ -f "$SOURCE_PROJECT/keys/validators/validator-$i/key" ] && echo "  ✓ Private key exists" || echo "  ✗ Private key MISSING"
    [ -f "$SOURCE_PROJECT/keys/validators/validator-$i/key.pub" ] && echo "  ✓ Public key exists" || echo "  ✗ Public key MISSING"
    [ -f "$SOURCE_PROJECT/keys/validators/validator-$i/address" ] && echo "  ✓ Address exists" || echo "  ✗ Address MISSING"
done
```

---

## 3. Besu Node Keys

### Location (if using node-specific configs)
**Directory**: `/home/intlc/projects/smom-dbis-138/config/nodes/<node-name>/`

### Files
- `nodekey` - Node identification key

### Destination
- Container path: `/data/besu/nodekey`

### Security
- ✅ Node keys are less sensitive than validator keys
- ✅ Still should not be committed to public repositories
- ✅ Ownership set to `besu:besu` user

---

## 4. Application-Specific Secrets

### Blockscout Explorer

**Required Secrets**:
```bash
SECRET_KEY_BASE     # Rails secret (auto-generated if not provided)
POSTGRES_PASSWORD   # Database password (default: blockscout)
DATABASE_URL        # Full database connection string
```

**Configuration**:
- Location: Environment variables in `install/blockscout-install.sh`
- `SECRET_KEY_BASE`: Generated via `openssl rand -hex 64` if not provided
- `POSTGRES_PASSWORD`: Set via `DB_PASSWORD` environment variable (default: `blockscout`)

**Example**:
```bash
export DB_PASSWORD="your-secure-password-here"
export SECRET_KEY="$(openssl rand -hex 64)"
```

---

### Firefly

**Required Secrets**:
```bash
POSTGRES_PASSWORD   # Database password (default: firefly)
FF_DATABASE_URL     # Database connection string
```

**Configuration**:
- Location: Environment variables in `install/firefly-install.sh`
- `POSTGRES_PASSWORD`: Set via `DB_PASSWORD` environment variable (default: `firefly`)

**Example**:
```bash
export DB_PASSWORD="your-secure-password-here"
```

---

### Monitoring Stack (Grafana)

**Required Secrets**:
```bash
GRAFANA_PASSWORD    # Admin password (default: admin)
```

**Configuration**:
- Location: Environment variable in `install/monitoring-stack-install.sh`
- Default: `admin` (⚠️ **CHANGE THIS IN PRODUCTION**)

**Example**:
```bash
export GRAFANA_PASSWORD="your-secure-grafana-password"
```

---

### Financial Tokenization

**Required Secrets**:
```bash
FIREFLY_API_KEY     # Firefly API key (if needed)
```

**Configuration**:
- Location: Environment variable in `install/financial-tokenization-install.sh`
- Optional: Only needed if integrating with Firefly

**Example**:
```bash
export FIREFLY_API_KEY="your-firefly-api-key-here"
```

---

## 5. Environment Variables Summary

### Setting Environment Variables

**Option 1: Export in shell session**
```bash
export PROXMOX_TOKEN_VALUE="your-token"
export DB_PASSWORD="your-password"
export GRAFANA_PASSWORD="your-password"
```

**Option 2: Add to `~/.env` file**
```bash
# Proxmox API
PROXMOX_HOST="192.168.11.10"
PROXMOX_PORT="8006"
PROXMOX_USER="root@pam"
PROXMOX_TOKEN_NAME="mcp-server"
PROXMOX_TOKEN_VALUE="your-token-secret"

# Application Secrets
DB_PASSWORD="your-database-password"
GRAFANA_PASSWORD="your-grafana-password"
SECRET_KEY="$(openssl rand -hex 64)"
```

**Option 3: Create `.env.local` file in project root**
```bash
# .env.local (gitignored)
PROXMOX_TOKEN_VALUE="your-token"
DB_PASSWORD="your-password"
```

---

## 6. Secrets Management Best Practices

### ✅ DO:
- Store secrets in `~/.env` file with restrictive permissions (`chmod 600`)
- Use environment variables for secrets
- Generate strong passwords and keys
- Rotate secrets periodically
- Use API tokens instead of passwords where possible
- Document which secrets are required

### ❌ DON'T:
- Commit secrets to git repositories
- Hardcode secrets in scripts
- Share secrets via insecure channels
- Use default passwords in production
- Store secrets in plain text files in project directory

---

## 7. Secrets Verification Checklist

### Pre-Deployment
- [ ] Proxmox API token configured in `~/.env`
- [ ] Validator keys exist and are secure
- [ ] Application passwords are set (if not using defaults)
- [ ] Database passwords are configured (if using databases)
- [ ] All required environment variables are set

### During Deployment
- [ ] Secrets are loaded from `~/.env` correctly
- [ ] Validator keys are copied securely to containers
- [ ] Application secrets are passed via environment variables
- [ ] No secrets appear in logs

### Post-Deployment
- [ ] Verify services can authenticate (Proxmox API, databases, etc.)
- [ ] Verify validators are using correct keys
- [ ] Verify application passwords are working
- [ ] Audit logs for any secret exposure

---

## 8. Troubleshooting

### Proxmox API Token Not Working
**Error**: `401 Unauthorized`

**Solution**:
1. Verify token exists in Proxmox: Check API Tokens in Web UI
2. Verify token secret is correct in `~/.env`
3. Check token permissions
4. Verify token hasn't expired
5. Test token manually:
   ```bash
   curl -H "Authorization: PVEAPIToken=root@pam=mcp-server=your-token-secret" \
        https://192.168.11.10:8006/api2/json/version
   ```

### Validator Keys Not Found
**Error**: `Validator keys directory not found`

**Solution**:
1. Verify keys directory exists: `ls -la /home/intlc/projects/smom-dbis-138/keys/validators/`
2. Check key files exist for all validators
3. Verify file permissions: `ls -la keys/validators/validator-*/key`

### Database Password Issues
**Error**: `Authentication failed for user`

**Solution**:
1. Verify `DB_PASSWORD` environment variable is set
2. Check password matches in database
3. Verify password doesn't contain special characters that need escaping
4. Check application logs for detailed error messages

---

## 9. References

- **Proxmox API Documentation**: https://pve.proxmox.com/pve-docs/api-viewer/
- **Besu Validator Keys**: https://besu.hyperledger.org/en/stable/Reference/CLI/CLI-Subcommands/#validator-key
- **Environment Variables**: `lib/common.sh` - `load_env_file()` function
- **Configuration**: `config/proxmox.conf`