d-bis/proxmox
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor code for improved readability and performance

Browse Source
This commit is contained in:
defiQUG
2025-12-21 22:32:09 -08:00
parent 79e3c02f50
commit b45c2006be
2259 changed files with 380318 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

254
docs/05-network/CLOUDFLARE_NGINX_INTEGRATION.md Normal file
View File

@@ -0,0 +1,254 @@
# Cloudflare and Nginx Integration
## Overview
Integration of Cloudflare (via cloudflared tunnel on VMID 102) with nginx-proxy-manager (VMID 105) for routing to RPC nodes.
---
## Architecture
```
Internet → Cloudflare → cloudflared (VMID 102) → nginx-proxy-manager (VMID 105) → RPC Nodes (2500-2502)
```
### Components
1. **Cloudflare** - Global CDN, DDoS protection, SSL termination
2. **cloudflared (VMID 102)** - Cloudflare tunnel client
3. **nginx-proxy-manager (VMID 105)** - Reverse proxy and routing
4. **RPC Nodes (2500-2502)** - Besu RPC endpoints
---
## VMID 102: cloudflared
**Status**: Existing container (running)
**Purpose**: Cloudflare tunnel client
**Configuration**: Routes Cloudflare traffic to nginx-proxy-manager
### Configuration Requirements
The cloudflared tunnel should be configured to route to nginx-proxy-manager (VMID 105):
```yaml
# Example cloudflared config (config.yml)
tunnel: <your-tunnel-id>
credentials-file: /etc/cloudflared/credentials.json
ingress:
# RPC Core
- hostname: rpc-core.yourdomain.com
service: http://192.168.11.105:80 # nginx-proxy-manager
# RPC Permissioned
- hostname: rpc-perm.yourdomain.com
service: http://192.168.11.105:80 # nginx-proxy-manager
# RPC Public
- hostname: rpc.yourdomain.com
service: http://192.168.11.105:80 # nginx-proxy-manager
# Catch-all (optional)
- service: http_status:404
```
---
## VMID 105: nginx-proxy-manager
**Status**: Existing container (running)
**Purpose**: Reverse proxy and routing to RPC nodes
### Proxy Host Configuration
Configure separate proxy hosts for each RPC type:
#### 1. Core RPC Proxy
- **Domain Names**: `rpc-core.yourdomain.com`
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.250`
- **Forward Port**: `8545`
- **Websockets**: Enabled (for WS-RPC on port 8546)
- **SSL**: Handle at Cloudflare level (or configure SSL here)
- **Access**: Restrict to internal network if needed
#### 2. Permissioned RPC Proxy
- **Domain Names**: `rpc-perm.yourdomain.com`
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.251`
- **Forward Port**: `8545`
- **Websockets**: Enabled
- **SSL**: Handle at Cloudflare level
- **Access**: Configure authentication/authorization
#### 3. Public RPC Proxy
- **Domain Names**: `rpc.yourdomain.com`, `rpc-public.yourdomain.com`
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.252`
- **Forward Port**: `8545`
- **Websockets**: Enabled
- **SSL**: Handle at Cloudflare level
- **Cache Assets**: Disabled (RPC responses shouldn't be cached)
- **Block Common Exploits**: Enabled
- **Rate Limiting**: Configure as needed
---
## Network Flow
### Request Flow
1. **Client** makes request to `rpc.yourdomain.com`
2. **Cloudflare** handles DNS, DDoS protection, SSL termination
3. **cloudflared (VMID 102)** receives request via Cloudflare tunnel
4. **nginx-proxy-manager (VMID 105)** receives request from cloudflared
5. **nginx-proxy-manager** routes based on domain to appropriate RPC node:
- `rpc-core.*` → 192.168.11.250:8545 (Core RPC)
- `rpc-perm.*` → 192.168.11.251:8545 (Permissioned RPC)
- `rpc.*` → 192.168.11.252:8545 (Public RPC)
6. **RPC Node** processes request and returns response
### Response Flow (Reverse)
1. **RPC Node** returns response
2. **nginx-proxy-manager** forwards response
3. **cloudflared** forwards to Cloudflare tunnel
4. **Cloudflare** delivers to client
---
## Benefits
1. **DDoS Protection**: Cloudflare provides robust DDoS mitigation
2. **Global CDN**: Faster response times worldwide
3. **SSL/TLS**: Automatic SSL certificate management via Cloudflare
4. **Rate Limiting**: Cloudflare rate limiting + nginx-proxy-manager controls
5. **Centralized Routing**: Single point (nginx-proxy-manager) to manage routing logic
6. **Type-Based Routing**: Clear separation of RPC node types
7. **Security**: Validators remain behind firewall, only RPC nodes exposed
---
## Configuration Checklist
### Cloudflare (Cloudflare Dashboard)
- [ ] Create Cloudflare tunnel
- [ ] Configure DNS records (CNAME) for each RPC type:
- `rpc-core.yourdomain.com` → tunnel
- `rpc-perm.yourdomain.com` → tunnel
- `rpc.yourdomain.com` → tunnel
- [ ] Enable SSL/TLS (Full or Full (strict))
- [ ] Configure DDoS protection rules
- [ ] Set up rate limiting rules (optional)
- [ ] Configure WAF rules (optional)
### cloudflared (VMID 102)
- [ ] Install/configure cloudflared
- [ ] Set up tunnel configuration
- [ ] Configure ingress rules to route to nginx-proxy-manager (192.168.11.105:80)
- [ ] Test tunnel connectivity
- [ ] Enable/start cloudflared service
### nginx-proxy-manager (VMID 105)
- [ ] Access web UI (typically port 81)
- [ ] Create proxy host for Core RPC (rpc-core.* → 192.168.11.250:8545)
- [ ] Create proxy host for Permissioned RPC (rpc-perm.* → 192.168.11.251:8545)
- [ ] Create proxy host for Public RPC (rpc.* → 192.168.11.252:8545)
- [ ] Enable WebSocket support for all proxy hosts
- [ ] Configure access control/authentication for Permissioned RPC
- [ ] Configure rate limiting for Public RPC (optional)
- [ ] Test routing to each RPC node
### RPC Nodes (2500-2502)
- [ ] Ensure RPC nodes are running and accessible
- [ ] Verify RPC endpoints respond on ports 8545/8546
- [ ] Test direct access to each RPC node
- [ ] Verify correct config files are deployed:
- 2500: `config-rpc-core.toml`
- 2501: `config-rpc-perm.toml`
- 2502: `config-rpc-public.toml`
---
## Testing
### Test Direct RPC Access
```bash
# Test Core RPC
curl -X POST http://192.168.11.250:8545 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
# Test Permissioned RPC
curl -X POST http://192.168.11.251:8545 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
# Test Public RPC
curl -X POST http://192.168.11.252:8545 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```
### Test Through nginx-proxy-manager
```bash
# Test Core RPC via nginx-proxy-manager
curl -X POST http://192.168.11.105/rpc-core \
-H "Host: rpc-core.yourdomain.com" \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```
### Test Through Cloudflare
```bash
# Test Public RPC via Cloudflare
curl -X POST https://rpc.yourdomain.com \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
```
---
## Security Considerations
1. **SSL/TLS**: Cloudflare handles SSL termination (Full mode recommended)
2. **Access Control**:
- Core RPC: Restrict to internal network IPs
- Permissioned RPC: Require authentication/authorization
- Public RPC: Rate limiting and DDoS protection
3. **Firewall Rules**: Ensure only necessary ports are exposed
4. **Rate Limiting**: Configure at both Cloudflare and nginx-proxy-manager levels
5. **WAF**: Enable Cloudflare WAF for additional protection
---
## Troubleshooting
### Cloudflare Tunnel Not Connecting
- Check cloudflared service status: `systemctl status cloudflared`
- Verify tunnel configuration: `cloudflared tunnel info`
- Check Cloudflare dashboard for tunnel status
- Verify network connectivity from VMID 102 to VMID 105
### nginx-proxy-manager Not Routing
- Check proxy host configuration in web UI
- Verify domain names match Cloudflare DNS records
- Check nginx-proxy-manager logs
- Test direct connection to RPC nodes
### RPC Nodes Not Responding
- Check Besu service status: `systemctl status besu-rpc`
- Verify RPC endpoints are enabled in config files
- Check firewall rules on RPC nodes
- Test direct connection from nginx-proxy-manager to RPC nodes
---
## References
- **Cloudflare Tunnels**: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/
- **nginx-proxy-manager**: https://nginxproxymanager.com/
- **RPC Node Types**: `docs/RPC_NODE_TYPES_ARCHITECTURE.md`
- **Nginx Architecture**: `docs/NGINX_ARCHITECTURE_RPC.md`

128
docs/05-network/NETWORK_STATUS.md Normal file
View File

@@ -0,0 +1,128 @@
# Network Status Report
**Date**: 2025-12-20
**Network**: Chain ID 138 (QBFT Consensus)
**Status**: ✅ OPERATIONAL
---
## Executive Summary
The network is **fully operational** and producing blocks. The root cause issue (ethash conflicting with QBFT in genesis.json) has been resolved.
---
## 1. Block Production
- **Current Block Height**: Blocks 83-85 (actively increasing)
- **Block Period**: ~2 seconds (as configured)
- **Status**: ✅ Blocks are being produced consistently
### Block Production by Node
- VMID 1000 (validator-1): Block 83+
- VMID 1001 (validator-2): Block 84+
- VMID 1002 (validator-3): Block 85+
---
## 2. Validator Recognition
- **Total Validators**: 5
- **Status**: ✅ All validators recognized by QBFT consensus
### Validator Addresses (from QBFT)
1. `0x1c25c54bf177ecf9365445706d8b9209e8f1c39b` (VMID 1000)
2. `0xc4c1aeeb5ab86c6179fc98220b51844b74935446` (VMID 1001)
3. `0x22f37f6faaa353e652a0840f485e71a7e5a89373` (VMID 1002)
4. `0x573ff6d00d2bdc0d9c0c08615dc052db75f82574` (VMID 1003)
5. `0x11563e26a70ed3605b80a03081be52aca9e0f141` (VMID 1004)
---
## 3. Service Status
### Validators (5 nodes)
- VMID 1000 (besu-validator-1): ✅ active
- VMID 1001 (besu-validator-2): ✅ active
- VMID 1002 (besu-validator-3): ✅ active
- VMID 1003 (besu-validator-4): ✅ active
- VMID 1004 (besu-validator-5): ✅ active
### Sentries (4 nodes)
- VMID 1500 (besu-sentry-1): ✅ active
- VMID 1501 (besu-sentry-2): ✅ active
- VMID 1502 (besu-sentry-3): ✅ active
- VMID 1503 (besu-sentry-4): ✅ active
### RPC Nodes (3 nodes)
- VMID 2500 (besu-rpc-1): ✅ active
- VMID 2501 (besu-rpc-2): ✅ active
- VMID 2502 (besu-rpc-3): ✅ active
**Total Nodes**: 12 (5 validators + 4 sentries + 3 RPC)
---
## 4. Network Connectivity
- **Peer Connections**: All validators showing healthy peer counts (10+ peers)
- **Status**: ✅ Network topology is functioning correctly
---
## 5. Consensus Configuration
- **Consensus Algorithm**: QBFT (Quorum Byzantine Fault Tolerance)
- **Block Period**: 2 seconds
- **Epoch Length**: 30,000 blocks
- **Request Timeout**: 10 seconds
- **Status**: ✅ QBFT consensus is active and functioning
---
## 6. Recent Changes Applied
### Critical Fix Applied
- **Issue**: Genesis file contained both `ethash: {}` and `qbft: {...}`, causing Besu to default to ethash instead of QBFT
- **Solution**: Removed `ethash: {}` from genesis.json config
- **Result**: QBFT consensus now active, validators recognized, blocks being produced
### Previous Fixes
1. ✅ Key rotation completed (all validator and node keys regenerated)
2. ✅ Configuration files updated (removed deprecated options)
3. ✅ RPC enabled on validators (with QBFT API)
4. ✅ Permissioning configured correctly
5. ✅ Static nodes and permissioned nodes files updated
---
## 7. Network Health
### Overall Status: 🟢 HEALTHY
- ✅ All services running
- ✅ Validators recognized and producing blocks
- ✅ Blocks being produced consistently
- ✅ Network connectivity operational
- ✅ Consensus functioning correctly
---
## Next Steps / Recommendations
1. **Monitor Block Production**: Continue monitoring to ensure consistent block production
2. **Monitor Validator Participation**: Ensure all 5 validators continue to participate
3. **Network Metrics**: Consider setting up metrics collection for long-term monitoring
4. **Backup Configuration**: Archive the working genesis.json and key configurations
---
## Troubleshooting History
This network has been successfully restored from a state where:
- Validators were not recognized
- Blocks were not being produced
- Consensus was defaulting to ethash instead of QBFT
All issues have been resolved through systematic troubleshooting and configuration fixes.

242
docs/05-network/NGINX_ARCHITECTURE_RPC.md Normal file
View File

@@ -0,0 +1,242 @@
# Nginx Architecture for RPC Nodes
## Overview
There are two different nginx use cases in the RPC architecture:
1. **nginx-proxy-manager (VMID 105)** - Centralized reverse proxy/load balancer
2. **nginx on RPC nodes (2500-2502)** - Local nginx on each RPC container
---
## Current Architecture
### VMID 105: nginx-proxy-manager
- **Purpose**: Centralized reverse proxy management with web UI
- **Status**: Existing container (running)
- **Use Case**: Route traffic to multiple services, SSL termination, load balancing
- **Advantages**:
- Centralized management via web UI
- Easy SSL certificate management
- Can load balance across multiple RPC nodes
- Single point of configuration
### nginx on RPC Nodes (2500-2502)
- **Purpose**: Local nginx on each RPC container
- **Current Status**: Installed but not necessarily configured
- **Use Case**: SSL termination, local load balancing, rate limiting per node
- **Advantages**:
- Node-specific configuration
- Redundancy (each node has its own nginx)
- Can handle local routing needs
---
## Recommendation: Use VMID 105 for RPC
### ✅ YES - VMID 105 can and should be used for RPC
**Recommended Architecture**:
```
Clients → nginx-proxy-manager (VMID 105) → Besu RPC Nodes (2500-2502:8545)
```
**Benefits**:
1. **Centralized Management**: Single web UI to manage all RPC routing
2. **Type-Based Routing**: Route requests to appropriate RPC node type (Public, Core, Permissioned, etc.)
3. **SSL Termination**: Handle HTTPS at the proxy level
4. **Access Control**: Different access rules per RPC node type
5. **Simplified RPC Nodes**: Remove nginx from RPC nodes (they just run Besu)
6. **Better Monitoring**: Central point to monitor RPC traffic
**Note**: RPC nodes 2500-2502 are **different types**, not redundant instances. Therefore, load balancing/failover between them is NOT appropriate. See `docs/RPC_NODE_TYPES_ARCHITECTURE.md` for details.
---
## Implementation Options
### Option 1: Use VMID 105 Only (Recommended)
**Remove nginx from RPC nodes** and use nginx-proxy-manager exclusively:
**Steps**:
1. Remove nginx package from `install/besu-rpc-install.sh` ✅ **DONE**
2. Configure nginx-proxy-manager (VMID 105) with **separate proxy hosts** for each RPC node type:
- **Core RPC**: `rpc-core.besu.local` → `192.168.11.250:8545` (VMID 2500)
- **Permissioned RPC**: `rpc-perm.besu.local` → `192.168.11.251:8545` (VMID 2501)
- **Public RPC**: `rpc.besu.local` → `192.168.11.252:8545` (VMID 2502)
3. Configure access control per proxy host (public vs internal)
4. Expose appropriate endpoints based on RPC node type
**Important**: Do NOT set up load balancing between these nodes, as they are different types serving different purposes.
**Configuration in nginx-proxy-manager** (separate proxy host per type):
**Public RPC Proxy**:
- **Domain**: `rpc.besu.local` (or `rpc-public.chainid138.local`)
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.250` (Public RPC node)
- **Forward Port**: `8545`
- **Websockets**: Enabled (for WS-RPC on port 8546)
- **Access**: Public (as appropriate for public RPC)
**Core RPC Proxy**:
- **Domain**: `rpc-core.besu.local` (or `rpc-core.chainid138.local`)
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.251` (Core RPC node)
- **Forward Port**: `8545`
- **Websockets**: Enabled
- **Access**: Restricted to internal network IPs
**Permissioned RPC Proxy**:
- **Domain**: `rpc-perm.besu.local` (or `rpc-perm.chainid138.local`)
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.252` (Permissioned RPC node)
- **Forward Port**: `8545`
- **Websockets**: Enabled
- **Access**: Additional authentication/authorization as needed
---
### Option 2: Hybrid Approach
**Keep both** but use them for different purposes:
- **nginx-proxy-manager (VMID 105)**:
- Public-facing entry point
- SSL termination
- Load balancing across RPC nodes
- **nginx on RPC nodes**:
- Optional: Local rate limiting
- Optional: Node-specific routing
- Can be used for internal routing within the container
**Use Case**: If you need per-node rate limiting or complex local routing
---
## Configuration Details
### nginx-proxy-manager Configuration (VMID 105)
**Proxy Host Setup**:
1. Access nginx-proxy-manager web UI (typically port 81)
2. Add Proxy Host:
- **Domain Names**: `rpc.besu.local`, `rpc.chainid138.local` (or your domain)
- **Scheme**: `http`
- **Forward Hostname/IP**: Use load balancer with:
- `192.168.11.250:8545`
- `192.168.11.251:8545`
- `192.168.11.252:8545`
- **Forward Port**: `8545`
- **Cache Assets**: Disabled (RPC responses shouldn't be cached)
- **Websockets**: Enabled
- **Block Common Exploits**: Enabled
- **SSL**: Configure Let's Encrypt or custom certificate
**Type-Based Routing Configuration**:
Since RPC nodes are different types (not redundant instances), configure **separate proxy hosts** rather than load balancing:
1. **Core RPC Proxy**: Routes to `192.168.11.250:8545` only (VMID 2500)
2. **Permissioned RPC Proxy**: Routes to `192.168.11.251:8545` only (VMID 2501)
3. **Public RPC Proxy**: Routes to `192.168.11.252:8545` only (VMID 2502)
**Health Checks**: Enable health checks for each proxy host to detect if the specific node type is down
**Note**: If you deploy multiple instances of the same type (e.g., 2 Public RPC nodes), THEN you can configure load balancing within that type's proxy host.
**WebSocket Support**:
- Add separate proxy host for WebSocket:
- **Forward Port**: `8546`
- **Websockets**: Enabled
- **Domain**: `rpc-ws.besu.local` (or subdomain)
---
### Removing nginx from RPC Nodes (Option 1)
**Update `install/besu-rpc-install.sh`**:
Remove nginx from apt packages:
```bash
apt-get install -y -qq \
openjdk-17-jdk \
wget \
curl \
jq \
netcat-openbsd \
iproute2 \
iptables \
ca-certificates \
gnupg \
lsb-release
# nginx <-- REMOVE THIS LINE
```
**Update documentation**:
- Remove nginx from `docs/APT_PACKAGES_CHECKLIST.md` for RPC nodes
- Update architecture diagrams to show nginx-proxy-manager as entry point
---
## Network Flow
### Current Flow (with nginx on RPC nodes):
```
Internet → nginx-proxy-manager (VMID 105) → [Optional] nginx on RPC node → Besu (8545)
```
### Recommended Flow (nginx-proxy-manager only):
```
Internet → nginx-proxy-manager (VMID 105) → Besu RPC Node (2500-2502:8545)
```
---
## Verification
### Test RPC through nginx-proxy-manager:
```bash
# Test HTTP RPC
curl -X POST http://rpc.besu.local:8080 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
# Test WebSocket RPC (if configured)
wscat -c ws://rpc-ws.besu.local:8080
```
### Verify Load Balancing:
```bash
# Check which backend is serving requests
# (nginx-proxy-manager logs will show backend selection)
```
---
## Recommendation Summary
✅ **Use VMID 105 (nginx-proxy-manager) for RPC**
**Benefits**:
- Centralized management
- Load balancing across RPC nodes
- SSL termination
- High availability
- Simplified RPC node configuration
**Action Items**:
1. Remove nginx package from `install/besu-rpc-install.sh` (if going with Option 1)
2. Configure nginx-proxy-manager to proxy to RPC nodes (2500-2502)
3. Update documentation to reflect architecture
4. Test load balancing and failover
---
## References
- **nginx-proxy-manager**: https://nginxproxymanager.com/
- **Besu RPC Configuration**: `install/besu-rpc-install.sh`
- **Network Configuration**: `config/network.conf`

25
docs/05-network/README.md Normal file
View File

@@ -0,0 +1,25 @@
# Network Infrastructure
This directory contains network infrastructure documentation.
## Documents
- **[NETWORK_STATUS.md](NETWORK_STATUS.md)** ⭐⭐ - Current network status and configuration
- **[NGINX_ARCHITECTURE_RPC.md](NGINX_ARCHITECTURE_RPC.md)** ⭐ - NGINX RPC architecture
- **[CLOUDFLARE_NGINX_INTEGRATION.md](CLOUDFLARE_NGINX_INTEGRATION.md)** ⭐ - Cloudflare + NGINX integration
- **[RPC_NODE_TYPES_ARCHITECTURE.md](RPC_NODE_TYPES_ARCHITECTURE.md)** ⭐ - RPC node architecture
- **[RPC_TEMPLATE_TYPES.md](RPC_TEMPLATE_TYPES.md)** ⭐ - RPC template types
## Quick Reference
**Network Components:**
- NGINX RPC architecture and configuration
- Cloudflare + NGINX integration
- RPC node types and templates
## Related Documentation
- **[../02-architecture/NETWORK_ARCHITECTURE.md](../02-architecture/NETWORK_ARCHITECTURE.md)** - Complete network architecture
- **[../04-configuration/ER605_ROUTER_CONFIGURATION.md](../04-configuration/ER605_ROUTER_CONFIGURATION.md)** - Router configuration
- **[../04-configuration/CLOUDFLARE_ZERO_TRUST_GUIDE.md](../04-configuration/CLOUDFLARE_ZERO_TRUST_GUIDE.md)** - Cloudflare setup

219
docs/05-network/RPC_NODE_TYPES_ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,219 @@
# RPC Node Types Architecture
## Overview
RPC nodes 2500-2502 represent **different types** of RPC nodes, not redundant instances of the same type. Each node serves a specific purpose and cannot be used as a failover for another type.
---
## RPC Node Types
### Type 1: Public RPC Node (`config-rpc-public.toml`)
- **Purpose**: Public-facing RPC endpoints for dApps and external users
- **APIs**: ETH, NET, WEB3 (read-only)
- **Access**: Public (CORS enabled, host allowlist: "*")
- **Use Cases**:
- Public dApp connections
- Blockchain explorers
- External tooling access
- General-purpose RPC queries
### Type 2: Core RPC Node (`config-rpc-core.toml`)
- **Purpose**: Internal/core infrastructure RPC endpoints
- **APIs**: May include ADMIN, DEBUG (if needed)
- **Access**: Restricted (internal network only)
- **Use Cases**:
- Internal service connections
- Core infrastructure tooling
- Administrative operations
- Restricted API access
### Type 3: Permissioned RPC Node (`config-rpc-perm.toml`)
- **Purpose**: Permissioned RPC with account-level access control
- **APIs**: Custom based on permissions
- **Access**: Permissioned (account-based allowlist)
- **Use Cases**:
- Enterprise/private access
- Permissioned dApps
- Controlled API access
### Type 4/5: (Additional types as defined in your source project)
- **Purpose**: Additional specialized RPC node types
- **Use Cases**: Depends on specific requirements
---
## Current Deployment (2500-2502)
**RPC Node Type Mapping**:
| VMID | IP Address | Node Type | Config File | Purpose |
|------|------------|-----------|-------------|---------|
| 2500 | 192.168.11.250 | **Core** | `config-rpc-core.toml` | Internal/core infrastructure RPC endpoints |
| 2501 | 192.168.11.251 | **Permissioned** | `config-rpc-perm.toml` | Permissioned RPC (Requires Auth, select APIs) |
| 2502 | 192.168.11.252 | **Public** | `config-rpc-public.toml` | Public RPC (none or minimal APIs) |
**Notes**:
- These are 3 of 4 or 5 total RPC node types
- Additional RPC nodes will be added later for load balancing and High Availability/Failover
- Each type serves a distinct purpose and cannot substitute for another type
---
## nginx-proxy-manager Architecture (Corrected)
Since these are **different types**, not redundant instances, nginx-proxy-manager should route based on **request type/purpose**, not load balance:
### Recommended Architecture
```
Public Requests → nginx-proxy-manager → Public RPC Node (2502:8545)
Core/Internal Requests → nginx-proxy-manager → Core RPC Node (2500:8545)
Permissioned Requests → nginx-proxy-manager → Permissioned RPC Node (2501:8545)
```
**With Cloudflare Integration (VMID 102: cloudflared)**:
```
Internet → Cloudflare → cloudflared (VMID 102) → nginx-proxy-manager (VMID 105) → RPC Nodes
```
### nginx-proxy-manager Configuration
**Separate Proxy Hosts for Each Type**:
1. **Core RPC Proxy** (VMID 2500):
- Domain: `rpc-core.besu.local` or `rpc-core.chainid138.local`
- Forward to: `192.168.11.250:8545` (Core RPC node)
- Purpose: Internal/core infrastructure RPC endpoints
- Access: Restrict to internal network IPs
- APIs: Full APIs (ADMIN, DEBUG, ETH, NET, WEB3, etc.)
2. **Permissioned RPC Proxy** (VMID 2501):
- Domain: `rpc-perm.besu.local` or `rpc-perm.chainid138.local`
- Forward to: `192.168.11.251:8545` (Permissioned RPC node)
- Purpose: Permissioned RPC (Requires Auth, select APIs)
- Access: Authentication/authorization required
- APIs: Select APIs based on permissions
3. **Public RPC Proxy** (VMID 2502):
- Domain: `rpc.besu.local` or `rpc-public.chainid138.local`
- Forward to: `192.168.11.252:8545` (Public RPC node)
- Purpose: Public RPC (none or minimal APIs)
- Access: Public (with rate limiting recommended)
- APIs: Minimal APIs (ETH, NET, WEB3 - read-only)
**Cloudflare Integration** (VMID 102: cloudflared):
- Cloudflare tunnels route through cloudflared (VMID 102) to nginx-proxy-manager (VMID 105)
- Provides DDoS protection, SSL termination, and global CDN
- See `docs/CLOUDFLARE_NGINX_INTEGRATION.md` for configuration details
---
## High Availability Considerations
### ❌ NO Failover Between Types
You **cannot** failover from one type to another because:
- Different APIs exposed
- Different access controls
- Different use cases
- Clients expect specific functionality
### ✅ HA Options (If Needed)
**Option 1: Deploy Multiple Instances of Same Type**
- If you need HA for Public RPC, deploy multiple Public RPC nodes (e.g., 2500, 2503)
- Then nginx-proxy-manager can load balance between them
- Same for Core RPC (2501, 2504) and Permissioned RPC (2502, 2505)
**Option 2: Accept Single-Instance Risk**
- For non-critical types, accept single instance
- Only deploy HA for critical types (e.g., Public RPC)
**Option 3: Different VMID Ranges for Same Types**
- Public RPC: 2500-2502 (if all 3 are public)
- Core RPC: 2503-2504 (2 instances)
- Permissioned RPC: 2505 (1 instance)
---
## Future Expansion
**Additional RPC Nodes for HA/Load Balancing**:
- Additional instances of existing types (Core, Permissioned, Public) will be deployed
- Load balancing and failover will be configured within each type
- VMID ranges: 2503+ (within the 2500-3499 RPC range)
**Example Future Configuration**:
- Core RPC: 2500, 2503, 2504 (3 instances for HA)
- Permissioned RPC: 2501, 2505 (2 instances for HA)
- Public RPC: 2502, 2506, 2507 (3 instances for HA/load distribution)
---
## Updated Recommendation
### If RPC Nodes 2500-2502 are Different Types:
**nginx-proxy-manager should route by type**, not load balance:
1. **Configure separate proxy hosts** for each type
2. **Route requests based on domain/subdomain** to appropriate node
3. **No load balancing** (since they're different types)
4. **SSL termination** for all types
5. **Access control** based on type (internal vs public)
### Benefits:
- ✅ Proper routing to correct node type
- ✅ SSL termination
- ✅ Centralized management
- ✅ Access control per type
- ✅ Clear separation of concerns
### NOT Appropriate:
- ❌ Load balancing across different types
- ❌ Failover from one type to another
- ❌ Treating them as redundant instances
---
## Next Steps
1. ✅ **RPC node types identified**:
- 2500 → Core (`config-rpc-core.toml`)
- 2501 → Permissioned (`config-rpc-perm.toml`)
- 2502 → Public (`config-rpc-public.toml`)
2. **Update deployment scripts**: Ensure each node gets the correct config file type
- Update `scripts/copy-besu-config-with-nodes.sh` to map VMID to correct config file
- Ensure node-specific configs in `config/nodes/rpc-*/` are properly identified
3. **Configure nginx-proxy-manager (VMID 105)**: Set up type-based routing
- Core RPC: `rpc-core.*` → 192.168.11.250:8545
- Permissioned RPC: `rpc-perm.*` → 192.168.11.251:8545
- Public RPC: `rpc.*` or `rpc-public.*` → 192.168.11.252:8545
4. **Configure Cloudflare Integration**: Set up cloudflared (VMID 102) to route through nginx-proxy-manager
- See `docs/CLOUDFLARE_NGINX_INTEGRATION.md` for details
---
## Script Updates Required
### Updated: `scripts/copy-besu-config-with-nodes.sh`
The script has been updated to map each VMID to its specific RPC type and config file:
```bash
# RPC Node Type Mapping
2500 → core → config-rpc-core.toml
2501 → perm → config-rpc-perm.toml
2502 → public → config-rpc-public.toml
```
**File Detection Priority** (for each RPC node):
1. Node-specific config: `config/nodes/rpc-N/config.toml` (if nodes/ structure exists)
2. Node-specific type config: `config/nodes/rpc-N/config-rpc-{type}.toml`
3. Flat structure: `config/config-rpc-{type}.toml`
4. Fallback (backwards compatibility): May use alternative config if exact type not found

228
docs/05-network/RPC_TEMPLATE_TYPES.md Normal file
View File

@@ -0,0 +1,228 @@
# RPC Template Types Reference
This document describes the different RPC configuration template types used in the deployment.
## RPC Template Types
### 1. `config-rpc-public.toml` (Primary)
**Location**:
- Source: `config/config-rpc-public.toml` (in source project)
- Destination: `/etc/besu/config-rpc-public.toml` (on RPC nodes)
**Purpose**: Public-facing RPC node configuration with full RPC API access
**Characteristics**:
- HTTP RPC enabled on port 8545
- WebSocket RPC enabled on port 8546
- Public API access (CORS enabled, host allowlist: "*")
- Read-only APIs: `ETH`, `NET`, `WEB3`
- Metrics enabled on port 9545
- Full sync mode
- Discovery enabled
- P2P enabled on port 30303
**Used For**:
- Public RPC endpoints
- dApp connections
- External tooling access
- Blockchain explorers
**Scripts That Use It**:
- `besu-rpc-install.sh` - Creates template at installation
- `copy-besu-config.sh` - Copies from source project (primary)
- `copy-besu-config-with-nodes.sh` - Copies from source project or nodes/ directories
---
### 2. `config-rpc-core.toml` (Alternative/Fallback)
**Location**:
- Source: `config/config-rpc-core.toml` (in source project)
- Destination: `/etc/besu/config-rpc-public.toml` (on RPC nodes - renamed during copy)
**Purpose**: Alternative RPC configuration, typically with more restricted access
**Characteristics**:
- Similar to `config-rpc-public.toml` but may have different security settings
- Used as fallback if `config-rpc-public.toml` is not found
- Renamed to `config-rpc-public.toml` when copied to containers
**Used For**:
- Internal RPC nodes with restricted access
- Core infrastructure RPC endpoints
- Alternative configuration option
**Scripts That Use It**:
- `copy-besu-config.sh` - Fallback if `config-rpc-public.toml` not found
- `copy-besu-config-with-nodes.sh` - Checks both types
---
### 2b. `config-rpc-perm.toml` (Permissioned RPC)
**Location**:
- Source: `config/config-rpc-perm.toml` (in source project)
- Destination: Not currently used in deployment scripts (would need to be manually copied)
**Purpose**: Permissioned RPC configuration with account permissioning enabled
**Characteristics**:
- May have account permissioning enabled
- Different access controls than public RPC
- Currently not automatically deployed by scripts
**Used For**:
- Permissioned RPC endpoints
- Account-restricted access
- Enhanced security configurations
**Scripts That Use It**:
- Currently not used in deployment scripts
- Available in source project for manual configuration if needed
**Note**: This file exists in the source project but is not currently integrated into the deployment scripts. To use it, you would need to manually copy it or modify the deployment scripts.
---
### 3. Template from Install Script (Fallback)
**Location**:
- Source: Created by `besu-rpc-install.sh` at `/etc/besu/config-rpc-public.toml.template`
- Destination: `/etc/besu/config-rpc-public.toml` (copied if no source config found)
**Purpose**: Default template created during Besu installation
**Characteristics**:
- Basic RPC configuration
- Public access enabled
- Full API access
- Created automatically during installation
**Used For**:
- Fallback if no source configuration is provided
- Initial setup before configuration copy
**Scripts That Use It**:
- `besu-rpc-install.sh` - Creates the template
- `copy-besu-config.sh` - Uses as last resort fallback
---
## Template Selection Priority
The deployment scripts use the following priority order:
1. **Primary**: `config/config-rpc-public.toml` from source project
2. **Alternative**: `config/config-rpc-core.toml` from source project (renamed to `config-rpc-public.toml`)
3. **Node-Specific**: `config/nodes/rpc-*/config.toml` (if using nodes/ structure)
4. **Fallback**: Template from install script (`config-rpc-public.toml.template`)
**Note**: `config-rpc-perm.toml` exists in the source project but is **not currently used** by deployment scripts. It's available for manual configuration if permissioned RPC is needed.
---
## Script Behavior
### `copy-besu-config.sh`
```bash
# Priority 1: config-rpc-public.toml
RPC_CONFIG="$SOURCE_PROJECT/config/config-rpc-public.toml"
# Priority 2: config-rpc-core.toml (fallback)
if not found:
RPC_CONFIG="$SOURCE_PROJECT/config/config-rpc-core.toml"
# Copies as config-rpc-public.toml
# Priority 3: Install script template (last resort)
if not found:
pct exec "$vmid" -- cp /etc/besu/config-validator.toml.template /etc/besu/config-rpc-public.toml
```
### `copy-besu-config-with-nodes.sh`
```bash
# For each RPC node:
# Priority 1: config/nodes/rpc-*/config.toml (if nodes/ structure exists)
# Priority 2: config/config-rpc-public.toml
# Priority 3: config/config-rpc-core.toml
for name in "config-rpc-public.toml" "config-rpc-core.toml"; do
# Try to find in nodes/ directory or flat structure
done
```
---
## Configuration Differences
### `config-rpc-public.toml` (Typical)
```toml
# Public RPC Configuration
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["ETH","NET","WEB3"]
rpc-http-cors-origins=["*"]
rpc-http-host-allowlist=["*"]
rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
rpc-ws-api=["ETH","NET","WEB3"]
rpc-ws-origins=["*"]
```
### `config-rpc-core.toml` (Typical)
```toml
# Core/Internal RPC Configuration
# May have:
# - Restricted host allowlist
# - Additional APIs enabled (ADMIN, DEBUG, etc.)
# - Different security settings
# - Internal network access only
```
---
## Location Summary
| Template Type | Source Location | Container Location | Priority | Status |
|--------------|----------------|-------------------|----------|--------|
| `config-rpc-public.toml` | `config/config-rpc-public.toml` | `/etc/besu/config-rpc-public.toml` | 1 | ✅ Active |
| `config-rpc-core.toml` | `config/config-rpc-core.toml` | `/etc/besu/config-rpc-public.toml` | 2 | ✅ Active (fallback) |
| `config-rpc-perm.toml` | `config/config-rpc-perm.toml` | (Manual copy) | N/A | ⚠️ Available but not used |
| Node-specific | `config/nodes/rpc-*/config.toml` | `/etc/besu/config-rpc-public.toml` | 1 (if nodes/ exists) | ✅ Active |
| Install template | Created by install script | `/etc/besu/config-rpc-public.toml.template` | 3 | ✅ Fallback |
---
## Validation
The comprehensive validation script (`validate-deployment-comprehensive.sh`) checks that:
- RPC nodes (2500-2502) have type-specific config files:
- VMID 2500: `config-rpc-core.toml`
- VMID 2501: `config-rpc-perm.toml`
- VMID 2502: `config-rpc-public.toml`
- No incorrect config files exist on RPC nodes (e.g., validator or sentry configs)
---
## Current Usage
**Active Configuration**:
- All RPC nodes (2500-2502) use type-specific config files (see `docs/RPC_NODE_TYPES_ARCHITECTURE.md`)
- Scripts check for both `config-rpc-public.toml` and `config-rpc-core.toml` from source project
- If neither exists, uses install script template as fallback
**Recommended**:
- Use `config-rpc-public.toml` from source project
- `config-rpc-core.toml` is available as alternative if needed
- Both are copied as `config-rpc-public.toml` to containers
---
**Last Updated**: $(date)