proxmox/docs/05-network/NGINX_ARCHITECTURE_RPC.md

# Nginx Architecture for RPC Nodes

**Last Updated:** 2025-01-20  
**Document Version:** 1.0  
**Status:** Active Documentation

---

## Overview

There are two different nginx use cases in the RPC architecture:

1. **nginx-proxy-manager (VMID 105)** - Centralized reverse proxy/load balancer
2. **nginx on RPC nodes (2500-2502)** - Local nginx on each RPC container

---

## Current Architecture

### VMID 105: nginx-proxy-manager
- **Purpose**: Centralized reverse proxy management with web UI
- **Status**: Existing container (running)
- **Use Case**: Route traffic to multiple services, SSL termination, load balancing
- **Advantages**:
  - Centralized management via web UI
  - Easy SSL certificate management
  - Can load balance across multiple RPC nodes
  - Single point of configuration

### nginx on RPC Nodes (2500-2502)
- **Purpose**: Local nginx on each RPC container
- **Current Status**: Installed but not necessarily configured
- **Use Case**: SSL termination, local load balancing, rate limiting per node
- **Advantages**:
  - Node-specific configuration
  - Redundancy (each node has its own nginx)
  - Can handle local routing needs

---

## Recommendation: Use VMID 105 for RPC

### ✅ YES - VMID 105 can and should be used for RPC

**Recommended Architecture**:
```
Clients → nginx-proxy-manager (VMID 105) → Besu RPC Nodes (2500-2502:8545)
```

**Benefits**:
1. **Centralized Management**: Single web UI to manage all RPC routing
2. **Type-Based Routing**: Route requests to appropriate RPC node type (Public, Core, Permissioned, etc.)
3. **SSL Termination**: Handle HTTPS at the proxy level
4. **Access Control**: Different access rules per RPC node type
5. **Simplified RPC Nodes**: Remove nginx from RPC nodes (they just run Besu)
6. **Better Monitoring**: Central point to monitor RPC traffic

**Note**: RPC nodes 2500-2502 are **different types**, not redundant instances. Therefore, load balancing/failover between them is NOT appropriate. See `docs/RPC_NODE_TYPES_ARCHITECTURE.md` for details.

---

## Implementation Options

### Option 1: Use VMID 105 Only (Recommended)

**Remove nginx from RPC nodes** and use nginx-proxy-manager exclusively:

**Steps**:
1. Remove nginx package from `install/besu-rpc-install.sh` ✅ **DONE**
2. Configure nginx-proxy-manager (VMID 105) with **separate proxy hosts** for each RPC node type:
   - **Core RPC**: `rpc-core.besu.local` → `192.168.11.250:8545` (VMID 2500)
   - **Permissioned RPC**: `rpc-perm.besu.local` → `192.168.11.251:8545` (VMID 2501)
   - **Public RPC**: `rpc.besu.local` → `192.168.11.252:8545` (VMID 2502)
3. Configure access control per proxy host (public vs internal)
4. Expose appropriate endpoints based on RPC node type

**Important**: Do NOT set up load balancing between these nodes, as they are different types serving different purposes.

**Configuration in nginx-proxy-manager** (separate proxy host per type):

**Public RPC Proxy**:
- **Domain**: `rpc.besu.local` (or `rpc-public.chainid138.local`)
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.250` (Public RPC node)
- **Forward Port**: `8545`
- **Websockets**: Enabled (for WS-RPC on port 8546)
- **Access**: Public (as appropriate for public RPC)

**Core RPC Proxy**:
- **Domain**: `rpc-core.besu.local` (or `rpc-core.chainid138.local`)
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.251` (Core RPC node)
- **Forward Port**: `8545`
- **Websockets**: Enabled
- **Access**: Restricted to internal network IPs

**Permissioned RPC Proxy**:
- **Domain**: `rpc-perm.besu.local` (or `rpc-perm.chainid138.local`)
- **Scheme**: `http`
- **Forward Hostname/IP**: `192.168.11.252` (Permissioned RPC node)
- **Forward Port**: `8545`
- **Websockets**: Enabled
- **Access**: Additional authentication/authorization as needed

---

### Option 2: Hybrid Approach

**Keep both** but use them for different purposes:

- **nginx-proxy-manager (VMID 105)**: 
  - Public-facing entry point
  - SSL termination
  - Load balancing across RPC nodes
  
- **nginx on RPC nodes**: 
  - Optional: Local rate limiting
  - Optional: Node-specific routing
  - Can be used for internal routing within the container

**Use Case**: If you need per-node rate limiting or complex local routing

---

## Configuration Details

### nginx-proxy-manager Configuration (VMID 105)

**Proxy Host Setup**:
1. Access nginx-proxy-manager web UI (typically port 81)
2. Add Proxy Host:
   - **Domain Names**: `rpc.besu.local`, `rpc.chainid138.local` (or your domain)
   - **Scheme**: `http`
   - **Forward Hostname/IP**: Use load balancer with:
     - `192.168.11.250:8545`
     - `192.168.11.251:8545`
     - `192.168.11.252:8545`
   - **Forward Port**: `8545`
   - **Cache Assets**: Disabled (RPC responses shouldn't be cached)
   - **Websockets**: Enabled
   - **Block Common Exploits**: Enabled
   - **SSL**: Configure Let's Encrypt or custom certificate

**Type-Based Routing Configuration**:
Since RPC nodes are different types (not redundant instances), configure **separate proxy hosts** rather than load balancing:

1. **Core RPC Proxy**: Routes to `192.168.11.250:8545` only (VMID 2500)
2. **Permissioned RPC Proxy**: Routes to `192.168.11.251:8545` only (VMID 2501)
3. **Public RPC Proxy**: Routes to `192.168.11.252:8545` only (VMID 2502)

**Health Checks**: Enable health checks for each proxy host to detect if the specific node type is down

**Note**: If you deploy multiple instances of the same type (e.g., 2 Public RPC nodes), THEN you can configure load balancing within that type's proxy host.

**WebSocket Support**:
- Add separate proxy host for WebSocket:
  - **Forward Port**: `8546`
  - **Websockets**: Enabled
  - **Domain**: `rpc-ws.besu.local` (or subdomain)

---

### Removing nginx from RPC Nodes (Option 1)

**Update `install/besu-rpc-install.sh`**:

Remove nginx from apt packages:
```bash
apt-get install -y -qq \
    openjdk-17-jdk \
    wget \
    curl \
    jq \
    netcat-openbsd \
    iproute2 \
    iptables \
    ca-certificates \
    gnupg \
    lsb-release
    # nginx  <-- REMOVE THIS LINE
```

**Update documentation**:
- Remove nginx from `docs/APT_PACKAGES_CHECKLIST.md` for RPC nodes
- Update architecture diagrams to show nginx-proxy-manager as entry point

---

## Network Flow

### Current Flow (with nginx on RPC nodes):
```
Internet → nginx-proxy-manager (VMID 105) → [Optional] nginx on RPC node → Besu (8545)
```

### Recommended Flow (nginx-proxy-manager only):
```
Internet → nginx-proxy-manager (VMID 105) → Besu RPC Node (2500-2502:8545)
```

---

## Verification

### Test RPC through nginx-proxy-manager:
```bash
# Test HTTP RPC
curl -X POST http://rpc.besu.local:8080 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Test WebSocket RPC (if configured)
wscat -c ws://rpc-ws.besu.local:8080
```

### Verify Load Balancing:
```bash
# Check which backend is serving requests
# (nginx-proxy-manager logs will show backend selection)
```

---

## Recommendation Summary

✅ **Use VMID 105 (nginx-proxy-manager) for RPC**

**Benefits**:
- Centralized management
- Load balancing across RPC nodes
- SSL termination
- High availability
- Simplified RPC node configuration

**Action Items**:
1. Remove nginx package from `install/besu-rpc-install.sh` (if going with Option 1)
2. Configure nginx-proxy-manager to proxy to RPC nodes (2500-2502)
3. Update documentation to reflect architecture
4. Test load balancing and failover

---

## Related Documentation

### Network Documents
- **[CENTRAL_NGINX_ROUTING_SETUP.md](CENTRAL_NGINX_ROUTING_SETUP.md)** ⭐⭐⭐ - Central Nginx routing setup
- **[CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md](CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md)** ⭐⭐⭐ - Cloudflare tunnel routing
- **[CLOUDFLARE_NGINX_INTEGRATION.md](CLOUDFLARE_NGINX_INTEGRATION.md)** ⭐⭐ - Cloudflare + NGINX integration
- **[RPC_NODE_TYPES_ARCHITECTURE.md](RPC_NODE_TYPES_ARCHITECTURE.md)** ⭐⭐ - RPC node architecture

### Configuration Documents
- **[../04-configuration/cloudflare/CLOUDFLARE_DNS_TO_CONTAINERS.md](../04-configuration/cloudflare/CLOUDFLARE_DNS_TO_CONTAINERS.md)** - DNS mapping to containers

### External References
- [nginx-proxy-manager](https://nginxproxymanager.com/) - Official documentation

---

**Last Updated:** 2025-01-20  
**Document Version:** 1.0  
**Review Cycle:** Quarterly