# Parallel Execution Limits

**Last Updated**: 2025-01-11  
**Purpose**: Document recommended parallel execution limits for deployment operations

---

## Overview

Deployment operations can be parallelized to reduce total deployment time, but must respect Proxmox host resources to prevent exhaustion and instability.

---

## Recommended Limits

### Container Creation

**Operation**: Creating LXC containers  
**Recommended**: **10 containers simultaneously**  
**Rationale**:
- Container creation is I/O intensive
- Proxmox can handle ~10 concurrent container creations
- More than 10 can cause resource contention

**Configuration**:
```bash
# In proxmox.conf
MAX_PARALLEL_CREATE=10
```

---

### Package Installation

**Operation**: Installing packages (apt, npm, etc.)  
**Recommended**: **8 containers simultaneously**  
**Rationale**:
- Package installation is CPU and RAM intensive
- Downloads can saturate network
- Compilation (if any) requires CPU
- 8 containers balances speed vs. resource usage

**Configuration**:
```bash
# In proxmox.conf
MAX_PARALLEL_PACKAGES=8
```

**CCIP Nodes**: Use **8 nodes per batch** (16 nodes = 2 batches)
- CCIP nodes have complex dependencies
- Chainlink installation is resource-intensive
- Batching prevents resource exhaustion

---

### Service Startup

**Operation**: Starting services (systemd, Docker, etc.)  
**Recommended**: **15 containers simultaneously**  
**Rationale**:
- Service startup is less resource-intensive
- Mostly I/O and network operations
- Can handle more concurrent startups
- Services initialize independently

**Configuration**:
```bash
# In proxmox.conf
MAX_PARALLEL_START=15
```

---

### OS Template Installation

**Operation**: Installing OS templates to containers  
**Recommended**: **10-20 containers simultaneously**  
**Rationale**:
- Template installation is I/O intensive
- Can parallelize more than creation
- Depends on storage performance
- 10-20 is safe for most systems

**Configuration**:
```bash
# In proxmox.conf
MAX_PARALLEL_TEMPLATE=15
```

---

## Service-Specific Recommendations

### Besu Nodes

**Container Creation**: 5 validators + 4 sentries + 3 RPC = 12 containers
- Can create all 12 in parallel (within 10 limit)
- Or create by type: 5 validators → 4 sentries → 3 RPC

**Package Installation**: Can install on all 12 in parallel (within 8 limit)

**Service Startup**: Can start all 12 in parallel (within 15 limit)

---

### CCIP Nodes (41-43 containers)

**Container Creation**: Batch by type
- CCIP-OPS (2): Parallel ✓
- CCIP-MON (2): Parallel ✓
- CCIP-COMMIT (16): 8 per batch, 2 batches
- CCIP-EXEC (16): 8 per batch, 2 batches
- CCIP-RMN (5-7): All parallel ✓

**Package Installation**: Same batching strategy
- Install packages in same batches as creation
- 8 nodes per batch prevents resource exhaustion

**Service Startup**: Can start more in parallel
- CCIP-OPS (2): Parallel
- CCIP-MON (2): Parallel
- CCIP-COMMIT (16): 8 per batch or all parallel
- CCIP-EXEC (16): 8 per batch or all parallel
- CCIP-RMN (5-7): All parallel

---

### Other Services (14 containers)

**Container Creation**: Can create all in parallel (within 10 limit)
- Or create by type for organization

**Package Installation**: Varies by service complexity
- Simple services (Besu services): Can install in parallel
- Complex services (Blockscout, Fabric): Install sequentially or 2-3 at a time

**Service Startup**: Can start most in parallel
- Database services (Blockscout, Firefly) may need sequential startup
- Other services can start in parallel

---

## Resource Considerations

### CPU

**Impact**: Package installation, compilation, service startup  
**Limiting Factor**: Number of CPU cores  
**Recommendation**: 
- For 8-core host: MAX_PARALLEL_PACKAGES=6
- For 16-core host: MAX_PARALLEL_PACKAGES=8
- For 32+ core host: MAX_PARALLEL_PACKAGES=12

---

### RAM

**Impact**: Package installation, service startup, container overhead  
**Limiting Factor**: Available RAM  
**Recommendation**:
- Ensure sufficient RAM for parallel operations
- Each container uses RAM for OS + packages + services
- CCIP nodes may need more RAM during installation

---

### Storage I/O

**Impact**: Container creation, template installation, package installation  
**Limiting Factor**: Storage type and performance  
**Recommendation**:
- Local SSD: Can handle higher parallelism
- Network storage: Lower parallelism recommended
- HDD: Even lower parallelism

---

### Network Bandwidth

**Impact**: Package downloads, file transfers  
**Limiting Factor**: Network connection speed  
**Recommendation**:
- 1 Gbps: Can handle full parallelism
- 100 Mbps: Reduce parallelism by 50%
- Slower: Reduce further or use local package mirror

---

## Configuration Example

```bash
# In proxmox.conf

# Parallel deployment enabled
PARALLEL_DEPLOY=true

# General limits
MAX_PARALLEL=10              # Default for operations without specific limit

# Operation-specific limits
MAX_PARALLEL_CREATE=10       # Container creation
MAX_PARALLEL_PACKAGES=8      # Package installation (CPU/RAM intensive)
MAX_PARALLEL_START=15        # Service startup (less intensive)
MAX_PARALLEL_TEMPLATE=15     # OS template installation

# Service-specific limits
MAX_PARALLEL_CCIP=8          # CCIP nodes (resource intensive)
MAX_PARALLEL_BESU=12         # Besu nodes (can handle more)

# CCIP batching
CCIP_BATCH_SIZE=8            # Process 8 CCIP nodes at a time
```

---

## Monitoring During Deployment

**Watch for**:
- High CPU usage (>90% sustained)
- High RAM usage (>90% used)
- Storage I/O wait times
- Network saturation
- Container creation failures

**If issues occur**:
- Reduce MAX_PARALLEL values
- Increase delays between batches
- Deploy in smaller batches
- Check Proxmox host resource usage

---

## Best Practices

1. **Start Conservative**: Begin with lower limits, increase if system handles well
2. **Monitor Resources**: Watch CPU, RAM, I/O during deployment
3. **Batch Large Deployments**: CCIP nodes should always be batched
4. **Respect Service Dependencies**: Some services need to start in order
5. **Test Limits**: On test environment before production deployment

---

## References

- **Deployment Time Estimate**: `docs/DEPLOYMENT_TIME_ESTIMATE.md`
- **Optimization Recommendations**: `docs/DEPLOYMENT_OPTIMIZATION_RECOMMENDATIONS.md`
- **Configuration**: `smom-dbis-138-proxmox/config/proxmox.conf`