249 lines
6.2 KiB
Markdown
249 lines
6.2 KiB
Markdown
# Parallel Execution Limits
|
|
|
|
**Last Updated**: 2025-01-11
|
|
**Purpose**: Document recommended parallel execution limits for deployment operations
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Deployment operations can be parallelized to reduce total deployment time, but must respect Proxmox host resources to prevent exhaustion and instability.
|
|
|
|
---
|
|
|
|
## Recommended Limits
|
|
|
|
### Container Creation
|
|
|
|
**Operation**: Creating LXC containers
|
|
**Recommended**: **10 containers simultaneously**
|
|
**Rationale**:
|
|
- Container creation is I/O intensive
|
|
- Proxmox can handle ~10 concurrent container creations
|
|
- More than 10 can cause resource contention
|
|
|
|
**Configuration**:
|
|
```bash
|
|
# In proxmox.conf
|
|
MAX_PARALLEL_CREATE=10
|
|
```
|
|
|
|
---
|
|
|
|
### Package Installation
|
|
|
|
**Operation**: Installing packages (apt, npm, etc.)
|
|
**Recommended**: **8 containers simultaneously**
|
|
**Rationale**:
|
|
- Package installation is CPU and RAM intensive
|
|
- Downloads can saturate network
|
|
- Compilation (if any) requires CPU
|
|
- 8 containers balances speed vs. resource usage
|
|
|
|
**Configuration**:
|
|
```bash
|
|
# In proxmox.conf
|
|
MAX_PARALLEL_PACKAGES=8
|
|
```
|
|
|
|
**CCIP Nodes**: Use **8 nodes per batch** (16 nodes = 2 batches)
|
|
- CCIP nodes have complex dependencies
|
|
- Chainlink installation is resource-intensive
|
|
- Batching prevents resource exhaustion
|
|
|
|
---
|
|
|
|
### Service Startup
|
|
|
|
**Operation**: Starting services (systemd, Docker, etc.)
|
|
**Recommended**: **15 containers simultaneously**
|
|
**Rationale**:
|
|
- Service startup is less resource-intensive
|
|
- Mostly I/O and network operations
|
|
- Can handle more concurrent startups
|
|
- Services initialize independently
|
|
|
|
**Configuration**:
|
|
```bash
|
|
# In proxmox.conf
|
|
MAX_PARALLEL_START=15
|
|
```
|
|
|
|
---
|
|
|
|
### OS Template Installation
|
|
|
|
**Operation**: Installing OS templates to containers
|
|
**Recommended**: **10-20 containers simultaneously**
|
|
**Rationale**:
|
|
- Template installation is I/O intensive
|
|
- Can parallelize more than creation
|
|
- Depends on storage performance
|
|
- 10-20 is safe for most systems
|
|
|
|
**Configuration**:
|
|
```bash
|
|
# In proxmox.conf
|
|
MAX_PARALLEL_TEMPLATE=15
|
|
```
|
|
|
|
---
|
|
|
|
## Service-Specific Recommendations
|
|
|
|
### Besu Nodes
|
|
|
|
**Container Creation**: 5 validators + 4 sentries + 3 RPC = 12 containers
|
|
- Can create all 12 in parallel (within 10 limit)
|
|
- Or create by type: 5 validators → 4 sentries → 3 RPC
|
|
|
|
**Package Installation**: Can install on all 12 in parallel (within 8 limit)
|
|
|
|
**Service Startup**: Can start all 12 in parallel (within 15 limit)
|
|
|
|
---
|
|
|
|
### CCIP Nodes (41-43 containers)
|
|
|
|
**Container Creation**: Batch by type
|
|
- CCIP-OPS (2): Parallel ✓
|
|
- CCIP-MON (2): Parallel ✓
|
|
- CCIP-COMMIT (16): 8 per batch, 2 batches
|
|
- CCIP-EXEC (16): 8 per batch, 2 batches
|
|
- CCIP-RMN (5-7): All parallel ✓
|
|
|
|
**Package Installation**: Same batching strategy
|
|
- Install packages in same batches as creation
|
|
- 8 nodes per batch prevents resource exhaustion
|
|
|
|
**Service Startup**: Can start more in parallel
|
|
- CCIP-OPS (2): Parallel
|
|
- CCIP-MON (2): Parallel
|
|
- CCIP-COMMIT (16): 8 per batch or all parallel
|
|
- CCIP-EXEC (16): 8 per batch or all parallel
|
|
- CCIP-RMN (5-7): All parallel
|
|
|
|
---
|
|
|
|
### Other Services (14 containers)
|
|
|
|
**Container Creation**: Can create all in parallel (within 10 limit)
|
|
- Or create by type for organization
|
|
|
|
**Package Installation**: Varies by service complexity
|
|
- Simple services (Besu services): Can install in parallel
|
|
- Complex services (Blockscout, Fabric): Install sequentially or 2-3 at a time
|
|
|
|
**Service Startup**: Can start most in parallel
|
|
- Database services (Blockscout, Firefly) may need sequential startup
|
|
- Other services can start in parallel
|
|
|
|
---
|
|
|
|
## Resource Considerations
|
|
|
|
### CPU
|
|
|
|
**Impact**: Package installation, compilation, service startup
|
|
**Limiting Factor**: Number of CPU cores
|
|
**Recommendation**:
|
|
- For 8-core host: MAX_PARALLEL_PACKAGES=6
|
|
- For 16-core host: MAX_PARALLEL_PACKAGES=8
|
|
- For 32+ core host: MAX_PARALLEL_PACKAGES=12
|
|
|
|
---
|
|
|
|
### RAM
|
|
|
|
**Impact**: Package installation, service startup, container overhead
|
|
**Limiting Factor**: Available RAM
|
|
**Recommendation**:
|
|
- Ensure sufficient RAM for parallel operations
|
|
- Each container uses RAM for OS + packages + services
|
|
- CCIP nodes may need more RAM during installation
|
|
|
|
---
|
|
|
|
### Storage I/O
|
|
|
|
**Impact**: Container creation, template installation, package installation
|
|
**Limiting Factor**: Storage type and performance
|
|
**Recommendation**:
|
|
- Local SSD: Can handle higher parallelism
|
|
- Network storage: Lower parallelism recommended
|
|
- HDD: Even lower parallelism
|
|
|
|
---
|
|
|
|
### Network Bandwidth
|
|
|
|
**Impact**: Package downloads, file transfers
|
|
**Limiting Factor**: Network connection speed
|
|
**Recommendation**:
|
|
- 1 Gbps: Can handle full parallelism
|
|
- 100 Mbps: Reduce parallelism by 50%
|
|
- Slower: Reduce further or use local package mirror
|
|
|
|
---
|
|
|
|
## Configuration Example
|
|
|
|
```bash
|
|
# In proxmox.conf
|
|
|
|
# Parallel deployment enabled
|
|
PARALLEL_DEPLOY=true
|
|
|
|
# General limits
|
|
MAX_PARALLEL=10 # Default for operations without specific limit
|
|
|
|
# Operation-specific limits
|
|
MAX_PARALLEL_CREATE=10 # Container creation
|
|
MAX_PARALLEL_PACKAGES=8 # Package installation (CPU/RAM intensive)
|
|
MAX_PARALLEL_START=15 # Service startup (less intensive)
|
|
MAX_PARALLEL_TEMPLATE=15 # OS template installation
|
|
|
|
# Service-specific limits
|
|
MAX_PARALLEL_CCIP=8 # CCIP nodes (resource intensive)
|
|
MAX_PARALLEL_BESU=12 # Besu nodes (can handle more)
|
|
|
|
# CCIP batching
|
|
CCIP_BATCH_SIZE=8 # Process 8 CCIP nodes at a time
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring During Deployment
|
|
|
|
**Watch for**:
|
|
- High CPU usage (>90% sustained)
|
|
- High RAM usage (>90% used)
|
|
- Storage I/O wait times
|
|
- Network saturation
|
|
- Container creation failures
|
|
|
|
**If issues occur**:
|
|
- Reduce MAX_PARALLEL values
|
|
- Increase delays between batches
|
|
- Deploy in smaller batches
|
|
- Check Proxmox host resource usage
|
|
|
|
---
|
|
|
|
## Best Practices
|
|
|
|
1. **Start Conservative**: Begin with lower limits, increase if system handles well
|
|
2. **Monitor Resources**: Watch CPU, RAM, I/O during deployment
|
|
3. **Batch Large Deployments**: CCIP nodes should always be batched
|
|
4. **Respect Service Dependencies**: Some services need to start in order
|
|
5. **Test Limits**: On test environment before production deployment
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- **Deployment Time Estimate**: `docs/DEPLOYMENT_TIME_ESTIMATE.md`
|
|
- **Optimization Recommendations**: `docs/DEPLOYMENT_OPTIMIZATION_RECOMMENDATIONS.md`
|
|
- **Configuration**: `smom-dbis-138-proxmox/config/proxmox.conf`
|
|
|