smom-dbis-138/runbooks/node-add-remove.md

# Node Add/Remove Runbook

## Adding a Validator Node

### Prerequisites

1. Generate validator key
2. Update genesis extraData
3. Configure network access
4. Prepare node infrastructure

### Steps

1. **Generate Validator Key**
   ```bash
   ./scripts/key-management/generate-validator-keys.sh 1
   ```

2. **Update Genesis extraData**
   - Add validator address to genesis
   - Generate new extraData with Besu tooling
   - Update genesis.json

3. **Update static-nodes.json**
   - Add validator enode to static-nodes.json
   - Update on all validators and sentries

4. **Deploy Validator Node**
   ```bash
   kubectl apply -f k8s/base/validators/statefulset.yaml
   ```

5. **Verify Node**
   - Check node is peering
   - Verify block production
   - Monitor metrics

### Rollback

If issues occur:
1. Remove validator from static-nodes.json
2. Update genesis extraData
3. Restart all nodes
4. Verify network stability

## Removing a Validator Node

### Steps

1. **Update static-nodes.json**
   - Remove validator enode
   - Update on all validators and sentries

2. **Update Genesis extraData**
   - Remove validator address from genesis
   - Generate new extraData
   - Update genesis.json

3. **Stop Validator Node**
   ```bash
   kubectl delete statefulset besu-validator-<index> -n besu-network
   ```

4. **Verify Network**
   - Check remaining validators are working
   - Verify block production
   - Monitor metrics

### Rollback

If issues occur:
1. Add validator back to static-nodes.json
2. Update genesis extraData
3. Restart validator node
4. Verify network stability

## Adding a Sentry Node

### Steps

1. **Generate Node Key**
   ```bash
   ./scripts/key-management/generate-validator-keys.sh 1
   ```

2. **Update static-nodes.json**
   - Add sentry enode to static-nodes.json
   - Update on all validators and sentries

3. **Deploy Sentry Node**
   ```bash
   helm install besu-sentry-<index> ./helm/besu-network -f helm/besu-network/values-sentries.yaml -n besu-network
   ```

4. **Verify Node**
   - Check node is peering
   - Verify P2P connections
   - Monitor metrics

## Removing a Sentry Node

### Steps

1. **Update static-nodes.json**
   - Remove sentry enode
   - Update on all validators and sentries

2. **Stop Sentry Node**
   ```bash
   helm uninstall besu-sentry-<index> -n besu-network
   ```

3. **Verify Network**
   - Check remaining sentries are working
   - Verify P2P connections
   - Monitor metrics

## Adding an RPC Node

### Steps

1. **Deploy RPC Node**
   ```bash
   helm install besu-rpc-<index> ./helm/besu-network -f helm/besu-network/values-rpc.yaml -n besu-network
   ```

2. **Update Load Balancer**
   - Add RPC node to load balancer pool
   - Update health checks

3. **Verify Node**
   - Check RPC endpoint is responding
   - Verify sync status
   - Monitor metrics

## Removing an RPC Node

### Steps

1. **Remove from Load Balancer**
   - Remove RPC node from load balancer pool
   - Wait for connections to drain

2. **Stop RPC Node**
   ```bash
   helm uninstall besu-rpc-<index> -n besu-network
   ```

3. **Verify Network**
   - Check remaining RPC nodes are working
   - Verify load balancer health
   - Monitor metrics

## Troubleshooting

### Node Not Peering

1. Check network connectivity
2. Verify static-nodes.json configuration
3. Check firewall rules
4. Review node logs

### Node Not Syncing

1. Check genesis file matches
2. Verify node configuration
3. Check disk space
4. Review node logs

### Node Performance Issues

1. Check resource usage
2. Verify node configuration
3. Review metrics
4. Consider scaling resources