92 lines
2.2 KiB
Markdown
92 lines
2.2 KiB
Markdown
|
# Deployment Fix Plan
|
||
|
|
|
||
|
|
## Problem Summary
|
||
|
|
|
||
|
|
**Failed Clusters (7)**: Stopped during Terraform updates - cannot be fixed, must be deleted and recreated
|
||
|
|
**Canceled Clusters (16)**: Deployment interrupted - exist in Azure but not in Terraform state - must be deleted or imported
|
||
|
|
|
||
|
|
## Fix Strategy
|
||
|
|
|
||
|
|
### Option 1: Clean Slate (Recommended)
|
||
|
|
**Delete all problematic clusters and recreate with Terraform**
|
||
|
|
|
||
|
|
**Pros**:
|
||
|
|
- Clean state, no import complexity
|
||
|
|
- Ensures consistent configuration
|
||
|
|
- Faster than importing 17 clusters
|
||
|
|
|
||
|
|
**Cons**:
|
||
|
|
- Temporary loss of any existing workloads
|
||
|
|
- Requires full redeployment
|
||
|
|
|
||
|
|
### Option 2: Import Existing (Complex)
|
||
|
|
**Import canceled clusters into Terraform state**
|
||
|
|
|
||
|
|
**Pros**:
|
||
|
|
- Preserves existing clusters
|
||
|
|
- No downtime
|
||
|
|
|
||
|
|
**Cons**:
|
||
|
|
- Complex import process (17 clusters)
|
||
|
|
- May have configuration drift
|
||
|
|
- Still need to delete 7 failed clusters
|
||
|
|
|
||
|
|
## Recommended Fix: Option 1 - Clean Slate
|
||
|
|
|
||
|
|
### Step 1: Delete All Failed Clusters (7)
|
||
|
|
Failed clusters are in terminal error state and must be deleted.
|
||
|
|
|
||
|
|
### Step 2: Delete All Canceled Clusters (16)
|
||
|
|
Canceled clusters cause state mismatch and should be deleted for clean recreation.
|
||
|
|
|
||
|
|
### Step 3: Clean Up Terraform State
|
||
|
|
Remove any references to deleted clusters from Terraform state.
|
||
|
|
|
||
|
|
### Step 4: Re-run Terraform Deployment
|
||
|
|
Deploy all clusters fresh with proper configuration.
|
||
|
|
|
||
|
|
## Implementation Scripts
|
||
|
|
|
||
|
|
### Script 1: Delete Failed Clusters
|
||
|
|
```bash
|
||
|
|
./scripts/azure/delete-failed-clusters.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
### Script 2: Delete Canceled Clusters
|
||
|
|
```bash
|
||
|
|
./scripts/azure/delete-canceled-clusters.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
### Script 3: Delete All Problematic Clusters
|
||
|
|
```bash
|
||
|
|
./scripts/azure/delete-all-problematic-clusters.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
### Script 4: Re-run Terraform
|
||
|
|
```bash
|
||
|
|
cd terraform/well-architected/cloud-sovereignty
|
||
|
|
terraform apply -parallelism=128 -auto-approve
|
||
|
|
```
|
||
|
|
|
||
|
|
## Quick Fix Command
|
||
|
|
|
||
|
|
Run the automated fix script:
|
||
|
|
```bash
|
||
|
|
./scripts/azure/fix-deployment-issues.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
This will:
|
||
|
|
1. Delete all 7 failed clusters
|
||
|
|
2. Delete all 16 canceled clusters
|
||
|
|
3. Clean Terraform state
|
||
|
|
4. Re-run Terraform deployment
|
||
|
|
5. Monitor progress
|
||
|
|
|
||
|
|
## Prevention
|
||
|
|
|
||
|
|
After fix, implement:
|
||
|
|
1. Prevent manual cluster stops during deployment
|
||
|
|
2. Check power state before updates
|
||
|
|
3. Use proper state management
|
||
|
|
4. Monitor during deployment
|