loc_az_hci/docs/operations/runbooks/gitops-workflow.md

# GitOps Workflow Runbook

## Overview

This runbook describes the GitOps workflow using Flux for managing Kubernetes deployments.

## GitOps Architecture

```
Git Repository (Gitea/GitLab)
    │
    │ (Poll/Sync)
    │
    ▼
Flux Controller (Kubernetes)
    │
    │ (Apply)
    │
    ▼
Kubernetes Cluster
    │
    │ (Deploy)
    │
    ▼
Application Pods
```

## Workflow

### 1. Making Changes

#### Update Application Configuration

1. Clone Git repository:
```bash
git clone http://git.local:3000/user/gitops-repo.git
cd gitops-repo
```

2. Edit Helm chart values:
```bash
# Edit values.yaml
vim gitops/apps/besu/values.yaml
```

3. Commit and push:
```bash
git add gitops/apps/besu/values.yaml
git commit -m "Update Besu configuration"
git push origin main
```

#### Add New Application

1. Add Helm chart to repository:
```bash
cp -r /path/to/new-chart gitops/apps/new-app/
```

2. Create Flux Kustomization:
```bash
# Create gitops/apps/new-app/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: new-app
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/new-app
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
```

3. Commit and push:
```bash
git add gitops/apps/new-app/
git commit -m "Add new application"
git push origin main
```

### 2. Monitoring Sync Status

#### Check Flux Status

```bash
# Check Flux pods
kubectl get pods -n flux-system

# Check Git repository status
kubectl get gitrepository -n flux-system
kubectl describe gitrepository flux-system -n flux-system

# Check Kustomization status
kubectl get kustomization -n flux-system
kubectl describe kustomization <app-name> -n flux-system
```

#### View Sync Events

```bash
# Watch Flux events
kubectl get events -n flux-system --sort-by='.lastTimestamp'

# View Flux logs
kubectl logs -n flux-system -l app=flux -f
```

### 3. Troubleshooting

#### Sync Not Happening

**Check Git repository access**:
```bash
kubectl get gitrepository flux-system -n flux-system -o yaml
kubectl describe gitrepository flux-system -n flux-system
```

**Check authentication**:
```bash
# For HTTPS with token
kubectl get secret -n flux-system

# For SSH
kubectl get secret flux-system -n flux-system -o yaml
```

#### Application Not Deploying

**Check Kustomization**:
```bash
kubectl get kustomization <app-name> -n flux-system
kubectl describe kustomization <app-name> -n flux-system
```

**Check Helm release**:
```bash
kubectl get helmrelease -n <namespace>
kubectl describe helmrelease <app-name> -n <namespace>
```

#### Manual Sync Trigger

```bash
# Trigger immediate sync
flux reconcile source git flux-system
flux reconcile kustomization <app-name>
```

### 4. Best Practices

#### Repository Structure

```
gitops-repo/
├── infrastructure/
│   ├── namespace.yaml
│   ├── ingress-controller.yaml
│   └── cert-manager.yaml
└── apps/
    ├── besu/
    │   ├── Chart.yaml
    │   ├── values.yaml
    │   └── templates/
    ├── firefly/
    └── ...
```

#### Branch Strategy

- **main**: Production deployments
- **staging**: Staging environment
- **develop**: Development environment

#### Change Management

1. Create feature branch
2. Make changes
3. Test in development
4. Merge to staging
5. Promote to production

### 5. Common Operations

#### Suspend Sync

```bash
# Suspend specific application
flux suspend kustomization <app-name>

# Resume
flux resume kustomization <app-name>
```

#### Rollback Changes

```bash
# Revert Git commit
git revert <commit-hash>
git push origin main

# Or manually edit and push
```

#### Update Helm Chart

```bash
# Update chart version in values.yaml
# Commit and push
git add gitops/apps/<app>/values.yaml
git commit -m "Update <app> to version X.Y.Z"
git push origin main
```

### 6. Azure Arc GitOps Integration

#### Configure GitOps in Azure Portal

1. Navigate to: Azure Arc → Kubernetes → Your cluster
2. Go to "GitOps" section
3. Add configuration:
   - Repository URL
   - Branch
   - Path
   - Authentication

#### View GitOps Status in Azure

```bash
az k8s-extension show \
  --resource-group HC-Stack \
  --cluster-name proxmox-k3s-cluster \
  --cluster-type connectedClusters \
  --name flux
```

### 7. Security

#### Secret Management

**Option 1: Kubernetes Secrets** (not recommended for production):
```bash
kubectl create secret generic app-secret \
  --from-literal=password=secret-value \
  -n <namespace>
```

**Option 2: Sealed Secrets**:
```bash
# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Create sealed secret
kubeseal < secret.yaml > sealed-secret.yaml
```

**Option 3: External Secrets Operator**:
- Integrate with Azure Key Vault
- Use External Secrets Operator

#### RBAC

Configure Flux RBAC:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: flux-<namespace>
  namespace: <namespace>
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]
```

### 8. Monitoring

#### Set Up Alerts

```bash
# Create alert for sync failures
kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: flux-sync-alerts
spec:
  groups:
  - name: flux
    rules:
    - alert: FluxSyncFailed
      expr: flux_kustomization_status_condition{status="False"} == 1
      annotations:
        summary: "Flux sync failed"
EOF
```

### 9. Disaster Recovery

#### Backup Git Repository

```bash
# Clone repository
git clone --mirror http://git.local:3000/user/gitops-repo.git

# Backup to external location
tar -czf gitops-backup-$(date +%Y%m%d).tar.gz gitops-repo.git
```

#### Restore from Backup

```bash
# Restore repository
tar -xzf gitops-backup-YYYYMMDD.tar.gz
cd gitops-repo.git
git remote set-url origin http://git.local:3000/user/gitops-repo.git
git push --mirror
```