loc_az_hci/docs/operations/runbooks/gitops-workflow.md

GitOps Workflow Runbook

Overview

This runbook describes the GitOps workflow using Flux for managing Kubernetes deployments.

GitOps Architecture

Git Repository (Gitea/GitLab)
    │
    │ (Poll/Sync)
    │
    ▼
Flux Controller (Kubernetes)
    │
    │ (Apply)
    │
    ▼
Kubernetes Cluster
    │
    │ (Deploy)
    │
    ▼
Application Pods

Workflow

1. Making Changes

Update Application Configuration

1. Clone Git repository:

git clone http://git.local:3000/user/gitops-repo.git
cd gitops-repo

2. Edit Helm chart values:

# Edit values.yaml
vim gitops/apps/besu/values.yaml

3. Commit and push:

git add gitops/apps/besu/values.yaml
git commit -m "Update Besu configuration"
git push origin main

Add New Application

1. Add Helm chart to repository:

cp -r /path/to/new-chart gitops/apps/new-app/

2. Create Flux Kustomization:

# Create gitops/apps/new-app/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: new-app
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/new-app
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

3. Commit and push:

git add gitops/apps/new-app/
git commit -m "Add new application"
git push origin main

2. Monitoring Sync Status

Check Flux Status

# Check Flux pods
kubectl get pods -n flux-system

# Check Git repository status
kubectl get gitrepository -n flux-system
kubectl describe gitrepository flux-system -n flux-system

# Check Kustomization status
kubectl get kustomization -n flux-system
kubectl describe kustomization <app-name> -n flux-system

View Sync Events

# Watch Flux events
kubectl get events -n flux-system --sort-by='.lastTimestamp'

# View Flux logs
kubectl logs -n flux-system -l app=flux -f

3. Troubleshooting

Sync Not Happening

Check Git repository access:

kubectl get gitrepository flux-system -n flux-system -o yaml
kubectl describe gitrepository flux-system -n flux-system

Check authentication:

# For HTTPS with token
kubectl get secret -n flux-system

# For SSH
kubectl get secret flux-system -n flux-system -o yaml

Application Not Deploying

Check Kustomization:

kubectl get kustomization <app-name> -n flux-system
kubectl describe kustomization <app-name> -n flux-system

Check Helm release:

kubectl get helmrelease -n <namespace>
kubectl describe helmrelease <app-name> -n <namespace>

Manual Sync Trigger

# Trigger immediate sync
flux reconcile source git flux-system
flux reconcile kustomization <app-name>

4. Best Practices

Repository Structure

gitops-repo/
├── infrastructure/
│   ├── namespace.yaml
│   ├── ingress-controller.yaml
│   └── cert-manager.yaml
└── apps/
    ├── besu/
    │   ├── Chart.yaml
    │   ├── values.yaml
    │   └── templates/
    ├── firefly/
    └── ...

Branch Strategy

• main: Production deployments
• staging: Staging environment
• develop: Development environment

Change Management

1. Create feature branch
2. Make changes
3. Test in development
4. Merge to staging
5. Promote to production

5. Common Operations

Suspend Sync

# Suspend specific application
flux suspend kustomization <app-name>

# Resume
flux resume kustomization <app-name>

Rollback Changes

# Revert Git commit
git revert <commit-hash>
git push origin main

# Or manually edit and push

Update Helm Chart

# Update chart version in values.yaml
# Commit and push
git add gitops/apps/<app>/values.yaml
git commit -m "Update <app> to version X.Y.Z"
git push origin main

6. Azure Arc GitOps Integration

Configure GitOps in Azure Portal

1. Navigate to: Azure Arc → Kubernetes → Your cluster
2. Go to "GitOps" section
3. Add configuration:
   • Repository URL
   • Branch
   • Path
   • Authentication

View GitOps Status in Azure

az k8s-extension show \
  --resource-group HC-Stack \
  --cluster-name proxmox-k3s-cluster \
  --cluster-type connectedClusters \
  --name flux

7. Security

Secret Management

Option 1: Kubernetes Secrets (not recommended for production):

kubectl create secret generic app-secret \
  --from-literal=password=secret-value \
  -n <namespace>

Option 2: Sealed Secrets:

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Create sealed secret
kubeseal < secret.yaml > sealed-secret.yaml

Option 3: External Secrets Operator:

• Integrate with Azure Key Vault
• Use External Secrets Operator

RBAC

Configure Flux RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: flux-<namespace>
  namespace: <namespace>
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]

8. Monitoring

Set Up Alerts

# Create alert for sync failures
kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: flux-sync-alerts
spec:
  groups:
  - name: flux
    rules:
    - alert: FluxSyncFailed
      expr: flux_kustomization_status_condition{status="False"} == 1
      annotations:
        summary: "Flux sync failed"
EOF

9. Disaster Recovery

Backup Git Repository

# Clone repository
git clone --mirror http://git.local:3000/user/gitops-repo.git

# Backup to external location
tar -czf gitops-backup-$(date +%Y%m%d).tar.gz gitops-repo.git

Restore from Backup

# Restore repository
tar -xzf gitops-backup-YYYYMMDD.tar.gz
cd gitops-repo.git
git remote set-url origin http://git.local:3000/user/gitops-repo.git
git push --mirror