GitOps & Progressive Delivery

GitOps treats Git as the single source of truth for infrastructure and application state. Controllers continuously reconcile the desired state (in Git) with the actual state (in the cluster), eliminating manual kubectl apply in production.

When to Activate

• Setting up ArgoCD or Flux for Kubernetes deployments
• Designing GitOps repository structure (mono-repo vs. config-repo)
• Configuring multi-cluster GitOps with ApplicationSets
• Managing secrets in a GitOps workflow (Sealed Secrets, External Secrets Operator)
• Migrating from imperative kubectl workflows to declarative GitOps
• Configuring drift detection and automated remediation

For canary/blue-green deployments and automated rollback with Argo Rollouts — see skill progressive-delivery.

---

GitOps Core Principles

Four principles from the OpenGitOps standard:

Principle	Meaning
Declarative	System state fully described in YAML/Helm/Kustomize — no "scripts that do things"
Versioned & Immutable	Git history is the audit log; every change is a commit
Pulled Automatically	Cluster pulls changes from Git (not CI pushes to cluster)
Continuously Reconciled	Controller loops compare desired vs. actual state, fix drift automatically

---

ArgoCD

Application CRD

# apps/my-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: my-team
  source:
    repoURL: https://github.com/myorg/config-repo
    targetRevision: main
    path: apps/my-app/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app-prod
  syncPolicy:
    automated:          # Remove for manual sync in production
      prune: true       # Delete resources no longer in Git
      selfHeal: true    # Revert manual changes automatically
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
  revisionHistoryLimit: 10

Sync Strategies

# Manual sync (recommended for production)
syncPolicy: {}   # No automated block — requires manual Sync in UI or CLI

# Automated sync (good for dev/staging)
syncPolicy:
  automated:
    prune: true
    selfHeal: true

# Sync with resource hooks (e.g., run migrations before deploy)
syncPolicy:
  syncOptions:
    - RespectIgnoreDifferences=true

App of Apps Pattern

One root Application manages all other Applications:

# root-app.yaml — bootstraps the whole cluster
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  source:
    path: clusters/production/apps   # Directory of Application manifests
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

ApplicationSet — Dynamic App Generation

# Generate one Application per directory in apps/
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/config-repo
        revision: main
        directories:
          - path: apps/*/overlays/production
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/config-repo
        targetRevision: main
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Health Checks

ArgoCD checks resource health automatically. Custom health checks for CRDs:

-- health-check.lua (in argocd-cm ConfigMap)
hs = {}
if obj.status ~= nil then
  if obj.status.phase == "Running" then
    hs.status = "Healthy"
    hs.message = "Running"
    return hs
  end
  if obj.status.phase == "Failed" then
    hs.status = "Degraded"
    hs.message = obj.status.message
    return hs
  end
end
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs

RBAC: Projects Isolate Teams

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: my-team
  namespace: argocd
spec:
  description: My Team applications
  sourceRepos:
    - 'https://github.com/myorg/*'
  destinations:
    - namespace: 'my-team-*'
      server: https://kubernetes.default.svc
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
  roles:
    - name: developer
      description: Can sync apps but not delete
      policies:
        - p, proj:my-team:developer, applications, get, my-team/*, allow
        - p, proj:my-team:developer, applications, sync, my-team/*, allow
      groups:
        - my-team-developers

Image Updater

Automatically commit new image tags to Git when a new image is pushed:

# Annotation on ArgoCD Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: my-app=myorg/my-app
    argocd-image-updater.argoproj.io/my-app.update-strategy: semver
    argocd-image-updater.argoproj.io/my-app.allow-tags: regexp:^v[0-9]+\.[0-9]+\.[0-9]+$
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main

---

Flux

Core Controllers

# GitRepository — what to watch
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: config-repo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/config-repo
  ref:
    branch: main
  secretRef:
    name: flux-system   # SSH or HTTPS credentials

---
# Kustomization — what to apply
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/my-app/overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: config-repo
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: my-app
      namespace: my-app-prod
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

HelmRelease

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cert-manager
  namespace: flux-system
spec:
  interval: 1h
  chart:
    spec:
      chart: cert-manager
      version: '>=1.14.0 <2.0.0'
      sourceRef:
        kind: HelmRepository
        name: jetstack
  values:
    installCRDs: true
    replicaCount: 2
  valuesFrom:
    - kind: ConfigMap
      name: cert-manager-values

Flux Notification Controller

# Alert to Slack on sync failure
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: slack-alert
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: '*'
    - kind: HelmRelease
      name: '*'
  summary: "Flux sync failure in production cluster"

Flux vs. ArgoCD Decision Guide

Criteria	Flux	ArgoCD
UI	None (CLI only)	Rich web UI
Multi-cluster	Via separate Flux instances	Built-in with ApplicationSet
Kubernetes-native feel	Yes (pure CRDs)	Yes but heavier
Helm support	Via HelmRelease CRD	Native, with live rendering
Complexity	Lower baseline	Higher, more features
Best for	Platform teams who prefer GitOps-native	Product teams who want UI

---

Kustomize in GitOps

Repository Structure

config-repo/
├── base/
│   └── my-app/
│       ├── deployment.yaml
│       ├── service.yaml
│       └── kustomization.yaml
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml      # patches + resources
    │   └── replica-patch.yaml
    ├── staging/
    │   └── kustomization.yaml
    └── production/
        ├── kustomization.yaml
        └── hpa-patch.yaml

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base/my-app
namePrefix: prod-
nameSuffix: ""
commonLabels:
  environment: production
patches:
  - path: hpa-patch.yaml
images:
  - name: myorg/my-app
    newTag: v1.2.3   # Updated by CI or Image Updater
configMapGenerator:
  - name: app-config
    literals:
      - LOG_LEVEL=warn
      - REPLICA_COUNT=5

---

Secrets Management in GitOps

NEVER commit plaintext secrets to Git. Choose one:

Option 1: Sealed Secrets (simple, self-contained)

# Encrypt a secret for the cluster's public key
kubeseal --fetch-cert --controller-namespace kube-system > pub-cert.pem
kubectl create secret generic my-secret \
  --from-literal=API_KEY=supersecret \
  --dry-run=client -o yaml | \
  kubeseal --cert pub-cert.pem -o yaml > sealed-secret.yaml
# Commit sealed-secret.yaml safely — only the cluster can decrypt
git add sealed-secret.yaml && git commit -m "feat: add sealed API key"

Option 2: External Secrets Operator (ESO) — recommended for production

# ExternalSecret — pulls from AWS Secrets Manager, Vault, etc.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secrets
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: my-app-secrets          # Creates a native Kubernetes Secret
    creationPolicy: Owner
  data:
    - secretKey: API_KEY           # Key in the K8s Secret
      remoteRef:
        key: prod/my-app           # Path in Secrets Manager
        property: api_key

# ClusterSecretStore — one per cluster, points to the secret backend
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

---

Multi-Cluster GitOps

Hub-Spoke Architecture

Management Cluster (ArgoCD/Flux installed)
  ├── dev-cluster    (ApplicationSet target)
  ├── staging-cluster
  └── prod-cluster

# ApplicationSet targeting all clusters in ArgoCD
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app-all-clusters
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production   # Only prod clusters
  template:
    metadata:
      name: 'my-app-{{name}}'
    spec:
      project: default
      source:
        path: apps/my-app/overlays/production
      destination:
        server: '{{server}}'
        namespace: my-app

---

GitOps Repository Patterns

Option A: Mono-Repo (small teams)

repo/
├── src/           # Application code
├── k8s/
│   ├── base/
│   └── overlays/
└── .github/workflows/
    └── ci.yml     # Build image, update k8s/overlays/*/kustomization.yaml

Option B: Separate Config Repo (recommended for teams)

app-repo/          # Source code + CI (builds image, opens PR to config-repo)
config-repo/       # Kubernetes manifests only — GitOps controller watches this
  ├── clusters/
  │   ├── dev/
  │   ├── staging/
  │   └── prod/
  └── apps/
      └── my-app/
          ├── base/
          └── overlays/

CI → GitOps Bridge

# In app-repo CI: after successful image build, update config-repo
- name: Update image tag in config-repo
  run: |
    git clone https://github.com/myorg/config-repo
    cd config-repo
    cd apps/my-app/overlays/staging
    kustomize edit set image myorg/my-app=myorg/my-app:${{ github.sha }}
    git config user.email "ci@myorg.com"
    git config user.name "CI Bot"
    git add -A
    git commit -m "chore: update my-app to ${{ github.sha }}"
    git push

---

Common Pitfalls

Problem	Solution
Manual kubectl apply bypasses GitOps	Enable selfHeal: true + RBAC to prevent direct cluster access
Secrets in Git	Use ESO or Sealed Secrets — never commit plaintext
All environments auto-sync	Production must require manual sync approval
No drift detection alerts	Configure ArgoCD Notifications / Flux Alerts
Helm values spread across repos	Keep values files in config-repo alongside the HelmRelease
Image tag drift	Use ArgoCD Image Updater or Flux Image Automation