Kubernetes GitOps Crash Course

Kustomize · FluxCD · Flux Operator

A one-week ramp-up for a new Kubernetes DevOps engineer joining the Platform/SRE team. Assumes solid Kubernetes fundamentals (kubectl, Deployments, Services, ConfigMaps, RBAC, namespaces). If any of that is shaky, fix it first — none of what follows will land otherwise.

How to use this document

The course is laid out as five days, each ~3–5 focused hours. Don't skim. Each day ends with an exercise you should actually do against a real cluster (kind, k3d, or minikube locally; a dev cluster if we've given you one). Push your exercise repo to GitHub so the team can review.

There is also a cheat sheets section at the end. Bookmark it.

Day 0  · Mental model — GitOps, declarative ops, the "why"
Day 1  · Kustomize — config management without templating hell
Day 2  · FluxCD core — controllers, CRDs, the reconciliation loop
Day 3  · Flux Operator — declarative Flux, ResourceSets, AI tooling
Day 4  · Putting it together — repo layout, environments, secrets
Day 5  · Operations — debugging, drift, alerts, on-call

Day 0 — Mental Model

The GitOps contract

GitOps says: the desired state of the cluster lives in Git, and a controller in the cluster continuously reconciles the cluster to match it. Four properties:

Declarative. You describe what, not how.
Versioned & immutable. Git is the source of truth.
Pulled, not pushed. Cluster agents pull from Git; no CI pipeline runs kubectl apply against prod.
Continuously reconciled. Drift is detected and corrected automatically.

Why this matters operationally:

Rollback = git revert. No bespoke pipeline state.
Audit = git log. Who changed what, when, why.
Disaster recovery is trivial. Lose the cluster, bootstrap a new one, point it at the repo, walk away.
Reviews happen on PRs, not in Slack messages saying "I'm applying this now".

Where each tool fits

┌────────────────────────────────────────────────────────────────────┐
│  Developer writes YAML  →  Kustomize composes & overlays it        │
│              ↓                                                     │
│       Git repository  ──(optional CI step)──▶  OCI registry        │
│              ↓                                  ↓                  │
│       FluxCD (in cluster)  ◀────pulls from either source           │
│              ↑                                                     │
│      Flux Operator manages Flux itself + ResourceSets              │
└────────────────────────────────────────────────────────────────────┘

Kustomize is config authoring: producing the right YAML for the right environment, cloud, region, or tenant without copy-paste or string templating.
FluxCD is the delivery engine: pulling its source (Git or signed OCI artifacts), applying the rendered Kustomize output, reconciling, alerting.
Flux Operator is Flux's own management plane: it installs/upgrades Flux declaratively and adds higher-level abstractions (ResourceSets, ephemeral environments, status UI, gitless OCI sync, MCP).

Helm exists too. We use HelmReleases (via Flux) for third-party charts (Prometheus, cert-manager, etc.). For our own services we prefer Kustomize. The reasoning: our services don't need the value-templating power of Helm; they need clean overlays per environment, which Kustomize does better.

Exercise (Day 0)

Spin up a local cluster: kind create cluster --name training.
Verify: kubectl cluster-info, kubectl get nodes.
Read the Flux GitOps Toolkit overview once through.
Write a one-paragraph answer in your notes: "What changes for me as an engineer when I stop running kubectl apply against prod?"

Day 1 — Kustomize

The problem Kustomize solves

You have a Deployment. You want it slightly different in dev, staging, prod (replicas, resource limits, image tags, ingress hostnames). The bad ways to handle this:

Three separate copies of the YAML — drift guaranteed.
A Helm chart with 40 if/else blocks in templates — readable by no one.
A shell script that seds YAML — please no.

Kustomize's answer: a base + overlays. The base is a plain set of Kubernetes manifests. Overlays patch the base. No templating, no DSL — just YAML transforming YAML.

Core concepts

`kustomization.yaml`

Every Kustomize directory has one. It declares what manifests to include and how to transform them.

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app: orders-api

Build it:

kustomize build base/
# or, since kubectl ships with kustomize:
kubectl kustomize base/

This emits the composed YAML to stdout. Pipe to kubectl apply -f - for an out-of-Flux test, but in production Flux does this for you.

Bases and overlays

An overlay is just a directory with a kustomization.yaml that references another directory and modifies it. The dev/staging/prod split is the textbook example, but overlays are a general composition mechanism. They are not tied to environments. You split along whichever dimensions of variation your platform actually has. Common ones:

Dimension	Example values	What varies
Lifecycle / environment	`dev`, `staging`, `prod`	replicas, resource limits, log levels, ingress hostnames
Cloud platform	`aws`, `gcp`, `azure`, `on-prem`	storage classes, load-balancer annotations, IAM/identity bindings, image registry
Cluster flavor	`eks`, `gke`, `aks`, `kind`, `openshift`	controller-specific annotations, security context, SCC, default service account
Region	`eu-west-1`, `us-east-1`, `ap-southeast-2`	endpoint URLs, KMS key ARNs, GDPR-sensitive flags
Cluster tier	`small`, `medium`, `large`	replica counts, HPA bounds, requests/limits
Tenant / customer	`customer-a`, `customer-b`	namespace, branding config, feature toggles

Real platforms layer multiple of these. A prod EKS cluster in eu-west-1 isn't just "prod" — it's prod ∩ AWS ∩ EKS ∩ eu-west-1. Two patterns for handling this:

Pattern 1 — stacked overlays. Each overlay extends one parent.

base → overlays/aws → overlays/aws/eks → overlays/aws/eks/prod-eu-west-1

Simple to reason about, but rigid: if you also need a GCP prod overlay, you duplicate the prod-specific bits.

Pattern 2 — Kustomize components (preferred for cross-cutting concerns). A Component is a reusable transformation module that can be mixed into any overlay. This is exactly what they were designed for.

apps/orders-api/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── serviceaccount.yaml
├── components/
│   ├── aws/                    # AWS-specific cross-cutting concerns
│   │   ├── kustomization.yaml
│   │   ├── irsa-patch.yaml             # eks.amazonaws.com/role-arn annotation
│   │   ├── lb-annotations-patch.yaml   # AWS Load Balancer Controller annotations
│   │   └── ebs-storage-class-patch.yaml
│   ├── gcp/
│   ├── observability/          # Prometheus scrape annotations, log sidecar
│   └── network-policy-strict/  # production-grade NetworkPolicies
└── overlays/
    ├── dev-kind/                       # local kind cluster
    │   └── kustomization.yaml
    ├── staging-eks-eu-west-1/
    │   └── kustomization.yaml          # mixes in aws + observability
    └── prod-eks-eu-west-1/
        ├── kustomization.yaml          # mixes in aws + observability + network-policy-strict
        ├── replicas-patch.yaml
        └── resources-patch.yaml

An overlay composes a base with components:

# overlays/prod-eks-eu-west-1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: orders-prod
resources:
  - ../../base
components:
  - ../../components/aws
  - ../../components/observability
  - ../../components/network-policy-strict
patches:
  - path: replicas-patch.yaml
  - path: resources-patch.yaml
images:
  - name: ghcr.io/acme/orders-api
    newTag: v1.42.0

And the component itself:

# components/aws/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component                          # note: Component, not Kustomization
patches:
  - path: irsa-patch.yaml
    target:
      kind: ServiceAccount
      name: orders-api
  - path: lb-annotations-patch.yaml
    target:
      kind: Service
      name: orders-api

# components/aws/irsa-patch.yaml — bind the workload to an AWS IAM role via IRSA
apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders-api
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/orders-api

# components/aws/lb-annotations-patch.yaml — use the AWS Load Balancer Controller (NLB)
apiVersion: v1
kind: Service
metadata:
  name: orders-api
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

The same components/aws is now reusable across every overlay that runs on AWS, regardless of environment. A GCP overlay swaps in components/gcp (Workload Identity annotation, GCE Ingress annotations, pd-ssd storage class). The base stays cloud-neutral.

Rules of thumb:

If a variation is one-axis (only environment varies), plain overlays are fine.
If a variation is cross-cutting (applies to many overlays — "all our AWS clusters need IRSA"), make it a component.
Components apply in order and after the base; later components override earlier ones. Test the rendered output.
Don't nest too deep. Three layers (base + component + overlay) is the sweet spot; four is workable; five and you've built something nobody can debug at 3 AM.

Patches: strategic merge vs JSON 6902

Strategic merge patch — the easy one. Write a partial Kubernetes object; Kustomize merges fields by their semantics (lists keyed by name get merged, not replaced).

# overlays/prod/replicas-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 6

JSON 6902 patch — for when you need surgical precision (deleting an item from a list, modifying a specific array index).

patches:
  - target:
      kind: Deployment
      name: orders-api
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/image
        value: ghcr.io/acme/orders-api:v1.42.0

Default to strategic merge. Reach for 6902 only when strategic merge won't express what you need.

Generators

Kustomize can generate ConfigMaps and Secrets from files or literals, producing a hash-suffixed name so a content change forces a Deployment rollout:

configMapGenerator:
  - name: orders-api-config
    files:
      - config/app.yaml
    options:
      disableNameSuffixHash: false  # default; keep hashing on

secretGenerator:
  - name: orders-api-secrets
    envs:
      - .env.prod

Note: do not check real secrets into Git. Real secrets are pulled from our secret manager via the External Secrets Operator (Day 4) — generators are fine for non-sensitive config or for examples.

Common gotchas

namespace: in kustomization.yaml rewrites all resources into that namespace — including ClusterRoleBindings' subjects[].namespace, which is usually what you want, but be aware.
commonLabels applies to selectors too. Changing them on an existing Deployment is a breaking change because selectors are immutable. Use labels (with includeSelectors: false) for cosmetic-only labels.
Image overrides only match by name, not full reference. name: must match what's in the base manifest exactly.
Patches that don't match anything fail silently in older versions; modern Kustomize errors out. Always kustomize build locally before pushing.

Exercise (Day 1)

In a fresh repo, build the following structure for a toy hello-api service:

A base with a Deployment (1 replica, image nginxdemos/hello:latest), a Service, a ServiceAccount, and a ConfigMap.
A component at components/aws/ that adds an IRSA annotation to the ServiceAccount and an AWS Load Balancer Controller annotation to the Service. The values can be fake ARNs — we're practicing the pattern, not deploying it.
Overlays for dev-kind (local), staging-eks, and prod-eks with:
- Different namespaces.
- Different replica counts (1, 2, 4).
- Different image tags via the images: directive.
- The two EKS overlays mix in components/aws; the kind overlay does not.
- Prod-only: resource requests/limits added via a patch.
Validate each overlay: kustomize build overlays/prod-eks | kubectl apply --dry-run=server -f -.
Diff two overlays: diff <(kustomize build overlays/dev-kind) <(kustomize build overlays/prod-eks). Convince yourself the only differences are what you intended.

Push this repo to GitHub. You'll wire it to Flux tomorrow.

Day 2 — FluxCD

What FluxCD actually is

FluxCD (Flux v2) is a set of Kubernetes controllers — the GitOps Toolkit — that together implement the GitOps loop. The major ones:

Controller	What it does
source-controller	Fetches artifacts from Git, OCI registries, S3 buckets, Helm repos. Caches them as internal artifacts other controllers consume.
kustomize-controller	Takes a Kustomize build from a source and applies the result to the cluster. Reconciles continuously.
helm-controller	Renders and installs Helm charts; reconciles drift.
notification-controller	Sends events outward (Slack, MS Teams, generic webhooks) and receives inbound webhooks (GitHub push triggers).
image-reflector-controller + image-automation-controller	Watches container registries and writes back to Git to bump image tags.

Each controller watches its own CRDs, reconciles on a configurable interval, and exposes Prometheus metrics.

The reconciliation model

For a typical Git → Kustomize flow:

GitRepository CR    ───▶  source-controller fetches commit X        ───▶  internal artifact
                                                                                │
Kustomization CR    ───▶  kustomize-controller pulls that artifact,             │
                          runs `kustomize build`, applies to cluster, ◀─────────┘
                          records inventory of applied objects.

Every N seconds (default 1m for Kustomization, 1m for GitRepository):
   - re-check source for new commit
   - re-apply (server-side apply with field manager) to correct drift

Critically: kustomize-controller does server-side apply with an inventory. It knows what it applied last time and can prune objects you've removed from Git. Manual kubectl edit on a Flux-managed object will be reverted on the next reconciliation. This is a feature.

The CRDs you'll touch daily

`GitRepository`

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/acme/platform-config
  ref:
    branch: main
  secretRef:
    name: github-deploy-key

`Kustomization` (Flux's, not Kustomize's)

Important distinction: Kustomize has a kustomization.yaml file. Flux has a Kustomization Kubernetes CRD. Different things, same name. The Flux CRD references a path in a source, and that path contains a Kustomize kustomization.yaml file.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: orders-api-prod
  namespace: flux-system
spec:
  interval: 5m
  retryInterval: 1m
  timeout: 3m
  sourceRef:
    kind: GitRepository
    name: platform-config
  path: ./apps/orders-api/overlays/prod
  prune: true              # delete objects removed from Git
  wait: true               # wait for health check before marking ready
  targetNamespace: orders-prod
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: orders-api
      namespace: orders-prod

Key fields to internalize:

prune: true — without this, removing manifests from Git leaves orphans in the cluster.
wait: true — block reconciliation success on objects becoming healthy. Essential for dependent Kustomizations.
dependsOn: — order Kustomizations. E.g., apply cert-manager before any app that depends on Issuers.
postBuild.substituteFrom: — variable substitution into the rendered manifests (useful but use sparingly; prefer pure Kustomize).

`HelmRepository` + `HelmRelease`

For third-party charts:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 1h
  url: https://charts.bitnami.com/bitnami
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redis
  namespace: cache
spec:
  interval: 10m
  chart:
    spec:
      chart: redis
      version: "20.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    architecture: standalone

The `flux` CLI

Install it (brew install fluxcd/tap/flux or from releases). Daily-driver commands:

flux check                              # cluster prerequisites + components health
flux get sources git -A                 # list all GitRepositories
flux get kustomizations -A              # list all Flux Kustomizations and their status
flux reconcile source git platform-config -n flux-system
flux reconcile kustomization orders-api-prod -n flux-system --with-source
flux suspend kustomization orders-api-prod -n flux-system
flux resume kustomization orders-api-prod -n flux-system
flux trace deployment/orders-api -n orders-prod   # who manages this object?
flux events -n flux-system --watch       # live event stream

flux trace is the killer command. Given any Kubernetes object, it walks the chain back to the Git commit that produced it.

Repo layout — the monorepo pattern we use

platform-config/
├── clusters/
│   ├── dev/
│   │   ├── flux-system/        # Flux's own bootstrap (managed by Flux Operator)
│   │   ├── infrastructure.yaml # Kustomization → ./infrastructure/dev
│   │   └── apps.yaml           # Kustomization → ./apps/dev
│   ├── staging/
│   └── prod/
├── infrastructure/
│   ├── base/                   # cert-manager, ingress-nginx, monitoring stack
│   ├── dev/                    # overlay
│   ├── staging/
│   └── prod/
└── apps/
    ├── base/
    ├── dev/
    └── prod/

Each clusters/<env>/apps.yaml is one Flux Kustomization pointing at apps/<env>. That overlay itself includes per-app overlays. Flux follows the chain.

Exercise (Day 2)

Install Flux on yesterday's kind cluster using the bootstrap method — temporarily, just to feel it: flux bootstrap github --owner=YOU --repository=YOUR_REPO --branch=main --path=clusters/dev --personal.
Add yesterday's hello-api overlays to the repo at apps/dev/, apps/staging/, apps/prod/.
Write a Flux Kustomization CR in clusters/dev/apps.yaml that points at apps/dev/, with prune: true and a health check on the Deployment.
Commit, push, wait. Verify with flux get kustomizations and kubectl get pods -n <namespace>.
Change the replica count in Git, push, time how long until the cluster reflects it.
Manually kubectl scale the Deployment. Watch Flux revert it within one reconciliation interval.
Tear it all down before tomorrow — we're going to re-install Flux the Flux Operator way.

Day 3 — Flux Operator

Why an operator on top of Flux?

flux bootstrap is convenient for one cluster. It is awkward at scale because:

Upgrading Flux means re-running bootstrap or editing the bootstrap path's manifests by hand.
The bootstrap output is a flat dump of Flux's own manifests in your repo — noisy and tightly coupled.
Multi-cluster fleets need consistent Flux versions, components, and configuration; this is hard to express in plain bootstrap.

Flux Operator (fluxoperator.dev, built by ControlPlane and the CNCF Flux core maintainers) is a Kubernetes operator that manages Flux itself declaratively. Its features:

Declarative install/upgrade of Flux via the FluxInstance CRD.
ResourceSet API — a higher-level abstraction for templated, parameterized, multi-environment deployments and ephemeral environments tied to PRs.
Gitless GitOps — store desired state as signed OCI artifacts in a container registry, decoupling production clusters from Git access entirely (covered below).
Built-in web UI / status page for visibility into reconciliation.
Flux MCP server — exposes Flux state to AI assistants (Claude, Cursor) for natural-language troubleshooting.
Multi-tenancy lockdown, controller sharding, persistent storage for fleet operators.

We use it because it makes Flux itself a Kubernetes-native concern that can be managed the same way we manage everything else.

Installation

brew install controlplaneio-fluxcd/tap/flux-operator

(Other methods: Helm chart, Terraform, OperatorHub. We standardize on the Helm chart for cluster bootstrap.)

The `FluxInstance` CRD

This is where you describe what Flux you want running. Minimal example:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
spec:
  distribution:
    version: "2.x"                                          # pin a major; operator picks latest minor/patch
    registry: "ghcr.io/fluxcd"
    artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller
  cluster:
    type: kubernetes
    size: medium
    multitenant: false
    networkPolicy: true
    domain: "cluster.local"
  sync:
    kind: GitRepository
    url: "https://github.com/acme/platform-config"
    ref: "refs/heads/main"
    path: "clusters/dev"
    pullSecret: "flux-system"

The operator reads this, installs the listed controllers at the requested version, and creates the root GitRepository + Kustomization pointing at the sync.path. From that point on, Flux is bootstrapped and everything else flows from Git as before.

Upgrades: change distribution.version, commit, push. The operator rolls Flux forward.

Key things you can declare on a FluxInstance that you previously had to hand-roll:

Controller sharding for very large fleets.
Image pull credentials.
Persistent storage for the source-controller's artifact cache.
Strict multi-tenancy (cross-namespace source references denied by default).
Kustomize patches applied to Flux's own controller manifests (resource limits, node selectors, etc.).

Gitless GitOps with OCI artifacts

So far, the chain has been Git → Flux. There's an alternative model that Flux supports natively and that Flux Operator embraces: OCI → Flux. Manifests are packaged as OCI artifacts and pushed to a container registry (the same registries that already hold your images), and Flux pulls from the registry instead of cloning Git.

This is the "Gitless GitOps" pattern. The full picture:

Developer  ──push──▶  Git (still source of truth for humans)
                       │
                       ▼
                      CI/CD pipeline
                       │
                       │  flux push artifact oci://registry/configs:v1.42.0 \
                       │      --path ./manifests \
                       │      --source <git-url> --revision <sha>
                       │  cosign sign  ◀──── signs the artifact
                       ▼
                Container registry (ECR / GHCR / Artifact Registry)
                       │
                       ▼
                Flux in cluster ──pulls signed OCI artifact──▶ reconciles

Git is still the human source of truth — developers commit and review there. But production clusters never talk to Git. They pull pre-packaged, signed artifacts from a registry.

Why this matters

No Git credentials in production clusters. A compromised production cluster can't reach Git. ECR/GHCR auth (often via IRSA / Workload Identity) is enough.
Air-gapped / restricted environments. OCI artifacts can be mirrored into a private registry; clusters without internet access can still pull them.
Stronger supply-chain guarantees. Artifacts are immutable by digest, can be Cosign-signed, and can carry SLSA provenance attestations. Flux can be configured to refuse anything unsigned or unverified.
Faster reconciliation. Pulling a 100 KB OCI blob is faster than cloning a Git repo that may have grown large.
Better for monorepos. Each component publishes its own artifact; clusters pull only what they need, with independent versioning per artifact.
Promotion = digest update. Promoting from staging to prod is bumping a single immutable digest reference, not a Git merge.

The `OCIRepository` source

Instead of GitRepository, use OCIRepository:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: platform-config
  namespace: flux-system
spec:
  interval: 1m
  url: oci://ghcr.io/acme/platform-config
  ref:
    semver: ">=1.0.0"        # or tag: v1.42.0, or digest: sha256:...
  verify:                    # optional but recommended in prod
    provider: cosign
    secretRef:
      name: cosign-pub

Then a Flux Kustomization references it the same way it referenced a GitRepository:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: orders-api-prod
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: OCIRepository       # the only thing that changes
    name: platform-config
  path: ./apps/orders-api/overlays/prod-eks-eu-west-1
  # ... rest identical to the GitRepository-backed version

Everything downstream — Kustomize overlays, components, patches, health checks, prune, External Secrets — works identically. The only thing that changed is where the manifests come from.

Bootstrapping Flux Operator from OCI

FluxInstance can sync itself from an OCI source:

spec:
  sync:
    kind: OCIRepository
    url: oci://ghcr.io/acme/platform-config
    ref: "latest"
    path: "clusters/prod-eu-west-1"
    pullSecret: "ghcr-pull"

Now nothing in the cluster — not even Flux's own bootstrap config — touches Git. This is the configuration the Flux Operator docs call out as the production target for serious fleets; ControlPlane has a reference architecture and the RBC Capital Markets case study describing it at scale.

Publishing OCI artifacts from CI

The flux CLI ships the publisher:

# In CI, after a merge to main
flux push artifact \
  oci://ghcr.io/acme/platform-config:$(git rev-parse --short HEAD) \
  --path ./clusters \
  --source $(git config --get remote.origin.url) \
  --revision $(git rev-parse HEAD) \
  --provider generic

flux tag artifact oci://ghcr.io/acme/platform-config:$(git rev-parse --short HEAD) \
  --tag latest

# Sign with Cosign (keyless OIDC in GitHub Actions)
cosign sign --yes ghcr.io/acme/platform-config:$(git rev-parse --short HEAD)

The artifact is a small tarball of your manifests, addressable by tag or digest. Flux's OCIRepository pulls it on the configured interval.

When to choose which

Use Git source	Use OCI source
Small team, one or few clusters	Fleet of clusters, especially across networks/accounts
You want clusters to reflect main branch immediately	You want explicit, immutable releases promoted via CI
Hobby / dev / training	Production, compliance-bound, or air-gapped environments
You're learning Flux	You've operated Flux for a while and want stronger guarantees

Our team uses OCI source (immutability, signing, no Git creds in prod).

Related: `ArtifactGenerator` and gitless image automation

Flux Operator extends this further with the ArtifactGenerator CRD (build artifacts in-cluster from other sources) and gitless image automation via ResourceSets — bumping image versions without writing back to Git. Read Gitless image automation when you're comfortable with the basics.

`ResourceSet` — the differentiator

A ResourceSet is a templated bundle of Kubernetes resources, parameterized by inputs. Where Kustomize gives you static overlays, ResourceSets give you dynamic instantiation: one definition, many instances, driven by external state.

Conceptual example — one ResourceSet, one instance per developer:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: dev-sandboxes
  namespace: flux-system
spec:
  inputs:
    - tenant: alice
      replicas: 1
    - tenant: bob
      replicas: 1
    - tenant: carol
      replicas: 2
  resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: sandbox-<< inputs.tenant >>
    - apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: hello
        namespace: sandbox-<< inputs.tenant >>
      spec:
        replicas: << inputs.replicas >>
        # ...

The operator expands this into one set of resources per input row.

`ResourceSetInputProvider` — dynamic inputs

The real power: drive the input list from external systems. Built-in providers:

GitHub Pull Requests — one ephemeral environment per open PR, torn down when the PR merges or closes.
GitLab Merge Requests — same model.
GitLab Environments — sync with GitLab's environments API.
Git feature branches — one environment per long-lived branch.

Example provider:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  name: orders-api-prs
  namespace: flux-system
spec:
  type: GitHubPullRequest
  url: https://github.com/acme/orders-api
  secretRef:
    name: github-token
  filter:
    labels:
      - "preview"

A ResourceSet then references this provider, and the operator creates/destroys preview environments as PRs labeled preview open and close. This is the "self-service environments" feature on the Flux Operator homepage.

Read Application definitions and GitHub PRs Integration when you're ready to use this for real.

Web UI

The operator ships a lightweight status page:

kubectl -n flux-system port-forward svc/flux-operator 9080:9080
# open http://localhost:9080

It shows reconciliation status across all Flux objects, recent events, and lets you trigger reconciliations from the browser. For shared team access, expose it via Ingress with SSO (Dex, Keycloak, Microsoft Entra — see the Web UI guides). On our team this is the first place we look when someone says "is my deploy stuck?"

MCP Server (AI-assisted ops)

The Flux Operator MCP server exposes Flux state and (optionally) actions through the Model Context Protocol, so AI assistants can answer questions like "Why is the orders-api HelmRelease failing in staging?" by querying the cluster directly.

brew install controlplaneio-fluxcd/tap/flux-operator-mcp

Then add it to your assistant's MCP config:

{
  "flux-operator-mcp": {
    "command": "flux-operator-mcp",
    "args": ["serve", "--read-only=true"],
    "env": { "KUBECONFIG": "/path/to/.kube/config" }
  }
}

Default to --read-only=true until you're confident. The full tools and prompts reference is on the docs site under MCP Server.

Exercise (Day 3)

Tear down yesterday's bootstrap-installed Flux.
Install Flux Operator on the kind cluster (Helm chart).
Create a FluxInstance that installs Flux 2.x with the four core controllers, pointed at the same platform-config repo you used yesterday. Apply and verify with kubectl get fluxinstance -A and the status page.
Confirm that yesterday's apps still reconcile.
Write a trivial ResourceSet that creates a Namespace and a ConfigMap for each of three "tenant" inputs. Apply and verify.
Gitless mode (stretch): publish your platform-config repo as an OCI artifact to GHCR using flux push artifact. Add a second FluxInstance (or modify the existing one) to sync from OCIRepository instead of GitRepository. Confirm the same workloads reconcile from the registry.
(Stretch) Wire up the MCP server to your local AI assistant in read-only mode and ask it to summarize the cluster's Flux state.

Day 4 — Putting It Together

Multi-environment promotion

Our model: same Git repo, environment as a directory. A change moves through environments by being merged from a dev-pointing branch to a prod-pointing branch, or by editing the overlay's image tag.

Three common promotion strategies:

A) Tag-based promotion (recommended for Git-source app deployments). The image tag in apps/<env>/<service>/kustomization.yaml is the source of truth for what's deployed.

images:
  - name: ghcr.io/acme/orders-api
    newTag: v1.42.0

A merge to main bumps the dev overlay's tag. Promotion to staging/prod is a separate PR that bumps those overlays. Each promotion is an auditable Git change.

B) Branch-based promotion. Different Flux GitRepository refs per environment: refs/heads/dev, refs/heads/staging, refs/heads/main. Promotion is a merge between branches.

C) OCI digest-based promotion (recommended for OCI-source production). Each environment's OCIRepository is pinned to an immutable digest or tag. Promotion is a PR that updates prod-eu-west-1's artifact reference to the digest that's been running cleanly in staging.

spec:
  ref:
    digest: sha256:7f3d8c...e91     # promoted from staging unchanged

The strong guarantee: the artifact prod runs is byte-for-byte the one staging ran. No re-rendering, no surprises from a Kustomize change that landed between staging and prod runs.

We prefer C for OCI-sourced staging/prod — flatter Git history, immutable releases, no long-lived branches.

Image automation (optional)

For dev environments, manually editing tags on every push is friction. Flux's image-reflector-controller and image-automation-controller can watch a registry and write commits back to Git that bump tags automatically.

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: orders-api
  namespace: flux-system
spec:
  image: ghcr.io/acme/orders-api
  interval: 5m
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: orders-api
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: orders-api
  policy:
    semver:
      range: ">=1.0.0 <2.0.0"
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: orders-api-dev
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: platform-config
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxbot@acme.io
        name: fluxbot
      messageTemplate: "chore(images): {{range .Updated.Images}}{{.}} {{end}}"
    push:
      branch: main
  update:
    path: ./apps/dev
    strategy: Setters

Enable this for dev only. Never auto-bump prod.

Flux Operator also offers gitless image automation via ResourceSets — image bumps without writing back to Git. See Gitless image automation.

Secrets — External Secrets Operator

Plaintext secrets in Git are not acceptable. Our pattern: secrets live in a real secret manager (AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault, depending on the cluster), and the External Secrets Operator (ESO) projects them into Kubernetes Secret objects on demand.

Why ESO rather than committing encrypted secrets to Git:

Rotation works. Rotate the value in the secret manager; ESO refreshes the in-cluster Secret on its interval. No Git change required.
Single source of truth for secrets. The same vault feeds CI, infra (Terraform), and Kubernetes. No second copy of the secret material lying around in a Git repo.
No key management ceremony. No team-wide encryption keys to provision, rotate, or revoke; cluster identity (IRSA on AWS, Workload Identity on GCP) is what authorizes access.
Auditability. The secret manager logs every access.

The shape of what goes in Git:

# A SecretStore tells ESO where to fetch from. One per namespace, or a
# ClusterSecretStore for cluster-wide. Authorization is via the cluster's
# workload identity — no static credentials in the manifest.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: orders-prod
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa     # IRSA-bound to an IAM role with read access
---
# An ExternalSecret declares: "fetch these keys from the store and produce
# a Kubernetes Secret of this shape." This manifest is safe to commit.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: orders-api-db
  namespace: orders-prod
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: orders-api-db                  # the resulting K8s Secret
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: prod/orders-api/db
        property: url
    - secretKey: DATABASE_PASSWORD
      remoteRef:
        key: prod/orders-api/db
        property: password

The application Deployment consumes the resulting Secret the normal way — envFrom.secretRef or volumeMounts. The application doesn't know or care that ESO populated it.

Operational notes:

ESO itself is installed via a Flux HelmRelease (it's a chart). Bootstrap order: install ESO before any app that has ExternalSecret resources, using a Kustomization dependsOn: chain.
The IAM role / Workload Identity that ESO uses must be scoped tightly — read-only, prefix-restricted to the keys that namespace needs. We do not give cluster-wide read access to the entire vault.
For high-churn secrets (database passwords on a short rotation), tune refreshInterval accordingly; the default 1h is fine for most things.
If a Secret is required before a workload can start (e.g. image pull credentials), make sure the dependent Deployment can tolerate the few seconds ESO needs after the namespace is created. Helm hooks and wait: true on the parent Flux Kustomization usually handle this cleanly.

For full docs: https://external-secrets.io/.

Multi-tenancy basics

If teams share a cluster, you want:

Each team scoped to its own namespace(s).
A team's Flux Kustomization impersonates a ServiceAccount in their namespace via spec.serviceAccountName, so RBAC enforces what they can apply.
FluxInstance.spec.cluster.multitenant: true to disable cross-namespace source references by default.

This pushes the question "who can deploy what" out of Flux and into standard Kubernetes RBAC.

Exercise (Day 4)

In your training repo, add a real ImageRepository + ImagePolicy + ImageUpdateAutomation for the hello-api dev overlay (use a real public image like nginxdemos/hello and pin to a semver range).
Install External Secrets Operator on your kind cluster (Helm chart, via a Flux HelmRelease). Stand up a fake "vault" using ESO's fake provider so you don't need real cloud credentials. Define an ExternalSecret that materializes a Secret from the fake store, and consume it from the hello-api Deployment via envFrom. Verify the env var lands in the pod.
Write a short design note in your repo's README.md describing how you'd structure overlays for three real services with different teams. Bring it to your next 1:1.

Day 5 — Operations

Debugging the GitOps loop

When something hasn't deployed, the question is where in the chain it broke. Walk it in order:

1. Is the source healthy?

flux get sources git -A
# Look at READY and STATUS columns. If not ready, the URL or auth is wrong.
kubectl describe gitrepository platform-config -n flux-system

2. Is the Kustomization healthy?

flux get kustomizations -A
kubectl describe kustomization orders-api-prod -n flux-system
# Common failures:
#   - Build error (Kustomize syntax)
#   - Dry-run error (CRD missing, namespace missing)
#   - Health check timeout (deployment didn't roll out)

3. Are the workloads actually running?

kubectl get pods -n orders-prod
kubectl describe pod <name> -n orders-prod
kubectl logs <pod> -n orders-prod

4. Who manages this object anyway?

flux trace deployment/orders-api -n orders-prod

This walks back: Deployment → managed by Kustomization X → which uses GitRepository Y → at commit Z.

5. Force a reconciliation.

flux reconcile source git platform-config -n flux-system
flux reconcile kustomization orders-api-prod -n flux-system --with-source

--with-source chains: pull Git first, then re-apply.

6. Inspect events in real time.

flux events -n flux-system --watch
kubectl get events -n flux-system --sort-by=.lastTimestamp

Drift

Flux corrects drift on every reconciliation. If someone insists on a manual change, two options:

Suspend the relevant Kustomization (flux suspend kustomization X). The cluster will hold your manual change until you flux resume, at which point the Git state wins again.
Make the change in Git. Always the right answer for anything you want to keep.

Add a Prometheus alert on gotk_reconcile_condition{type="Ready",status="False"} so persistent failures page someone.

Notifications

Wire notification-controller to Slack so the team sees deploys and failures:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack-deploys
  namespace: flux-system
spec:
  type: slack
  channel: deploys
  secretRef:
    name: slack-webhook  # contains an "address" key with the webhook URL
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: prod-alerts
  namespace: flux-system
spec:
  providerRef:
    name: slack-deploys
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: '*'
      namespace: flux-system

Set eventSeverity: info on a separate alert if you also want successful reconciliations announced (noisy but useful at first).

Common failure modes

Symptom	Likely cause	Fix
`Kustomization` stuck `Reconciling`, never `Ready`	A health-checked object hasn't gone Ready (CrashLoop, ImagePullBackOff)	Fix the workload; check `kubectl describe pod`
`prune: true` deleted something unexpected	A resource was removed from the rendered output (often by an accidental Kustomize change)	`git revert`; consider adding the object to `commonAnnotations: kustomize.toolkit.fluxcd.io/prune: disabled` if it must persist outside Flux
`Kustomize build failed: accumulating resources`	Path wrong or missing `kustomization.yaml`	`kustomize build <path>` locally to repro
Image not updating	Image pull policy `IfNotPresent` with a mutable tag	Use immutable tags; let imageautomation bump the tag in Git
Drift not corrected	The object isn't actually owned by Flux (created by `kubectl apply` instead of Flux)	Adopt it: add to Git, let Flux apply with server-side apply force conflicts
`Kustomization` says ready but cluster state is wrong	Looking at the wrong Kustomization, or a child Kustomization is failing	`flux get kustomizations -A` to find the broken one; `flux trace` on the object

On-call expectations

The first response to a Flux-related page is not kubectl apply. It's flux get kustomizations -A and flux events.
Never edit a Flux-managed object directly except as a deliberate flux suspend-protected hotfix, and only with a follow-up PR within the same on-call shift.
All hotfixes land in Git within 24 hours. No exceptions. This is the deal we make with GitOps.

Exercise (Day 5)

Deliberately break things in your training repo and practice diagnosis:
- Bad YAML syntax in a Kustomize overlay.
- A health-checked Deployment with an image that doesn't exist.
- A deleted base resource that the overlay still references.
- A manual kubectl edit on a Flux-managed Deployment.
For each, write down: (a) what the symptom looked like, (b) which command surfaced the root cause, (c) how you fixed it.
Set up the Slack notification provider against the team's #training channel. Trigger a reconciliation failure and confirm you see it.

Cheat Sheets

Kustomize commands

kustomize build <dir>                              # render
kubectl kustomize <dir>                            # same, via kubectl
kustomize build <dir> | kubectl apply -f -         # render + apply (DON'T do this against prod; let Flux do it)
kustomize build <dir> --enable-helm                # render Helm charts referenced from kustomization.yaml
kustomize edit set image foo=bar:v2                # mutate kustomization.yaml programmatically
kustomize edit add resource thing.yaml             # add a resource

Flux CLI

# Health & visibility
flux check                                          # cluster + components health
flux check --pre                                    # pre-flight before install
flux get all -A                                     # everything Flux manages
flux get sources git -A
flux get sources oci -A
flux get kustomizations -A
flux get helmreleases -A
flux events -A --watch                              # live event stream
flux stats -A                                       # reconciliation counts

# Force reconciliation
flux reconcile source git <name> -n <ns>
flux reconcile source oci <name> -n <ns>
flux reconcile kustomization <name> -n <ns> --with-source
flux reconcile helmrelease <name> -n <ns>

# Pause / resume
flux suspend kustomization <name> -n <ns>
flux resume kustomization <name> -n <ns>

# Trace ownership
flux trace <kind>/<name> -n <ns>

# Export / diff
flux diff kustomization <name> --path <local-path>  # what would change if you applied this local checkout
flux export source git <name> -n <ns>

# OCI artifacts (publishing from CI)
flux push artifact oci://<registry>/<repo>:<tag> \
  --path ./manifests \
  --source <git-url> --revision <sha>
flux tag artifact oci://<registry>/<repo>:<tag> --tag latest
flux pull artifact oci://<registry>/<repo>:<tag> --output ./out

Flux Operator CLI

flux-operator install -f flux-instance.yaml         # install operator + apply FluxInstance
flux-operator create secret basic-auth flux-system --namespace=flux-system --username=git --password=$TOKEN
kubectl get fluxinstance -A
kubectl get fluxreport -A                            # operator's view of Flux's health
kubectl get resourceset -A
kubectl get resourcesetinputprovider -A

Kustomize `kustomization.yaml` quick reference

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: my-ns                  # set namespace on all resources
namePrefix: dev-                  # prefix all resource names
nameSuffix: -v2

commonLabels:                     # applied to objects AND selectors (be careful)
  app: orders-api

labels:                           # applied to objects only
  - pairs:
      tier: backend
    includeSelectors: false

commonAnnotations:
  owner: platform-team

resources:                        # include other manifests or kustomization dirs
  - deployment.yaml
  - ../../base
  - github.com/acme/shared//k8s/base?ref=v1.0.0

patches:                          # strategic merge or JSON 6902
  - path: replicas-patch.yaml
  - target:
      kind: Deployment
      name: orders-api
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 6

images:                           # override container images
  - name: ghcr.io/acme/orders-api
    newName: ghcr.io/acme/orders-api  # optional
    newTag: v1.42.0

configMapGenerator:
  - name: app-config
    files: [app.yaml]
    literals: [LOG_LEVEL=info]

secretGenerator:
  - name: app-secrets
    envs: [.env]

replicas:
  - name: orders-api
    count: 6

components:                       # reusable transformation modules
  - ../../components/monitoring

Flux `Kustomization` quick reference

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: orders-api-prod
  namespace: flux-system
spec:
  interval: 5m                    # how often to reconcile
  retryInterval: 1m               # on failure
  timeout: 3m
  sourceRef:                      # where the manifests come from
    kind: GitRepository           # or OCIRepository or Bucket
    name: platform-config
  path: ./apps/orders-api/overlays/prod
  prune: true                     # delete what's removed from Git
  wait: true                      # block on health checks
  force: false                    # force replace on immutable field conflicts (use sparingly)
  targetNamespace: orders-prod    # override namespace on all objects
  serviceAccountName: orders-deployer  # impersonate for multi-tenancy
  dependsOn:
    - name: cert-manager
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: orders-api
      namespace: orders-prod
  postBuild:
    substitute:
      cluster_name: prod-eu-1
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

`OCIRepository` quick reference (Gitless GitOps)

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: platform-config
  namespace: flux-system
spec:
  interval: 1m
  url: oci://ghcr.io/acme/platform-config
  ref:
    # exactly one of:
    tag: v1.42.0
    # semver: ">=1.0.0 <2.0.0"
    # digest: sha256:7f3d8c...e91
  layerSelector:
    mediaType: "application/vnd.cncf.flux.config.v1+yaml"
    operation: extract
  secretRef:                      # registry auth (basic / docker config)
    name: ghcr-pull
  # or use cloud workload identity:
  # provider: aws | azure | gcp
  verify:                         # cosign signature / SLSA provenance verification
    provider: cosign
    secretRef:
      name: cosign-pub

Useful links

Flux docs: https://fluxcd.io/flux/
Flux GitOps Toolkit components: https://fluxcd.io/flux/components/
Kustomize docs: https://kubectl.docs.kubernetes.io/references/kustomize/
Kustomize Components reference: https://kubectl.docs.kubernetes.io/guides/config_management/components/
Flux Operator docs: https://fluxoperator.dev/docs/
Flux Operator FluxInstance reference: https://fluxoperator.dev/docs/crd/fluxinstance/
Flux Operator ResourceSet reference: https://fluxoperator.dev/docs/crd/resourceset/
Flux Operator Gitless GitOps overview: https://fluxoperator.dev/gitless-gitops/
Flux Operator OCIRepository reference: https://fluxoperator.dev/docs/crd/ocirepository/
ControlPlane reference architecture (OCI + Flux Operator): https://fluxcd.control-plane.io/guides/d2-architecture-reference/
Flux example repo (kustomize + helm): https://github.com/fluxcd/flux2-kustomize-helm-example
Flux Operator local-dev (OCI registry on kind): https://github.com/controlplaneio-fluxcd/flux-operator-local-dev
External Secrets Operator: https://external-secrets.io/

Sign-off checklist

By the end of Week 1, you should be able to, without help:

project0/flux-course.md