Note: This video and podcast was generated using AI, adapting the original content and technical insights created by the author of this article.

Why does GitOps stop scaling before the tools do?

One repository, one cluster, a controller reconciling the two. Every tutorial shows it, every proof of concept passes, and that shape quietly becomes the shape of everything after it.

Then the fleet grows and the same four cracks appear, in roughly the same order, in almost every team I talk to.

Repository structure goes one of two bad ways. A repo per cluster, where a baseline change becomes fifty pull requests. Or everything in one repo, where every team contends in the same merge queue and a bad merge reaches every cluster at once.

Then, promotion becomes folklore. “Deploy to dev, then staging, then production” sounds obvious until you ask what exactly moves between them. Nobody can say: a branch, a directory copy, a version bump. Most teams improvise it once, and the improvisation turns load-bearing.

Rendering accumulates layers, until answering “what YAML actually applies to cluster 37?” means running the toolchain locally and hoping your flags match CI’s. And drift stops being a technical question. Someone edits production during an incident: crime or necessity. A fleet needs one answer to that, not fifty.

These are design debts. The cost of each one scales with cluster count.

My position on this is specific, and it should change how you read the rest. Most weeks I am talking to the people who run these fleets, and working hands-on in the development and QA environments where their problems surface first. I have not operated a fifty-cluster production fleet myself. That seat is more useful than it sounds, because most of what follows gets found in preproduction by someone who did not know the rules yet.

Which repository topology should you choose?

There are three pure options, and a hybrid one.

Option 1

The monorepo puts every cluster, environment and team in one place. You get atomicity and easy discovery: one commit changes the baseline everywhere, and nobody hunts for where something lives. You pay in blast radius, merge contention and CI time that grows with the fleet. Permissions get awkward too, since path-based access control through CODEOWNERS and equivalents is coarser than per-repo authorization.

The CI cost deserves a caveat. A naive pipeline renders and diffs everything on every pull request and does get slow, but that is a configuration choice rather than a property of monorepos. Path-based change detection keeps the check fast enough that reviewers wait for it. A gate people can bypass stops being a gate.

Option 2

A repo-per-team gives each application team its own repository, syncing into namespaces the platform grants. It matches how organizations actually work, and blast radius stays team-sized. Fleet-wide changes are the cost: anything crossing every team needs coordination machinery, and consistency rests on convention rather than one enforcing commit.

Option 3

A repo-per-cluster mirrors infrastructure exactly and appeals where an auditor reviews one repository per certified system. At scale it is the weakest pure option: there is duplication everywhere, baseline updates must be done as bulk PR campaigns, there is steady divergence between clusters that were supposed to be identical.

Option 4

The hybrid teams I talk to keep converging on this option: one platform repository holding cluster definitions, baseline add-ons, and policy, organized by cluster group rather than by individual cluster plus per-team application repositories. The platform repo drives ApplicationSets or Kustomizations that fan out by label, so a baseline change is one commit with an explicit rollout order.

By cluster group I mean a set of clusters you treat identically for delivery, production in one region, or all internal tooling clusters. That way, there are two collisions to watch. Cluster API uses ClusterClass for something different, a template for provisioning topology, so if you run CAPI, pick a word that does not collide. And Rancher Fleet ships a ClusterGroup CRD that is close to what I mean here, which I should flag rather than let pass, because Fleet is my employer’s product.

The fan-out is what makes grouping work as an organizing idea:

Listing 1

#Platform repo: one ApplicationSet delivers the monitoring baseline

#to every EU production cluster, present and future

apiVersion: argoproj.io/v1alpha1

kind: ApplicationSet

metadata:

name: baseline-monitoring

spec:

goTemplate: true

goTemplateOptions: ["missingkey=error"]

generators:

- clusters:

selector:

matchLabels:

environment: production

region: eu

template:

metadata:

name: 'monitoring-{{.nameNormalized}}'

spec:

project: platform

source:

repoURL: https://git.example.com/platform/baseline

targetRevision: refs/heads/main

path: monitoring/overlays/production

destination:

server: '{{.server}}'

namespace: monitoring

syncPolicy:

automated:

prune: true

selfHeal: true

syncOptions:

- CreateNamespace=true

Registering cluster 51 with the right labels is the whole onboarding step for the baseline. Nobody writes51 Applications by hand, and nobody forgets one.

Four of those lines are easy to leave out and each one quietly breaks that promise. Drop syncPolicy and the generator creates an Application that sits OutOfSync forever. Drop CreateNamespace=true and the sync fails wherever monitoring doesn’t exist yet. Drop targetRevision and it means HEAD, which is the opposite of what the promotion section will argue for. And nameNormalized is not decoration: register a cluster without naming it explicitly and you inherit the kubeconfig context, which on EKS looks like arn:aws:eks:eu-west-1:123456789012:cluster/prod-eu. Colons and slashes are illegal in an object name, so {{.name}} builds an Application the API server rejects.

Copy missingkey=error into everything else you template. It makes a mistyped variable fail instead of rendering as an empty string, which is the silent-failure class the next section is built on.

The principle underneath is that repository boundaries should follow ownership, not infrastructure. Clusters get replaced. The people who own things are the stable unit.

One caveat before you consolidate onto that hybrid. A shared platform repository means a shared blast radius and a shared change freeze, so if two groups of clusters answer to different regulators, run on separate on-call rotations and share no workloads, running them as one fleet buys a consistency nobody asked for. Consolidation is the default recommendation in this space partly because it is what tooling vendors are organized to sell, and I work for one of them. It is often right. It is not automatically right, and splitting a fleet later is harder than never merging it (Fig. 1).

Fig. 1: Repository topologies and how each one fails. Four options side by side, each with its dominant failure mode.

Helm, Kustomize, or plain YAML: what actually renders?

Every GitOps engine applies rendered YAML in the end. Go back to the sentence I opened with, about which file wins. That question is always about this layer.

Plain YAML is explicit and diff-friendly, and it stops working past a handful of variants, because every variant is a copy and copies drift.

Kustomize keeps everything as YAML and layers patches through overlays. It handles environment variance well, dev patching replicas down while production patches resources up, and stays readable to about three levels. Past that, tracing a value’s origin becomes archaeology.

Why does cluster 37 mount an extra CA bundle? The base does not define it. The production overlay patches trust configuration fleet-wide. A cluster-level patch removes it again, because cluster 37 predates the shared PKI. And an admission webhook injects a different one at deploy time, which Git never sees. Four layers, three repositories, one afternoon gone (Fig. 2).

Fig. 2: Where did this value come from? Four layers touch the CA bundle on cluster 37, in order of application. Git can explain three.

Helm brings real templating and a large chart library, at the cost of logic living in templates. Conditionals, loops, values files that interact in ways only rendering reveals. Three hundred configurable values stop being configuration. It is a template language you now maintain, without the tooling a language would give you.

Two practices matter more than the choice itself. Cap rendering depth on purpose: one shared base, one environment layer, one cluster-specific layer, and treating a request for a fourth as a design smell. Then render in CI and diff the output, so every pull request shows reviewers the rendered delta against what each affected cluster runs today.

Pin the toolchain versions too. Argo CD ships its own bundled Helm and Kustomize, so if CI runs different ones, a manifest can render cleanly in review and differently in the controller. Helm 3.18.5 also swapped the JSON Schema library underneath values validation as part of a CVE fix, changing reference resolution and type checking, so charts that validated before can fail after an upgrade for reasons that have nothing to do with the chart. That skew has not bitten me personally, and I would rather say so than borrow someone else’s war story, but the failure mode is real and the fix costs nothing.

How do you test the delivery pipeline itself?

Application code gets tests. The chart that deploys it gets a review and a hope. At two clusters that asymmetry survives. At fifty it does not, and preproduction is where these failures actually get caught.

A team I worked with was converting an application’s deployment into a Helm chart. Somebody mistyped a variable name in the values file. One wrong character in a key. Helm did what Helm does with a key it does not recognize: nothing. No error, no warning. The value never reached the template, the chart fell back to its own default, and the manifest went out configured differently than anyone intended. QA caught it during testing.

Everybody already says “render your manifests.” What matters here is what the rendered diff would have shown. Nothing. An empty diff, because the key never landed and the default had not moved. A reviewer sees that and reads it as safe. What it means is that the change does not exist.

So the rule I would give now is that an empty rendered diff after an intentional change is an alarm, not an all-clear. If you edited configuration and the output did not move, find out why before you merge.

The specific defense against that failure is a values schema. Helm validates values files against values.schema.json if the chart ships one, and setting additionalProperties: false makes an unrecognized key a hard error instead of a silent shrug:

Listing 2

{

"$schema": "https://json-schema.org/draft-07/schema#",

"type": "object",

"additionalProperties": false,

"properties": {

"replicaCount": { "type": "integer", "minimum": 1 },

"resources": {

"type": "object",

"additionalProperties": false,

"properties": {

"limits": { "type": "object" },

"requests": { "type": "object" }

}

}

}

}

Now look at what that schema does not do, because it is the same trick played on you a second time. additionalProperties is not recursive. The schema above rejects limits, since resources is closed. It accepts memory inside limits, since limits is not. You added the guard, the guard went green, and the exact typo you were afraid of walked through it. An empty diff, one layer down.

Do not fix that by closing limits. Kubernetes resource names are extensible, so hugepages-2Mi and nvidia.com/gpu are valid keys, and a closed list breaks every GPU user of your chart at install time. Close the levels whose vocabulary really is fixed, and be able to name the ones that are not.

Two smaller caveats. Schemas interact badly with chart dependencies, since subchart values and the global key need declaring or validation rejects them. And a schema is only a guarantee while nobody skips it: Helm has –skip-schema-validation and Argo CD exposes the same escape hatch as spec.source.helm.skipSchemaValidation, so that flag belongs in your policy checks like everything else.

Around that sits the rest of the pyramid. Static validation on every pull request: schema checks with kubeconform or a server-side dry run against the Kubernetes versions you actually run, which catches the manifest that is valid on your newest cluster and rejected by your oldest, plus linting for missing limits, missing probes, floating tags.

Watch how that check gets configured, because it is the same failure again. kubeconform ships upstream Kubernetes schemas only, so on a platform repo full of custom resources it fails loudly on every one it cannot find. That wall of red is irritating, so pipelines reach for -ignore-missing-schemas, and from then on the check is green and most of the repo is unvalidated. Generate schemas from your CRDs instead, and print the skipped count somewhere a human sees it. Skipped is not valid.

Then policy testing, in both directions. Run your admission policies against rendered manifests in CI, so a workload production would reject the pull request instead of the sync. Kyverno and Gatekeeper both ship CLI modes for this, and conftest covers the same ground in CI if you would rather write the rules in Rego than run an admission controller. Test the policies themselves: a policy change that accidentally blocks every deployment is a fleet outage written in YAML. A corpus of known-good and known-bad manifests, wired as a required check on the policy repo, catches most of that.

At the top, an ephemeral cluster pointed at the proposed commit to verify the world converges. This is the only layer that catches ordering problems, the CRD that must exist before the resource using it. It costs minutes and money, so reserve it for platform-repo changes and run static checks on everything else (Fig. 3).

Fig. 3: The delivery pipeline test pyramid. Three tiers by cost and coverage, with what each one catches and where it runs.

How should changes promote across environments?

Two promotion patterns do not survive scale. Git branches per environment turn every promotion into conflict resolution, and the branches drift. Manual directory copies are unauditable, and somebody always forgets a file.

What works is promotion as a version pin change. Environments are directories, each pinning explicit versions: an image tag, a chart version, a Git SHA. Promotion is a commit updating the pin in the next environment’s directory, written by automation and reviewed by whoever owns that gate. Git history becomes the audit log at no extra cost.

Automate the mechanical part. When a version has soaked in dev without tripping your error and latency objectives, a bot opens the pull request against staging. The human approves the gate and never writes the diff.

Soak times don’t transfer. Teams copy whatever number an article suggests and then discover it encodes a release cadence they don’t have. If you deploy twice a day, a twenty-four hour dev soak is a change advisory board with extra steps. Set it long enough for your slowest realistic failure to surface, which for memory leaks and connection pool exhaustion is longer than most people guess. Then shorten it when nothing has caught anything in a quarter. A CI job that bumps a pin and opens a pull request covers this with no new component. Kargo formalizes the same pattern, and Argo CD Image Updater or Flux image automation cover narrower slices.

At fleet scale, promotion gains a second dimension: not just dev to production, but across the production fleet. Roll out in waves, low-risk clusters first, then the standard ones, then the revenue-critical ones, with soak time and health checks between. ApplicationSet progressive syncs express this directly, with a maxUpdate cap on how many clusters move at once, though they remain beta and opt-in behind –enable-progressive-syncs. Flux gives you the ordering half through dependsOn and no concurrency cap, and there is a sharp edge in it: a Kustomization defaults to wait: false, so unless the dependency sets wait: true or explicit health checks, you are sequencing on successful apply rather than on healthy workload. That distinction stays invisible until a wave rolls forward over crash-looping pods. Either way the wave order belongs in code, not in a runbook (Fig. 4).

Fig. 4: Promotion has two dimensions. A version pin moves right through environments, then down through fleet waves. Both directions are commits.

Who is allowed to kubectl, and what happens to drift?

The usual version of this story is about discipline: someone bypasses the pipeline during an incident and fights the controller. The one I actually watched says something that framing misses.

In 2024 I was advising a blockchain infrastructure project, running Kubernetes with Flux managing the web application layer. During load testing, a regular team member fell ill and someone else stepped in to cover. That person went looking for Argo CD, did not find it, and reasonably concluded the environment had no GitOps configured yet.

So they started changing the Deployment’s replica count by hand, watching how the application behaved under load. Each change held for a while and then quietly reverted, because Flux reconciled on a five minute interval and put the declared value back. Five minutes is long enough to scale up, watch the application respond, write down a result, and come back later to a number nobody changed. The results looked inconsistent rather than obviously wrong, which is why it ran for a while before anyone asked why the values kept moving on their own.

Once we understood what had happened, we stopped touching the cluster directly, changed the replica count in Git, and re-ran the tests properly.

Flux was working exactly as designed. The failure was that a person joining the environment had no way to find out which GitOps tool governed it, and nobody had thought to tell them.

Reconciliation is invisible by design, and it was also, in that moment, the most important fact about the environment. In Flux v2 spec.interval is a required field with no silent default, so somebody typed that five minute window, most likely by copying a scaffold rather than by reasoning about it. Which is the point. The configuration was correct. It just was not discoverable, and nobody had ever needed it to be.

Make the controller discoverable: a ConfigMap at a known name in kube-system, an annotation on every namespace naming the reconciler and its source, a line in the repo README, anything that answers “what reconciles this?” without asking a human. And when reconciliation is about to fight someone, say so, which is what an annotation or an admission warning on manual edits buys you.

The rest of the drift policy is less about incidents than most articles suggest. Self-heal on for platform baseline components everywhere, because nobody should be hand-editing the policy engine. For application workloads in production, either self-heal with a documented break-glass path, or alert-on-drift with a deadline you enforce. Break-glass means a named role, a logged justification, a scoped and time-bounded pause, and a page to the owning team to codify whatever they did.

Mechanically it is unglamorous, with one trap worth finding before the night you need it. Disabling automated sync works on a standalone Application. On one generated by the ApplicationSet you built earlier, the controller reverts your pause on its next pass, because the template is the source of truth. That needs ignoreApplicationDifferences on /spec/syncPolicy, configured in advance, which is the real lesson: a break-glass path has to be built before it is used. Otherwise use a deny sync window with manualSync enabled, freezing automation while a named human can still act. In Flux, suspend the Kustomization. Whichever you pick, have the automation that flips it post to chat.

Drift that outlives the incident is the real problem, because it quietly makes Git wrong about production, and the first time Git is wrong, people stop trusting the system.

So what should you fix first?

If your fleet is small, you are in the cheap window. Pick the hybrid topology, cap rendering depth at three layers, adopt version-pin promotion with automated pull requests, and write down the drift policy before an incident writes it for you. Each costs a few times more to retrofit once fifty clusters depend on the old shape.

If you are already at 50 with the cracks showing, start with promotion. It is the highest-friction daily pain and the change is additive rather than structural. Repository topology comes next and is the hardest, since it means moving live workloads between repositories without a freeze, one team at a time with both paths valid during the transition. Rendering depth after that. Drift policy you can fix this week with a document and a controller flag.

One more thing, because the sequence is the easy half.

Both failures I described here were caught in preproduction, and neither was found by the people who built the system. A QA engineer found the typo. A stand-in who didn’t know which tool was running found the drift. Two cases don’t make a pattern, but the mechanism behind them is easy to state and you can test it on your own team this month.

Conclusion

Whoever built your delivery pipeline knows which repository wins and which overlay applies. That knowledge is exactly what makes the missing documentation invisible to them. The person who joined last week is the only one who doesn’t know, which makes them the only honest audit you have. Most teams treat that first confused week as an onboarding cost. It is closer to free penetration testing of your platform’s assumptions.

So: put new people in preproduction, which you were going to do anyway, and write down every question they ask that nobody could answer from a document. That list is your backlog, it is more accurate than the one you wrote yourselves, and it costs nothing to collect.

The tools scale. Whether the design around them deserves it is the part you have to decide before it gets expensive.