Argo CD is a cornerstone of GitOps for Kubernetes, continuously reconciling the desired state defined in Git (or Helm/OCI sources) with the live cluster state. At the heart of this reconciliation lies diffing—the process of comparing manifests to detect drifts, mutations, or manual changes. When diffing fails to spot differences, applications can appear “OutOfSync” unexpectedly or miss real updates, leading to frustration and debugging headaches.
In the talk “What the Diff?” by Red Hat’s Gerald Nunn (Technical Marketing Manager for GitOps) and Andrew Block (Distinguished Architect), presented at ArgoCon NA 2025 and available on the CNCF YouTube channel, the speakers unpack Argo CD’s diffing engine in depth. They explain the strategies, common pitfalls, and practical ways to make diffing reliable. This article distills those insights for practitioners working with declarative deployments.
How Argo CD Performs Diffs
Unlike kubectl diff, which is client-side only, Argo CD uses its internal libraries and GitOps engine to compare:
- Desired state: Rendered from Git/Helm/OCI.
- Live state: Fetched from the Kubernetes API.
- Argo CD strips non-essential metadata (timestamps, generations, etc.) before comparison to focus on meaningful changes.
The system supports multiple diff strategies, each with trade-offs in accuracy, performance, and compatibility.
1. Legacy (Client-Side) Diff – The Default
This is the out-of-the-box behavior, mimicking Kubernetes’ three-way diff:
- Compares live state, desired state, and the last-applied-configuration annotation.
- Relies on the annotation (added by kubectl apply –save-config or equivalent).
- Limitations:
- Misses changes from imperative commands like kubectl edit (no annotation update).
- Ignores fields added/owned by controllers (e.g., auto-added labels, annotations, or status fields).
- Doesn’t detect schema violations or deep mutations reliably.
It’s lightweight but brittle in mixed imperative/declarative environments.
2. Server-Side Diff – Beta and More Accurate
Enabled via:
- Global: controller.diff.serverSide=true in the Argo CD ConfigMap.
- Per-application: Annotation like argocd.argoproj.io/sync-options: ServerSideApply=true (often paired with apply mode).
How it works:
- Performs a dry-run server-side apply.
- Compares the predicted result against live state.
- Leverages field managers (Kubernetes ownership tracking) for granular control.
- Better at detecting mutations, schema issues (e.g., invalid values), and webhook-applied changes.
Trade-offs:
- Higher performance cost (extra API calls).
- Still beta; potential for edge cases.
- Requires consistent use with server-side apply—mixing with client-side apply can hide removals or updates.
3. Structured Merge Diff – Tied to Server-Side (Being Deprecated)
- Automatically used when server-side apply is enabled.
- Powered by Kubernetes’ structured-merge-diff library.
- Handles field ownership via managers.
- Currently default for server-side, but future releases plan to shift to traditional server-side diff (per ongoing PRs).
Common Pitfalls and Real-World Gotchas
The talk highlights scenarios that trip up even experienced users:
- Imperative changes (e.g., kubectl patch or edit) → Legacy diff misses them unless annotations are updated.
- Controller mutations (e.g., HPA scaling, image digests, or admission webhooks) → Legacy often ignores added labels/annotations; server-side catches more but may still miss deletions.
- Mixed apply modes → Using client-side apply with server-side diff hides label removals (operation type mismatch).
- Field defaults and schema → Fields like revisionHistoryLimit: 10 (default) are ignored if omitted in manifests.
- External tools → Secrets managers or aggregators adding fields cause persistent “OutOfSync” noise.
Best Practices for Reliable Diffing
To minimize surprises:
- Standardize on one mode: Pair server-side diff with server-side apply for best results. Avoid mixing.
- Use ignoreDifferences wisely:
- Configure via annotations (per-app) or ConfigMap (global).
- Supports JSONPath (RFC 6902 patches) or JQ expressions.
- Example: Ignore dynamic fields in secrets or CRDs.
- Set argocd.argoproj.io/respect-ignore-differences: “true” to enforce during sync (not default).
- Test in staging → Reproduce drifts in a lab before production.
- Define clear ownership → Use field managers to control which controller “owns” which fields.
- Leverage annotations strategically → Per-resource ignores are preferable over global for multi-tenant setups.
- Monitor upcoming changes → Server-side diff is maturing; watch for it becoming default.
Why Mastering Diffing Matters
In GitOps, accurate diffing is what enables true self-healing and drift detection. Misunderstandings lead to manual interventions, broken automations, or false positives. By choosing the right strategy, configuring ignores thoughtfully, and avoiding mixed modes, teams can achieve more predictable, production-grade Argo CD deployments.
Watch the full session “What the Diff?” on the CNCF channel for demos, deeper examples, and Q&A insights from the experts. It’s an essential resource for anyone troubleshooting sync issues or scaling GitOps workflows.
