ArgoCD Application Stuck on Refresh - Issue Summary

Problem

ArgoCD applications were stuck in a permanent refresh state and unable to reconcile.

Root Cause

Known bug in ArgoCD v3.x + k8s.io/apimachinery v0.34.0

Introduced by PR #24197, tracked in Issue #25199.

During 3-way diff, strategicpatch.mergeKeyValueEqual panics when encountering []interface{} types:

panic: runtime error: comparing uncomparable type []interface {}
k8s.io/apimachinery/pkg/util/strategicpatch/patch.go:955

The controller recovers from the panic but the app remains stuck in refresh, retrying in an infinite loop.

Trigger

Deployments with podAnnotations containing nested JSON arrays, such as Datadog openmetrics:

podAnnotations:
  ad.datadoghq.com/app.checks: |
    {
      "openmetrics": {
        "instances": [
          {
            "metrics": [{".*": ""}]  # nested array triggers panic
          }
        ]
      }
    }

Attempted Workarounds That Did Not Work

• Restarting repo-server / application-controller / redis
• Running FLUSHALL on Redis cache
• Deleting and recreating the application
• Adding ignoreDifferences to the ApplicationSet (panic occurs before the ignore logic is reached)
• Patching Application status directly

Solution

Add the following annotation to the affected ApplicationSet template, which enables Server-Side Diff and bypasses the client-side strategicpatch logic that causes the panic:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: <app-name>
  namespace: argocd
spec:
  template:
    metadata:
      annotations:
        argocd.argoproj.io/compare-options: "ServerSideDiff=true,IncludeMutationWebhook=true"

After applying, hard refresh the application:

kubectl apply -f applicationset.yaml
argocd app get <app-name> --hard-refresh

Permanent Fix

Upgrade ArgoCD to a version containing the fix from PR #25294:

helm repo update
helm upgrade argocd argo/argo-cd -n argocd --reuse-values