cluster wide secret "<argocd-name>-default-cluster-config" being created causes "Out of Sync" about gitops-operator HOT 7 CLOSED

I think I know what's happening. When Operators create their resources, they add labels/annotations from their parent resource (in this case, probably the ArgoCD CR) to the resources. When the CR was added via ArgoCD from Git, it will get the app.kubernetes.io/instance label attached to it that Argo CD uses to identify the resources it manages. I have seen this happening in the past with a multitude of different operators.

To answer your question, this secret is created for putting the ArgoCD instance in namespace-scoped mode. It contains information about the cluster it is running on along with a list of namespaces that the instance is allowed to manage. Without this secret, the instance would become a cluster-scoped instance and also would require appropriate cluster admin RBAC. So no, this secret shouldn't be deleted (and it would also be recreated by the Operator).

I think to fix this, we need to apply the following additional annotations on the secret when it's being created by the Operator:

argocd.argoproj.io/compare-options: IgnoreExtraneous
argocd.argoproj.io/sync-options: Prune=false

You can try to add the two annotations manually as a workaround to the secret. They shouldn't be removed by the operator once the secret is created, but I haven't tested it.

from gitops-operator.

Also, can you please copy&paste the complete metadata section of the secret please?

from gitops-operator.

We commited a fix at the upstream operator that will prevent these situations from occuring. Will be released with v1.3.

from gitops-operator.

@jannfis - Thank you for your response - I think this may be fixed now!

Follow-up Question: I noticed a similar case was raised and the resolution states that this issue is fixed in Red Hat OpenShift GitOps 1.2. I've tried this out and I don't see the same issue now - although I don't see the labels mentioned in your original message on the secret, it is no longer considered "Out of Sync" by ArgoCD. Do you know why this might be?

Please find below the requested metadata section in case it's useful:

metadata:
  creationTimestamp: '2021-08-18T14:23:38Z'
  labels:
    app.kubernetes.io/managed-by: test-tenant
    app.kubernetes.io/name: test-tenant-default-cluster-config
    app.kubernetes.io/part-of: argocd
    argocd.argoproj.io/secret-type: cluster
    mo/parent-argo: management-test-tenant-tooling
  managedFields:
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          .: {}
          'f:config': {}
          'f:name': {}
          'f:namespaces': {}
          'f:server': {}
        'f:metadata':
          'f:labels':
            .: {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.kubernetes.io/part-of': {}
            'f:argocd.argoproj.io/secret-type': {}
            'f:mo/parent-argo': {}
        'f:type': {}
      manager: gitops-operator
      operation: Update
      time: '2021-08-18T14:23:38Z'

from gitops-operator.

What is the actual diff for this resource?

from gitops-operator.

@jannfis - Below is an image of part of the diff to hide some internal metadata. The full diff shows the cluster-config secret on the left as "live manifest", and an empty "desired manifest" on the right. The status is "OutOfSync (requires pruning)".

from gitops-operator.

This is great news - Thank you for update!

from gitops-operator.