add custom annotations on deployment and apiservice in order to prevent infinite progressing loop in argocd

[mardonner]

Is your feature request related to a problem? Please describe.

Hi,
so, I install cert-manager and this webhook with argocd applications.

I have a rather complicated setup where I rely on argocd's reconciliation loop and the eventual consistency model of k8s.

Essentially, it means that until cert-manager and the webhook are properly able to sync, some other applications need to first provide the imagepullsecrets, etc.. During that, argo constantly fails/retries until eventually every dependency is deployed. Then, the apps will eventually sync successfully.

My problem is that when cert-manager and the webhook start syncing at the same time, argo tries to sync everything but fails on the certificates and the issuers, since cert-manager webhook is not available in time. (ske gardener forces the cert-manager webhook timeout down from 30 to 15 seconds btw which increases the chances of running into this btw).
This results in the pod not being able to mount the certs and therefore to come up, which in turn makes the APIService stuck in this state:

      message: >-
        endpointslices for service/stackit-cert-manager-webhook in
        "cert-manager" have no addresses with port name "https"
      reason: MissingEndpoints
      status: 'False'
      type: Available

(the deployment's status will change to failed after 10 minutes due to progressDeadlineSeconds but the apiservice will never fail)
Unfortunately, in argo, this resource is now in "progressing" state. Meaning, the sync is never considered a failure and is now stuck indefinetely. The sync operation will never enter a failed state and therefore, no retry on the certs and issuers is done, which would fix this problem, since cert-manager webhook will come up in the meantime.
A manual terminate and sync will solve the issue but I don't want to need to do that. The automatic retry mechanism should take care of that.

This is what it looks like in the UI:

Describe the solution you'd like

Annotation support on the APIService and Deployment:
With this, I could and add an argocd annotation to accommodate for this.

# put them into a separate sync wave with
  annotations:
    argocd.argoproj.io/sync-wave: "1" # default is 0

This prevents the infinite loop, since the apiservice and deployment are not synced until the certs and issuers are there. With this, the argo retry mechanism would trigger properly. I tested it.

Annotation support is already present on the serviceAccount.

stackit-cert-manager-webhook/deploy/stackit/templates/rbac.yaml

Lines 15 to 18 in f323016

	{{- with .Values.serviceAccount.annotations}}
	annotations:
	{{- . | toYaml | nindent 4 }}
	{{- end }}

I'll let you be the judge but I think this is preferable to the other alternatives below. Do you have a better idea?
There seems to be nothing like progressDeadlineSeconds for the apiService.

Describe alternatives you've considered

helm hook-weights
We could add hook weights to force helm to deploy everything before the deployment and apiservice. However, I don't know if it is a good practice to rely on them. It least not by default. Users could add them via the annotations, mentioned above.

other alternatives
One alternative would be to put the cert-manager and webhook argo app into separate sync waves. However, I'm trying to avoid sync waves if possible.

The other would be writing a custom definition of the status/health of apiservices/deployments for argocd, which I also try to avoid.

Search

• I did search for other open and closed issues before opening this.

Code of Conduct

• I agree to follow this project's Code of Conduct

Additional context

No response