cert-manager randomly crashes with leader election lost about cert-manager HOT 2 OPEN

Hey,

I'd recommend everyone to not use --leader-elect=false although it has been talked about in #4918.

The reason you don't want to use --leader-elect=false in production is because even though the number of replicas for the cert-manager controller is 1, you will have multiple pods when upgrading cert-manager from one version to another. For a couple of seconds or minutes, two cert-manager instances will be running simultaneously, which will lead to the two cert-manager pods fighting with each other and behaving erratically.

*v1.Ingress ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost")

This makes me think that the connection to the kube-apiserver got lost for at least 40 seconds (as per renewDeadline in the Helm chart), and cert-manager crashed as expected. The reason we want to crash (instead of hanging indefinitely) is makes it easier to diagnose.

from cert-manager.

I remember a few years ago we had a similar reconnect issue with the kubernetes control panel in Azure.
My memory is a bit hazy but I think it came down to something like the reconnect try needs to refresh DNS.

golang/go#23427

from cert-manager.