Kong ingress controller failed to fetch secrets for kong consumers if restarted (either due to some crash or planned) about kubernetes-ingress-controller HOT 11 CLOSED

The user reported that he tested a nightly image and the issue doesn't occur. Can we close it? WDYT?

cc: @rainest & @randmonkey

from kubernetes-ingress-controller.

Also, to update I am using latest CRDS as stated below

controller-gen.kubebuilder.io/version: v0.13.0

from kubernetes-ingress-controller.

looks like we have same issue
#5710

from kubernetes-ingress-controller.

Do you see any difference when using the latest nightly when your Secrets have the konghq.com/credential label (instead of the kongCredType field)?

Other than the known (if not fully explained) issues when using older CRDs, I'm not sure what would cause this at startup. AFAIK the other issue with Secrets not being found occurs when creating a new Secret and KongConsumer simultaneously and would show a different error than the "failed to fetch" error you're seeing.

6db609c does at least simplify when we load Secrets, so it's worth a try in case something odd is happening with the older reference behavior.

Do you see any API server errors when those retrieve attempts fire? Sync delays between replicas also seem unlikely for a Secret that old, but it's worth checking in absence of another obvious explanation for why those Secrets are returning 404s to KIC's requests.

from kubernetes-ingress-controller.

I can try this but I am still using the KIC v2.12.3. Is this annotation not introduced in KIC v3.0.0?

I don't see any API server errors during these attempts. And of course, all my secrets are created long time back, never recreated but may get rotated credentials. In fact, the issue resolves as soon I recreate Kong consumers...so seems like Kong and k8s resources are not getting synched but re-initialization of Kong resources (i.e. for consumer) helps to process the k8s secret resource

from kubernetes-ingress-controller.

ok this doesn't work since replacing kongCredType results in another error:

time="2024-04-09T06:03:39Z" level=error msg="resource processing failed: credential "XXXXX" failure: failed to provision credential: invalid kongCredType: type 'string' not string" GVK="configuration.konghq.com/v1, Kind=KongConsumer" name=XXXX namespace=QA

from kubernetes-ingress-controller.

The annotation requires 3.0 yes, and the simplified Secret loading behavior has not been released yet, hence why it requires a nightly image.

Currently released versions relied on tracking references between KongConsumers and Secrets to determine which Secrets to load. If touching the KongConsumer loads the Secret properly, it's likely that reference isn't being handled properly. The unreleased changes just load any Secret with a credential label, so there's no opportunity for any race condition there.

#5175 is the other report that describes that race condition, and debug logs on your instance will probably mention the Secret name before the KongConsumer, indicating that it hadn't built the reference when it determined whether to ingest the Secret.

On 2.12 you can try increasing --init-cache-sync-duration (aka CONTROLLER_INIT_CACHE_SYNC_DURATION in the environment or ingressController.env.init_cache_sync_duration in the chart) to a value higher than the default (5) to see if that makes any difference.

Due to some limitations in how the controller generates configuration, we can't use the underlying Kubernetes controller libraries as we should. Increasing that value pauses some controller tasks at startup to give the lower level Kubernetes configuration ingest time to pull in more objects. This isn't 100% reliable (it's a workaround; we're trying to find a more reliable way of handling it), but may help.

from kubernetes-ingress-controller.

ok but I have already tried implemented init-cache-sync-duration and specifically upgrade from 2.11 to 2.12.3 to use this value but seems like as you said it's not reliable and didn't work in my case.

I am currently in process of upgrading KIC to v3.0.x. Once it is done, I will try to use nightly image and may be at the time, I will release v3.0.x in production , Simplified Secret loading behavior may release before that.

from kubernetes-ingress-controller.

I have tested with the nightly update image and the issue is fixed now. No more resource processing errors are seen and consumer is working correctly

from kubernetes-ingress-controller.

Hi everyone, we're also observing this issue after upgrading the controller to 2.12.3. we're running proxy version 2.8.4.4.

will the patch on the nightly build be released soon? i'm not seeing a fix related to this on latest release v2.12.4.

from kubernetes-ingress-controller.

To the best of my knowledge, the resolution to this will be released with 3.2. The revised ingest method does not exhibit the same issue. There are no plans to backport this change to 2.x at present. If we decide to do so, that will be handled in a follow-up issue.

from kubernetes-ingress-controller.