Flaky test: TestValidationWebhook/verify_validation_webhook_on_creating_KongVaults/should_fail_the_validation_if_the_spec.config_does_not_pass_the_schema_check_of_Kong_gateway about kubernetes-ingress-controller HOT 2 CLOSED

Seems that only the tests needed to send configuration to Kong gateway fails. I will try to separate the Consumer part which creates many consumers to a new test case. related: #5375

from kubernetes-ingress-controller.

Observed again in https://github.com/Kong/kubernetes-ingress-controller/actions/runs/7788877111/job/21239413941?pr=5562

Also failed on the unknown backend/name check.

The test creates these resources as usual and expects the webhook to catch and reject them. When this occurs, the resources do fail at apply time even though they ostensibly validate fine:

$ grep 400 integration-tests-kic-logs | tail -2
2024-02-05T19:48:24Z	error	dataplane-synchronizer	Could not update kong admin	{"error": "performing update for http://172.18.128.2:8001 failed: 2 errors occurred:\n\twhile processing event: Create vault unsupported-backend failed: HTTP status 400 (message: \"schema violation (name: vault 'env1' is not installed)\")\n\twhile processing event: Create vault invalid-config failed: HTTP status 400 (message: \"schema violation (config.foo: unknown field)\")\n"}
2024-02-05T19:48:27Z	error	dataplane-synchronizer	Could not update kong admin	{"error": "performing update for http://172.18.128.2:8001 failed: 2 errors occurred:\n\twhile processing event: Create vault unsupported-backend failed: HTTP status 400 (message: \"schema violation (name: vault 'env1' is not installed)\")\n\twhile processing event: Create vault invalid-config failed: HTTP status 400 (message: \"schema violation (config.foo: unknown field)\")\n"}

The test used a very outdated cleanup system though, so it wasn't creating diag dumps to get proxy logs.

We could potentially be failing to actually validate at all in the webhook. Although we do ensure the Service is available and has configuration before running tests, our failure policy is Ignore. It's a bit odd that the failure tests wouldn't be affected en masse, but it's not inconceivable that the runner's slow enough to exceed the timeout on some requests.

I don't know if client-go has the ability to indicate when a webhook check was requested and failed, but was ignored. Even if it does, we'd need a larger refactor of the checks to introduce retries. Currently they use a single Create() call and don't delete resources they create if they shouldn't have passed.

We could swap to Fail, but then we'd have problems if we encounter timeouts on the valid items, and wouldn't actually be testing what we want for the invalid ones.

from kubernetes-ingress-controller.