Comments (6)
randmonkey
commented on June 26, 2024
@oleksandrs-adorama Looks like the connection inside your k8s cluster ( connection between KIC pod and k8s apiserver, and connection between KIC and Kong gateway admin API) is not very stable so the cache inside KIC's controller runtime may not be synced with k8s apiserver. Do you know the what pods own IPs 10.50.56.159 and 10.50.57.190? This can help us to locate the problems.
from kubernetes-ingress-controller.
oleksandrs-adorama
commented on June 26, 2024
10.50.56.159 and 10.50.57.190 - konnectivity-agent
from kubernetes-ingress-controller.
oleksandrs-adorama
commented on June 26, 2024
also i can see errors
2024-03-20T12:05:21Z error controllers.KongConsumer Reconciler error {"reconcileID": "ce309f6f-d757-48ac-a553-f6722ecbd207", "error": "Operation cannot be fulfilled on kongconsumers.configuration.konghq.com \"****.public-user\": the object has been modified; please apply your changes to the latest version and try again"}
2024-03-20T12:05:22Z error controllers.KongConsumer Reconciler error {"reconcileID": "bd3cf33d-d5b4-4c97-9c4a-b749b9950d06", "error": "Operation cannot be fulfilled on kongconsumers.configuration.konghq.com \"****.public-user\": the object has been modified; please apply your changes to the latest version and try again"}
from kubernetes-ingress-controller.
randmonkey
commented on June 26, 2024
also i can see errors
2024-03-20T12:05:21Z error controllers.KongConsumer Reconciler error {"reconcileID": "ce309f6f-d757-48ac-a553-f6722ecbd207", "error": "Operation cannot be fulfilled on kongconsumers.configuration.konghq.com \"****.public-user\": the object has been modified; please apply your changes to the latest version and try again"} 2024-03-20T12:05:22Z error controllers.KongConsumer Reconciler error {"reconcileID": "bd3cf33d-d5b4-4c97-9c4a-b749b9950d06", "error": "Operation cannot be fulfilled on kongconsumers.configuration.konghq.com \"****.public-user\": the object has been modified; please apply your changes to the latest version and try again"}
This seems to be that the KongConsumer in the cache is outdated. While based on k8s's eventual consistency mechanism, it will be translated and applied on Kong gateway finally. It might take longer time for controller cache to be synced with k8s apiserver if your cluster is heavy loaded or network is not stable.
from kubernetes-ingress-controller.
oleksandrs-adorama
commented on June 26, 2024
we have three separate environments test, dev and prod. In three env ea have the same behavior.
from kubernetes-ingress-controller.
stale
commented on June 26, 2024
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
from kubernetes-ingress-controller.
Related Issues (20)
- Resolve conflicts in GRPCRoute conformance testing. HOT 4
- Action Required: Fix Renovate Configuration
- Release v3.1.6 HOT 3
- Add support for `konghq.com/rewrite` in `HTTPRoute`
- Add `--disable-fallback-configuration` CLI flag after graduating the feature gate
- Do Konnect and Gateway reconciliation in different goroutines
- Add feature flag to skip consumer sync to Konnect HOT 1
- Improve observability around config sync HOT 1
- Add documentation around sync failure modes
- Prometheus metrics at Path level inside rules
- Efficient Kong upstream slots allocation
- OpenAPI spec for diagnostics server
- Subdomain wildcard certificate domains not working with HTTPS
- Fallback configuration is triggered on config update errors that should not cause fallback (e.g. connection refused)
- KIC does not detect newly spawned Gateway Pods and prevents them from getting ready by not sending configuration HOT 4
- Same client certificate in different `Secret`s used in different services causes invalid configuration HOT 2
- Release 2.12.5 HOT 1
- KIC did not release memory in a timely manner when DP was downsized HOT 1
- KIC on Konnect bulletproofing
- Log metadata for Konnect requests and provide KIC metadata with requests to Konnect HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes-ingress-controller.