Comments (30)
Is it supported now? If not, is there any other way of achieving HAproxy ingress controller redundancy when using stick tables for routing?
from kubernetes-ingress.
Is there any implementation of a way to ensure that the stick table data propagate among all HAProxy pods?
from kubernetes-ingress.
Now, I realize you may mean "dns name to resolve others HAProxy peers". Can you confirm?
Correct, that's preventing us from running multiple HAProxies with stick tables in k8s so far.
I thought you meant DNS name of the server in the HAProxy backend.
That works with the dns provider and a server template, so not much that would need to be done here.
If you deploy your ingress controller as daemonset for north/south traffic this should be pretty static.
I would propose another solution: we make it a statefulset with static dns names for each pod and the peering should support it. The other k8s native solution would be via labels/annotation selectors and the k8s APIs.
I understand this is a bit stretchy, but would be cool to finally have a k8s native solution working for this case.
from kubernetes-ingress.
@Ivaka we didn't, we have a single haproxy pod running with the normal docker container. So the configuration is pretty much normal haproxy.cfg 👍
If that's of interest, we use Ambassador as an ingress controller. And on top we use haproxy for some stateful pod affinitization.
from kubernetes-ingress.
Hi @dbarvitsky, I believe we are using the same architecture in our project. We are deploying our cluster on AWS EKS. We have multiple HAProxy pods deployed that are sticking to our application pods by extracting a custom header from the responses and storing them in stick tables. Since the stick table entries are not shared between all haproxy pods, we needed to find a way to stick the user's connection to the same haproxy pod. We have done that by adding an AWS ALB in front of the haproxy pods. So the ALB load balances the haproxy pods. By enabling client affinity on the ALB, we ensure that the user (browser session) always sticks to the same haproxy pod which has the relevant stick table entries.
Now of course, when an haproxy pod is terminated due to automated scaling, some users will lose their session, which is bad... but so far it was the only convenient approach. Hope this helps
from kubernetes-ingress.
Some years ago, during an hackathon I developed a POC for peer syncing as an HAProxy sidecar in k8s.
I'll share it here so maybe it is useful for others. Please consider this is just a proof-of-concept and not high quality, production code. So please just use it as an example:
https://github.com/damdo/k8s-haproxy-peer-sync
from kubernetes-ingress.
Hi Thomas,
Currently peers are not enabled and stick tables are used for rate-limiting.
What would you like to do with your stick table? what's your use case?
from kubernetes-ingress.
thanks for the fast reply! we have a somewhat stateful backend that we would need to route to a pod by some url path fragment. In vanilla haproxy cfg that looks like this:
balance roundrobin
stick-table type string size 100m
stick on path, word(3,/)
And on top of that we would need to peer several HAProxy instances to ensure fault tolerance. Currently in k8s that's a bit painful, because the peering only works on static IPs.
from kubernetes-ingress.
an other issue you'll have to fix as well is the order of the servers in different HAProxy server.
in k8s, there is no guarantee that 2 ingress controllers will get the list of endpoint in the same order (furthermore if they use DNS). So basically, server id would be different and so persistence would be broken between 2 haproxys.
That said, in HAProxy 2.0, we can now rely on servername
@oktalz how are the server names computed? my guess is pure random :)
If pure random, this means that the only way to persist would be on serverIP and port which is not doable with HAProxy for now.
from kubernetes-ingress.
@bedis yes, they are pure random.
from kubernetes-ingress.
yeah we have tons of ordering issues with the DNS provider already and also with consistent hashing in some scenarios.
Our alternative solution would be to fallback to consistent hashing, but then again we need to ensure the ordering somehow (maybe through a statefulset?). It would be enough if we could use the pod id and use the natural string ordering, so at least it would stay consistent across container restarts.
Would be curious if others have similar issues/requirements using HAProxy.
from kubernetes-ingress.
Usually, workloads in kubernetes are supposed to be statefull :)
@oktalz is there a chance to make the server name less random, IE hash based on the backend name + position in the backend?
I am asking because stick table will be able to support server name as an identifier soon (if not already available in 2.0, will check this point). So 2 or more HAProxy would be able to generate the same server name and so stick table would work in such situation.
from kubernetes-ingress.
if we can run haproxy as a statefulset then we would get stable network identifiers for each of them. Does the peering support dns names nowadays?
from kubernetes-ingress.
it won't support DNS name, but server name in the configuration. That said, I can't see anything in the documentation, so I am currently getting this information and come back to you asap.
Can we scale out statefulset? (not very familiar with them, sorry)
from kubernetes-ingress.
it won't support DNS name, but server name in the configuration.
then we might have a problem :)
Can we scale out statefulset? (not very familiar with them, sorry)
yep it's possible.
from kubernetes-ingress.
I think I misunderstood your question about peering and DNS names. I thought you meant DNS name of the server in the HAProxy backend.
Now, I realize you may mean "dns name to resolve others HAProxy peers". Can you confirm?
If the later, the answer is still "no" :)
That said, I think we could do it with the following way for now:
- provide the ingress controller a config map where each haproxy belonging to a "cluster" could register its own IP
- make sure the ingress controller monitors this config map
- each time the config map is updated, ensure that the ingress controller update each HAProxy's peers section
If you deploy your ingress controller as daemonset for north/south traffic this should be pretty static.
from kubernetes-ingress.
👍 for this, it would be great to have stick table & peering available
@thomasjungblut How did you manage to configure haproxy ingress controller with stick tables, as I cannot find anything in the documentation about this?
from kubernetes-ingress.
I would propose another solution: we make it a statefulset with static dns names for each pod and the peering should support it.
I am not a fan of it because it requires persistent volumes which are not necessary from my point of view in the case of an Ingress controller / LB / HAProxy.
The other k8s native solution would be via labels/annotation selectors and the k8s APIs.
I would prefer this option, more dynamic and not imposing any limitations in term of deployment.
It's like the ConfigMap I proposed but in a different manner.
The idea is that you could provide a "cluster id" as an argument to the ingress controller and this cluster id could be used as a selector to find "peers" (IE HAProxy forwarding the same traffic).
I understand this is a bit stretchy, but would be cool to finally have a k8s native solution working for this case.
Well, the Selectors solution is more elegant than the ConfigMap. With ConfigMap you must find a way to allow each Ingress Controller to register / deregister and this could be painful.
As you said, Selectors is more "k8s native".
from kubernetes-ingress.
To come back to peers, since HAProxy 2.0, I got confirmation that stick-tables will use server name automatically. Server IP:port will be available much later, maybe in 2.1 or even later.
@oktalz do you think it's possible to find a way to make the server name generation less random and predictive when multiple HAProxy run in the same cluster?
If you can fix this behavior, then peering could be enabled.
from kubernetes-ingress.
server names can be SRV_00001.... or some similar predictive name, but I'm not totally for it right now,
however, I'm also open to discuss it further :)
from kubernetes-ingress.
from kubernetes-ingress.
current SRV_v8KyB
does not imply anything,
its not first, oldest, its not linked to any pod, so i can without any problem disable it and reuse for another pod, or even delete it completely on scaling.
If i have SRV_0001
at one point someone will ask why it has 001 and its the youngest pod (in theory).
I'm not saying its not possible, but that would require some coding and logic for all that,
but with enough requests/wishes, that feature will move up on roadmap.
from kubernetes-ingress.
Sorry to hijack my own issue here, having this server name discussion makes me curious though:
Most of the issues we observe are coming from one of the haproxy pods that has a different view/ordering of the backend servers and that makes the forwarding inconsistent (eg with consistent hashing via dns provider). Do you guys have a solution for these kind of problems?
from kubernetes-ingress.
Very much interested in the peering support. Sharing our journey in FYI capacity in case someone finds it helpful or inspirational.
Our goal was to cluster a third-party service that we run in our Kubernetes cluster. The service in question is stateful and keeps a lot of data in memory. We wanted to be able to scale horizontally and decided to try HAProxy. Our idea was to use a custom header computed on the caller and route requests based on the consistent hash of that header to actual pods. We hoped that consistent hashing will produce identical sticky tables. We expected that re-scaling will cause a bunch of misrouting intermittently, but assumed it will eventually stabilize.
We have this in production at the moment:
kind: Service
metadata:
...
labels:
...
haproxy-ingress.github.io/config-backend: |
# Backend works in HTTP mode, e.g. parses HTTP requests.
mode http
balance hdr(X-CUSTOM-HEADER-WHATEVER)
hash-type consistent
stick on hdr(X-CUSTOM-HEADER-WHATEVER)
stick-table type string size 30M
Note that we are running it with --sort-backends
option.
Initially it looked like it worked - all our bench tests confirmed we are sticking based on the header and the whole contraption tolerates deployments, restarts and scaling the backend pods up and down. We tested it with two haproxy pods under synthetic load.
However, when we moved to the prod, we started seeing a lot of misses continuously. We are talking ~ 10% easily and it does not go down over time. Sticky tables clearly diverge somehow and stay out of sync. So we had to scale haproxy down to a single pod and call it a partial success. Looks like our synthetic load was not random enough, or not long enough, or our deployments were not erratic enough to cause the sticky tables to diverge under testing. We just got (un)-lucky with our experiments.
At this point we don't really have a good way of scaling haproxy itself. It is a single-pod with limited resources, with all performance and reliability risks that come with it. It looks like peering would be the only reasonable option. I will be very grateful for workaround proposals. I may also arrange for a reasonable bounty for solving the problem. Thanks!
from kubernetes-ingress.
HI @dbarvitsky , I guess you're refering to this product not ours. But you're welcome to try. The product you're currently using is an other project based on our open source project.
from kubernetes-ingress.
@ivanmatmati you are right, I somehow confused the two. Apologies. I will take a look. Thank you.
from kubernetes-ingress.
@YanaKay23 would you please elaborate on what you meant by an ALB in front of the HAproxy pods?
Are you creating a Loadbalancer service using annotations to make it an ALB?
Thanks
from kubernetes-ingress.
Hi experts! Have you any ideas to ensure backend endpoints consistency between ingress haproxy instances ? Currently it's looks like that each instance has own backend server list without even sorting (by ip/port for example) - this cause 90% misrouting for statefull requests. Moreover, I suspect that even local state table is ignored after haproxy reload due this startup errors like this "[WARNING] ( config: Can't get version of the global server state file '/var/state/haproxy/global'" Do you have any suggestions and workaround while haproxy peers are not supported in ingress ?
Thanks.
from kubernetes-ingress.
Hello.
I would like to achieve the sharing of stick table between a multi-pod Kubernetes deployment.
Is there any way to achieve that today?
Is there maybe a way to synx the daya with a third-party memory database like Redis?
Thank you.
from kubernetes-ingress.
Any news ?
from kubernetes-ingress.
Related Issues (20)
- DOC: Remove link to 'external mode' blog post HOT 1
- FR: http-request set-priority-class HOT 7
- FR: slowstart in annotations HOT 3
- Fails to start when run as non-root HOT 1
- crd-check.go invocation should be idempotent in case of exising CRDs HOT 4
- Prometheus metrics of the pods in MAINT state HOT 1
- annotation "haproxy.org/load-balance: uri path-only" causes configuration reloads HOT 2
- Access logs settings get overridden if using a Global Custom Resource HOT 5
- constant prometheus reloading HOT 4
- default backend not working when --ingress.class defined HOT 2
- Regular expressions in "path" do not appear to work HOT 1
- docs: `haproxy.org/server-ssl` tip is misleading HOT 1
- Ingress reloading almost all the time with v1.11.1 HOT 1
- Add ability to use oauth2 authentication. HOT 3
- Use existing NLB for HAProxy ingress HOT 2
- questions about request headers HOT 3
- http -> https default redirect not working by default HOT 10
- Deployment yaml still point to privileged port after s6 upgrade HOT 3
- frontend tcp-514: cannot bind socket (Permission denied) for [0.0.0.0:514] HOT 2
- haproxy ingress 1.11.2 reloads when number of backend pods scales HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes-ingress.