Comments (26)
GCE Load Balancers don't support proxy protocol. To get the real IP addresses with an nginx ingress, you have to rely on this alpha feature introduced in kubernetes 1.4:
https://kubernetes.io/docs/user-guide/load-balancer/#loss-of-client-source-ip-for-external-traffic
Closing this as it's not a kube-lego bug/limitation. Let me know how it goes...
from kube-lego.
I would check out this documentation:
https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer.
kubectl patch svc loadbalancer -p '{"spec":{"externalTrafficPolicy":"Local"}}'
replace loadbalancer with your nginx service and add --namespace=xxx if you need to.
from kube-lego.
At aws I would add this to my config map
use-proxy-protocol: "true"
If I try that with google container engine I get this error.
while reading PROXY protocol, client: 10.0.0.1, server: 0.0.0.0:443
2016-11-15T21:24:41.460205161Z 2016/11/15 21:24:41 [error] 198#198: *69 broken header
This is probably a gke issue, so I will head over there.
from kube-lego.
Did somebody also get it working when using a GCLB through an ingress in front of Nginx? The service.beta.kubernetes.io/external-traffic: OnlyLocal
doesn't seem to work, because the service is just a NodePort.
Sorry for using this thread, but best way to get a solution :-)
from kube-lego.
@DocBradfordSoftware if you are using NGINX example then you are actually using Nginx Ingress Controller, not GCE Ingress Controller. But traffic is forwarded to your Nginx Ingress Controller via GCE load balancer, I suppose.
If so, take a look at Proxy protocol usage for Nginx Ingress Controller.
from kube-lego.
have the same error once the use-proxy-protocol
is enabled. Does it means that we have to setup haproxy to use it properly if the nginx service is set as NodePort?
from kube-lego.
Work like a charm!
need to set the type as LoadBalancer.
@simonswine so there is no way to avoid using LoadBalancer if we need to forward the client IP to the ingress? As the Network Load Balancing rule is kinda expensive for small project.
I have already disabled the httpLoadBalancing, still saw one loadbalancing rule got created once the type type is set as LoadBalancer.
gcloud container clusters describe my-project
addonsConfig:
httpLoadBalancing:
disabled: true
from kube-lego.
@kenng have you found a solution? My nginx ingress still shows 127.0.0.1 only as client ip
from kube-lego.
@alex88, per the reference above, you need to add an annotation to the nginx controller service
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: nginx-ingress
annotations:
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
type: LoadBalancer
ports:
- port: 80
name: http
- port: 443
name: https
selector:
app: nginx
from kube-lego.
@DocBradfordSoftware unfortunately that didn't help. :(
I've a domain pointing to the GCE load balancer IP with an A record, if I use curl with the domain name, nginx logs shows: 127.0.0.1 - [127.0.0.1]
, if I use curl with the load balancer IP address it correctly shows my client IP :/
from kube-lego.
I used the echo service in the example. and got these values
X-Forwarded-For=70.197.xxx.xxx
X-Real-IP=70.197.xxx.xxx
double checking with whatsmyip
http://www.whatsmyip.org/
i got the same ip: 70.197.xxx.xxx
from kube-lego.
Lucky you :) I still can't get my real ip if I use a domain to connect to the load balancer
from kube-lego.
I am not sure if what i am doing is correct, but the way I find my ip to hook up to the dns record, is that I go to gcloud console networking -> loadbalancing. there should be one instance, click on that and it will show the external ip.
Also if you look at the External IP Addresses tab, there should be one IP that points to a Forwarding Rule. Clicking on the forwarding rule, should bring you to the loadbalancing page above.
from kube-lego.
@DocBradfordSoftware I've found the discriminant, if I use port 443, I get 127.0.0.1 in the nginx ingress logs, if I use port 80 I get the real client IP
from kube-lego.
@alex88 Did you find a solution to your issue? I'm seeing the same problem. What version of the Nginx Ingress are you on?
from kube-lego.
@artushin nope, still the same issue, maybe it's because of the ports that the tcp load balancer can use the proxy protocol with, I don't really know.
from kube-lego.
Just upgraded to 0.9.0-beta.8 (https://github.com/kubernetes/ingress/releases). Looks good now.
Looks like it was probably this: kubernetes/ingress-nginx#233
from kube-lego.
@artushin Sorry forgot to mention I was using beta-5, I'll try with the beta 8. Anway, do you have any special config in the service? Like the service.beta.kubernetes.io/external-traffic: OnlyLocal
annotation?
from kube-lego.
I do, but it's a one node dev cluster, and I'm running the nginx pods in a daemonset, so it would probably work even without it. But yeah, based on the k8s docs, if you have nodes that are not running an nginx pod, you should use that annotation if you need source IPs.
from kube-lego.
Oh yeah it works! That's awesome! :D
from kube-lego.
Now it's just a matter of having nginx trust the forwarded ip and that's it :)
from kube-lego.
@DocBradfordSoftware Problem is that our services is a type:NodePort
, because we have an ingress in front of it. But I am now assuming that nobody in this thread has it setup that way.
PS. We use the ingress controller with GCP Http Loadbalancer, so we can use:
- TLS termination at the HTTP loadbalancer at Google
ingress.gcp.kubernetes.io/pre-shared-cert
kubernetes.io/ingress.global-static-ip-name
from kube-lego.
You are correct, mine is on GKE, but I followed the kube-lego/examples/nginx and so the service is Type:LoadBalancer
from kube-lego.
@bviolier we have the same setup and issue and been talking to Google support for about a week to find a solution. So far the best they have been able to offer is to setup an L2 load balancer with type:LoadBalancer
and host nginx ourselves including all the ingress routing, SSL termination, etc in the nginx configuration.
from kube-lego.
@bviolier we are using the same setup as you Ingress + NodePort
as the GKE document suggested. with externalTrafficPolicy: Local
set to the NotePort config, we no longer get source NAT'd as this document promised.
But the problem is, the health check of GCE loadbalancer simply not responding fast enough to catch up with the pod change while scaling, So it keeps sending traffic to nodes that has no healthy pods on it and then the requets get dropped directly 🙁
Does anyone managed to workaround this issue?
Update:
It turns out using Google L7 LB > Ingress > NodePort > nginx
the IP was correctly written in the X-Forwared-For correctly, all I have to do is filter the GKE node IP address.
from kube-lego.
See my solution posted in kubernetes/ingress-nginx#808 (comment)
from kube-lego.
Related Issues (20)
- The tls-sni challenge has been disabled due to strong credibility of a vulnerability report HOT 4
- Adding heptio/contour support HOT 1
- Wildcard Certificate Support HOT 2
- Pull the complete certificate chain HOT 1
- Unsupported ingress class HOT 1
- renewal expiry date is incorrect HOT 1
- Support for Letsencrypt wildcard certificate HOT 1
- Let's Encrypt Wildcard Support HOT 9
- How safe is it to use Kube Lego in producation on v1.9+ of Kubernetes? HOT 2
- If one of the domains in an ingress fails reachability, kube-lego should not try to authorize any of the domains
- Pod kube-lego not starting HOT 4
- read udp i/o timeout HOT 4
- Does not seem to work on k8s 1.8.8-gke.0 HOT 6
- Failed to list *v1beta1.Ingress HOT 1
- creating new secret
- Auto-renewal of certificates is not being triggered in 0.1.6 HOT 2
- Memory Leak?
- kubernetes 1.10 on GCP cant create a GCE loadbalancer ingress without secret
- Add: kubernetes.io/tls-acme: 'true' annotation
- Archive the kube-lego repository
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kube-lego.