Thredds do not behave well behind Nginx proxy serving as SSL termination about thredds HOT 4 OPEN

Thank you for passing along your nginx config - I think we're getting close to being able to document this setup, but not quite yet.

Here is my understanding of the overall situation:

When obtaining information about incoming requests (ip addresses, for example), the TDS uses the Java Servlet API, so it only "sees" what the servlet container passes it. This gives server administrators the ability to choose a servlet container of their choice (e.g. Jetty, GlassFish, WebSphere, JBoss, etc.), as long as it supports the proper version of the servlet specification (3.0 in the case of the TDS). The TDS Docker image uses Tomcat as the servlet container.

When you add a proxy, you have to make sure that the proxy server and the servlet container are configured properly so that the correct information makes it from the proxy through the servlet container to the application (i.e. the TDS), and that they servlet container can get the response information from the application back to the proxy. Because there are a number of combinations of proxy server / servlet container combinations, there isn't one single configuration that can be used. We've documented the one we know and use.

The solution you found for getting https to work adds a Connector to Tomcat to properly interface with nginx to make sure the protocol makes it to the servlet container (this approach was mentioned here. We modify the Tomcat server.xml config in a similar way to get the apache proxy working, too (see here). To ease the process, perhaps the TDS Docker image should allow one to supply their own Tomcat configuration (although I think it sort of does, as the docs hint at it: "Tomcat configuration can be done by mounting over the appropriate directories in CATALINA_HOME (/usr/local/tomcat).")

In terms of logging, the IP address logged by the TDS is the one passed in by Tomcat via the servlet API. The question is, how do you configure Tomcat to pass along the "correct" IP when nginx is the proxy? We use mod_jk, which provides a smooth interaction between the Apache proxy and the servlet container, and it does not require us to consider that question, as it's handled under-the-hood, so to say. With other proxies (like nginx), a more careful consideration of the configuration is needed to make sure the proxy and Tomcat can fully "talk".

It looks like everything could be done by adding a special <Valve> configuration to Tomcat's server.xml (no need for the <Connector>), but I'm not certain quite yet. I do not manage the proxy / servlet setup on our machines, and locally I only ever run Tomcat, but at this point I think I have enough information to get started and give things a try on my local machine. I'll update this ticket as things progress.

from thredds.

I found out how to get Thredds to handle the X-Forwarded-Proto and X-Forwarded-For http headers from Nginx properly. It's just one more line to add in the server.xml file !

In the unidata/thredds-docker:4.6.14 docker image, in file /usr/local/tomcat/conf/server.xml, there is only one line <Engine name="Catalina" defaultHost="localhost">, just add this line right below <Valve className="org.apache.catalina.valves.RemoteIpValve" protocolHeader="X-Forwarded-Proto" />. Doc for that Valve https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valves/RemoteIpValve.html

I now have the original httpS proto in the URL displayed by Thredds and the original real client IP in my logs.

Below are the logs from 1 refresh in the browser:

$ docker exec thredds tail -f /usr/local/tomcat/logs/localhost_access_log.`date '+%Y-%m-%d'`.txt /usr/local/tomcat/content/thredds/logs/threddsServlet.log

==> /usr/local/tomcat/content/thredds/logs/threddsServlet.log <==
2019-09-20T17:22:07.990 +0000 [     15358][       2] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1"
2019-09-20T17:22:07.999 +0000 [     15367][       1] INFO  - threddsServlet - Remote host: 172.16.0.60 - Request: "GET /twitcher/ows/proxy/thredds/catalog/birdhouse/catalog.html HTTP/1.0"
2019-09-20T17:22:08.109 +0000 [     15477][       2] INFO  - threddsServlet - Request Completed - 200 - 1603 - 126
2019-09-20T17:22:08.122 +0000 [     15490][       1] INFO  - threddsServlet - Request Completed - 200 - 3070 - 138
2019-09-20T17:22:08.213 +0000 [     15581][       3] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/ HTTP/1.1"
2019-09-20T17:22:08.223 +0000 [     15591][       3] INFO  - threddsServlet - Request Completed - 302 - -1 - 10
2019-09-20T17:22:08.231 +0000 [     15599][       4] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1"
2019-09-20T17:22:08.235 +0000 [     15603][       4] INFO  - threddsServlet - Request Completed - 200 - 1603 - 4
2019-09-20T17:22:08.269 +0000 [     15637][       5] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/threddsIcon.gif HTTP/1.1"
2019-09-20T17:22:08.284 +0000 [     15652][       6] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/folder.gif HTTP/1.1"
2019-09-20T17:22:08.296 +0000 [     15664][       5] INFO  - threddsServlet - Request Completed - 304 - -1 - 27
2019-09-20T17:22:08.297 +0000 [     15665][       6] INFO  - threddsServlet - Request Completed - 304 - -1 - 13

==> /usr/local/tomcat/logs/localhost_access_log.2019-09-20.txt <==
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1" 200 1603
172.21.0.22 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/catalog/birdhouse/catalog.html HTTP/1.0" 200 3070
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds HTTP/1.1" 302 -
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/ HTTP/1.1" 302 -
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1" 200 1603
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/threddsIcon.gif HTTP/1.1" 304 -
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/folder.gif HTTP/1.1" 304 -

The IP 172.16.0.60 is the original real client IP. 172.21.0.1 is the network gateway from docker-compose and 172.21.0.22 is the IP of the Nginx proxy container.

Not sure why Thredds/Tomcat recorded to be hit by the network gateway so much. I would have expected to see the original real client IP or the Nginx proxy IP at all the places we see the network gateway IP.

Also it's funny that Thredds logs have the good client IP while the Tomcat logs have the Nginx proxy IP.

Will submit a PR for the extra needed Valve line in server.xml for the docker image unidata/thredds-docker. This way I can just use the stock docker image without having to customize the server.xml file.

from thredds.

Excellent!

Have you ran across this post at https://serverfault.com/questions/514551/make-tomcat-use-x-real-ip?

It sounds like you could use

<Valve className="org.apache.catalina.valves.RemoteIpValve"
               protocolHeader="X-Forwarded-Proto"
               remoteIpHeader="X-Forwarded-For"
               requestAttributesEnabled="true"
               internalProxies="172\.21\.0\.1"  />

and that would take care of everything you needed, including the IP address in the tomcat logs.

from thredds.

Have you ran across this post at https://serverfault.com/questions/514551/make-tomcat-use-x-real-ip?

It sounds like you could use

<Valve className="org.apache.catalina.valves.RemoteIpValve"
               protocolHeader="X-Forwarded-Proto"
               remoteIpHeader="X-Forwarded-For"
               requestAttributesEnabled="true"
               internalProxies="172\.21\.0\.1"  />

and that would take care of everything you needed, including the IP address in the tomcat logs.

No did not saw that post, thanks for the reference. That post seems to be applicable to older version of Tomcat.

requestAttributesEnabled do not exist anymore on that RemoteIpValve for Tomcat 8 and later. The default value for remoteIpHeader is already the wanted one so no need to set.

internalProxies probably a good idea but in the context of a full docker stack, the proxy is also running in a container so its IP is not known in advance which make it harder to set in the config file, but still possible.

The fact that the real client IP is not in the Tomcat logs is not too much annoying. The Thredds logs have all the important bits (date time, real client IP, http verb, url path, return code, content lenght).

The proposed minimally customized RemoteIpValve works well enough.

Also, from what I understand, X-Forwarded-For is the superset of X-Real-IP. X-Real-IP contains only 1 IP. X-Forwarded-For is actually a list of IPs so if the customer goes through 3 proxies before reaching Thredds, the X-Forwarded-For header received by Thredds would contain original client IP, proxy1 IP, proxy2 IP.

My PR has been declined by the way. Don't know if it's possible to appeal the decision. I've re-explained in the PR why I think the PR is generic and will contribute to the out of the box user experience with Thredds.

from thredds.