Missing connection reset on reconfiguration with wildcard domain about contour HOT 10 OPEN

yep, only allowing one request per connection is definitely not a full solution

we'll have to do some work on an appropriate solution to this, my suggestion above on adding routes to the wildcard routeconfig might work, but needs more thinking/testing

contributions of course welcome!

from contour.

Note: The issue seems to appear with HTTP 1.1, too. So setting default-http-versions to support HTTP 1.1 exclusively does not work as a workaround.

from contour.

Contour should be detecting mismatched SNI and Host headers (and send 421 responses), we use some custom Lua to configure Envoy to catch this, see:

Mentioning this since it is the main focus of the linked Envoy issue

from contour.

@sunjayBhatia Can I support you in debugging this issue? I definitely get responses from the wrong backend after an initial connection. I even still receive responses from the wrong backend while I get responses from the correct backend in an incognito window.

I am not sure what you mean with mismatched SNI and Host headers. I guess there will be never a mismatch, as the wildcard domain is also valid for the specific domain, right? The issue is that the target backend changes when a new specific ingress is configured. So as soon as the specific ingress is configured, the old wildcard backend becomes invalid and the new specific backend becomes the only valid backend.

from contour.

(I only mentioned the mismatched SNI and Host feature to make sure that was covered for posterity, since the linked Envoy issue talks about that case heavily)

The issue here generally is due to connection reuse/coalescing from the browser, I can reproduce this in chrome on mac with HTTP/2 and HTTP/1.1 (HTTP/2 disabled in contour).

If a browser reuses an existing connection for additional requests, Envoy does not "re-triage" further requests on the connection since by design this connection is already tied to a Listener filter chain. In this case that means that subsequent requests on that same connection will be processed by the wildcard *.example.com filter chain and routed to the default app rather than flow through the new specific.example.com filter chain.

By default we have an idle timeout on downstream connections set to 60s so you can observe the requests being routed to the correct backends if you let the connection get timed out.

As a workaround as well, you could use the max-requests-per-connection Listener setting https://projectcontour.io/docs/1.27/configuration/#listener-configuration (though this is a global setting so maybe not ideal)

from contour.

The structure of our Envoy configuration in this situation is set up as the following hierarchy:

• HTTPS Listener on secure port (default 8443)
• Listener has multiple filter chains for different domains, in this case *.example.com and specific.example.com
• each of these filter chains will match connections with the appropriate SNI
• each filter chain has an HTTP filter chain that points to a route config
• Each route config has a list of domain matches and a list of routes with matches

One thing we could do to help with this case is for each specific FQDN, add a domain match and route (that would match the specific host header) to any wildcard routeconfigs that exist (and match the specific route) as a catch all

from contour.

cc @projectcontour/maintainers @projectcontour/contour-reviewers for any thoughts here

from contour.

e.g. here is an example wildcard routeconfig

    {
     "version_info": "4",
     "route_config": {
      "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
      "name": "https/*.example.com",
      "virtual_hosts": [
       {
        "name": "*.example.com",
        "domains": [
         "*.example.com"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/",
           "headers": [
            {
             "name": ":authority",
             "string_match": {
              "safe_regex": {
               "regex": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.example\\.com(:[0-9]+)?"
              }
             }
            }
           ]
          },
          "route": {
           "cluster": "default/echo-1/80/da39a3ee5e"
          },
          "metadata": {
           "filter_metadata": {
            "envoy.access_loggers.file": {
             "io.projectcontour.name": "echo-wildcard",
             "io.projectcontour.namespace": "default",
             "io.projectcontour.kind": "HTTPProxy"
            }
           }
          }
         }
        ]
       }
      ],
      "request_headers_to_add": [
       {
        "header": {
         "key": "x-request-start",
         "value": "t=%START_TIME(%s.%3f)%"
        }
       }
      ],
      "ignore_port_in_host_matching": true
     },
     "last_updated": "2023-12-11T21:30:04.295Z"
    },

in this case, we could duplicate each route on specific.example.com into this config, each of which is augmented with a host/:authority header match on specific.example.com

from contour.

As a workaround as well, you could use the max-requests-per-connection Listener setting https://projectcontour.io/docs/1.27/configuration/#listener-configuration (though this is a global setting so maybe not ideal)

This works for us as a workaround. Thank you!

Edit: We could further improve performance by disabling HTTP/2 and only supporting HTTP/1.1. I guess chrome is expecting short-living connections more on HTTP/1.1 than on HTTP/2.

from contour.

@sunjayBhatia as expected the workaround is really adding a lot of overhead, especially on many small requests, especially as chrome allows only 6 concurrent connections per origin. So it would still be important for us to get a fix without the need to set max-requests-per-connection to 1.

from contour.