Comments (16)
TimWolla
commented on September 7, 2024
@r-tock From your configuration the frontend and backend configuration is missing. Are you using HTTP/2, the new HTX representation or something else that might affect the routing / HTTP handling?
from haproxy.
robin-anil
commented on September 7, 2024
No htx, and no http2. I will post the backend configuration, it is pretty complex with multiple environments and need sanitizing before posting.
from haproxy.
lukastribus
commented on September 7, 2024
Configuration is need as well as log files showing every request of a frontend session, including request routed correctly before the issue. Unfortunately, with the information provided we don't even know how you route between your backends.
from haproxy.
robin-anil
commented on September 7, 2024
@lukastribus @TimWolla Let me know which logs you need. We have our backend logs which shows how requests are incorrectly being routed /api requests going to our assets backend and consumer backends and vice versa (regardless of environment)
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy.stat mode 600 level admin
daemon
hard-stop-after 30s
tune.ssl.default-dh-param 1024
tune.ssl.cachesize 100000
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
tune.bufsize 16384
tune.maxrewrite 1024
ssl-engine rdrand
ssl-mode-async
nbthread 4
defaults
log global
mode http
compression algo gzip
compression type text/html text/plain text/css application/javascript application/octet-stream application/json
option httplog
option dontlognull
option redispatch
option tcp-smart-accept
option tcp-smart-connect
option forwardfor if-none
option splice-auto
option socket-stats
option http-buffer-request
timeout check 5s
timeout client 300s
timeout tunnel 60000s
timeout connect 20s
timeout http-keep-alive 300s
timeout http-request 30s
timeout queue 20s
timeout server 50s
hash-balance-factor 125
balance hdr(Cookie)
hash-type consistent djb2
stats enable
stats hide-version
stats uri /statz
default-server inter 5s fall 3 rise 1
errorfile 503 /etc/haproxy/errors/503.http
frontend health_check
bind *:8081
timeout client 5s
monitor-uri /
frontend http
bind :::80 v4v6
maxconn 4096
redirect scheme https if !{ ssl_fc }
default_backend assets_backend
... backends
from haproxy.
TimWolla
commented on September 7, 2024
@r-tock
Let me know which logs you need.
Ideally a combination of HAProxy logs (where does HAProxy claim to send the traffic) and backend logs (where did the traffic actually arrive). You can use unique-id-header to add a header identifying the request to the backend, logging this header in the backend and then add %ID to the HAProxy log format to correlate the logs.
You appear to be routing using the HTTP Host header only. It might be useful to add this header to the HAProxy logs as well: capture request header Host len 25.
from haproxy.
B-Reiter
commented on September 7, 2024
I can confirm that this is an issue in 1.9.2. Have not tested 1.9.3, we reverted to 1.8.17.
No errors in the proxy logs, but application errors, one very clear indication that we are hitting this bug is a reload of a page which loaded a favicon from a different backend.
from haproxy.
TimWolla
commented on September 7, 2024
@B-Reiter Are you able to provide your configuration and the logs I requested in the comment above yours?
from haproxy.
B-Reiter
commented on September 7, 2024
Logs will be tricky, I will need to set up a test-instance for 1.9.2 and try to replicate.
Config I can provide right now:
global
nbproc 2
cpu-map 1 0
cpu-map 2 1
daemon
maxconn 10000
tune.ssl.default-dh-param 4096
stats socket /var/lib/haproxy/admin.sock mode 660 level admin expose-fd listeners process 1
stats socket /var/lib/haproxy/process2.sock mode 660 process 2
stats timeout 30s
user haproxy
group haproxy
chroot /var/lib/haproxy
tune.ssl.cachesize 100000
tune.ssl.lifetime 600
tune.ssl.maxrecord 2859
defaults
mode http
retries 3
option http-server-close
option http-keep-alive
option http-ignore-probes
option httplog
option dontlognull
option forwardfor
log global
maxconn 10000
timeout connect 10s
timeout http-request 10s
timeout client 3600s
timeout server 3600s
errorfile 400 /etc/haproxy/errorpages/400.http
errorfile 500 /etc/haproxy/errorpages/500.http
errorfile 502 /etc/haproxy/errorpages/502.http
errorfile 503 /etc/haproxy/errorpages/503.http
errorfile 504 /etc/haproxy/errorpages/504.http
frontend http-in-all-frontends
bind 0.0.0.0:80
http-request set-header X-Forwarded-For %[src]
redirect scheme https if !{ ssl_fc }
http-response del-header Server
http-response del-header X-AspNet-Version
http-response del-header X-Powered-By
http-response del-header X-FEServer
http-response set-header ReferrerPolicy origin
http-response set-header X-XSS-Protection "1; mode=block;"
http-response set-header X-Frame-Options sameorigin
http-response set-header X-Content-Type-Options nosniff
frontend https-in-frontend-secure
bind 10.220.14.200:443 alpn h2,http/1.1 ssl no-tls-tickets no-sslv3 no-tlsv10 no-tlsv11 strict-sni ecdhe secp384r1 crt /etc/haproxy/ssl/wildcard.domain.net.pem crt /etc/haproxy/ssl/wildcard.prod.domain.net.pem crt /etc/haproxy/ssl/wildcard.dev.domain.net.pem crt /etc/haproxy/ssl/wildcard.test.domain.net.pem crt /etc/haproxy/ssl/random.domain.net.pem crt /etc/haproxy/ssl/random2.domain.net.pem ciphers ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
use_backend %[req.hdr(host),lower,map_str(/etc/haproxy/backendmapping.cfg,not_found)]
http-response del-header Server
http-response del-header X-AspNet-Version
http-response del-header X-Powered-By
http-response del-header X-FEServer
http-response set-header Referrer-Policy same-origin
http-response set-header X-XSS-Protection "1; mode=block;"
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
http-response set-header X-Frame-Options sameorigin
http-response set-header X-Content-Type-Options nosniff
listen stats1
bind 0.0.0.0:1936
bind-process 1
mode http
stats enable
stats refresh 60s
stats show-node
stats show-desc "Process 1"
stats uri /
listen stats2
bind 0.0.0.0:1937
bind-process 2
mode http
stats enable
stats refresh 60s
stats show-node
stats show-desc "Process 2"
stats uri /
# Default Backend - Invalid Service Page
backend not_found
errorfile 503 /etc/haproxy/errorpages/invalid.http
# IP Backend - Domain Website
backend domain.com
http-request set-header Host domain.com
http-response set-header Content-Security-Policy "default-src 'self' domain.com domain.net; form-action 'self'; base-uri 'self'; report-uri https://csp-report.domain.net"
server domain.com domain.com:80 maxconn 256 check verify none
backend dms01.prod.domain.net
server dms01.prod.domain.net dms01.prod.domain.net:443 maxconn 128 check ssl verify none
backend projmgmt01.prod.domain.net
server projmgmt01.prod.domain.net projmgmt01.prod.domain.net:8080 maxconn 128 check
backend cts01.prod.domain.net
server cts01.prod.domain.net cts01.prod.domain.net:443 maxconn 128 check ssl verify none
backend wawision01.prod.domain.net
server wawision01.prod.domain.net wawision01.prod.domain.net:80 maxconn 128 check
backend wiki01.prod.domain.net
server wiki01.prod.domain.net wiki01.prod.domain.net:8090 maxconn 128 check
backend collab01.prod.domain.net
server collab01.prod.domain.net collab01.prod.domain.net:8065 maxconn 128 check
... Lots more backends
from haproxy.
B-Reiter
commented on September 7, 2024
~$ haproxy -vv
HA-Proxy version 1.9.2 2019/01/16 - https://haproxy.org/
Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -m64 -march=native -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -fPIE -pie -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now
OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built without multi-threading support (USE_THREAD not set).
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
<default> : mode=HTX side=FE|BE
<default> : mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
from haproxy.
B-Reiter
commented on September 7, 2024
uname -a
Linux proxy2a.dc.domain.net 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
from haproxy.
wtarreau
commented on September 7, 2024
Thank you, I managed to reproduce it. It's related to http-reuse. There definitely is a bug there. It is enabled by default in 1.9, and something's definitely weird in it. As a workaround you can add "http-reuse never" in your config, it will work like in 1.8. Note that 1.8's reuse works however.
from haproxy.
wtarreau
commented on September 7, 2024
OK it was a stupid thing, when switching between idle conns, the previous connection was put back into the new conn's target list instead of the old one. Now fixed with commit 3c4e19f and backported to 1.9.
from haproxy.
B-Reiter
commented on September 7, 2024
Thank you! Fix will be in 1.9.4?
from haproxy.
haproxy-mirror
commented on September 7, 2024
yes, definitely!
from haproxy.
robin-anil
commented on September 7, 2024
Thanks for the fix! I wish i could have given a better repro case earlier
from haproxy.
EmericBr
commented on September 7, 2024
Just for note: ssl-mode-async is useless on engine rdrand, since this engine doesn't use a device and was not blocking. ssl-mode-async was designed to work using intel quick assist engine.
from haproxy.
Related Issues (20)
- server-state-file causing SEGV in 3.0-c0ee2d7 HOT 9
- "internal error" stream state when using gzip compression on 3.0-dev7 HOT 9
- src/ssl_ckch.c: dead code suspected by coverity HOT 1
- Enhanced observability for TLS ClientHello: ciphers, supported groups, key shares and sigalgs HOT 2
- if there are any issues with the code here? HOT 4
- OCSP stapling support with WolfSSL HOT 2
- Unexpected and random content-length in HTTP/1.0 responses with filter bwlim-out and 3.0-dev7-50d8c1-78 HOT 17
- infinite loop when shutting down HOT 7
- Support rewriting username in MQTTS CONNECT HOT 1
- Additional websocket connections cause exponential increase in CPU usage HOT 3
- cannot build on MacOS-14 Silicon HOT 13
- Lua ACL does not work HOT 3
- x86 build failed
- 3.0-dev8 runtime API "clear map" not clearing the whole map HOT 5
- quic http/3 limited-quic version 2.9.7 HOT 2
- Optimised code in h1.c doesn't do what it says it does HOT 16
- add `cap_sys_admin` to setcap supported caps HOT 7
- couple of Y2K38_SAFETY issues suspected by coverity HOT 1
- Stream state processing errors caused by special characters HOT 3
- QUIC: stalled connection rejection causing stalled connection reopening HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from haproxy.