Comments (9)
Thanks so much for your help @ptaoussanis! Unfortunately after some deliberation we decided to move to Jetty 11 and put together a working Sente adapter for it. Took a bit of tweaking but works great now!
from http-kit.
Cross-posting taoensso/sente#426 (comment)
from http-kit.
@alexandergunnarson Hi Alex,
A few questions so long-
- What version of http-kit are you running?
- I presume this behaviour is new? Did anything obvious change prior to the new behaviour?
- Are you able to easily check if http-kit is receiving https "Upgrade", and "Connection: upgrade" headers? Off-hand, I seem to recall having seen this error before when behind a proxy that wasn't forwarding the above headers.
Thanks
from http-kit.
Hey! Thanks for your quick response.
- Running
2.8.0-beta3
. I believe there was some reason why we're using a beta version, but I don't remember precisely why. Could have been because the client used virtual threads, and that was a desirable feature for us? I'm going to upgrade now to2.8.0-RC1
to rule out issues. - This behavior has happened occasionally ever since we introduced WebSockets to our system back in December. So not new, just newly reported here :)
- CloudFront is forwarding all headers as far as I can tell (based on config and logs). However, it just occurred to me that because
Upgrade
, andConnection: upgrade
is specific to HTTP/1.1 (not HTTP/2), maybe what's happening here is thatUpgrade
, andConnection: upgrade
aren't actually being handled correctly somehow, and that the issue is being masked. That is, maybe it's the case that >99% of our clients are using HTTP/2 in the browser, not HTTP/1.1, and so when a user uses HTTP/1.1, it yields the "Unmasked client to server frame" error. But this is highly conjectural. I will see whether I can check in our logs.
from http-kit.
If CloudFront logs are to be believed, the following headers were forwarded for the most recent offending WS requests (all such requests yield 401 or 502 errors):
Accept-Encoding
Accept-Language
Cache-Control
Connection
Cookie
Host
Origin
Pragma
Sec-WebSocket-Extensions
Sec-WebSocket-Key
Sec-WebSocket-Version
Upgrade
User-Agent
The load balancer we have is "dumb" in that it forwards everything and is unlikely to affect anything.
from http-kit.
Just checked and the vast majority of our WebSocket requests (both failed and successful) are HTTP/1.1, so we can seemingly rule out the headers issue.
from http-kit.
@alexandergunnarson Hi Alex, apologies for the delay replying!
Unfortunately this isn't an area I'm familiar with so it's not clear to me what might be going on here.
From some very light digging, what I gather is:
-
During initial WebSocket handshake/upgrade, the client shares a masking key with the server. As a security measure, subsequent requests from the client are meant to be masked with that key to help protect the content.
-
So http-kit server is expecting subsequent WebSocket requests to be so masked.
-
But the error you're seeing appears to indicate that http-kit server is seeing a request that isn't masked as expected (?).
-
It seems possible causes for an unmasked or incorrectly masked request from the client would include (in ~order of expected likelihoood):
- Proxy or firewall interference (i.e. modifying WS traffic in a way that's interfering with masking)
- An issue with http-kit server (presumably some edge-case?)
- An issue with the sending WebSocket client
- Malicious activity
I'm just engaged with some other work atm so won't have the opportunity to investigate further right now, but perhaps you or someone else might be able to?
Some quick ideas-
Re: possible cause 1
- Is it possible in your setup that a proxy or firewall may be interfering? Is there perhaps some pattern in the logs for affected requests (e.g. geographical source, etc.)?
Re: possible cause 2
- Someone would need to dig into the relevant http-kit code, and/or protocols to better understand what might be happening. Ideally some sort of reproducible example would be ideal, but I understand that that might be very difficult to identify.
Re: possible cause 3
- Do you control the client sending the requests? Is it always the same client? Is it always configured the same way?
- If the client varies and/or isn't under your control, might it be possible to identify the client in your logs in the cases with the error?
Re: possible cause 4
- Again I'd probably start by looking for patterns in the logs of affected requests. Perhaps something stands out? Related to cause 3 above, I'd especially look for unexpected clients, etc. Is it possible to tell from the logs whether the affected requests appear to be properly authenticated?
Next steps
I've marked this issue as help wanted
. Hopefully someone is available to help investigate this.
Otherwise I'll try take a closer look myself when I'm next on batched http-kit work.
In the meantime any additional info you could provide (esp. re: questions above) would be helpful!
from http-kit.
@alexandergunnarson You're very welcome, I'm sorry that you needed to put effort into a move! But happy it worked out (and happy for Sente to get Jetty 11 support!). Thanks for updating 👍
Will keep this issue open until someone can investigate further and hopefully find a fix on http-kit's side if necessary/possible.
from http-kit.
No worries — appreciate your support :)
from http-kit.
Related Issues (20)
- Ring websocket API support HOT 4
- Problem in native-image and HttpUtils when using virtual threads HOT 11
- 2.8.0-beta2 has CIDER dependencies HOT 3
- CURL and finagle failing to parse :set-cookies with '\n' HOT 13
- Unix socket benchmark HOT 1
- It will turn headers into camel format HOT 2
- logger-warn gets rebound HOT 1
- logger-warn and error-warn are passed in wrong order to HttpServer constructor HOT 1
- Requests which throw java.net.ConnectException may actually succeed HOT 3
- Request Map doesn't contain information about authority HOT 8
- http-kit v2.8.0-RC1 HOT 1
- http-kit v2.8.0 final HOT 1
- Consider adding `Content-Type: text/plain` to HTTP 500 response in `org.httpkit.server.HttpHandler#run` HOT 6
- v2.7.0 SNI change broke connections to plain IP addresses with SSLHandshakeException "Hostname or IP address is undefined." HOT 8
- Add options for encoding nested form and query params a la clj-http HOT 7
- WebTransport support HOT 2
- NoSuchMethodError on projects AOT'd on java 21 but run on earlier java versions HOT 13
- [Proposal][Client] Consider more idiomatic bridges with JVM async paradigms HOT 5
- [client][performance] Regression after #446 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from http-kit.