Compression question about strfry HOT 6 CLOSED

Or perhaps it is allowed to send multiple nostr messages in a single websocket message payload? I did not think this was the case, but the NIP-01 document is a bit vague on the details.

from strfry.

In websockets terminology, "per-message" means that you can optionally compress some messages and not others. Each message has a flag that says whether it's compressed, basically. This is partly because it's not worthwhile to compress small messages, and also partly because of security reasons, where mixing attacker-provided data with secret data in the same compression context can leak the secret data (ie see the CRIME attack: https://security.stackexchange.com/questions/19911/crime-how-to-beat-the-beast-successor/19914#19914 )

The actual compression is specified as the DEFLATE (aka LZ77) algorithm (basically gzip without some headers and checksums).

When the websocket connection is started, clients and servers negotiate how the compression will happen. Either each message can use an empty DEFLATE buffer, or the buffer can be populated by previous messages over the same websocket connection. This is called a "sliding window" and achieves cross-message compression. Whoever told you this is not possible hasn't read the websocket compression RFC. It's is all described in detail here:

https://www.rfc-editor.org/rfc/rfc7692#section-7.1

The websocket sliding window system is not perfect though. First of all there are much better algorithms nowdays (ie zstd). The sliding window is limited to 32k max, and there is no way to pre-populate a dictionary and/or use a static dictionary for each message. Still, IMO the calculus for nostr is pretty clear that this is beneficial, which is why strfry enables it by default. You can see in the logs how effective the compression is for each connection.

from strfry.

Cool thanks! I read a bit more and learned that the "context takeover" headers are the relevant ones. I was a bit confused. Thanks for the answer!

from strfry.

No worries! FWIW, I find the *_context_takeover terminology confusing too. I think that RFC could've benefited from some significant editorial work.

from strfry.

Yes I was testing strfry but I was sending the server_no_context_takeover extension parameter. The compression was still pretty good (according to the helpful strfry logs). I am excited to see what happens if I don't send the server_no_context_takeover extension parameter. I need to figure out how to handle the received messages though. Anyways, thanks again for taking the time to answer my silly question.

PS. I was playing around with quadrable in the strfry directory:

quadb --db=./strfry-db dumpTree

Pretty cool stuff!

from strfry.

Thanks! The merkle tree stuff is going to have some pretty fun applications, I'm looking forward to writing up some more details on that. :)

For testing the sliding windows, I mostly use web browsers since this isn't a widely implemented feature for command-line clients. However, pretty much every browser negotiates the full window. The easiest way to see the effect is to do a query with sliding window enabled in the strfry config, then turn it off and do the same query. I think you have to restart the relay for this currently. That's not an inherent limitation, I just haven't got around to making that hot-changeable.

from strfry.