Chunk compression at rest about mimir HOT 3 CLOSED

• Compression can be composed on-top of existing file formats, if random access is not required. With some additional work, it can be implemented even if random access to uncompressed access is needed. (And a bit more work, if mmap(2) is needed, though mmap should not be used with golang in the first place due to goroutine stalls.)

Mimir does use random access to objects in the object store. Fortunately we removed most mmap in store-gateways for reasons you're mentioning. (However TSDB library still uses mmap, so blocks in ingesters do use this)

• As for CPU usage, this one is unlikely to be a bottleneck if proper algorithms and compression levels are chosen. For example, lz4 decompression operates pretty close to memmove(3) speeds. Not to mention, that if IO and CPU usage is overlapped then compression may even come for free. And given that compression reduces network, memory, and disk pressure, it may likely reduce end-to-end latencies.

Finally, transfer costs (and NVMe/memory on store gateways) are fairly expensive so it may be worth optimizing.

I agree there would be benefits to this. At my previous job I've implemented file-level compression of files on long-term storage with support for random-access, using lz4. It was a fun task, and I'd personally be happy to work on this again :) However it is a non-trivial engineering effort, and it's not addressing our biggest pain points today.

from mimir.

Thank you for your suggestion. We're not going to pursue this anytime soon.

First of all, we use TSDB file format from Prometheus, and we don't plan to change this (if changes are done in Prometheus first, Mimir would adapt to such changes).

Second, Mimir fetches many segments of the index and chunks on demand, and adding additional compression to these files would require lot of changes -- not just a redesign of the format, but changes in the code to adapt to the new format.

Finally, our storage costs are tiny compared to cost of other resources (especially cpu).

For these reasons, implementing this change is a very low priority for us.

from mimir.

I would agree to disagree here, mostly due to:

• Compression can be composed on-top of existing file formats, if random access is not required. With some additional work, it can be implemented even if random access to uncompressed access is needed. (And a bit more work, if mmap(2) is needed, though mmap should not be used with golang in the first place due to goroutine stalls.)
• As for CPU usage, this one is unlikely to be a bottleneck if proper algorithms and compression levels are chosen. For example, lz4 decompression operates pretty close to memmove(3) speeds. Not to mention, that if IO and CPU usage is overlapped then compression may even come for free. And given that compression reduces network, memory, and disk pressure, it may likely reduce end-to-end latencies.

Finally, transfer costs (and NVMe/memory on store gateways) are fairly expensive so it may be worth optimizing.

That said, let me seems like prometheus community was looking into this some time ago, there there is indeed not much traction:

• prometheus/prometheus#5865 (comment)

from mimir.