hs-zstd's Introduction

Zstandard bindings for Haskell

This library provides Haskell bindings to the Zstandard compression library.

Note that is now the official repositoryc for the zstd Haskell package. The original authors (Facebook) are no longer interested in maintaining it.

The library is structured to provide several layers of abstraction.

For the simplest use cases, the top-level Zstd module is the best place to get started.
If you need to stream a large amount of data with a constant memory footprint, use the Zstd.Streaming module. See also the conduit-zstd package which provides a very thin wrapper to integrate with the conduit library. If you need to use lazy bytestrings instead, see the Zstd.Lazy module. This is built using the abstractions from the Zstd.Streaming module.
When your usage is dominated by lots of small messages (presumably using pre-computed compression dictionaries), use the Zstd.Efficient module to amortize the cost of allocating and initializing context and dictionary values.

Join in

If you'd like to help improve the code, please read the contribution guidelines. This discusses how to file bugs and submit changes to the code itself.

API documentation

The APIs should be easy to understand and work with, and you can find documentation on Hackage.

hs-zstd's People

Contributors

Stargazers

Watchers

hs-zstd's Issues

multi-core compression

Looking at zstd code enabling ZSTD_MULTITHREAD would enable that, but it probably needs testing if it works with the bindings.

Also needs to set ZSTD_c_nbWorkers.

Context memory inflates memory usage beyond RTS heap limit

hs-zstd allocates significant memory that isn't in the Haskell heap. This has implications:

The heap size limit becomes less effective (RTS -M flag).
The RTS is unaware of memory pressure when multiple contexts are created, especially when multiple contexts are created while there is little Haskell activity to trigger garbage collection.

I've lost a lot of time troubleshooting these issues, so I hope these can be prevented in the future.
Some ideas:

Codec.Zstd.Streaming: explicitly deallocate the stream context when Done. This will reduce severity of the memory amplification from O(number of streams between GCs) to O(number of concurrent streams). I'll give this a shot.
The C library seems to support the use of a custom allocator. If hs-zstd were to use Haskell-provided memory regions in pinned memory, the RTS will be able to collect when appropriate and enforce the heap size limit.
Improve documentation to acknowledge these intricacies.

Streaming Compression and Decompression using Dictionary

I need to compress using dictionary but the inputs are being consumed from network as lazy bytestrings. I want to keep it that way. Is there support for streaming operations with dictionary?

Recommend Projects