This time with support for LZMA_SYNC_FLUSH 🎉
Works directly with liblzma.so via ctypes. No other dependencies.
LZMA is a compression algorithm - like gzip or bzip2.
The default configuration (of both Python lzma module and this library)
is to use the LZMA2 filter and XZ container format.
So you can directly save the compressed data in a file with the .xz suffix and it will work with many other
programs, utilities and systems. For example xzcat or xzgrep.
Wikipedia: Lempel–Ziv–Markov chain algorithm
Lot of software uses LZMA compression internally. For example many software package managers.
Yes, but it does not support the SYNC FLUSH operation.
There is LZMACompressor.flush()
but it does something different - it
finishes the compression process
and closes the compressor.
It is not possible to compress more data after flush().
For some of my use cases I need to use "sync flush". The constant LZMA_SYNC_FLUSH
does not even appear in the CPython source code.
From lzma/base.h (by Lasse Collin, public domain):
LZMA_SYNC_FLUSH = 1,
/**<
* \brief Make all the input available at output
*
* Normally the encoder introduces some latency.
* LZMA_SYNC_FLUSH forces all the buffered data to be
* available at output without resetting the internal
* state of the encoder. This way it is possible to use
* compressed stream for example for communication over
* network.
*
* Only some filters support LZMA_SYNC_FLUSH. Trying to use
* LZMA_SYNC_FLUSH with filters that don't support it will
* make lzma_code() return LZMA_OPTIONS_ERROR. For example,
* LZMA1 doesn't support LZMA_SYNC_FLUSH but LZMA2 does.
*
* Using LZMA_SYNC_FLUSH very often can dramatically reduce
* the compression ratio. With some filters (for example,
* LZMA2), fine-tuning the compression options may help
* mitigate this problem significantly (for example,
* match finder with LZMA2).
*
* Decoders don't support LZMA_SYNC_FLUSH.
*/Tested on Debian Linux and macOS.
You need to have liblzma installed - that means there should be a file
liblzma.so (for Linux) or liblzma.dylib (for macOS) somewhere in a library
directory (/usr/lib or similar). Usually it is already installed.
Install yalzma from current Github master:
$ pip install git+https://github.com/messa/yalzmaInstall specific version:
$ pip install git+https://github.com/messa/[email protected]Or add this line to your requirements.txt:
git+https://github.com/messa/[email protected]#egg=yalzma==0.0.4
from yalzma import LZMAEncoder
import lzma
text = b'Hello, World!'
enc = LZMAEncoder()
xz_data = enc.run(text)
xz_data += enc.finish()
assert lzma.decompress(xz_data) == textDemonstration of the flush functionality:
from io import BytesIO
enc = LZMAEncoder()
xz_data = enc.run(b'first line\n')
xz_data += enc.sync_flush()
assert lzma.open(BytesIO(xz_data), mode='rb').readline() == b'first line\n'
xz_data += enc.run(b'second line\n')
xz_data += enc.finish()
assert lzma.decompress(xz_data) == b'first line\nsecond line\n'
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent