Hello guys,

I recently found out a bug regarding decompressing a file which does not have frame-crc.
If I decompress the file without frame crc, the last block is not decompressed, and is ignored.

Here is a simple example,

>echo "Hello World" | ./lz4 -c | ./lz4 -d -c --no-sparse
Hello World

>echo "Hello World" | ./lz4 -c --no-frame-crc | ./lz4 -d -c --no-sparse
(nothing is printed)

I looked into the problem.
And, I found out that LZ4F_decompress behaves differently depending on whether there is frame-crc or not. If there is the crc, LZ4F_decompress returns "4" after reading the last block. However, if there is not the crc, LZ4F_decompress returns "0" after reading the last block. This makes LZ4IO_decompressLZ4F return earlier before writing last block into the output file.

I think there are two solutions.

1. Making LZ4F_decompress behave the same way regardless of whether there is crc or not.
   This is, making it return "4" even if there is not frame crc.
2. Switching the order of if-statement in LZ4IO_decompressLZ4F in lz4io.c like this.

diff --git a/programs/lz4io.c b/programs/lz4io.c
index 209f5ed..bfa3488 100644
--- a/programs/lz4io.c
+++ b/programs/lz4io.c
@@ -844,7 +844,6 @@ static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE
             nextToLoad = LZ4F_decompress(ress.dCtx, ress.dstBuffer, &decodedBytes, (char*)(ress.srcBuffer)+pos, &remaining, NULL);
             if (LZ4F_isError(nextToLoad)) EXM_THROW(66, "Decompression error : %s", LZ4F_getErrorName(nextToLoad));
             pos += remaining;
-            if (!nextToLoad) break;

             if (decodedBytes)
             {
@@ -853,6 +852,8 @@ static unsigned long long LZ4IO_decompressLZ4F(dRess_t ress, FILE* srcFile, FILE
                 DISPLAYUPDATE(2, "\rDecompressed : %u MB  ", (unsigned)(filesize>>20));
                 storedSkips = LZ4IO_fwriteSparse(dstFile, ress.dstBuffer, decodedBytes, storedSkips);
             }
+
+            if (!nextToLoad) break;
         }
     }

Best regards,
Yongwoon Cho

I once wrote a small compression/decompression tool, which was RLE based.
While doing so, I found a lot of redundancy in current implementations and came up with making some values that had special meanings. This reduced the size of the compressed files dramatically.

One idea I never tried, was to use inverted MIDI bytes. Inverted, because there's an advantage in having values 0 ... 127 mean "more data bytes" and values 128 ... 255 "last data byte".

Imagine you have a length, which is 8387 bytes. For this, you'd need a 0x0f in the token and 32 runs of 0xff followed by the value 212.

Instead, I think it would pay to still have the 0x0f in the token, but the length section could be processed by the decoder like the following:
1: zero the length variable
2: shift the length variable left by 7
3: read a data byte
4: add the data byte to the length variable
5: If bit 7 of the data byte is zero, then go back to step 3
6: add 15-128 to the length (15 is the value from the token, -128 for zeroing the last data byte's bit 7)

Values from 15 to 142 will still only use a single data byte. Values from 143 to 16398 would use two bytes; values from 16399 to 2097166 would use three bytes; higher values are not very likely, but still they would not use a lot of space.
The only place I think the old system might be able to win, is with length values between 143 and 269 (eg. a single byte). If a compressed stream contains a lot of those, then my suggested method will cause the file to be larger.

A different approach could be to analyze the length values. If for instance the length '18' is used thousands of times, then it might be a good idea to tokenize it, thus values from 0 ... 251 would look up length values; a value of 252 would mean "add a new byte sized value", a value of 253 would mean "add new 16-bit sized value", etc...

Hello,

I'm using your compression library for my engine:
http://esenthel.com/
and it works great. Since last updates though I'm getting crashes when running on Linux (Ubuntu x64).
The crash occurs only when LZ4 is compiled in "Performance Release" setting in NetBeans.
There it generates:
Signal received: SIGSEGV (Segmentation fault)

This occurs around LZ4HC_compress_generic
with instruction:
movdqa (%rdi,%rax,1),%xmm0
$rdi=0x7fffea4b53e6
$rax=0x0

I've read about movdqa and it requires aligned accesses to 16 bytes, while it looks like $rdi has ..e6 ending.

I've narrowed the problem down to LZ4_wildCopy, which I see uses
static void LZ4_copy8(void* dstPtr, const void* srcPtr)
{
if (LZ4_UNALIGNED_ACCESS)
{
if (LZ4_64bits())
(U64)dstPtr = (U64)srcPtr;
else
((U32_)dstPtr)[0] = ((U32_)srcPtr)[0],
((U32_)dstPtr)[1] = ((U32_)srcPtr)[1];
return;
}
memcpy(dstPtr, srcPtr, 8);
}
if I replace it with:
static void LZ4_copy8(void* dstPtr, const void* srcPtr)
{
memcpy(dstPtr, srcPtr, 8);
}
Then I have no crashes, which looks like LZ4_UNALIGNED_ACCESS should be disabled for this platform.

I'm using latest Linux Ubuntu 64-bit, 14.10, installed on one of the latest MacBook Air, Intel® Core™ i5-4250U CPU @ 1.30GHz × 4, Intel® Haswell Mobile.
Note that when I reboot into Mac OS X on the same laptop, then the LZ4 works just fine with release and all default settings (no crashes).
So I'm assuming that Linux makes extra requirements for alignment, or perhaps the C compiler generates invalid code.
Not sure, but I've found 3 working solutions:

Disable LZ4_UNALIGNED_ACCESS altogether.

if (defined(CPU_HAS_EFFICIENT_UNALIGNED_MEMORY_ACCESS) \

|| defined(__ARM_FEATURE_UNALIGNED) \
|| defined(__i386__) || defined(__x86_64__) \
|| defined(_M_IX86) || defined(_M_X64) \
|| defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_8__) \
|| (defined(_M_ARM) && (_M_ARM >= 7)))\

&& (defined ANDROID || !defined linux) // this was added

define LZ4_UNALIGNED_ACCESS 1

else

define LZ4_UNALIGNED_ACCESS 0

endif

Note the extra check for ANDROID, because Android is based on Linux, and when compiling for Android, then linux is always defined, however I want to disable unaligned access just for linux, without the android.

Second solution is to:
compile LZ4 with "Release" in NetBeans, instead of "Performance Release"

Third solution is to:
static void LZ4_copy8(void* dstPtr, const void* srcPtr)
{

if defined ANDROID || !defined linux

if (LZ4_UNALIGNED_ACCESS)
{
    if (LZ4_64bits())
        *(U64*)dstPtr = *(U64*)srcPtr;
    else
        ((U32*)dstPtr)[0] = ((U32*)srcPtr)[0],
        ((U32*)dstPtr)[1] = ((U32*)srcPtr)[1];
    return;
}

endif

memcpy(dstPtr, srcPtr, 8);

}

I've done some benchmarks, for compressing 128MB using LZ4 HC, and all methods have similar results.

My GCC_VERSION is 409

In the end I'm suspecting compiler error, since movdqa appears, it means that the compiler tries to somehow merge 2x64-bit operations into 1x128bit, because movdqa is 128-bit.

I'm choosing to go with solution number 3.

Recently, Joffrey LABORDE (implicitly) pointed out, there is no clear evidence for interoperability between releases. Also there is a similar FAQ : "Are 64bit (lz4c) and 32bit (lz4c32) interoperable ?"
The following Python3 script is initial answer for these questions :

• https://gist.github.com/t-mat/3c5e1e4df3b250691caf

This automated test compress and decompress between

• All git tagged releases
• lz4c and lz4c32
• -1 and -9

I hope this test will guarantee basic backward and forward compatibility for all LZ4 releases.

Usage

cd
mkdir lz4-test
cd lz4-test
curl -O https://gist.githubusercontent.com/t-mat/3c5e1e4df3b250691caf/raw/test-lz4-releases.py
chmod +x ./test-lz4-releases.py
./test-lz4-releases.py

[edit] : Add chmod

Details

• Checkout all tagged release code (pattern r[0-9][0-9][0-9])
  • Compiles lz4c and lz4c32
  • Rename these executables. Add tag as suffix to these executables (eg lz4c32.r129)
  • Copy all executables to grand parent directory
• Copy README.md as test file (test.dat)
• Compress test.dat by all executables
  • Use lz4c.<TAGS> and lz4c32.<TAGS>
  • Specify -1zf and -9zf
• Remove dupilcated .lz4 files
• Decompress all .lz4 files by all executables
  • Compare decompressed data with test.dat
  • Remove the file if there is no difference
  • Report the error if difference is found

Remained Problems

• More test data is needed for serious test.
• Test must be running on TravisCI.
  • Also VC++ and appveyor ?
• Dependency to python is not fit for LZ4 project.
• Is make cc=clang needed ?

Suggestion by @jorangreef :

Test new hash formula, based on Tabulation hashing

Expectation : slower on x86/x64.
Maybe competitive on ARM, depending on cache pressure

I've got the following warnings when I build LZ4 on ARM platform (RaspberryPi Model B+ with 2015-02-16-raspbian-wheezy) :

../lib/lz4.c: In function ‘LZ4_slideInputBuffer’:
../lib/lz4.c:1324:33: warning: cast increases required alignment of target type [-Wcast-align]
../lib/lz4hc.c: In function ‘LZ4_compressHC_continue_generic’:
../lib/lz4hc.c:646:24: warning: cast increases required alignment of target type [-Wcast-align]

Line #1324 in lib/lz4.c is

char* LZ4_slideInputBuffer (void* LZ4_Data)
{
    LZ4_stream_t_internal* ctx = (LZ4_stream_t_internal*)LZ4_Data;
    int dictSize = LZ4_saveDict((LZ4_stream_t*)ctx, (char*)ctx->bufferStart, 64 KB); // Line#1324
    return (char*)(ctx->bufferStart + dictSize);
}

-Wcast-align is warning for alignment mismatch. Here, alignment of LZ4_stream_t_internal is 4 (sizeof(U32)), but alignment of LZ4_stream_t is 8 (sizeof(long long)).

So we should maintain structure's alignment more strictly like this :

#if defined(__GNUC__) || defined(__INTEL_COMPILER) || defined(__clang__)
#  define GCC_ATTR_ALIGNED(x) __attribute__ ((aligned (x)))
#endif

typedef struct {
    ...
} LZ4_stream_t_internal GCC_ATTR_ALIGNED(8);

Perhaps, adding union { long long hashTableU64[HASH_SIZE_U32/2]; ... }; is more generic and LZ4 way.

Build Log

$ uname -a
Linux raspberrypi 3.18.7+ #755 PREEMPT Thu Feb 12 17:14:31 GMT 2015 armv6l GNU/Linux
$ cd
$ git clone https://github.com/Cyan4973/lz4.git
$ cd lz4
$ git checkout dev
$ git rev-parse HEAD
80e71c6e8b0cbcd3b9976ded45cef1474a34b40c -> https://github.com/Cyan4973/lz4/commit/80e71c6e8b0cbcd3b9976ded45cef1474a34b40c
$ make
make[1]: Entering directory '/home/pi/lz4/programs'
cc      -I../lib  -O3 -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"r128\"  ../lib/lz4.c ../lib/lz4hc.c ../lib/lz4frame.c ../lib/xxhash.c bench.c lz4io.c lz4cli.c -o lz4
../lib/lz4.c: In function ‘LZ4_slideInputBuffer’:
../lib/lz4.c:1324:33: warning: cast increases required alignment of target type [-Wcast-align]
../lib/lz4hc.c: In function ‘LZ4_compressHC_continue_generic’:
../lib/lz4hc.c:646:24: warning: cast increases required alignment of target type [-Wcast-align]
cc      -I../lib  -O3 -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"r128\"  -DENABLE_LZ4C_LEGACY_OPTIONS ../lib/lz4.c ../lib/lz4hc.c ../lib/lz4frame.c ../lib/xxhash.c bench.c lz4io.c lz4cli.c -o lz4c
../lib/lz4.c: In function ‘LZ4_slideInputBuffer’:
../lib/lz4.c:1324:33: warning: cast increases required alignment of target type [-Wcast-align]
../lib/lz4hc.c: In function ‘LZ4_compressHC_continue_generic’:
../lib/lz4hc.c:646:24: warning: cast increases required alignment of target type [-Wcast-align]
make[1]: Leaving directory '/home/pi/lz4/programs'

For microcontrollers which have very little SRAM, it would be possible to decompress large files, if the maximum match offset was limited.
This would of course increase the size of the output files, but it would result in making it possible to decompress more files on devices which has for instance 16K RAM available.

Limiting the maximum size of the match offset does not necessarily mean that a new compressed block will be started; it just means that it's not allowed to use data, which is beyond a certain distance.
Sometimes it would pay to start a new block of compressed data, sometimes it would pay to just find another position where the compressed data could be read from.

(Note: I have a suggestion for reducing the number of 0xff in the compressed stream; I'll file a suggestion on this later)

See also my comment here: http://community.arm.com/docs/DOC-10570#comment-15939

Hello I can't compile a program link against lz4 static library on 64bit linux,
command is
g++ -fPIE -pie test.cpp /usr/lib64/liblz4.a -o test

error is
/usr/bin/ld: /usr/lib64/liblz4.a(lz4.o): relocation R_X86_64_PC32 against undefined symbol `memcpy@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC

I try to recompile lz4 use env CFLAGS="-fPIC" make, but still have that error when I static linking with liblz4.a.

Sony uses for their android firmwares ext4 images which are LZ4 compressed. I am trying to create an application for arm (based on C) to decompress those images on a device directly using your library.
I have a similiar application for Windows based on C# (using lz4net).

The input file is ~2GB and the output file ~5GB. However my arm app can not correctly decompress the image. For my tests I fill the output file with 00 before decompressing it and after running my app, some bytes still remain 00. The locations are random and change with every execution. At some location it's just one byte in the middle of the decompressed block which is still 00.

But sometimes a few bytes are wrong followed by 00 until the next page (?). (Note that this is basically in the middle of the block.)

I have tried LZ4_decompress_fast

if(LZ4_decompress_fast(source, dest, decompressedsize) != compressedsize)
        perror("Error decompressing LZ4 block");

and also LZ4_decompress_safe

if(LZ4_decompress_safe(source, dest, compressedsize, decompressedsize) != decompressedsize)
        perror("Error decompressing LZ4 block");

I have tried gcc-4.8, gcc-4.9 and clang3.6 (those provided within the Android NDK). Tried various optimization flags from -O0, -O2, -O3, -Os but the issue remains and I don't get any error during decompression.

My C# app and also some other app (which uses the Java implementation of LZ4) can decompress the image correctly.

I can't reproduce the problem if I try to decompress a single block which occured to be faulty once.

Do you have any idea how to fix this problem?

Hi to all,

AFAIK the windows installer shipped through
http://fastcompression.blogspot.de/p/lz4.html
but that one only installs the executable binaries.
Is there any plan (or any way) to build liblz4 (dynamic and/or static) under windows as well?

Best,
Peter

Description

If the program is compiled using a custom CFLAGS, then lz4 reports the wrong release version:

$ lz4 --version
*** LZ4 command line interface 64-bits r128, by Yann Collet (Sep  1 2015) ***

How to reproduce

CFLAGS="-Wall" make
make install
lz4 --version

Analisys

The problem seems to be that the release version is set in programs/Makefile trough the variable CFLAGS:

CFLAGS ?= -O3
CFLAGS += -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-qual -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"$(RELEASE)\"
FLAGS  := -I../lib $(CPPFLAGS) $(CFLAGS) $(LDFLAGS)

However, here make is invoked from the main Makefile using the option -e which cause the variables inside the Makefile to be overridden by the environment variables. This means that LZ4_VERSION is never defined so the default release version is used. The latter is defined in programs/lz4cli.c but is still r128.

frametest.c and fuzzer.c need to be adapted in the same way that fullbench and bench already are wrt. sys/timeb.h and the ftime syscall.

Currently, they fail to build e.g. on NetBSD with:

# gmake frametest
cc      -I../lib  -O3 -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"r126\"  ../lib/lz4frame.c ../lib/lz4.c ../lib/lz4hc.c ../lib/xxhash.c frametest.c -o frametest
/var/tmp//ccDdJ0OZ.o: In function `fuzzerTests':
frametest.c:(.text+0x1152): undefined reference to `ftime'
frametest.c:(.text+0x13e4): undefined reference to `ftime'
/var/tmp//ccDdJ0OZ.o: In function `main':
frametest.c:(.text.startup+0x2a5): undefined reference to `ftime'
Makefile:90: recipe for target 'frametest' failed
gmake: *** [frametest] Error 1
# gmake fuzzer
cc      -I../lib  -O3 -std=c99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -pedantic -DLZ4_VERSION=\"r126\"  ../lib/lz4.c ../lib/lz4hc.c ../lib/xxhash.c fuzzer.c -o fuzzer
/var/tmp//ccMMIjtu.o: In function `FUZ_test':
fuzzer.c:(.text+0x539): undefined reference to `ftime'
fuzzer.c:(.text+0x585): undefined reference to `ftime'
/var/tmp//ccMMIjtu.o: In function `FUZ_unitTests':
fuzzer.c:(.text+0x2d4e): undefined reference to `ftime'
fuzzer.c:(.text+0x2d93): undefined reference to `ftime'
/var/tmp//ccMMIjtu.o: In function `main':
fuzzer.c:(.text.startup+0x3bc): undefined reference to `ftime'
Makefile:84: recipe for target 'fuzzer' failed
gmake: *** [fuzzer] Error 1

Since a few revisions, lz4 library now integrates lz4frame,
which needs xxHash for checksum validation.
xxHash is therefore integrated into lz4 library, but not exposed (no *.h is provided), since it's for internal library use only.

The problem is, from an ABI perspective, xxHash functions are nonetheless there.
So when someone try to link lz4 library on top of their own xxHash library, it conflicts, since C has no namespace separation.

This use case should be better supported.

Ideally, there would be a way to create lz4 library with its own private version of xxHash library, which would remain invisible outside of lz4 library.

In lib/Makefile, the -a switch is used for cp. As I do not know what -a does and it's not available for my platform, I'd like to request that cp -a is not used.
I'm hoping that you will consider using GNU autotools to generate the Makefile for the platform, which is being targeted; eg. so one could type ./configure && make && [sudo] make install

Hi:

I got a compile error when I try to develop with lz4 in linux kernel(ver. 3.14):
typedef redefinition with different type: types ('long' vs 's64' (aka 'long long'))
typedef __INT64_TYPE int64_t;
****linux/types.h:312:17 note: previous definition is here
typedef __s64 int64_t;

I think there need another MACRO switch in lz4.c:

if defined (STDC_VERSION) && (STDC_VERSION >=199901L) && (!defined (KERNEL))

include <stdint.h>

else

typedef unsigned ...

Waiting for any advice.
Thanks a lot!

I am testing the performance of the LZ4 library through the JNI binding provided by jpountz. Currently the Java binding uses r123. I have determined a quite bad degradation of performance when the decompressed data consists of a short byte sequence repeated many times (with a period of 1-32 bytes). In reviewing the memory operations used for decompression I noticed that unaligned memory access is applied, which at least my Intel Core i7 does quite slowly when source and target are close together. As a specific example, for a period length 101 decompression reaches 16 GB/s, but for periods of 3, 5, or 7 the speed drops to 1.8 GB/s. A simple stretch of the same byte value works with 5 GB/s.

To alleviate this issue I have written a wild copying routine which does no reading at all in the copy loop for periods up to 8 and for wider periods uses word-aligned writing. For periods of 3, 5, and 7 I get at least 6 GB/s; for periods of 1, 2, 4, 8 I get ~30 GB/s (vs. 5 for the JNI version). For wider periods, say 13, I get 3.6 GB/s vs. 1.8 GB/s native.

I have opened this issue in case there might be some interest in involving this kind of approach on the native LZ4 project, My code was presented at lz4/lz4-java#72. It is specific to a little-endian architecture, but is trivial to adapt to big-endian (swapping all >>> and << ops).

lz4 could be used as a tar compression program: it understands -d
May I set compression level with something like GZIP_OPT in gzip?

LZ4_OPT=-9 tar -I /usr/bin/lz4 "$@"

I've checked recent lz4 repo on Ubuntu 14.04 with the following commands :

cd
mkdir lz4-test && cd lz4-test
git clone https://github.com/Cyan4973/lz4.git
cd lz4


# Checking master branch
git checkout master
file * programs/* | grep LF

# result : nothing


# Checking dev branch
git checkout dev
file * programs/* | grep LF

# result:
programs/xxhash.c:          C source, ASCII text, with CRLF, LF line terminators
programs/xxhash.h:          C source, ASCII text, with CRLF, LF line terminators

It seems master branch is okay, but in dev branch, programs/xxhash.* has some bad EOLs.

Hello,
I can't seem to find any structured info on compression level used in LZ4. From the source code it seems that values from 0 to 16 should be used, and it seem that from level 9 and above HC compression will be used. However, I am not sure if I am correct on it. So I would be nice if devs provide some clear info about compression levels and how they relate to each other. How does computational complexity raise with raising compression level? Is it linear? Logarithmic? Are there any "standard" compression levels that should be used in benchmarks? Well, I think you got the idea.
Thank you!

When I compiled r128's programs/lz4io.c with MSVC, I got the following error :

1>..\..\programs\lz4io.c(239): warning C4013: 'S_ISREG' undefined; assuming extern returning int
1>lz4io.obj : error LNK2001: unresolved external symbol _S_ISREG

Since MSVC does not have S_ISREG, we should define equivalent macro. An answer on StackOverflow suggests Curl's macro

#if !defined(S_ISREG) && defined(S_IFMT) && defined(S_IFREG)
#define S_ISREG(m) (((m) & S_IFMT) == S_IFREG)
#endif

Perhaps, it is good time to consider some Windows based online CI environment which has MSVC, eg. AppVeyor.

Please consider adding those switches to lz4 command line utility to get a similar cmdline interface to other _ZIP_s variant in Unix.

It is openly painfull to make special case for only lz4 everywhere to get lz4 support in utility because lz4 require '-'/'- -' while all other compressor utilities do not require a '-' as input stream when piped into; or another '-' stdout stream.

Having a '-' for stdout is nice. However, adding those switches will permit to add easily lz4 support in utilities/scripts.

So every script/utility which make use '-c -d' or no '-' when stdin stream is piped has to make a special case... for lz4.

Please fix this to facilitate lz4 support in utility/script.

Note: all compressor utility that comes to mind bzip2, lzip, lzop, gzip, xz-utils have those switches. lzma utilities may have them too, although I don't remeber because I jumped to xz-utils which has lzma support. I don't use zip often but to unzip, so I have no idea what it has as cmdline switches.

Recently, we introduce new function naming convention. So I think it's good time to change other names.

I would like to add LZ4F_ prefix to the following names

// lz4frame.h
typedef enum { LZ4F_default=0, max64KB=4, max256KB=5, max1MB=6, max4MB=7 } blockSizeID_t;
//                             ^^^^^^^    ^^^^^^^^    ^^^^^^    ^^^^^^     ^^^^^^^^^^^^^

typedef enum { blockLinked=0, blockIndependent} blockMode_t;
//             ^^^^^^^^^^^    ^^^^^^^^^^^^^^^^  ^^^^^^^^^^^

typedef enum { noContentChecksum=0, contentChecksumEnabled } contentChecksum_t;
//             ^^^^^^^^^^^^^^^^^    ^^^^^^^^^^^^^^^^^^^^^^   ^^^^^^^^^^^^^^^^^

typedef enum { LZ4F_frame=0, skippableFrame } frameType_t;
//                           ^^^^^^^^^^^^^^   ^^^^^^^^^^^

// lz4frame_static.h
#define LZ4F_GENERATE_ENUM(ENUM) ENUM,
//                               ^^^^

Since replacing these enums will cause source code incompatibility, mark old enums as "obsolete" and append new one.

// New
typedef enum { LZ4F_blockLinked=0, LZ4F_blockIndependent } LZ4F_blockMode_t;

// Obsolete
typedef enum { blockLinked=0, blockIndependent} blockMode_t;

We also could introduce "deprecated enum" as the following notation

enum {
    en_a,
    en_b __attribute__((deprecated)), // gcc, clang
//  en_c __declspec(deprecated),      // MSVC does not allow this form
    en_d __pragma(deprecated(en_b)),  // MSVC (2010 or later)
};

• Clang : Attributs on enumerators

but I think it should be another issue in near future.

Can you please improve the documentation in the streaming section? At least you should add a link to a working example. This snippet was helping me a lot. A doxygen conform documentation would also be nice... :D

On FreeBSD; 'liblz4-129'; installed via portmaster.

This used to work, but fails with 129:

cat /dev/urandom | lz4c | lz4c -d | cat > /dev/null
Error 68 : Skip error (sparse file)

I can make the piped output version work again with --no-sparse on the decompression step; i.e. both of these work:

cat /dev/urandom | lz4c | lz4c -d --no-sparse | cat > /dev/null
cat /dev/urandom | lz4c | lz4c -d > /dev/null

It seems to me that --no-sparse should be the default when outputting to a pipe; and to be compatible with other (bzip2, gzip, etc) filters that don't require an additional flag. This could be accomplished either via stat() and S_ISREG() testing, or by gracefully recovering on a skip error. (My guess is the former will be more straightforward to implement, and the latter a more robust solution for unexpected cases.)

Following PR #114 and @t-mat suggestion on opening an issue ticket for a suggestion to improve licensing terms/contributions/contributors clarity, and simplify sub/same-license per contribution.

Please consider this suggestion to use a simple approach, meaning using COPYING.{BSD-2-clause,GPLv2} (or simplly COPYING.BSD-2) files addition and usage of dedicated header in files for authors/contributors and addition of AUTHORS file for the main/noticeable/regular contributors per contribution e.g.

AUTHORS:

Copyright (c) 2012-2015 Yann Collet <[email protected]>
Copyright (c) 2015 Takayuki MATSUOKA <[email protected]> for contrib/examples
[....]

HEADER (per file):

/*
 * Copyright (c) 2012-2015 Yann Collet <[email protected]>
 * Distributed under the 2-clause BSD license.
 */

Or a more verbose header variant...

Since we recently introduce long commands, I also want to see --no- prefix for boolean switches like gcc's -Wno- or Ruby OptionPraser's --no- convention.

What's the point ?

Since command line switches are evaluated left to right order, in some situations boolean switch orverriding is useful.
For example in some Makefile

# "Default" behaviour
LZ4=lz4 --quiet --force

...
some-target:
    # "Temporary" behaviour
    $(LZ4) --verbose --no-force ...

We already have --quiet and --verbose pair, but currently there is no opposite command for --force.

Problem : This may also introduce a lot of boilerplates and redundancy to the command line parser.

programs/xxhash.c and programs/xxhash.h are in r120 DOS formatted, they were UNIX formatted in r117.

This is first approximation to solve "Add list of LZ4 variants on Github".

Demo

https://t-mat.github.io/lz4/

My fork is https://github.com/t-mat/lz4/tree/gh-pages. Please see index.html. It is almost same as current README.md.

gh-pages branch and GitHub pages

gh-pages is special branch for GitHub. You can see how gh-pages works by the following commands.
Read "Creating Project Pages manually" for the details.

cd /your/workspace/gh-pages/
git clone https://github.com/Cyan4973/lz4.git
cd lz4
git checkout --orphan gh-pages
git rm -rf .
echo "LZ4 gh-pages testing 1,2,3" > index.html
git add index.html
git commit -a -m "First gh-pages commit"
git push origin gh-pages

Here, you can open https://Cyan4973.github.io/lz4/ in your browser.

After that, if you could merge my gh-pages branch.

Further evolution

For simplicity, I use Strapdown.js for Markdown rendering. But obviously it is js-heavy page.
So for long term, it would be nice to introduce some static page generators. For exmaple GitHub recommends Jekyll.

See also

• Setting up a custom domain with GitHub Pages

Hi I can't build lz4 r125 on my dev machine:

CentOS 6.5 x86_64
gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC)

With following error messages:

$ make
compiling static library
lz4hc.c:95: warning: declaration does not declare anything
lz4hc.c: In function 'LZ4HC_init':
lz4hc.c:124: error: 'LZ4HC_Data_Structure' has no member named 'hashTable'
lz4hc.c:124: error: 'LZ4HC_Data_Structure' has no member named 'hashTable'
lz4hc.c: In function 'LZ4HC_Insert':
lz4hc.c:140: error: 'LZ4HC_Data_Structure' has no member named 'hashTable'
lz4hc.c: In function 'LZ4HC_InsertAndFindBestMatch':
lz4hc.c:165: error: 'LZ4HC_Data_Structure' has no member named 'hashTable'
lz4hc.c: In function 'LZ4HC_InsertAndGetWiderMatch':
lz4hc.c:224: error: 'LZ4HC_Data_Structure' has no member named 'hashTable'
make: *** [liblz4] Error 1

Did I miss something?

 269 static void LZ4_copy8(void* dstPtr, const void* srcPtr)                                                                                                                  
 270 {                                                                                                                                                                        
 271 #if GCC_VERSION!=409  /* disabled on GCC 4.9, as it generates invalid opcode (crash) */                                                                                  
 272     if (LZ4_UNALIGNED_ACCESS)                                                                                                                                            
 273     {                                                                                                                                                                    
 274         if (LZ4_64bits())                                                                                                                                                
 275             *(U64*)dstPtr = *(U64*)srcPtr;                                                                                                                               
 276         else                                                                                                                                                             
 277             ((U32*)dstPtr)[0] = ((U32*)srcPtr)[0],                                                                                                                       
 278             ((U32*)dstPtr)[1] = ((U32*)srcPtr)[1];                                                                                                                       
 279         return;                                                                                                                                                          
 280     }                                                                                                                                                                    
 281 #endif                                                                                                                                                                   
 282     memcpy(dstPtr, srcPtr, 8);                                                                                                                                           
 283 }

gcc-4.9 and upcoming gcc-5 correctly assume that dstPtr and srcPtr are suitably aligned for U64,
otherwise you're invoking undefined behavior.
Since version 4.9 gcc's vectorizer uses that alignment info.
So please at least add gcc-5 to the version check.

See also: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65709

@Cyan4973,
Since there is no example code for LZ4F, I've created initial version of "Idiomatic lz4frame decompressor"

• https://gist.github.com/t-mat/c47868a00c60f93682d8

During writing this code, I have some questions and record these questions as comment which is beginning with !!!. My questions are

• Need some "safe" LZ4F error code value/symbols in lz4_frame.h
• Is 64KiB recommended source buffer size ?
• Should we use LZ4F_getFrameInfo() to determine "best" destination buffer size, or just always allocate 64KiB ?
  • Need public version of LZ4F_getBlockSize() to allocate "best" destination buffer
• Finally, is this sufficiently "idiomatic" ?

@funcodeio, since this PR is somewhat related #106 and you have enough knowledge and experience of all part of LZ4, could you also review this code ?

Proposed patch

Replace the current sed script

LIBVER_MAJOR=`sed -n '/LZ4_VERSION_MAJOR/s/.*\s\+\([0-9]\+\).*/\1/p' < lz4.h`
LIBVER_MINOR=`sed -n '/LZ4_VERSION_MINOR/s/.*\s\+\([0-9]\+\).*/\1/p' < lz4.h`
LIBVER_PATCH=`sed -n '/LZ4_VERSION_RELEASE/s/.*\s\+\([0-9]\+\).*/\1/p' < lz4.h`

with the following one

LIBVER_MAJOR=`sed -n '/define LZ4_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h`
LIBVER_MINOR=`sed -n '/define LZ4_VERSION_MINOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h`
LIBVER_PATCH=`sed -n '/define LZ4_VERSION_RELEASE/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h`

Error report

Yue Du reports the following problem on Mac OS X

• sed regex not working under Mac OS

The report is true. This error is caused by sed script which recently introduced in [9f42b09].

Problem (1) : Match multiple lines

# command line
sed -n '/LZ4_VERSION_MAJOR/s/\(.*\)/\1/p' < lz4.h

# output
#define LZ4_VERSION_MAJOR    1    /* for major interface/format changes  */
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE)

To prevent this multiple match, we should use more unique pattern.

# command line
sed -n '/define LZ4_VERSION_MAJOR/s/\(.*\)/\1/p' < lz4.h

# output
#define LZ4_VERSION_MAJOR    1    /* for major interface/format changes  */

Problem (2) : "Extended Regular Expression" is not supported

With "Basic Regular Expression" (BRE), we can not use \+. \+ is "Extended Regular Expression" (ERE).

Recent GNU sed and BSD sed allow ERE by command line switch -r, but Mac OS X's default sed is not support -r option.

Also \s is not supported by the standard regular expression.

So we should stick with BRE.

• Replace \s with [[:blank:]]
• Replace \+ with *

# command line
sed -n '/define LZ4_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < lz4.h

# output
1

Possible solution : Use Perl

Another possible solution is using the Perl. It is widely ported and its behaviour is consitent. But Another problem may occur by the extra dependency to non-standard command.

(1) In streaming branch, MSVC 2012 and 2013 show the following errors :

lz4.c(658): error C2054: expected '(' to follow 'inline'
lz4.c(822): error C2054: expected '(' to follow 'inline'

(2) Other branches have FORCE_INLINE macro, but it is omitted in stream branch. Maybe for simplification.

(3) It seems to me we have 2 options to solve this problem.

• (A) : Define inline macro for MSVC.
• (B) : Redefine FORCE_INLINE macro and use it.

But since inline is keyword in C99 and C++98, I think using FORCE_INLINE is verifed and better solution.

I was just looking at the fastMode branch, and I noticed that LZ4_compress_fast takes a maxOutputSize parameter, unlike compress and compressHC, and that there is no LZ4_compress_fast_limitedOutput.

I don't really mind there being no unsafe version, but you might want to consider renaming compress_fast to compress_fast_limitedOutput for two reasons: (1) consistency with existing APIs, and (2) gives you room to add an unsafe compress_fast in the future without weird naming (although LZ4_compress_faster would be funny…)

Please remove sudo from Makefile as this add an additional useless build dependency.
Morever, installing something on any OS always require admin access, so, this does not
improve anything (from installation stand point). Quite the contrary, I had to filter out sudo
from the Makefile(s) in order to successfully install from a snadbox environment (because
sudo was implying un-authorized filesystem accesses.)

Disclaimer : I'm the author of density

I know benchmarks are always a subject for debate, that's why I'm not really bothered when results vary from platform to platform, compiler to compiler or even between file types, but in this case I honestly think that your benchmark is grossly misleading, and for at least 2 reasons :

It is incomplete
What I don't understand is that you use fsbench (without providing the source code of the lz4 library version you're using) which includes much more codecs. Where are LZF, wfLZ, and density for example ? It's nice to add zlib as a reference but is it really useful ? That's not really what lz4 competes against.
If you want an up-to-date version of fsbench with source code of the codecs I maintain one here.

Also, you only benchmark the file silesia, and you omit text files like enwik8 or database files etc. In my case (density) people were concerned that I benchmarked against enwik8 only so I added silesia.

I know you're aware of the fantastic benchmark by @nemequ which can be seen here, it's probably the best there is today (lots of platforms, files, codecs)... why is there no link to it on your project's page ?

All of this gives the impression that lz4 is in a class of its own when one considers high-speed compression, but that is clearly not the case if you actually compare it to libraries designed with the same purpose in mind.

It is biased
That's actually a direct consequence of the previous problem. I suspect for example that lz4 was optimized against silesia, there's nothing wrong with that as long as you also benchmark it against something for which it wasn't optimized (enwik8 ?). Simple use case : when you run "lz4 fast 17" with enwik8 the compression ratio is terrible.

Another example : I think (maybe I'm mistaken) that you use lz4_decompress_fast in your tests, which is inherently unsafe whereas other libraries' decompress functions are safe... it's not really fair to compare apples with oranges.

Well that's it, enough ranting for me 😉 , but it really is disappointing to see this kind of things on such a classy project like lz4.

@Cyan4973 First off, great work on this library. We are getting good results using it.

I'm testing the latest HC streaming API on the streamHC branch, but decompression results are invalid. Couple of questions:

1. I understand the streamHC branch is still under development, that said is the new HC streaming API as it is implemented with latest source supposed to provide correct compression/decompression results at this stage? I'm not getting any exceptions or anything, the decompressed data just does not match the original data.
2. There is no LZ4_decompressHC_safe_continue or any other HC decompress API functions. I'm assumption that we should use the standard LZ4_decompress_safe_continue API for decompression?
3. My standard lz4 streaming code is working fine. Is my understanding correct that from the compression code, the only difference between the standard lz4 and lz4hc streaming API usage is changing to LZ4_streamHC_t and LZ4_compressHC_continue ? Or is there something else I need to implement to use lz4hc ?

Hello,

I recently made lz4.vim, just a simple vim plugin to read and write lz4frame files. It shows the original content of a compressed file, and enables you to modify the content.

If there are vim users, can you try it and give some feedback to me?
Here is the repository.

https://github.com/funcodeio/lz4.vim

Thank you!

If I attempt to compress an empty buffer (e.g., by using the lz4 example command-line utility on an empty file), getting the frame info fails upon decompression. However, decompressing without using LZ4F_getFrameInfo appears to work as expected.

LZ4F_preferences_t prefs;
LZ4F_decompressionContext_t dCtx = NULL;
LZ4F_compressionContext_t cctx = NULL;
LZ4F_frameInfo_t frame_info;

memset(&prefs, 0, sizeof(prefs));
memset(&frame_info, 0, sizeof(frame_info));
prefs.frameInfo.blockMode = LZ4F_blockIndependent;
prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled;

char buf[200000];
size_t errorCode;
errorCode = LZ4F_createCompressionContext(&cctx, LZ4F_VERSION);
if (LZ4F_isError(errorCode)) fprintf(stderr, "failed: %s\n", LZ4F_getErrorName(errorCode));
size_t written = 0;
errorCode = LZ4F_compressBegin(cctx, buf, 200000, &prefs);
if (LZ4F_isError(errorCode)) fprintf(stderr, "failed: %s\n", LZ4F_getErrorName(errorCode));
written += errorCode;
errorCode = LZ4F_compressEnd(cctx, buf + written, 200000 - written, NULL);
if (LZ4F_isError(errorCode)) fprintf(stderr, "failed: %s\n", LZ4F_getErrorName(errorCode));
errorCode = LZ4F_freeCompressionContext(cctx);
if (LZ4F_isError(errorCode)) fprintf(stderr, "failed: %s\n", LZ4F_getErrorName(errorCode));

errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION);
if (LZ4F_isError(errorCode)) fprintf(stderr, "failed: %s\n", LZ4F_getErrorName(errorCode));
size_t avail_in = written;
result = LZ4F_getFrameInfo(dCtx, &frame_info, static_cast<const void*>(buf), &avail_in);
if (LZ4F_isError(errorCode)) fprintf(stderr, "failed: %s\n", LZ4F_getErrorName(errorCode));

The above code snippet fails with:

failed: ERROR_frameHeader_incomplete

Apparently you're aware of this (based on the comment preceding the LZ4_wildCopy function), but during decompression (at least) LZ4 will attempt to access memory outside of the provided buffer. This causes crashes when doing things like decompressing to a memory-mapped file:

==20225== Process terminating with default action of signal 11 (SIGSEGV)
==20225==  General Protection Fault
==20225==    at 0x5E1D670: LZ4_copy8 (lz4.c:275)
==20225==    by 0x5E1D670: LZ4_wildCopy (lz4.c:291)
==20225==    by 0x5E1D670: LZ4_decompress_generic (lz4.c:1105)
==20225==    by 0x5E1D670: LZ4_decompress_safe (lz4.c:1123)
==20225==    by 0x5E18610: squash_lz4_decompress_buffer (squash-lz4.c:122)
==20225==    by 0x4E38BE6: squash_codec_decompress_with_options (codec.c:728)
==20225==    by 0x4E393CD: squash_codec_process_file_with_options (codec.c:1040)
==20225==    by 0x4E3974A: squash_codec_decompress_file_with_options (codec.c:1168)
==20225==    by 0x401719: benchmark_codec_with_options (benchmark.c:138)
==20225==    by 0x401BB0: benchmark_codec (benchmark.c:240)
==20225==    by 0x401E93: main (benchmark.c:324)

If you want I could probably put together a test case, but I doubt it's necessary. That backtrace is from a copy of enwik8 which was compressed with LZ4_compress_limitedOutput.

Our static analyzer (Coverity) is complaining about the following line in lz4frame.c:

 value64 += (srcPtr[3]<<24);

saying that the compiler will promote the rhs to a signed 32-bit integer with potential sign extension. Should probably cast this with something like:

 value64 += ((U64)srcPtr[3]<<24);

I use lz4 compression to improve speed of image loading. I was trying to migrate to new version of implementation and found compression performance fall down.
I put test case here:
https://drive.google.com/file/d/0B1PbczW6QMAyUWVpMGI5WTJnQW8/view?usp=sharing

/* LZ4F_compressBound() : gives the size of Dst buffer given a srcSize to handle worst case situations.

• The LZ4F_frameInfo_t structure is optional :
• you can provide NULL as argument, all preferences will then be set to default.
• /
  size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t preferencesPtr)
  {
  LZ4F_frameInfo_t* frameInfoPtr = (LZ4F_frameInfo_t_)preferencesPtr; /_ works because prefs starts with frameInfo /
  blockSizeID_t bid = (frameInfoPtr==NULL) ? LZ4F_BLOCKSIZEID_DEFAULT : frameInfoPtr->blockSizeID;
  size_t blockSize = LZ4F_getBlockSize(bid);
  unsigned nbBlocks = (unsigned)(srcSize / blockSize) + 1;
  size_t lastBlockSize = preferencesPtr->autoFlush ? srcSize % blockSize : blockSize;
  size_t blockInfo = 4; / default, without block CRC option _/
  size_t frameEnd = 4 + (frameInfoPtr->contentChecksumFlag_4);
  size_t result = (blockInfo * nbBlocks) + (blockSize * (nbBlocks-1)) + lastBlockSize + frameEnd;
  return result;
  }

size_t lastBlockSize = preferencesPtr->autoFlush ? srcSize % blockSize : blockSize;
if preferencesPtr is nullptr, fun will abort.

I've built lz4 executable for Android with NDK like this according to the Official Guide

$ /path/to/ndk/build/tools/make-standalone-toolchain.sh \
--arch=arm --platform=android-21 --install-dir=/tmp/my-android-toolchain
$ export PATH=/tmp/my-android-toolchain/bin:$PATH
$ export CC=arm-linux-androideabi-gcc 
$ make

Is it proper to add this in Makefile or provide a script about this?

Hello,
I found two minor errors:
lz4.c line 828:
//printf("offset %5i, matchLength%5i \n", (int)(ip-match), matchLength + MINMATCH);
and lz4hc.c line 106:
//#define DELTANEXTU16(p) chainTable[(p) & MAXD_MASK] /* flexible, MAXD dependent */
contain c++-style comments.
This fails to compile with Intel C++ Compiler 15.0.3 on Linux.
Thanks for your excellent work!

Hi,

It seems to me that in the current implementation, once a block is compressed, then it is written to the output, using the same thread. Could it be possible to add support to use two separate threads here: one does compression, and the other does disk writes? It will help overlap disk writes with compression and likely speedup the whole process. Here are some measurements I have taken.

input file is 3.3GB.

compression time (input and output are stored in mem): 15 seconds
$ lz4 /dev/shm/input /dev/shm/input.lz4

copy time (disk write): 12 seconds
$ cp /dev/shm/input.lz4 /dev/sda4/

compression + disk write time: 26 seconds
$ lz4 /dev/shm/input /dev/sda4/input.lz4

As we can see, there is almost no overlap in compression and disk write time (15+12 ~= 26). If we have two separate threads, one does compression and the other does disk writes, I think we can overlap these two tasks and get the end-to-end time (the 3rd command) reduced to ~15 seconds. The problem with current implementation is when we call a fwrite() to write a compressed block, it is likely cached in the page cache. The actual disk writes will happen only when we close the output stream. Thanks,

-Xing

When including compressed data in code; for instance when writing firmware for microcontrollers, it's a bit tedious to remove the initial (often 11) header bytes.
One really need to write a tool to make sure the correct number of bytes are removed.
Instead I'd like to request a command-line switch on the lz4 utility, which allows to write the raw lz4 data without checksum, starting from the first token.
The reason is that I've just finished writing a decompressor for ARM Cortex-M/Cortex-A (but it can be used by earlier ARM architectures as well) and made it public here for everyone to use:

http://community.arm.com/docs/DOC-10570

-Thus I think there might soon be a lot of people needing only the raw data.

Motivation

Need simple and safe version checking especially for dynamic link library.

Existing Implementation

• sqlite3_libversion()
• zlibVersion()

Possible Implementation (1) : Integer version number

// lz4.h
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR * 1000 * 1000 + LZ4_VERSION_MINOR * 1000 + LZ4_VERSION_RELEASE)

int LZ4_versionNumber (void);

// lz4.c
int LZ4_versionNumber (void)
{
    return LZ4_VERSION_NUMBER;
}

Possible Implementation (2) : Version string

// lz4.h
#define LZ4_STR_HELPER(x)  #x
#define LZ4_STR(x)         LZ4_STR_HELPER(x)
#define LZ4_VERSION_STRING LZ4_STR(LZ4_VERSION_MAJOR) "." LZ4_STR(LZ4_VERSION_MINOR) "." LZ4_STR(LZ4_VERSION_RELEASE)

const char* LZ4_versionString (void);

// lz4.c
const char* LZ4_versionString (void)
{
    return LZ4_VERSION_STRING;
}

Usage Example

void my_lz4_init()
{
    assert(LZ4_versionNumber() == LZ4_VERSION_NUMBER);
    assert(strcmp(LZ4_versionString(), LZ4_VERSION_STRING) == 0);

    // or
    if(LZ4_versionNumber() != LZ4_VERSION_NUMBER)
    {
        // ... show error ...
    }
}

Hi,

Please tag your project once you release a new version.

Thanks.

LZ4_decompress_safe returns

the number of bytes decompressed into destination buffer (necessarily <= maxDecompressedSize)
If destination buffer is not large enough, decoding will stop and output an error code (<0).
If the source stream is detected malformed, the function will stop decoding and return a negative result.
This function is protected against buffer overflow exploits, including malicious data packets.

I would like to be able to distinguish between the different errors. At least I would like to know if the input was malformed, incomplete, or the output buffer was just too small.

AFAICT the error code isn't really documented anywhere, only that <0 is an error. Maybe you could just return -1, -2, or -3 (or whatever) depending on the type of error…?