frommi / miniz_oxide Goto Github PK

View Code? Open in Web Editor NEW

167.0 10.0 48.0 1.21 MB

Rust replacement for miniz

License: MIT License

Shell 0.32% C 67.54% Rust 27.28% C++ 4.86%

miniz_oxide's Introduction

Main library

C API

miniz_oxide

Pure rust replacement for the miniz deflate/zlib encoder/decoder using no unsafe code. Builds in no_std mode, though requires the use of alloc and collections crates.

This project is organized into a C API shell and a rust crate. The Rust crate is found in the miniz_oxide subdirectory.

miniz_oxide 0.7.x requires at least Rust 1.50.0 Older versions 0.5.x and 0.6.x require at least rust 1.40.0, 0.3.x requires at least rust 0.36.0.

For a friendlier streaming API using readers and writers, flate2 can be used, which can use miniz_oxide as a rust-only back-end.

miniz_oxide_C_API

The C API is intended to replicate the API exported from miniz, and in turn also part of zlib. The C header is generated using cbindgen. The current implementation has not seen a lot of testing outside of automated tests, is a bit weak in documentation and should be seen as experimental.

The data structures do not share the exact same layout that is specified in miniz.h (from the original miniz), and should thus be allocated via the included functions.

API documentation

TODO

Testing

$ cargo test
$ ./test.sh

Benches

$ cargo bench --features=benching

or to compare to miniz

$ ./travis-after-success.sh

Including in C/C++ projects

Link against the libminiz_oxide_c_api.a generated by build.sh. The generated header that can be used is miniz.h (using the original miniz headers may or may not work), which currently also uses miniz_extra_defs.h for some static definitions.

Cargo-fuzz testing

Install fuzzer:

$ cargo install cargo-fuzz

Run fuzzer:

$ ./run_fuzz.sh

License

This library (excluding the miniz C code used for tests) is licensed under the MIT license. The library is based on the miniz C library, of which the parts used are dual-licensed under the MIT license and also the unlicense. The parts of miniz that are not covered by the unlicense is some Zip64 code which is only MIT licensed. This and other Zip functionality in miniz is not part of the miniz_oxidde and miniz_oxide_c_api rust libraries.

miniz_oxide's People

Contributors

Stargazers

Watchers

miniz_oxide's Issues

Evaluate unsafe code in encoder

There are still a number of uses of unsafe in the encoder code, more specifically in the match finding code to avoid bounds checks inside tight loops. While we've looked over them before, it would be a good idea to check them again and try to get rid of as many as we can if they have no or negligible effect on performance. In many of the unsafe calls the compiler may already be able to optimize them away anyhow.

The only unsafe code in the decoder are simple bytes to integer conversions, which should be sound.

Once 1.34 is reliably available in distros, we can switch to using the built-in functions for converting from bytes to integers.

Consider adding a changelog and using git tags for releases

This would help those who scrutinize changes before upgrading their dependencies.
No need to backfill the changelog with the past history: a changelog that starts now on is still way better than nothing.

I can add a changelog template if you like.

Port CVE and other tests from zlib-ng

zlib-ng has a bunch of tests to check for among other things previous zlib vulnerabilities, and to provide good coverage of the library.

It would be useful to do port these tests to check that miniz_oxide won't panic on these specific inputs.

As they are zlib-licensed we may want to put the tests in separate files if possible and maybe exclude them from the crate upload, not sure how zlib and mit licenses interact (though the tests won't be a part of the final library obviously).

'attempt to add with overflow' during deflate.

Found as part of the Rust compiler build.
miniz version - miniz_oxide-0.3.5
Rust version tag : release v1.41.1
Channel : nightly
config.toml

[llvm]
ninja = true
link-jobs = 1
version-suffix = "-rust"
release-debuginfo = true
use-libcxx = true
use-linker = "lld"

[rust]
optimize = true
debug = true
debuginfo-level = 2
debuginfo-level-rustc = 2
debuginfo-level-std = 2
debuginfo-level-tools = 2
channel = "nightly"

[build]
extended = true
verbose = 1

during x.py install

Dist rustc stage2 (x86_64-unknown-linux-gnu)
running: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
thread '<unnamed>' panicked at 'attempt to add with overflow', /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:1669:31
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1028
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:188
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:205
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:464
  11: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:373
  12: rust_begin_unwind
             at src/libstd/panicking.rs:302
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:139
  14: core::panicking::panic
             at src/libcore/panicking.rs:70
  15: miniz_oxide::deflate::core::compress_inner
           at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:0
  16: miniz_oxide::deflate::core::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:2070
  17: miniz_oxide::deflate::stream::deflate
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/stream.rs:52
  18: <flate2::ffi::rust::Deflate as flate2::ffi::DeflateBackend>::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/ffi/rust.rs:149
  19: flate2::mem::Compress::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:297
  20: flate2::mem::Compress::compress_vec
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:322
  21: <flate2::mem::Compress as flate2::zio::Ops>::run_vec
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:54
  22: flate2::zio::Writer<W,D>::write_with_status
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:217
  23: <flate2::zio::Writer<W,D> as std::io::Write>::write
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:256
  24: <flate2::gz::write::GzEncoder<W> as std::io::Write>::write
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/gz/write.rs:149
  25: std::io::Write::write_all
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/io/mod.rs:1341
  26: <installer::tarballer::RayonTee<A,B> as std::io::Write>::write_all::{{closure}}
             at src/tools/rust-installer/src/tarballer.rs:175
  27: rayon_core::join::join::call::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:103
  28: rayon_core::join::join_context::call_b::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:130
  29: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::call::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:113
  30: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:317
  31: std::panicking::try::do_call
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:287
  32: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  33: std::panicking::try
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
  34: std::panic::catch_unwind
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
  35: rayon_core::unwind::halt_unwinding
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/unwind.rs:17
  36: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:119
  37: rayon_core::job::JobRef::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:59
  38: rayon_core::registry::WorkerThread::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:735
  39: rayon_core::registry::WorkerThread::wait_until_cold
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:719
  40: rayon_core::registry::WorkerThread::wait_until
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:693
  41: rayon_core::registry::main_loop
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:813
  42: rayon_core::registry::ThreadBuilder::run
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:58
  43: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:103


command did not execute successfully: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
expected success, got: signal: 4

solana-labs/solana#5411 looks similar.

Bug in decompression.

Cargo built with miniz_oxide panics while unpacking clfft v0.3.1 package from crates.io. Found this bug using brand new tester.

Clarifications about streaming support

Hi, I'm developing a streaming .ZIP library that uses miniz_oxide for decompressing DEFLATE. Streaming is hard because you get the data in smaller chunks and want to decompress incrementally. Because of some peculiarities of the .ZIP format, you don't sometimes even know the size of the data beforehands. Here's my library (WIP): https://github.com/golddranks/stream_zipper

I've managed to get decompression to work when parsing files as one chunk. However, incremental decompression is giving me trouble. I'd like to ask some questions which I'd hope get eventually clarified in documentation:

What's the difference between FailedCannotMakeProgress and NeedsMoreInput states? If I hand miniz_oxide a partial buffer (try git cloning my repo and running cargo test test_partial_decompression -- --nocapture and see the debug printing; it gives 400 bytes instead of the full buffer), I get FailedCannotMakeProgress.
If I want to continue incrementally, what parameters am I supposed to pass? At the moment, I pass the rest of the input buffer (bytes after 400 EDIT: After making sure that it has indeed consumed 400 bytes up to that point!) and make a fresh output buffer, plus reuse the same DecompressorOxide struct, but it returns a Failure. If I pass the input buffer in one go, it manages to decompress the thing. (Edit: Is it already in a failed state after FailedCannotMakeProgress so that's why it returns Failure?)
If I understand right, DEFLATE stream format has a marker when the stream ends, so knowing the size beforehands isn't required. Is this correct?

Use smallvec lib to encapsulate work with uninitialized memory.

Right now in decompression we may use uninitialized memory without compiler-checked guarantees after with_init_state_only, addressing arrays C-style. More rust-y way would be to encapsulate these unsafe operations in another library: smallvec. Implementation should be pretty straightforward.

In compression miniz_oxide always 0-initializes all buffers, so small tests are slow compared to miniz (which has valgrind warnings here). I am not sure how hard this part would be. Maybe we would be forced to split HashBuffers's dict buffer.

Publish miniz_oxide on crates.io

Now that the C API stuff is split off, we should prepare and publish the rust crate to crates.io. Possibly also the C API.

Minimize unsafe usage in C API

There's still some unsafe usage in the C API glue that could be avoided, such as raw memory allocation.

Eventually we want to fully fix #16, but the innter C glue code may or may not be used by that.

Crashes 'index 15536 out of range for slice of length 15533'

  22: std::panic::catch_unwind
             at /checkout/src/libstd/panic.rs:365
  23: mz_inflate
             at /home/sanmai/.cargo/bin/registry/src/github.com-1ecc6299db9ec823/miniz_oxide_c_api-0.1.1/src/lib.rs:104
  24: flate2::mem::Decompress::decompress
             at /home/sanmai/.cargo/bin/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.1/src/mem.rs:368
  25: <flate2::mem::Decompress as flate2::zio::Ops>::run

Wasm: remove libc dependency

Hi,

Currently, miniz_oxide is not buildable with wasm32-unknown-unknown as target.
I would like to help to remove libc to be able to use flate2 inside a browser.
Unfortunately I have no knowledge in miniz/zlib api.

Do you think it's doable? Can you give me guidance to do it?
From what I understand it's highly related to #3 .

Thanks.

Performance

@oyvindln, can you please give a link for your performance tests?

adler32(crc) vectorization. I rewrote the shortest version.
Also, when rewriting I skipped on copying fields from compressor/decompressor to stack as it is in miniz. Check if it matters.
In find_match check if bit magic with u64 (instead of u16) can boost speed by finding first different spot in xor using leading zeros count.

Produces incorrect output on big-endian systems.

At least decompression of raw blocks and compression doesn't work.
Tested using cross
E.g tests run fine on mips64el, but not mips64 deflate::test::compress_small and inflate::tinfl_oxide::test::raw_block are failing with incorrect output. Haven't tested the C API yet.

Working on fixing it.

0.3.7 on crates.io

It looks like crates.io only has 0.3.6. I would like 0.3.7 for f0b0e8f

Please could you publish 0.3.7

Rustification

In miniz deflation usually consists of one tdefl_init and multiple tdefl_compress. Both calls carry pointers/references to data which is alive for different time, but in C is stored in one struct.
So I divided the struct in two: CallbackOxide for one tdefl_compress call, responsible for input buffers and output buffers and the rest in CompressorOxide.
But CallbackOut is a enum: it can be a function or a buffer and function is supposed to live as long as CompressorOxide and is given at tdefl_init (or not). So, I need to extract this Option of callback function and place it into CallbackOut which is unwieldy.

I mostly preserved all functions, but divided huge "global-struct" tdefl_compressor from C into sub-structs by meaning, so functions now carry a bundle of these structs. It is better, but now I must carry a list.

There is a mess of error codes, especially in high-level functions: like so.

write in Cursor takes a slice and so sometimes it produces something like this:

while rle.rle_repeat_count != 0 {
    rle.rle_repeat_count -= 1;
    packed_code_sizes.write(&[rle.prev_code_size][..])?;
}

There is an option to make a vec of size rle.rle_repeat_count, but it will be on heap and probably slow.

Hottest function seems to be this one.

decompress_to_vec should have a max_size parameter

Guarding against decompression bombs should be easy.

LZOxide too large for Windows

On Windows, the following code produces a stack overflow:

extern crate miniz_oxide;

use miniz_oxide::deflate::core::CompressorOxide;

struct Foo {
    bar: CompressorOxide,
}

fn main() {
    let foo = Foo { bar: CompressorOxide::new(0), };
}

Specifically, I get this error: thread 'main' has overflowed its stack.
This does not happen in release mode, however.

I think the structure LZOxide may be too large, as the overflow happens in it's new() funtion.

Perhaps it is a good idea to allocate this data on the heap?

Wasm: stub the libc dependency in miniz_oxide_c_api

Hi!

I read through #26, and we could add a small hack to stub the left-over libc calls to also be able to compile miniz_oxide_c_api to Wasm. This is all following what was done to use libsodium in alexcrichton/wasm-sodium (see the libc-stub folder).

The wasm32 branch in wehlutyk/miniz_oxide shows that, and it makes compiling flate2-rs to wasm32 possible (see wehlutyk/flate2-rs/tree/wasm32). I use it to read compressed pdf files using lopdf, all in the browser, and up to now it works like a charm.

Would you accept a PR for this? It would be great as we could then merge the changes for flate2-rs and lopdf and have them work on wasm32. But seeing as it's a bit of hack I thought you might want to wait till libc gets implementations for wasm32. Then again it's a useful shim in the meantime :)

Travis miniz_tester can't see linux files.

https://travis-ci.org/Frommi/miniz_oxide/builds/267227423
It seems that everything builds properly, all the folders contain right files, but

../bin/miniz_tester -v a linux-4.8.11

doesn't find the files:

pwd output:

/home/travis/build/Frommi/miniz_oxide/test_scratch

so we are in the right folder.

ls output:

linux-4.8.11  linux-4.8.11.tar.xz

ls linux-4.8.11 output:

arch	 crypto		include  kernel       net	      security
block	 Documentation	init	 lib	      README	      sound
certs	 drivers	ipc	 MAINTAINERS  REPORTING-BUGS  tools
COPYING  firmware	Kbuild	 Makefile     samples	      usr
CREDITS  fs		Kconfig  mm	      scripts	      virt

so linux is downloaded and extracted properly.

ls ../bin output:

example1  example2  example3  example4	example5  example6  miniz_tester

so miniz_tester is compiled and then it is executed ../bin/miniz_tester -v a linux-4.8.11, but can't detect files:

Total files processed: 0

inflate::core::init_tree() is slow

When opening small-ish images with png crate more than 50% of time is spent in inflate::core::init_tree function. This makes png crate 3x slower than spng according to spng crate benchmarks. Using larger images closes the gap.

Sample image demonstrating the issue (25Kb): https://github.com/aloucks/spng-rs/blob/master/spng/tests/test-002.png

Profile: https://profiler.firefox.com/public/d87cd7f7fde83f8b4ddd2ee3e5d0b4f20312ee42/flame-graph/?globalTrackOrder=0-1-2-3-4&localTrackOrderByPid=27730-0-1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-21-22-23-24-25-26-27-28-29-30-31-32-33-34-35-36-37-38-39-40~27839-0~27840-0-1-2-3-4~27841-0~27847-0-1-2-3-4~&thread=42&timelineType=stack&v=4

ARM testing

There is a place that probably will break on not x86 arch. As well as performance issues with read/write_unaligned.

decompress should return HasMoreOutput if output is full when input is empty.

See #68

include LICENSE to sub-crates

Archives of miniz_oxide (and probably others) on crates.io do not include LICENSE which is against MIT license terms. Please include it.

Use Rust API directly from flate2

An interesting problem was reported at https://gitter.im/rust-lang/rust?at=5a57cc7183152df26d5d5cfc.

Basically, miniz-sys and miniz_oxide_c_api expose the same C-API, so you can't link them both into the same process. So it might make sense to bypass C-API in flate2 and avoid polluting linker's namespace.

Add CI for 32-bit and big endian

As there is a little bit of endinaness and word-size dependent code, there should be CI tests for this to avoid any breakage.

Panic when passing a zero-sized output buffer.

miniz_oxide::inflate::core::decompress will panic if a output buffer of size 0 is passed in. The panic is due to an overflow in core.rs, on line 1020: out_cur.get_ref().len() - 1.

It says that "This function shouldn't panic pending any bugs." in the documentation so I presume this is a bug.

Zip broken for 0.3.4?

I was looking into a build failure on CI and noticed that a build command is panicking when trying to extract a zip archive. It seems to be failing at this assertion.

Looking at the diff that introduced this change, I found the following comment:

2a5703a#diff-1b531c5e405e0bc95890d52cf2f2f5b5R44

zip is broken right now due to struct difference between c and rust version

That makes it sound like ZIP is not supported in the latest release. Is this correct? In a non-breaking release this is likely to break a lot of things downstream, which will only be noticed at runtime.

Downgrading to 0.3.3 fixes this problem.

More than one person should be able to push releases to crates.io

ping @Frommi @matklad

adler32 check in fuzz target interferes with fuzzing

The fuzz target currently invokes miniz_oxide like this:

miniz_oxide/fuzz/fuzz_targets/inflate_nonwrapping.rs

Line 7 in 01a5712

let _ = miniz_oxide::inflate::decompress_to_vec(data);

It does not seem to disable alder32 checks. Because of checksum verification fuzzers cannot reach the actually interesting decoding code.

Rust fuzzer integration provides special configuration option fuzzing which is set when a binary is built with fuzzing instrumentation; you can use it to disable checksums via conditional compilation (example from lodepng-rust). Alternatively, you can adapt your fuzzing target to set the required configuration parameters to disable checksum verification (inflate crate provides a *_no_checksum function).

cc @oyvindln who has authored the fuzz target.

I can also provide you with an initial fuzzing corpus to kickstart fuzzing, both with and without zlib headers, obtained by fuzzing inflate crate with afl-fuzz, libfuzzer and honggfuzz. If you're interested, let me know and I'll attach it here.

honggfuzz-rs provides further documentation, including enabling sanitizers to catch bugs in code under unsafe. Or ping me or https://github.com/rust-fuzz if you have any questions regarding fuzzing.

Generate miniz.h with cbindgen for C export automatically

The internal compressor/decompressor structs do not match the layout of the C versions, so we want to generate headers with the correct layout rather than using the C headers for the original library. The C zip tests/functions segfault at the moment as the decompressor structure is put on the stack, and the init function in C is just a macro that zeroes a value.

CBindgen currently doesn't handle function pointers and boxes inside options (Whcih can be exported to c as pointers). Need to either be fixed upstream, or we will want to use raw pointers here which means a bit more unsafety.

Some Panics cause segfaults in jemalloc when called from flate2

Ideally we should never panic of course, but there could still be bugs, and if there are, it shouldn't end up in a segfault.
See this
I've tried with removing no_mangle and extern_c from mz_inflate, so that doesn't seem to be the issue.

Zlib generated does not include proper compression header

The generated fine does not include the referred compression level.

It should looks like:

Level | ZLIB  | GZIP 
  1   | 78 01 | 1F 8B 
  2   | 78 5E | 1F 8B 
  3   | 78 5E | 1F 8B 
  4   | 78 5E | 1F 8B 
  5   | 78 5E | 1F 8B 
  6   | 78 9C | 1F 8B 
  7   | 78 DA | 1F 8B 
  8   | 78 DA | 1F 8B 
  9   | 78 DA | 1F 8B

However, the second byte is always 0x01. This happens using the default feature. Using zlib backend it works as expected.

Originally reported at: rust-lang/flate2-rs#225

Plans for C API

Would it be okay with some slight differences in output from the compressor compared to miniz?
E.g I think it would be interesting to attempt implementing this matching algorithm as it looks like it could give significant performance improvements at higher compression levels.

There are also some minor compression level improvements that we could make, miniz doesn't do matches with the first byte, and does a very simple block type selection that doesn't always produce great results on very short input. (As opposed to zlib.)

Should we keep the compressor/decompressor structs exactly the same, or is it enough that inflate/deflateinit etc behave the same way?

I also propose we separate the C api part(the extern c exports) as a separate wrapper crate to avoid possible symbol conflicts when using it as a rust library and don't have to create the c static lib when we don't need it.

Support a foolproof(er) higher-level API for streaming

Hi, I have wrapped the streaming (ring-buffer using) API of miniz_oxide behind a higher level API here: https://github.com/golddranks/stream_zipper/blob/master/src/deflate.rs

Is there any interest in supporting such a higher level API within miniz_oxide itself?

I've experimented with a state machine-like API:

pub enum State<'a> {
    HasMoreOutput(ChunkIter<'a>),
    NeedsMoreInput(InputSink),
    Stop(FinishedStream<'a>),
}

The idea is that every function that operates on the state takes self as value, and as such, parsing the stream can proceed as a state machine alternating between states "HasMoreOutput" and "NeedsMoreInput". The states own the internal state, including the ring buffer, which makes this a higher level API as the one provided by miniz_oxide at the moment.

Here's a roughly how it would be used:

        let mut state = start_deflate_stream();

        for chunk in pure_deflate_stream.chunks(500) {
            if let State::NeedsMoreInput(sink) = state {
                state = sink.feed(chunk)?;
            };
            while let State::HasMoreOutput(out) = state {
                println!("Got {:?}", out.get());
                state = out.next()?;
            }
        }

Another experiment I did was to have just InputSink, and then passing a callback that gets the uncompressed output. This is a kind of an internal iterator. Again, here's an example

        let mut state = start_deflate_stream();

        for chunk in pure_deflate_stream.chunks(50) {
            if let State::NeedsMoreInput(sink) = state {
                state = sink.inner_iter(chunk, |out| {
                    println!("Got {:?}", out);
                })?;
            };
        }

What do you think about introducing such a high level API?

subtract-with-overflow or out-of-range panic decompressing invalid file

I've found and minified a file which causes the same symptoms as #13, but I don't think it's the same problem.

The file is pretty horribly invalid. Horribly, horribly invalid. Confuses the other tool I'm working on, too.

Test and data file:
FauxFaux@0f3b480

thread 'inf_issue_13' panicked at 'attempt to subtract with overflow', miniz_oxide/src/inflate/core.rs:958:25
...
  10: miniz_oxide::inflate::core::decompress_fast
             at miniz_oxide/src/inflate/core.rs:958
  11: miniz_oxide::inflate::core::decompress_inner
             at miniz_oxide/src/inflate/core.rs:1347
  12: miniz_oxide::inflate::core::decompress
             at miniz_oxide/src/inflate/core.rs:1003
  13: miniz_oxide::inflate::decompress_to_vec_inner
             at miniz_oxide/src/inflate/mod.rs:93
  14: miniz_oxide::inflate::decompress_to_vec
             at ./src/inflate/mod.rs:61
  15: test::inf_issue_13
             at miniz_oxide/tests/test.rs:20

`inflate::core::decompress` panics on input

The function seems to get confused when the vector is partially drained between calls. This contradicts the documentation which specifies it should never panic.

Reproduction

Execute the test render_images at this exact commit of image-png: https://github.com/image-rs/image-png/tree/cb9a340e6e1b9cd4dbcc1bb69368399d2354da77

I don't have more minimized test cases at the moment, sorry. It's getting attached as soon as one is found.

Note the manual streaming inflate implementation. It transfers 'fully decoded' data to a separate buffer provided by the caller, modifying the position of the constructed Cursor. According to documentation it leaves 32KB data in the intermediate buffer. Tracing shows that the second call to decompress panics already and skipping the call to transfer_finished_data also avoids it.

https://github.com/image-rs/image-png/blob/cb9a340e6e1b9cd4dbcc1bb69368399d2354da77/src/decoder/stream.rs#L681-L807

Expected behaviour

The PNG format is specifically designed for decoding a rough view of image in a small subset at the beginning of the image called pass extraction such as adam7 interlacing. The benefit is substantially reduced if the image data can only be decompressed as a whole.

It also puts the decoding party at risk of resource exhaustion since it may be impossible to progress without violating limits on buffer allocations.

Context: we would like to switch inflate implementation in image-png to reduce the amount of unsafe code in dependencies, better stability and performance.

Speed up adler32

When decoding lightly compressed data the majority of the time is spent in adler32::RollingAdler32::update_buffer, so even small speedups in it will translate to measurable speeupds in the entire decoding.

Profiling results from miniz-oxide branch of png crate to back this up: https://perfht.ml/2V3ml3q

Consider feature flagging the zlib bindings

It'd be nice to not include any bindings to zlib when unnecessary.

Unable to compile using miniz_oxide with 'use of unstable library' errors

I've been trying to compile a simple script that uses miniz_ozide and am running into a few errors when running cargo build. They mostly look like they're related to the use of unstable library features in deflate. Output below, with the path edited to $HOME.

   Compiling miniz_oxide v0.3.2                                                                                                                                                                                                    
   Compiling rand_chacha v0.1.1                                                                                                                                                                                                    
   Compiling rand_pcg v0.1.2                                                                                                                                                                                                       
   Compiling rand v0.6.5                                                                                                                                                                                                           
   Compiling rustc_version v0.2.3                                                                                                                                                                                                  
   Compiling proc-macro2 v0.4.30                                                                                                                                                                                                   
   Compiling rand_hc v0.1.0                                                                                                                                                                                                        
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)                                                                                                                                                        
 --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:3:5                                                                                                                          
  |                                                                                                                                                                                                                                
3 | use std::convert::TryInto;                                                                                                                                                                                                     
  |     ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                      
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)                                                                                                                                                        
 --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:6:5                                                                                                                          
  |                                                                                                                                                                                                                                
6 | use std::convert::TryInto;                                                                                                                                                                                                     
  |     ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                      
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)                                                                                                                                               
   --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:634:25                                                                                                                     
    |                                                                                                                                                                                                                              
634 |             let bytes = u64::to_le_bytes(self.bit_buffer);                                                                                                                                                                   
    |                         ^^^^^^^^^^^^^^^^                                                                                                                                                                                     
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)                                                                                                                                                        
    --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1124:52                                                                                                                   
     |                                                                                                                                                                                                                             
1124 |         let bytes: [u8; 4] = self.b.dict[pos..end].try_into().unwrap();                                                                                                                                                     
     |                                                    ^^^^^^^^                                                                                                                                                                 
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)                                                                                                                                               
    --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1125:9                                                                                                                    
     |                                                                                                                                                                                                                             
1125 |         u32::from_le_bytes(bytes)                                                                                                                                                                                           
     |         ^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                  
                                                                                                                                                                                                                                   
error[E0277]: the trait bound `[u8; 4]: std::convert::From<&[u8]>` is not satisfied                                                                                                                                                
    --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1124:52                                                                                                                   
     |                                                                                                                                                                                                                             
1124 |         let bytes: [u8; 4] = self.b.dict[pos..end].try_into().unwrap();                                                                                                                                                     
     |                                                    ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 4]`                                                                                          
     |                                                                                                                                                                                                                             
     = note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 4]`                                                                                                                      
     = note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 4]>` for `&[u8]`                                                                                                                      
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)                                                                                                                                                        
    --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1133:56                                                                                                                   
     |                                                                                                                                                                                                                             
1133 |         let bytes: [u8; 8] = self.b.dict[pos..pos + 8].try_into().unwrap();                                                                                                                                                 
     |                                                        ^^^^^^^^                                                                                                                                                             
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)                                                                                                                                               
    --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1134:9                                                                                                                    
     |                                                                                                                                                                                                                             
1134 |         u64::from_le_bytes(bytes)                                                                                                                                                                                           
     |         ^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                  
                                                                                                                                                                                                                                   
error[E0277]: the trait bound `[u8; 8]: std::convert::From<&[u8]>` is not satisfied                                                                                                                                                
    --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/deflate/core.rs:1133:56                                                                                                                   
     |                                                                                                                                                                                                                             
1133 |         let bytes: [u8; 8] = self.b.dict[pos..pos + 8].try_into().unwrap();                                                                                                                                                 
     |                                                        ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 8]`                                                                                      
     |                                                                                                                                                                                                                             
     = note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 8]`                                                                                                                      
     = note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 8]>` for `&[u8]`                                                                                                                      
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)                                                                                                                                                        
   --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:342:44                                                                                                                     
    |                                                                                                                                                                                                                              
342 |         let two_bytes = iter.as_ref()[..2].try_into().unwrap();                                                                                                                                                              
    |                                            ^^^^^^^^                                                                                                                                                                          
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)                                                                                                                                               
   --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:343:9                                                                                                                      
    |                                                                                                                                                                                                                              
343 |         u16::from_le_bytes(two_bytes)                                                                                                                                                                                        
    |         ^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                   
                                                                                                                                                                                                                                   
error[E0277]: the trait bound `[u8; 2]: std::convert::From<&[u8]>` is not satisfied                                                                                                                                                
   --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:342:44                                                                                                                     
    |                                                                                                                                                                                                                              
342 |         let two_bytes = iter.as_ref()[..2].try_into().unwrap();                                                                                                                                                              
    |                                            ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 2]`                                                                                                   
    |                                                                                                                                                                                                                              
    = note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 2]`                                                                                                                       
    = note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 2]>` for `&[u8]`                                                                                                                       
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'try_from' (see issue #33417)                                                                                                                                                        
   --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:357:54                                                                                                                     
    |                                                                                                                                                                                                                              
357 |         let four_bytes: [u8; 4] = iter.as_ref()[..4].try_into().unwrap();                                                                                                                                                    
    |                                                      ^^^^^^^^                                                                                                                                                                
                                                                                                                                                                                                                                   
error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)                                                                                                                                               
   --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:358:9                                                                                                                      
    |                                                                                                                                                                                                                              
358 |         u32::from_le_bytes(four_bytes)                                                                                                                                                                                       
    |         ^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                   
                                                                                                                                                                                                                                   
error[E0277]: the trait bound `[u8; 4]: std::convert::From<&[u8]>` is not satisfied                                                                                                                                                
   --> $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.2/src/inflate/core.rs:357:54                                                                                                                     
    |                                                                                                                                                                                                                              
357 |         let four_bytes: [u8; 4] = iter.as_ref()[..4].try_into().unwrap();                                                                                                                                                    
    |                                                      ^^^^^^^^ the trait `std::convert::From<&[u8]>` is not implemented for `[u8; 4]`                                                                                         
    |                                                                                                                                                                                                                              
    = note: required because of the requirements on the impl of `std::convert::TryFrom<&[u8]>` for `[u8; 4]`                                                                                                                       
    = note: required because of the requirements on the impl of `std::convert::TryInto<[u8; 4]>` for `&[u8]`                                                                                                                       
                                                                                                                                                                                                                                   
   Compiling rand_isaac v0.1.1                                                                                                                                                                                                     
error: aborting due to 15 previous errors                                                                                                                                                                                          
                                                                                                                                                                                                                                   
Some errors occurred: E0277, E0658.                                                                                                                                                                                                
For more information about an error, try `rustc --explain E0277`.                                                                                                                                                                  
error: Could not compile `miniz_oxide`.                                                                                                                                                                                            
warning: build failed, waiting for other jobs to finish...
error: build failed

Release a new version?

miniz_oxide deep in dependency tree makes my project uncompilable on WASM with stable Rust.

I wanted to make a PR, but noticed that it's actually already fixed on master >2 months ago, but wasn't released to crates.io.

Can you please publish a new release?

Make C API layout-compatible with miniz

The README mentions that exporting miniz-compatible headers is blocked on an upstream cbindgen issue. However, a workaround for that issue was devised: mozilla/cbindgen#326 (comment)

So it should be possible to export layout-compatible structures once again.

Expose crc32

Exposing mz_crc32_oxide() could be useful for writing a PNG encoder/decoder that uses miniz_oxide for zlib compression anyway.

I can see that it is part of miniz_oxide/src/lib.rs, but not of miniz_oxide/miniz_oxide/src/lib.rs. But I don't understand what the difference is, only that I could not access mz_crc32_oxide().

On a related note, how can you access the streaming decompression that is discussed in issue 24?

Add helpful error when trying to decompress gzip-wrapped data

When trying to decompress the following data using this Rust code:

use miniz_oxide::inflate::decompress_to_vec;

fn main() {
    let data = [31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 149, 147, 61, 75, 195, 80, 20, 134, 79, 62, 20, 84, 170, 1, 17, 28, 68, 28, 132, 110, 183, 247, 38, 185, 249, 154, 58, 10, 130, 116, 116, 170, 54, 109, 18, 11, 173, 169, 109, 90, 112, 210, 209, 81, 112, 112, 22, 127, 131, 56, 232, 232, 224, 143, 112, 112, 238, 159, 168, 55, 31, 234, 77, 91, 82, 26, 184, 57, 36, 79, 222, 115, 222, 251, 38, 9, 170, 27, 218, 209, 62, 128, 113, 76, 234, 181, 83, 82, 55, 31, 175, 223, 101, 0, 216, 249, 120, 217, 219, 102, 181, 244, 244, 186, 203, 10, 8, 108, 109, 18, 221, 70, 132, 234, 136, 152, 20, 81, 13, 222, 132, 148, 43, 11, 184, 144, 241, 178, 138, 49, 113, 176, 171, 90, 142, 175, 106, 45, 199, 79, 46, 217, 145, 63, 53, 126, 117, 241, 92, 49, 215, 215, 48, 17, 197, 185, 185, 179, 156, 252, 113, 49, 227, 91, 60, 39, 148, 240, 190, 196, 127, 95, 134, 217, 116, 176, 238, 89, 177, 47, 181, 200, 151, 180, 156, 206, 229, 247, 35, 229, 252, 176, 156, 8, 198, 252, 126, 138, 184, 144, 241, 57, 57, 106, 139, 114, 148, 167, 115, 178, 73, 46, 199, 34, 46, 100, 124, 206, 126, 245, 162, 185, 98, 166, 227, 242, 55, 16, 81, 49, 159, 227, 18, 125, 93, 222, 207, 202, 76, 14, 126, 172, 163, 69, 126, 148, 76, 87, 178, 9, 139, 213, 66, 148, 185, 177, 48, 0, 159, 211, 52, 167, 118, 198, 27, 189, 145, 134, 6, 145, 215, 65, 205, 176, 11, 240, 201, 158, 171, 150, 36, 104, 177, 90, 211, 37, 184, 99, 63, 11, 30, 2, 124, 63, 200, 73, 253, 98, 141, 206, 199, 233, 119, 18, 63, 11, 207, 34, 76, 38, 147, 155, 120, 193, 248, 48, 185, 23, 207, 186, 117, 214, 146, 26, 247, 57, 56, 1, 184, 63, 19, 18, 189, 82, 102, 51, 47, 162, 168, 55, 112, 42, 149, 8, 117, 189, 10, 123, 247, 65, 219, 95, 247, 195, 97, 127, 112, 53, 108, 244, 61, 144, 221, 246, 101, 192, 116, 171, 65, 24, 6, 29, 47, 13, 83, 73, 203, 15, 58, 186, 13, 141, 216, 3, 0, 0];
    assert!(decompress_to_vec(&data).is_ok());
}

I'm hitting the following error:

Err(TINFLStatus::Failed)

Python's gzip can decompress it just fine, though:

import gzip
data = [31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 149, 147, 61, 75, 195, 80, 20, 134, 79, 62, 20, 84, 170, 1, 17, 28, 68, 28, 132, 110, 183, 247, 38, 185, 249, 154, 58, 10, 130, 116, 116, 170, 54, 109, 18, 11, 173, 169, 109, 90, 112, 210, 209, 81, 112, 112, 22, 127, 131, 56, 232, 232, 224, 143, 112, 112, 238, 159, 168, 55, 31, 234, 77, 91, 82, 26, 184, 57, 36, 79, 222, 115, 222, 251, 38, 9, 170, 27, 218, 209, 62, 128, 113, 76, 234, 181, 83, 82, 55, 31, 175, 223, 101, 0, 216, 249, 120, 217, 219, 102, 181, 244, 244, 186, 203, 10, 8, 108, 109, 18, 221, 70, 132, 234, 136, 152, 20, 81, 13, 222, 132, 148, 43, 11, 184, 144, 241, 178, 138, 49, 113, 176, 171, 90, 142, 175, 106, 45, 199, 79, 46, 217, 145, 63, 53, 126, 117, 241, 92, 49, 215, 215, 48, 17, 197, 185, 185, 179, 156, 252, 113, 49, 227, 91, 60, 39, 148, 240, 190, 196, 127, 95, 134, 217, 116, 176, 238, 89, 177, 47, 181, 200, 151, 180, 156, 206, 229, 247, 35, 229, 252, 176, 156, 8, 198, 252, 126, 138, 184, 144, 241, 57, 57, 106, 139, 114, 148, 167, 115, 178, 73, 46, 199, 34, 46, 100, 124, 206, 126, 245, 162, 185, 98, 166, 227, 242, 55, 16, 81, 49, 159, 227, 18, 125, 93, 222, 207, 202, 76, 14, 126, 172, 163, 69, 126, 148, 76, 87, 178, 9, 139, 213, 66, 148, 185, 177, 48, 0, 159, 211, 52, 167, 118, 198, 27, 189, 145, 134, 6, 145, 215, 65, 205, 176, 11, 240, 201, 158, 171, 150, 36, 104, 177, 90, 211, 37, 184, 99, 63, 11, 30, 2, 124, 63, 200, 73, 253, 98, 141, 206, 199, 233, 119, 18, 63, 11, 207, 34, 76, 38, 147, 155, 120, 193, 248, 48, 185, 23, 207, 186, 117, 214, 146, 26, 247, 57, 56, 1, 184, 63, 19, 18, 189, 82, 102, 51, 47, 162, 168, 55, 112, 42, 149, 8, 117, 189, 10, 123, 247, 65, 219, 95, 247, 195, 97, 127, 112, 53, 108, 244, 61, 144, 221, 246, 101, 192, 116, 171, 65, 24, 6, 29, 47, 13, 83, 73, 203, 15, 58, 186, 13, 141, 216, 3, 0, 0]
assert gzip.decompress(bytes(data)) is not None

I presume this is the same as sile/libflate#45, which I'm linking because it provides a clearer error message which may be helpful:

btype 0x11 of DEFLATE is reserved(error) value

I'm not sure if this bug should be filled here, but I guess it can't hurt. Is there any way to make this library decompress the data successfully, or a way to work around it?

I have also tested flate2 as mentioned in the README, and that's able to decompress the data too.

Add fuzzing target for streaming API

Continuing from #29: inflate::decompress_to_vec() has its own fuzzing target and is covered pretty well. There should also be a fuzzing target for the streaming API.

tinfl_decompress

tinfl_decompress is implemented as a coroutine (state machine) and is a bit of a mess. tdefl is divided into functions so I was able to rewrite it function by function and check that nothing broke, but it won't work as straightforward for tinfl.
I propose to write a big match over enum of all the states and at first delegate them to C tinfl_decompress and rewrite them one by one in terms of match arms.

Infinite loop in deflate::compress_to_vec() on empty or one-byte slices

To reproduce, run the following program (with miniz_oxide 0.3.6):

fn main() {
    miniz_oxide::deflate::compress_to_vec(&[], 6);
}

Expected result: it should terminate instantly.
Actual result: it does not terminate.

Same for

fn main() {
    miniz_oxide::deflate::compress_to_vec(&[0], 6);
}

Releasing under the Zlib license?

Just wondering if you'd be interested in adding the Zlib license as a license option for this crate.

Mediocre deflate compression compared to Node.js's zlib

From rust-lang/flate2-rs#238

The pure rust implementation of deflate gives a somewhat unsatisfactory compression result, even using max compression level.

Somehow, the zlib binding from here is even worse, which is very strange since my benchmark is from Node.js's zlib, using zlib's default compression level.

Consider the following program - it shouldn't be too much of a stretch to wonder how this library isn't able to produce the same result as COMPRESSED at default, let alone max.

use miniz_oxide::deflate::compress_to_vec_zlib;
use miniz_oxide::deflate::compress_to_vec;

use miniz_oxide::inflate::decompress_to_vec;

const TEXT: &'static str = concat![
    "Chunk one\n",
    "data data\n",
    "\n",
    "Chunk two\n",
    "data data\n",
    "\n",
    "Chunk three\n",
    "data data\n",
];

// Generated in Node.js using it's zlib bindings.
// Zlib default compression level.
//
// Taken from my defunct project:
// https://github.com/Fishrock123/zlib-transform/blob/master/test/test-basic.js
const COMPRESSED: &'static [u8] = &[
    0x73, 0xce, 0x28, 0xcd, 0xcb, 0x56, 0xc8, 0xcf, 0x4b, 0xe5, 0x4a, 0x49, 0x2c, 0x49, 0x54, 0x00, 0x11, 0x5c, 0x5c, 0xce, 0x60, 0xc1, 0x92, 0xf2, 0x7c, 0x2c, 0x82, 0x19, 0x45, 0xa9, 0xc8, 0x6a, 0x01,
];

fn main() {
    let compressed = compress_to_vec(TEXT.as_bytes(), 10);
    println!("miniz_oxide : {:x?}", compressed);

    let compressed = compress_to_vec_zlib(TEXT.as_bytes(), 10);
    println!("zlib        : {:x?}", compressed);

    println!("node.js zlib: {:x?}", COMPRESSED);

    let text = decompress_to_vec(COMPRESSED).unwrap();
    assert_eq!(TEXT.as_bytes(), &text[..]);
    assert_eq!(TEXT, std::str::from_utf8(&text).unwrap());
}

Output

miniz_oxide : [6d, ca, b1, 9, 0, 30, 8, 5, d1, fe, 4f, e1, 2e, 4e, 22, 44, 10, 2, a, c1, 90, f5, 43, 52, 59, d8, 5c, f1, 38, b6, ed, 93, c2, 15, 43, 52, e8, 5, e0, 8f, 79, a2, 41, 5b, 5a, df, b]
zlib        : [78, da, 6d, ca, b1, 9, 0, 30, 8, 5, d1, fe, 4f, e1, 2e, 4e, 22, 44, 10, 2, a, c1, 90, f5, 43, 52, 59, d8, 5c, f1, 38, b6, ed, 93, c2, 15, 43, 52, e8, 5, e0, 8f, 79, a2, 41, 5b, 5a, df, b, ba, 21, 15, 4c]
node.js zlib: [73, ce, 28, cd, cb, 56, c8, cf, 4b, e5, 4a, 49, 2c, 49, 54, 0, 11, 5c, 5c, ce, 60, c1, 92, f2, 7c, 2c, 82, 19, 45, a9, c8, 6a, 1]

(This was noticed when building https://github.com/Fishrock123/tide-compress.)

0.2.4 broke the C API on 32-bit unix

On Fedora, we found that 0.2.4 broke for i686 and armv7hl, such as this build.

error[E0308]: mismatched types
   --> src/lib.rs:252:79
    |
252 |                 as_c_return_code(mz_compress2_oxide(&mut stream_oxide, level, dest_len))
    |                                                                               ^^^^^^^^ expected `u64`, found `u32`
    |
    = note: expected mutable reference `&mut u64`
               found mutable reference `&mut u32`
error[E0308]: mismatched types
   --> src/lib.rs:291:74
    |
291 |                 as_c_return_code(mz_uncompress2_oxide(&mut stream_oxide, dest_len))
    |                                                                          ^^^^^^^^ expected `u64`, found `u32`
    |
    = note: expected mutable reference `&mut u64`
               found mutable reference `&mut u32`
error[E0308]: mismatched types
   --> src/c_export.rs:150:23
    |
150 |             total_in: self.total_in,
    |                       ^^^^^^^^^^^^^ expected `u32`, found `u64`
error[E0308]: mismatched types
   --> src/c_export.rs:160:24
    |
160 |             total_out: self.total_out,
    |                        ^^^^^^^^^^^^^^ expected `u32`, found `u64`
error[E0308]: mismatched types
   --> src/c_export.rs:211:23
    |
211 |             total_in: stream.total_in,
    |                       ^^^^^^^^^^^^^^^ expected `u64`, found `u32`
error[E0308]: mismatched types
   --> src/c_export.rs:213:24
    |
213 |             total_out: stream.total_out,
    |                        ^^^^^^^^^^^^^^^^ expected `u64`, found `u32`
error: aborting due to 6 previous errors
For more information about this error, try `rustc --explain E0308`.
error: could not compile `miniz_oxide_c_api`.

I bisected the regression to commit 918aa73. For the mz_uncompress2_oxide errors, that parameter changed from &mut c_ulong to &mut u64. StreamOxide's total_in and total_out also changed from c_ulong to u64. Those are the same type on LP64 targets (most 64-bit Unix), but not on 32-bit targets, nor 64-bit Windows (LLP64).

roundtrip tests fail in published 0.3.5 crate

 Running `/var/tmp/portage/dev-rust/miniz_oxide-0.3.5/work/debug/deps/test-d07cae2084eaa2a5`

running 7 tests
test decompress_zero_code_len_2 ... ok
test decompress_zero_code_len_oom ... ok
test inf_issue_14 ... ok
test inf_issue_19 ... ok
test roundtrip_lvl_0 ... FAILED
test roundtrip_lvl_9 ... FAILED
test roundtrip_lvl_1 ... FAILED

failures:

---- roundtrip_lvl_0 stdout ----
thread 'roundtrip_lvl_0' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

---- roundtrip_lvl_9 stdout ----
thread 'roundtrip_lvl_9' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1084:5

---- roundtrip_lvl_1 stdout ----
thread 'roundtrip_lvl_1' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1084:5


failures:
    roundtrip_lvl_0
    roundtrip_lvl_1
    roundtrip_lvl_9

test result: FAILED. 4 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out

Full output with RUST_BACKTRACE=full

--- roundtrip_lvl_0 stdout ----
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5
stack backtrace:
   0:     0x55c5d8263be4 - backtrace::backtrace::libunwind::trace::h93069dd45e563114
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x55c5d8263be4 - backtrace::backtrace::trace_unsynchronized::ha81d63fb01c61f43
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x55c5d8263be4 - std::sys_common::backtrace::_print_fmt::h5800670bd4856d97
                               at src/libstd/sys_common/backtrace.rs:77
   3:     0x55c5d8263be4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8e952e7a864dce8d
                               at src/libstd/sys_common/backtrace.rs:61
   4:     0x55c5d8286c5c - core::fmt::write::ha6cd837d898e4a61
                               at src/libcore/fmt/mod.rs:1030
   5:     0x55c5d81e06d5 - std::io::Write::write_fmt::haa9265683e464bfb
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/io/mod.rs:1412
   6:     0x55c5d825ea91 - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::he7c7a5301251df0f
                               at src/libstd/io/impls.rs:141
   7:     0x55c5d8265e9e - std::sys_common::backtrace::_print::he810963362c39756
                               at src/libstd/sys_common/backtrace.rs:65
   8:     0x55c5d8265e9e - std::sys_common::backtrace::print::hde04e6d280846a89
                               at src/libstd/sys_common/backtrace.rs:50
   9:     0x55c5d8265e9e - std::panicking::default_hook::{{closure}}::h9d7792d5f2173a87
                               at src/libstd/panicking.rs:188
  10:     0x55c5d8265b38 - std::panicking::default_hook::h9970cb21e5e5b2d4
                               at src/libstd/panicking.rs:202
  11:     0x55c5d826659b - std::panicking::rust_panic_with_hook::hcbf43bd831dd8ef3
                               at src/libstd/panicking.rs:464
  12:     0x55c5d826613e - std::panicking::continue_panic_fmt::h47b6e25372f76c95
                               at src/libstd/panicking.rs:373
  13:     0x55c5d8266026 - rust_begin_unwind
                               at src/libstd/panicking.rs:302
  14:     0x55c5d8283a7e - core::panicking::panic_fmt::h24ff570098569537
                               at src/libcore/panicking.rs:141
  15:     0x55c5d8283b77 - core::result::unwrap_failed::hf7d72924a9a2d22c
                               at src/libcore/result.rs:1165
  16:     0x55c5d81dd300 - core::result::Result<T,E>::unwrap::h0830fd1b36ee5052
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/result.rs:933
  17:     0x55c5d81ddcbc - test::get_test_file_data::h65896587825f1fc9
                               at tests/test.rs:11
  18:     0x55c5d81dde04 - test::get_test_data::h65a752d85858b18c
                               at tests/test.rs:53
  19:     0x55c5d81dde5c - test::roundtrip::h91d60028ff561479
                               at tests/test.rs:57
  20:     0x55c5d81de698 - test::roundtrip_lvl_0::h2a97638a87236059
                               at tests/test.rs:81
  21:     0x55c5d81df0da - test::roundtrip_lvl_0::{{closure}}::he8364aa236ed0c19
                               at tests/test.rs:80
  22:     0x55c5d81deade - core::ops::function::FnOnce::call_once::h731f141a0b0e1982
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/ops/function.rs:227
  23:     0x55c5d81eb32f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h19a09fd199a0ff21
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/liballoc/boxed.rs:942
  24:     0x55c5d826d1aa - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:79
  25:     0x55c5d820690a - std::panicking::try::h1d1a1c5142fbfdd7
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panicking.rs:265
  26:     0x55c5d820690a - std::panic::catch_unwind::h200ca6210a82d128
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panic.rs:396
  27:     0x55c5d820690a - test::run_test_in_process::h766e5938de77f4da
                               at src/libtest/lib.rs:570
  28:     0x55c5d820690a - test::run_test::run_test_inner::{{closure}}::h3d3ac74133b541d4
                               at src/libtest/lib.rs:473
  29:     0x55c5d8205fe8 - test::run_test::run_test_inner::he77bc794d3ace3bd
                               at src/libtest/lib.rs:494
  30:     0x55c5d8204569 - test::run_test::hba77973327a2fc7f
                               at src/libtest/lib.rs:530
  31:     0x55c5d81fa954 - test::run_tests::h5a1270ddd5e0a473
                               at src/libtest/lib.rs:299
  32:     0x55c5d81fa954 - test::console::run_tests_console::h29d755f301130a08
                               at src/libtest/console.rs:295
  33:     0x55c5d82009e4 - test::test_main::hf2bc958b6a636d99
                               at src/libtest/lib.rs:120
  34:     0x55c5d8201961 - test::test_main_static::h90326c29e77e672b
                               at src/libtest/lib.rs:139
  35:     0x55c5d81de6b6 - test::main::h314b86d7399b1960
  36:     0x55c5d81ddc00 - std::rt::lang_start::{{closure}}::hbaebdb534a364c8b
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
  37:     0x55c5d8265fc3 - std::rt::lang_start_internal::{{closure}}::h040fbb64fb819fe8
                               at src/libstd/rt.rs:48
  38:     0x55c5d8265fc3 - std::panicking::try::do_call::hfb8f62e9781fa17e
                               at src/libstd/panicking.rs:287
  39:     0x55c5d826d1aa - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:79
  40:     0x55c5d8266a7d - std::panicking::try::h2aa48346968c7185
                               at src/libstd/panicking.rs:265
  41:     0x55c5d8266a7d - std::panic::catch_unwind::h8da95d989d5c8997
                               at src/libstd/panic.rs:396
  42:     0x55c5d8266a7d - std::rt::lang_start_internal::h298ac725745c4eb6
                               at src/libstd/rt.rs:47
  43:     0x55c5d81ddbd9 - std::rt::lang_start::hce40aa1f7d621390
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
  44:     0x55c5d81de6ea - main
  45:     0x7f5e423d1f2b - __libc_start_main
                               at ../csu/libc-start.c:308
  46:     0x55c5d81dc18a - _start
  47:                0x0 - <unknown>

---- roundtrip_lvl_1 stdout ----
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5
stack backtrace:
   0:     0x55c5d8263be4 - backtrace::backtrace::libunwind::trace::h93069dd45e563114
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x55c5d8263be4 - backtrace::backtrace::trace_unsynchronized::ha81d63fb01c61f43
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x55c5d8263be4 - std::sys_common::backtrace::_print_fmt::h5800670bd4856d97
                               at src/libstd/sys_common/backtrace.rs:77
   3:     0x55c5d8263be4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8e952e7a864dce8d
                               at src/libstd/sys_common/backtrace.rs:61
   4:     0x55c5d8286c5c - core::fmt::write::ha6cd837d898e4a61
                               at src/libcore/fmt/mod.rs:1030
   5:     0x55c5d81e06d5 - std::io::Write::write_fmt::haa9265683e464bfb
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/io/mod.rs:1412
   6:     0x55c5d825ea91 - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::he7c7a5301251df0f
                               at src/libstd/io/impls.rs:141
   7:     0x55c5d8265e9e - std::sys_common::backtrace::_print::he810963362c39756
                               at src/libstd/sys_common/backtrace.rs:65
   8:     0x55c5d8265e9e - std::sys_common::backtrace::print::hde04e6d280846a89
                               at src/libstd/sys_common/backtrace.rs:50
   9:     0x55c5d8265e9e - std::panicking::default_hook::{{closure}}::h9d7792d5f2173a87
                               at src/libstd/panicking.rs:188
  10:     0x55c5d8265b38 - std::panicking::default_hook::h9970cb21e5e5b2d4
                               at src/libstd/panicking.rs:202
  11:     0x55c5d826659b - std::panicking::rust_panic_with_hook::hcbf43bd831dd8ef3
                               at src/libstd/panicking.rs:464
  12:     0x55c5d826613e - std::panicking::continue_panic_fmt::h47b6e25372f76c95
                               at src/libstd/panicking.rs:373
  13:     0x55c5d8266026 - rust_begin_unwind
                               at src/libstd/panicking.rs:302
  14:     0x55c5d8283a7e - core::panicking::panic_fmt::h24ff570098569537
                               at src/libcore/panicking.rs:141
  15:     0x55c5d8283b77 - core::result::unwrap_failed::hf7d72924a9a2d22c
                               at src/libcore/result.rs:1165
  16:     0x55c5d81dd300 - core::result::Result<T,E>::unwrap::h0830fd1b36ee5052
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/result.rs:933
  17:     0x55c5d81ddcbc - test::get_test_file_data::h65896587825f1fc9
                               at tests/test.rs:11
  18:     0x55c5d81dde04 - test::get_test_data::h65a752d85858b18c
                               at tests/test.rs:53
  19:     0x55c5d81dde5c - test::roundtrip::h91d60028ff561479
                               at tests/test.rs:57
  20:     0x55c5d81de68b - test::roundtrip_lvl_1::h7254383e6a5eedb4
                               at tests/test.rs:76
  21:     0x55c5d81df0ba - test::roundtrip_lvl_1::{{closure}}::h1ea577cb5ee3e6cd
                               at tests/test.rs:75
  22:     0x55c5d81dec0e - core::ops::function::FnOnce::call_once::hc9e28689b5a1b24f
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/ops/function.rs:227
  23:     0x55c5d81eb32f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h19a09fd199a0ff21
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/liballoc/boxed.rs:942
  24:     0x55c5d826d1aa - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:79
  25:     0x55c5d820690a - std::panicking::try::h1d1a1c5142fbfdd7
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panicking.rs:265
  26:     0x55c5d820690a - std::panic::catch_unwind::h200ca6210a82d128
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panic.rs:396
  27:     0x55c5d820690a - test::run_test_in_process::h766e5938de77f4da
                               at src/libtest/lib.rs:570
  28:     0x55c5d820690a - test::run_test::run_test_inner::{{closure}}::h3d3ac74133b541d4
                               at src/libtest/lib.rs:473
  29:     0x55c5d8205fe8 - test::run_test::run_test_inner::he77bc794d3ace3bd
                               at src/libtest/lib.rs:494
  30:     0x55c5d8204569 - test::run_test::hba77973327a2fc7f
                               at src/libtest/lib.rs:530
  31:     0x55c5d81fa954 - test::run_tests::h5a1270ddd5e0a473
                               at src/libtest/lib.rs:299
  32:     0x55c5d81fa954 - test::console::run_tests_console::h29d755f301130a08
                               at src/libtest/console.rs:295
  33:     0x55c5d82009e4 - test::test_main::hf2bc958b6a636d99
                               at src/libtest/lib.rs:120
  34:     0x55c5d8201961 - test::test_main_static::h90326c29e77e672b
                               at src/libtest/lib.rs:139
  35:     0x55c5d81de6b6 - test::main::h314b86d7399b1960
  36:     0x55c5d81ddc00 - std::rt::lang_start::{{closure}}::hbaebdb534a364c8b
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
  37:     0x55c5d8265fc3 - std::rt::lang_start_internal::{{closure}}::h040fbb64fb819fe8
                               at src/libstd/rt.rs:48
  38:     0x55c5d8265fc3 - std::panicking::try::do_call::hfb8f62e9781fa17e
                               at src/libstd/panicking.rs:287
  39:     0x55c5d826d1aa - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:79
  40:     0x55c5d8266a7d - std::panicking::try::h2aa48346968c7185
                               at src/libstd/panicking.rs:265
  41:     0x55c5d8266a7d - std::panic::catch_unwind::h8da95d989d5c8997
                               at src/libstd/panic.rs:396
  42:     0x55c5d8266a7d - std::rt::lang_start_internal::h298ac725745c4eb6
                               at src/libstd/rt.rs:47
  43:     0x55c5d81ddbd9 - std::rt::lang_start::hce40aa1f7d621390
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
  44:     0x55c5d81de6ea - main
  45:     0x7f5e423d1f2b - __libc_start_main
                               at ../csu/libc-start.c:308
  46:     0x55c5d81dc18a - _start
  47:                0x0 - <unknown>

---- roundtrip_lvl_9 stdout ----
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1165:5
stack backtrace:
   0:     0x55c5d8263be4 - backtrace::backtrace::libunwind::trace::h93069dd45e563114
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x55c5d8263be4 - backtrace::backtrace::trace_unsynchronized::ha81d63fb01c61f43
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x55c5d8263be4 - std::sys_common::backtrace::_print_fmt::h5800670bd4856d97
                               at src/libstd/sys_common/backtrace.rs:77
   3:     0x55c5d8263be4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8e952e7a864dce8d
                               at src/libstd/sys_common/backtrace.rs:61
   4:     0x55c5d8286c5c - core::fmt::write::ha6cd837d898e4a61
                               at src/libcore/fmt/mod.rs:1030
   5:     0x55c5d81e06d5 - std::io::Write::write_fmt::haa9265683e464bfb
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/io/mod.rs:1412
   6:     0x55c5d825ea91 - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::he7c7a5301251df0f
                               at src/libstd/io/impls.rs:141
   7:     0x55c5d8265e9e - std::sys_common::backtrace::_print::he810963362c39756
                               at src/libstd/sys_common/backtrace.rs:65
   8:     0x55c5d8265e9e - std::sys_common::backtrace::print::hde04e6d280846a89
                               at src/libstd/sys_common/backtrace.rs:50
   9:     0x55c5d8265e9e - std::panicking::default_hook::{{closure}}::h9d7792d5f2173a87
                               at src/libstd/panicking.rs:188
  10:     0x55c5d8265b38 - std::panicking::default_hook::h9970cb21e5e5b2d4
                               at src/libstd/panicking.rs:202
  11:     0x55c5d826659b - std::panicking::rust_panic_with_hook::hcbf43bd831dd8ef3
                               at src/libstd/panicking.rs:464
  12:     0x55c5d826613e - std::panicking::continue_panic_fmt::h47b6e25372f76c95
                               at src/libstd/panicking.rs:373
  13:     0x55c5d8266026 - rust_begin_unwind
                               at src/libstd/panicking.rs:302
  14:     0x55c5d8283a7e - core::panicking::panic_fmt::h24ff570098569537
                               at src/libcore/panicking.rs:141
  15:     0x55c5d8283b77 - core::result::unwrap_failed::hf7d72924a9a2d22c
                               at src/libcore/result.rs:1165
  16:     0x55c5d81dd300 - core::result::Result<T,E>::unwrap::h0830fd1b36ee5052
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/result.rs:933
  17:     0x55c5d81ddcbc - test::get_test_file_data::h65896587825f1fc9
                               at tests/test.rs:11
  18:     0x55c5d81dde04 - test::get_test_data::h65a752d85858b18c
                               at tests/test.rs:53
  19:     0x55c5d81dde5c - test::roundtrip::h91d60028ff561479
                               at tests/test.rs:57
  20:     0x55c5d81de67b - test::roundtrip_lvl_9::hbbb589037d403d50
                               at tests/test.rs:71
  21:     0x55c5d81df09a - test::roundtrip_lvl_9::{{closure}}::hee71adf26a769dd9
                               at tests/test.rs:70
  22:     0x55c5d81debde - core::ops::function::FnOnce::call_once::had14d0d8939888d8
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libcore/ops/function.rs:227
  23:     0x55c5d81eb32f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h19a09fd199a0ff21
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/liballoc/boxed.rs:942
  24:     0x55c5d826d1aa - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:79
  25:     0x55c5d820690a - std::panicking::try::h1d1a1c5142fbfdd7
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panicking.rs:265
  26:     0x55c5d820690a - std::panic::catch_unwind::h200ca6210a82d128
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/panic.rs:396
  27:     0x55c5d820690a - test::run_test_in_process::h766e5938de77f4da
                               at src/libtest/lib.rs:570
  28:     0x55c5d820690a - test::run_test::run_test_inner::{{closure}}::h3d3ac74133b541d4
                               at src/libtest/lib.rs:473
  29:     0x55c5d8205fe8 - test::run_test::run_test_inner::he77bc794d3ace3bd
                               at src/libtest/lib.rs:494
  30:     0x55c5d8204569 - test::run_test::hba77973327a2fc7f
                               at src/libtest/lib.rs:530
  31:     0x55c5d81fa954 - test::run_tests::h5a1270ddd5e0a473
                               at src/libtest/lib.rs:299
  32:     0x55c5d81fa954 - test::console::run_tests_console::h29d755f301130a08
                               at src/libtest/console.rs:295
  33:     0x55c5d82009e4 - test::test_main::hf2bc958b6a636d99
                               at src/libtest/lib.rs:120
  34:     0x55c5d8201961 - test::test_main_static::h90326c29e77e672b
                               at src/libtest/lib.rs:139
  35:     0x55c5d81de6b6 - test::main::h314b86d7399b1960
  36:     0x55c5d81ddc00 - std::rt::lang_start::{{closure}}::hbaebdb534a364c8b
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
  37:     0x55c5d8265fc3 - std::rt::lang_start_internal::{{closure}}::h040fbb64fb819fe8
                               at src/libstd/rt.rs:48
  38:     0x55c5d8265fc3 - std::panicking::try::do_call::hfb8f62e9781fa17e
                               at src/libstd/panicking.rs:287
  39:     0x55c5d826d1aa - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:79
  40:     0x55c5d8266a7d - std::panicking::try::h2aa48346968c7185
                               at src/libstd/panicking.rs:265
  41:     0x55c5d8266a7d - std::panic::catch_unwind::h8da95d989d5c8997
                               at src/libstd/panic.rs:396
  42:     0x55c5d8266a7d - std::rt::lang_start_internal::h298ac725745c4eb6
                               at src/libstd/rt.rs:47
  43:     0x55c5d81ddbd9 - std::rt::lang_start::hce40aa1f7d621390
                               at /rustc/4f03f4a989d1c8346c19dfb417a77c09b34408b8/src/libstd/rt.rs:61
  44:     0x55c5d81de6ea - main
  45:     0x7f5e423d1f2b - __libc_start_main
                               at ../csu/libc-start.c:308
  46:     0x55c5d81dc18a - _start
  47:                0x0 - <unknown>


failures:
    roundtrip_lvl_0
    roundtrip_lvl_1
    roundtrip_lvl_9

test result: FAILED. 4 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out