alexcrichton / bzip2-rs Goto Github PK

View Code? Open in Web Editor NEW

93.0 93.0 57.0 1.85 MB

libbz2 (bzip2 compression) bindings for Rust

License: Apache License 2.0

Rust 11.74% Makefile 1.33% C 50.21% XSLT 2.62% Shell 1.77% CSS 0.37% Perl 0.36% HTML 26.80% Roff 4.81%

bzip2-rs's People

Contributors

Stargazers

Watchers

bzip2-rs's Issues

Filesize mismatch when decompressing multi-stream with sizes greater than 2GB (2^31)

Here is a self contained reproducer of the problem: bzip_bug.zip

To reproduce:

unzip bzip_bug.zip
cd bzip_bug
cargo run --release --bin bzip_bug

You will see the following output:

$ cargo run --release --bin bzip_bug
   Compiling pkg-config v0.3.18
   Compiling libc v0.2.72
   Compiling cc v1.0.58
   Compiling bzip2-sys v0.1.9+1.0.8
   Compiling bzip2 v0.4.1
   Compiling bzip_bug v0.1.0 (/private/tmp/foo/bzip_bug)
    Finished release [optimized] target(s) in 9.08s
     Running `target/release/bzip_bug`
Generating expected results
Decompressing stream made with bzip2
Decompressing stream made with pbzip2
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `405900000`,
 right: `3000000000`: decompressed length mismatch', src/main.rs:37:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The expected output is that the program complete's sucessfully

The test has two files:

raw.dat.bz2 is a 3GB generated file produced using the bzip2 tool on a mac
raw.dat.pbz2 is the same 3GB generated file produced using the pbzip2 tool on a mac

The data was generated using the generate.rs tool in the package, via the following commands:

$ cargo build --release --all
$ ./target/release/generate | nice bzip2 > raw.dat.bz2&
$ ./target/release/generate | nice pbzip2 > raw.dat.pbz2&

You can check the output byte counts using bzcat:

$ bzcat raw.dat.bz2 | wc -c
  3000000000

$ bzcat raw.dat.pbz2 | wc -c
 3000000000

The issue seems to affect files that are larger than 2^31 (which smells like a u32 overflow somewhere)

bzip2::Compression should derive Debug

It's a simple enum, no reason for it not to.

Stuck busy looping

First please take whit a grain of salt, and assume I might be doing something wrong, or the data is corrupted.

I am doing long running rdedup stressttest I caught it hanged, using 100% of one of the CPUs. I was able to attach gdb and turns out it looping on a drop/finish in this code of mine:

  8     fn decompress(&self, buf: SGData) -> io::Result<SGData> {                                                       
  7         let mut decompressor =                                                                                      
  6             bzip2::write::BzDecoder::new(Vec::with_capacity(buf.len()));                                            
  5                                                                                                                     
  4         for sg_part in buf.as_parts() {                                                                             
  3             decompressor.write_all(&sg_part)?;                                                                      
  2         }                                                                                                           
  1         Ok(SGData::from_single(decompressor.finish()?))                                                             
114     }

Note that 114 might be off, as I might have added/removed a line or two. I'm guessing it is finish() call.

#0  bzip2::write::{{impl}}::write<collections::vec::Vec<u8>> (self=0x7fff68365488, data=...)                            
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:254                           
#1  0x00005567a4127d20 in core::ptr::drop_in_place<bzip2::write::BzDecoder<collections::vec::Vec<u8>>> ()               
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:219                           
#2  0x00005567a4127952 in rdedup_lib::compression::{{impl}}::decompress (self=<optimized out>, buf=...)                 
    at lib/src/compression.rs:114                           
#3  0x00005567a4153913 in rdedup_lib::{{impl}}::read_chunk_into (self=0x7fff6837fd38, digest=...,                       
    chunk_type=<optimized out>, data_type=rdedup_lib::DataType::Data, writer=...) at lib/src/lib.rs:352                 
#4  0x00005567a4151807 in rdedup_lib::ReadContext::read_recursively (self=0x7fff68367aa0, accessor=..., digest=...)     
    at lib/src/lib.rs:262

If I keep doing next in my gdb, I get:

core::ptr::drop_in_place<bzip2::write::BzDecoder<collections::vec::Vec<u8>>> ()                                         
    at /home/dpc/lab/rust/rdedup/lib/<try macros>:3         
3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
(gdb)                         
218             while !self.done {                          
(gdb)                         
219                 try!(self.write(&[]));                  
(gdb)                         
3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
(gdb)                         
218             while !self.done {                          
(gdb)                         
219                 try!(self.write(&[]));                  
(gdb) n                       
3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
(gdb)                         
218             while !self.done {                          
(gdb)                         
219                 try!(self.write(&[]));                  
(gdb)                         
3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
(gdb)                         
218             while !self.done {                          
(gdb)                         
219                 try!(self.write(&[]));                  
(gdb)                         
3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.

If I do step, I get:

3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
(gdb) s                       
218             while !self.done {                          
(gdb)                         
219                 try!(self.write(&[]));                  
(gdb)                         
bzip2::write::{{impl}}::write<collections::vec::Vec<u8>> (self=0x7fff68365488, data=...)                                
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:254                           
254             if self.done {
(gdb)                         
258                 try!(self.dump());                      
(gdb)                         
258                 try!(self.dump());                      
(gdb)                         
bzip2::write::BzDecoder<collections::vec::Vec<u8>>::dump<collections::vec::Vec<u8>> (self=0x7fff68365488)               
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/write.rs:200                           
200             while self.buf.len() > 0 {                  
(gdb)                         
204             Ok(())        
(gdb)                         
205         }                 
(gdb)                         
bzip2::write::{{impl}}::write<collections::vec::Vec<u8>> (self=0x7fff68365488, data=...)                                
    at /home/dpc/lab/rust/rdedup/lib/<try macros>:3                                                                     
3       /home/dpc/lab/rust/rdedup/lib/<try macros>: No such file or directory.                                          
(gdb)                                                                                                                   
267             (self.raw.total_in_lo32 as u64) |                                                                       
(gdb)                                                                                                                   
268             ((self.raw.total_in_hi32 as u64) << 32)                                                                 
(gdb)                                                                                                                   
261                 let res = self.data.decompress_vec(data, &mut self.buf);                                            
(gdb)                                                                                                                   
bzip2::mem::Decompress::decompress_vec (self=0x7fff68365488, input=..., output=0x7fff683654a8)                          
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:243                             
243                 let before = self.total_out();                                                                      
(gdb)                                                                                                                   
bzip2::mem::Decompress::total_out (self=<optimized out>)                                                                
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:243                             
243                 let before = self.total_out();                                                                      
(gdb)                         
bzip2::mem::Stream<bzip2::mem::DirDecompress>::total_out<bzip2::mem::DirDecompress> (self=<optimized out>)              
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:272                             
272             (self.raw.total_out_lo32 as u64) |          
(gdb)                         
273             ((self.raw.total_out_hi32 as u64) << 32)    
(gdb)                         
bzip2::mem::Decompress::decompress_vec (self=0x7fff68365488, input=..., output=0x7fff683654a8)                          
    at /home/dpc/.cargo/registry/src/github.com-1ecc6299db9ec823/bzip2-0.3.2/src/mem.rs:245                             
245                     let ptr = output.as_mut_ptr().offset(len as isize);                                             
(gdb)                         
core::ptr::{{impl}}::offset<u8> (count=<optimized out>, self=<optimized out>) at /checkout/src/libcore/ptr.rs:617       
617     /checkout/src/libcore/ptr.rs: No such file or directory.                                                        
(gdb)         
(...)

Update for futures 0.3

Motivation: zip-rs/zip-old#124 (zip-rs async)

Error trying to install crate bzip2-sys

I'm trying to install the zip crate that depends on bzip2 but I'm getting the following error.

Using Rust nightly 1.31.0 ("2018 edition")

Compiling bzip2-sys v0.1.6
error: failed to run custom build command for `bzip2-sys v0.1.6`
process didn't exit successfully: `C:\Dev\[project]\target\debug\build\bzip2-sys-33f8cb5c7dfcf9ea\build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-pc-windows-msvc")
OPT_LEVEL = Some("0")
HOST = Some("x86_64-pc-windows-msvc")
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
HOST_CC = None
CC = Some("C:\\mingw-w64\\x86_64-7.3.0-posix-seh-rt_v5-rev0\\mingw64\\bin\\gcc.exe")
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
HOST_CFLAGS = None
CFLAGS = None
DEBUG = Some("true")
running: "C:\\mingw-w64\\x86_64-7.3.0-posix-seh-rt_v5-rev0\\mingw64\\bin\\gcc.exe" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-DBZ_NO_STDIO" "/FoC:\\Dev\\[project]\\target\\debug\\build\\bzip2-sys-351245291296f0d0\\out\\bzip2-1.0.6\\blocksort.o" "/c" "bzip2-1.0.6/blocksort.c"
cargo:warning=gcc.exe: error: /FoC:\Dev\[project]\target\debug\build\bzip2-sys-351245291296f0d0\out\bzip2-1.0.6\blocksort.o: Invalid argument
cargo:warning=gcc.exe: error: /c: No such file or directory
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "C:\\mingw-w64\\x86_64-7.3.0-posix-seh-rt_v5-rev0\\mingw64\\bin\\gcc.exe" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-DBZ_NO_STDIO" "/FoC:\\Dev\\[project]\\target\\debug\\build\\bzip2-sys-351245291296f0d0\\out\\bzip2-1.0.6\\blocksort.o" "/c" "bzip2-1.0.6/blocksort.c" with args "gcc.exe" did not execute successfully (status code exit code: 1).

', C:\Users\[user]\.cargo\registry\src\github.com-1ecc6299db9ec823\cc-1.0.25\src\lib.rs:2260:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Write::write_all is incompatible with non-blocking IO

Unfortunately std::io::Write::write_all is incompatible with non-blocking IO, because it can fail after a partial write and lose track of how much was written. We appear to use write_all in a couple of places. Consider replacing it with something like the write_from_offset I implemented in zstd-rs.

Is is possible to assume libbz2 is already linked?

I have a C static library that uses this crate. My static lib links with a C program that is already linked with libbz2 (static or dynamic). I don't need bzip2-sys to link with bzip2 - I just need it to assume it will be linked with bzip2. Is that possible?

Presently I'm looking at this build failure on Windows:

  bz2.lib(bz2.dll) : error LNK2005: BZ2_bzDecompressInit already defined in clamav_rust.lib(bzlib.o)
  bz2.lib(bz2.dll) : error LNK2005: BZ2_bzDecompress already defined in clamav_rust.lib(bzlib.o)
  bz2.lib(bz2.dll) : error LNK2005: BZ2_bzDecompressEnd already defined in clamav_rust.lib(bzlib.o)
     Creating library libclamav\clamav.lib and object libclamav\clamav.exp
  libclamav\libclamav.dll : fatal error LNK1169: one or more multiply defined symbols found

ref: https://github.com/Cisco-Talos/clamav/actions/runs/8347839964/job/22848401392?pr=1183

Implement tokio::io::AsyncBufRead for bufread::* types

Tokio not only provides the AsyncRead trait (mirroring std::io::Read but for async) but also AsyncBufRead (mirroring std::io::BufRead).

It would be great if it was implemented for the types in the bufread module if the tokio feature flag is enabled. Right now it appears that only AsyncRead is implemented in this case.

Potential undefined behavior from cyclic references between `bz_stream` and `EState`/`DState`.

I've noticed the following pattern is used in allocating a bz_stream object from Rust within src/mem.rs:

let mut raw = Box::new(mem::zeroed());
assert_eq!(
    ffi::BZ2_bzCompressInit(&mut *raw, lvl.level() as c_int, 0, work_factor as c_int),
    0
);

Then, the unique borrow created by &mut *raw is used to create a cyclic data structure between the initialized bz_stream and its EState member, state. For example, in BZ2_bzCompressInit in bzlib.c, you have the following:

int BZ_API(BZ2_bzCompressInit) 
                    ( bz_stream* strm, 
                     ... )
{
   ...
   EState* s;
   ...
   s->strm = strm;
   ...
   strm->state = s;
   ...
}

Every time the allocated bz_stream object is passed from Rust to C, it's done with &mut *raw. In Stacked Borrows, it seems like this would invalidate the SharedRW permission associated with the original pointer stored by s->strm = strm;. In Tree Borrows, it would initially transition the permission for s->strm to being Reserved the next time a &mut *raw is passed, so reading through it would still be allowed.

But, the root bz_stream allocation is both read and written to through its state member, so this would still lead to violations of the model eventually. For instance, in copy_input_until_stop (also in bzlib.c), you have both reads and writes from the EState pointer s to its parent bz_stream:

 ADD_CHAR_TO_BLOCK ( s, (UInt32)(*((UChar*)(s->strm->next_in))) ); 
 s->strm->next_in++;

It seems like the fix for this would be to convert the Box into a raw pointer with Box::into_raw and pass this raw pointer directly to the FFI. Then, it would be re-wrapped it for deallocation in the implementation of Drop for Stream. However, I'm totally unsure if this would be a situation where violating the Stacked Borrows/Tree Borrows models would be problematic. What do you think?

bz_internal_error exported

I develop rpm-sequoia, which is a Rust crate that implements rpm's OpenPGP API in terms of Sequoia PGP. (More details, unrelated to this issue, are described in this rpm issue).

rpm-sequoia should only export the rpm's PGP API. Unfortunately, a number of additional symbols are leaked by dependencies. Specifically, any symbols that have the #[no_mangle] attribute appear to be exported. bzip2-sys marks bz_internal_error like this, and thus it is exported by the library:

$ nm --defined-only --extern-only /tmp/rpm-sequoia/debug/librpm_sequoia.so | grep bz
0000000000361fa0 T bz_internal_error

In turns out that rpm-sequoia doesn't actually need compression support so I was able to workaround this issue by disabling that feature. But, this issue may trip up others.

failed to run custom build command for `bzip2-sys v0.1.8+1.0.8`

I stucked by this error, pls help me.

_Compiling bzip2-sys v0.1.8+1.0.8 Compiling bincode v1.2.1 Compiling serde_derive v1.0.110 Compiling chrono v0.4.11 error: failed to run custom build command forbzip2-sys v0.1.8+1.0.8`

Caused by:
process didn't exit successfully: D:\projects\write-teacher-master\miniprogram-write-teacher-master\teacher\target\release\build\bzip2-sys-a0175bb02427920c\build-script-build (exit code: 1)
--- stderr

error occurred: Command "cmd" "/c" "emcc.bat" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "bzip2-1.0.8" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "-o" "D:\projects\write-teacher-master\miniprogram-write-teacher-master\teacher\target\asmjs-unknown-emscripten\release\build\bzip2-sys-372a0e1a0c44a99a\out\lib\bzip2-1.0.8/blocksort.o" "-c" "bzip2-1.0.8/blocksort.c" with args "cmd" did not execute successfully (status code exit code: 1)._

bzip2 source is outdated

The bundled version of bzip2 is 1.0.6, which is behind two releases, that include CVE fixes.

Unclear how to cross-compile to Windows MSVC target from Linux

I saw an error in my larger project when compiling with --target=x86_64-pc-windows-msvc on my Linux x86_64 host system:

error: failed to run custom build command for `bzip2-sys v0.1.7`

Caused by:
  process didn't exit successfully: `/home/x/dev/rust/life/conwayste/target/debug/build/bzip2-sys-94bcdbfb8c02afbc/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-pc-windows-msvc")
OPT_LEVEL = Some("2")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
TARGET_CC = None
CC = None
CROSS_COMPILE = None
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("true")
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "/Fo/home/x/dev/rust/life/conwayste/target/x86_64-pc-windows-msvc/debug/build/bzip2-sys-79c995aba8afa54a/out/lib/bzip2-1.0.6/blocksort.o" "/c" "bzip2-1.0.6/blocksort.c"
cargo:warning=cc: error: /Fo/home/x/dev/rust/life/conwayste/target/x86_64-pc-windows-msvc/debug/build/bzip2-sys-79c995aba8afa54a/out/lib/bzip2-1.0.6/blocksort.o: No such file or directory
cargo:warning=cc: error: /c: No such file or directory
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "/Fo/home/x/dev/rust/life/conwayste/target/x86_64-pc-windows-msvc/debug/build/bzip2-sys-79c995aba8afa54a/out/lib/bzip2-1.0.6/blocksort.o" "/c" "bzip2-1.0.6/blocksort.c" with args "cc" did not execute successfully (status code exit code: 1).

', /home/x/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.28/src/lib.rs:2314:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1025
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:193
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:210
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:471
  11: rust_begin_unwind
             at src/libstd/panicking.rs:375
  12: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:326
  13: cc::fail
             at /home/x/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.28/src/lib.rs:2314
  14: cc::Build::compile
             at /home/x/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.28/src/lib.rs:951
  15: build_script_build::main
             at ./build.rs:17
  16: std::rt::lang_start::{{closure}}
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/rt.rs:67
  17: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  18: std::panicking::try::do_call
             at src/libstd/panicking.rs:292
  19: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  20: std::panicking::try
             at src/libstd/panicking.rs:270
  21: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  22: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  23: main
  24: __libc_start_main
  25: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

I tried to reproduce it with a minimal test case against bzip2 0.3.3 and got what appears to be an entirely different error:

$ cargo build --target=x86_64-pc-windows-msvc
   Compiling libc v0.2.66
   Compiling bzip2-sys v0.1.7
error: failed to run custom build command for `bzip2-sys v0.1.7`

Caused by:
  process didn't exit successfully: `/home/x/dev/rust/bzip2test/target/debug/build/bzip2-sys-86f222c7d48fea1e/build-script-build` (exit code: 1)
--- stdout
TARGET = Some("x86_64-pc-windows-msvc")
OPT_LEVEL = Some("0")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-pc-windows-msvc = None
CC_x86_64_pc_windows_msvc = None
TARGET_CC = None
CC = None
CROSS_COMPILE = None
CFLAGS_x86_64-pc-windows-msvc = None
CFLAGS_x86_64_pc_windows_msvc = None
TARGET_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "-Fo/home/x/dev/rust/bzip2test/target/x86_64-pc-windows-msvc/debug/build/bzip2-sys-1f1ad99dbefe35b6/out/lib/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c"
cargo:warning=In file included from bzip2-1.0.6/bzlib_private.h:33,
cargo:warning=                 from bzip2-1.0.6/blocksort.c:22:
cargo:warning=bzip2-1.0.6/bzlib.h:70: warning: "BZ_EXPORT" redefined
cargo:warning=   70 | #define BZ_EXPORT
cargo:warning=      | 
cargo:warning=<command-line>: note: this is the location of the previous definition
cargo:warning=In file included from bzip2-1.0.6/bzlib_private.h:33,
cargo:warning=                 from bzip2-1.0.6/blocksort.c:22:
cargo:warning=bzip2-1.0.6/bzlib.h:79:13: fatal error: windows.h: No such file or directory
cargo:warning=   79 | #   include <windows.h>
cargo:warning=      |             ^~~~~~~~~~~
cargo:warning=compilation terminated.
exit code: 1

--- stderr


error occurred: Command "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "-Fo/home/x/dev/rust/bzip2test/target/x86_64-pc-windows-msvc/debug/build/bzip2-sys-1f1ad99dbefe35b6/out/lib/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c" with args "cc" did not execute successfully (status code exit code: 1).



warning: build failed, waiting for other jobs to finish...
error: build failed

build android fail

buid command:
RUSTFLAGS='-C strip=symbols' cargo build -p 'lib_project' --release --target aarch64-linux-android
RUSTFLAGS='-C strip=symbols' cargo build -p 'lib_project' --release --target armv7-linux-androideabi
config file

[target.aarch64-linux-android]
ar = "NDK/aarch64/bin/aarch64-linux-android-ar"
linker = "NDK/aarch64/bin/aarch64-linux-android22-clang"

[target.armv7-linux-androideabi]
ar = "NDK/armv7/bin/arm-linux-androideabi-ar"
linker = "NDK/armv7/bin/armv7a-linux-androideabi22-clang"

code:

///zip compress
fn compress_zip(bytes: &Vec<u8>, level: u32) -> anyhow::Result<Vec<u8>> {
    use bzip2::write::{BzEncoder};
    let mut compressor = BzEncoder::new(vec![0x1, 0, 0, 0, 0], Compression::new(level.min(9)));
    compressor.write_all(bytes)?;
    let mut result = compressor.finish()?;
    Ok(result)
}

///zip decompress bytes buffer
fn depress_zip(bytes: &[u8], size: usize) -> anyhow::Result<Vec<u8>>{
    let mut decompressor = bzip2::read::BzDecoder::new(std::io::Cursor::new(bytes));
    let mut out_buffer = Vec::with_capacity(size);
    std::io::copy(&mut decompressor, &mut out_buffer)?;
    Ok(out_buffer)
}

error msg:

error: failed to run custom build command for bzip2-sys v0.1.11+1.0.8

Caused by:
process didn't exit successfully: /home/banagame/BattleServer/server/Rust/target/release/build/bzip2-sys-5c05721eef23dbb9/build-script-build (exit status: 1)
--- stdout
cargo:rerun-if-env-changed=BZIP2_NO_PKG_CONFIG
cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_armv7-linux-androideabi
cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_armv7_linux_androideabi
cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_ALLOW_CROSS
cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS
cargo:rerun-if-env-changed=PKG_CONFIG_armv7-linux-androideabi
cargo:rerun-if-env-changed=PKG_CONFIG_armv7_linux_androideabi
cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
cargo:rerun-if-env-changed=PKG_CONFIG
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_armv7-linux-androideabi
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_armv7_linux_androideabi
cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
TARGET = Some("armv7-linux-androideabi")
OPT_LEVEL = Some("3")
HOST = Some("x86_64-unknown-linux-gnu")
CC_armv7-linux-androideabi = None
CC_armv7_linux_androideabi = None
TARGET_CC = None
CC = None
CFLAGS_armv7-linux-androideabi = None
CFLAGS_armv7_linux_androideabi = None
TARGET_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("false")
running: "arm-linux-androideabi-clang" "-O3" "-DANDROID" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "bzip2-1.0.8" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "-o" "/home/banagame/BattleServer/server/Rust/target/armv7-linux-androideabi/release/build/bzip2-sys-16baeea8c12f7af8/out/lib/bzip2-1.0.8/blocksort.o" "-c" "bzip2-1.0.8/blocksort.c"

--- stderr

error occurred: Failed to find tool. Is arm-linux-androideabi-clang installed?

Could you publish a release?

The latest release to crates.io was in May 2018. There's newer MultiBz* functionality that would nice to have access to.

Thanks!

Build failure at gcc level: blocksort.c not found

bzip2-sys seems to refuse to build due to missing blocksort.c (it is indeed not present in the related build directory).

   Compiling bzip2-sys v0.1.6
error: failed to run custom build command for `bzip2-sys v0.1.6`
process didn't exit successfully: `c:\Local\poligon\extract_blobs\target\release\build\bzip2-sys-9937a6f77c12f052\build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-pc-windows-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-pc-windows-gnu")
HOST = Some("x86_64-pc-windows-gnu")
TARGET = Some("x86_64-pc-windows-gnu")
TARGET = Some("x86_64-pc-windows-gnu")
HOST = Some("x86_64-pc-windows-gnu")
CC_x86_64-pc-windows-gnu = Some("C:\\Local\\poligon\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\bin\\gcc.exe")
TARGET = Some("x86_64-pc-windows-gnu")
HOST = Some("x86_64-pc-windows-gnu")
CFLAGS_x86_64-pc-windows-gnu = None
CFLAGS_x86_64_pc_windows_gnu = None
HOST_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "C:\\Local\\poligon\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\bin\\gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "bzip2-1.0.6" "-D_WIN32" "-DBZ_EXPORT" "-DBZ_NO_STDIO" "-o" "c:\\Local\\poligon\\extract_blobs\\target\\release\\build\\bzip2-sys-f19fa19517d295e2\\out\\bzip2-1.0.6\\blocksort.o" "-c" "bzip2-1.0.6/blocksort.c"
cargo:warning=gcc.exe: error: CreateProcess: No such file or directory
exit code: 1

I'm using rustc 1.28.0-nightly-gnu-x64 with Win 7.

bzip2::write::{BzDecoder, BzEncoder} should implement get_ref/get_mut

Concrete use case:

For my own code I want to implement AsyncWrite (from tokio-io) for BzEncoder<W> if W: AsyncWrite. For that I need to implement the shutdown method, which (since this is a wrapper) should call shutdown on the wrapped writer. Going to need a get_mut for that.

(Hopefully I can get to this myself -- still waiting on legal approval...)

WASM support

Project: Add wasm32-unknown-unknown support (WASM) #93 is a candidate to bring WASM support

Compilation fails on i686-pc-windows-gnu with mingw64 compiler

Issue

When compiling (linking) a program on Windows x64, using Mingw64, with rustc target i686-pc-windows-gnu fails.

This does not occur when compiling with a x86_64-pc-windows-gnu compiler.

Unfortunately I have not been able to pinpoint the exact location of the error. Using rustc 1.1.0 also produces the error.

Since the flate2 crate does compile succesfully on this combination, I decided to report my findings here.

Compilers

PS D:\Programming\rust\bzip2-rs> rustc --version --verbose
rustc 1.3.0-nightly (0c052199b 2015-07-11)
binary: rustc
commit-hash: 0c052199b92104ba6d64886ff779cf89c3c384d9
commit-date: 2015-07-11
host: i686-pc-windows-gnu
release: 1.3.0-nightly

PS D:\Programming\rust\bzip2-rs> cargo --version --verbose
cargo 0.4.0-nightly (15b497b 2015-07-08) (built 2015-07-11)

PS D:\Programming\rust\bzip2-rs> gcc --version --verbose
Reading specs from c:/mingw-w64/bin/../lib64/gcc/x86_64-w64-mingw32/4.8.2/specs
COLLECT_GCC=C:\Mingw-w64\bin\gcc.exe
COLLECT_LTO_WRAPPER=c:/mingw-w64/bin/../libexec/gcc/x86_64-w64-mingw32/4.8.2/lto-wrapper.exe
gcc.exe (GCC) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Target: x86_64-w64-mingw32
(...)

Log

PS D:\Programming\rust\bzip2-rs> cargo test --verbose
   Compiling libc v0.1.9
     Running `rustc C:\Users\mathi_000\.cargo\registry\src\github.com-0a35038f75765ae4\libc-0.1.9\rust/src/liblibc/lib.rs --crate-name libc --crate-type lib -g --cfg "feature=\"default\"" --cfg "feature=\"cargo-build\"" -C metadata=a4f5a8ab4e211619 -C extra-filename=-a4f5a8ab4e211619 --out-dir D:\Programming\rust\bzip2-rs\target\debug\deps --emit=dep-info,link -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps -Awarnings`
   Compiling gcc v0.3.8
     Running `rustc C:\Users\mathi_000\.cargo\registry\src\github.com-0a35038f75765ae4\gcc-0.3.8\src\lib.rs --crate-name gcc --crate-type lib -g -C metadata=f054ee6639177d8d -C extra-filename=-f054ee6639177d8d --out-dir D:\Programming\rust\bzip2-rs\target\debug\deps --emit=dep-info,link -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps -Awarnings`
   Compiling bzip2-sys v0.1.2 (file:///D:/Programming/rust/bzip2-rs)
     Running `rustc bzip2-sys\build.rs --crate-name build_script_build --crate-type bin -C prefer-dynamic -g --out-dir D:\Programming\rust\bzip2-rs\target\debug\build\bzip2-sys-aabcada2f20b3aad --emit=dep-info,link -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps --extern gcc=D:\Programming\rust\bzip2-rs\target\debug\deps\libgcc-f054ee6639177d8d.rlib`
     Running `D:\Programming\rust\bzip2-rs\target\debug\build\bzip2-sys-aabcada2f20b3aad\build-script-build`
     Running `rustc bzip2-sys\lib.rs --crate-name bzip2_sys --crate-type lib -g -C metadata=c91d92cd1c0fbc6b -C extra-filename=-c91d92cd1c0fbc6b --out-dir D:\Programming\rust\bzip2-rs\target\debug\deps --emit=dep-info,link -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps --extern libc=D:\Programming\rust\bzip2-rs\target\debug\deps\liblibc-a4f5a8ab4e211619.rlib -L native=D:\Programming\rust\bzip2-rs\target\debug\build\bzip2-sys-aabcada2f20b3aad\out -l static=bz2`
   Compiling bzip2 v0.2.3 (file:///D:/Programming/rust/bzip2-rs)
     Running `rustc src\lib.rs --crate-name bzip2 --crate-type lib -g --out-dir D:\Programming\rust\bzip2-rs\target\debug --emit=dep-info,link -L dependency=D:\Programming\rust\bzip2-rs\target\debug -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps --extern bzip2_sys=D:\Programming\rust\bzip2-rs\target\debug\deps\libbzip2_sys-c91d92cd1c0fbc6b.rlib --extern libc=D:\Programming\rust\bzip2-rs\target\debug\deps\liblibc-a4f5a8ab4e211619.rlib -L native=D:\Programming\rust\bzip2-rs\target\debug\build\bzip2-sys-aabcada2f20b3aad\out`
     Running `rustc src\lib.rs --crate-name bzip2 --crate-type lib -g --test -C metadata=f8dc73e4479b0524 -C extra-filename=-f8dc73e4479b0524 --out-dir D:\Programming\rust\bzip2-rs\target\debug --emit=dep-info,link -L dependency=D:\Programming\rust\bzip2-rs\target\debug -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps --extern bzip2_sys=D:\Programming\rust\bzip2-rs\target\debug\deps\libbzip2_sys-c91d92cd1c0fbc6b.rlib --extern libc=D:\Programming\rust\bzip2-rs\target\debug\deps\liblibc-a4f5a8ab4e211619.rlib -L native=D:\Programming\rust\bzip2-rs\target\debug\build\bzip2-sys-aabcada2f20b3aad\out`
error: linking with `gcc` failed: exit code: 1
note: "gcc" "-Wl,--enable-long-section-names" "-fno-use-linker-plugin" "-Wl,--nxcompat" "-Wl,--large-address-aware" "-shared-libgcc" "-L" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib" "D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o" "-o" "D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.exe" "-Wl,--gc-sections" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\libtest-74fa456f.rlib" "D:\Programming\rust\bzip2-rs\target\debug\deps\libbzip2_sys-c91d92cd1c0fbc6b.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\libterm-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\libserialize-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\libgetopts-74fa456f.rlib" "D:\Programming\rust\bzip2-rs\target\debug\deps\liblibc-a4f5a8ab4e211619.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\liblog-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\libstd-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\libcollections-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\librustc_unicode-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\librand-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\liballoc-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\liblibc-74fa456f.rlib" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib\libcore-74fa456f.rlib" "-L" "D:\Programming\rust\bzip2-rs\target\debug" "-L" "D:\Programming\rust\bzip2-rs\target\debug\deps" "-L" "D:\Programming\rust\bzip2-rs\target\debug\build\bzip2-sys-aabcada2f20b3aad\out" "-L" "C:\Rust32\bin\rustlib\i686-pc-windows-gnu\lib" "-L" "D:\Programming\rust\bzip2-rs\.rust\bin\i686-pc-windows-gnu" "-L" "D:\Programming\rust\bzip2-rs\bin\i686-pc-windows-gnu" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-l" "kernel32" "-l" "ws2_32" "-l" "userenv" "-l" "advapi32" "-l" "compiler-rt"
note: D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o: In function `ZN5bzip23raw6Stream14new_decompressE':
D:\Programming\rust\bzip2-rs/<std macros>:45: undefined reference to `BZ2_bzDecompressInit'
D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o: In function `ZN5bzip23raw6Stream12new_compressE':
D:\Programming\rust\bzip2-rs/<std macros>:73: undefined reference to `BZ2_bzCompressInit'
D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o: In function `ZN5bzip23raw6Stream10decompressE':
D:\Programming\rust\bzip2-rs/src/raw.rs:85: undefined reference to `BZ2_bzDecompress'
D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o: In function `ZN5bzip23raw6Stream14decompress_vecE':
D:\Programming\rust\bzip2-rs/src/raw.rs:105: undefined reference to `BZ2_bzDecompress'
D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o: In function `ZN5bzip23raw6Stream8compressE':
D:\Programming\rust\bzip2-rs/src/raw.rs:121: undefined reference to `BZ2_bzCompress'
D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o: In function `ZN5bzip23raw6Stream12compress_vecE':
D:\Programming\rust\bzip2-rs/src/raw.rs:141: undefined reference to `BZ2_bzCompress'
D:\Programming\rust\bzip2-rs\target\debug\bzip2-f8dc73e4479b0524.0.o: In function `ZN5bzip23raw11Stream.Drop4dropE':
D:\Programming\rust\bzip2-rs/src/raw.rs:164: undefined reference to `BZ2_bzCompressEnd'
D:\Programming\rust\bzip2-rs/src/raw.rs:165: undefined reference to `BZ2_bzDecompressEnd'

error: aborting due to previous error
Could not compile `bzip2`.

Caused by:
  Process didn't exit successfully: `rustc src\lib.rs --crate-name bzip2 --crate-type lib -g --test -C metadata=f8dc73e4479b0524 -C extra-filename=-f8dc73e4479b0524 --out-dir D:\Programming\rust\bzip2-rs\target\debug --emit=dep-info,link -L dependency=D:\Programming\rust\bzip2-rs\target\debug -L dependency=D:\Programming\rust\bzip2-rs\target\debug\deps --extern bzip2_sys=D:\Programming\rust\bzip2-rs\target\debug\deps\libbzip2_sys-c91d92cd1c0fbc6b.rlib --extern libc=D:\Programming\rust\bzip2-rs\target\debug\deps\liblibc-a4f5a8ab4e211619.rlib -L native=D:\Programming\rust\bzip2-rs\target\debug\build\bzip2-sys-aabcada2f20b3aad\out` (exit code: 101)

write-side code isn't resilient to ErrorKind::Interrupted

We wrote https://crates.io/crates/partial-io to shake out bugs around partial reads/writes, and Interrupted and WouldBlock handling. I decided to throw it at bzip2-rs and found that while the read-side code is resilient to Interrupted errors, the write-side code isn't.

Here's the failure:

    Error: "called `Result::unwrap()` on an `Err` value: Error { repr: Custom(Custom { k
ind: Interrupted, error: StringError(\"error during write, generated by partial-io\") })
 }"', /home/sid0/.cargo/registry/src/github.com-1ecc6299db9ec823/quickcheck-0.4.1/src/te
ster.rs:147

See https://github.com/sid0/bzip2-rs/tree/write-bug for a demonstration of the bug.

(There's also a tokio feature you can enable on partial-io to get AsyncRead and AsyncWrite support.)

Write wrappers should only perform one write per write call

The documentation for Write::write says that

A call to write represents at most one attempt to write to any wrapped object.

The write wrappers at the moment do a write_all when write is called, which is in violation of this contract.

We would probably need to switch to trying to trying to write from the internal buffer and returning ErrorKind::Interrupted if it was successful.

Needs upgrade to new io

The library currently does not play well with the new Read and Write traits, for example.

Infinite loop after decoding error

Similar to behavior in #96 but different enough to warrant its own issue. Notably #97 does not fix this problem.

After a decoding error try_finish goes into an infinite loop:

#[test]
fn write_invalid() {
    let contents = b"BZh\xfb";
    let mut decoder = bzip2::write::BzDecoder::new(Vec::new());
    dbg!(decoder.write(&contents[..]));
    dbg!(decoder.try_finish());
}

I'm explicitly calling try_finish here to show the problem. But since the Drop implementation calls try_finish this cannot be avoided by not calling try_finish.

Readme has a bzip2 crate version that isn't published yet

I tried making an example program with bzip2 = "0.4" but it's not published:

$ cargo run
    Updating crates.io index
error: failed to select a version for the requirement `bzip2 = "^0.4"`
  candidate versions found which didn't match: 0.3.3, 0.3.2, 0.3.1, ...
  location searched: crates.io index
required by package `bzip2test v0.1.0 (/home/x/dev/rust/bzip2test)`

Reading Multiple Blocks

I'm encountering a slight issue when using the library: I'm only getting the first block of bz2 files. In my case, the block size is 900k, and when reading off BzDecoder, I get a total of 900k then EOF (size zero reads thereafter).

This happens on both the crates.io release and master here on github. Am I confused here? Would appreciate any suggestions. Thanks!

bzip2-sys build fails on Windows when its path contains non-latin characters

Using up-to-date master of bzip2-rs with i686 MSVC nightly Rust.
When bzip2-rs directory is in C:\Users\Александр\:

Александр@VAIO-FZ13 MINGW64 ~/bzip2-rs (master)
$ pwd
/c/Users/Александр/bzip2-rs

Александр@VAIO-FZ13 MINGW64 ~/bzip2-rs (master)
$ rustc -vV
rustc 1.5.0-nightly (168a23ebe 2015-10-01)
binary: rustc
commit-hash: 168a23ebe1729386138fa71643382fdd64fac205
commit-date: 2015-10-01
host: i686-pc-windows-msvc
release: 1.5.0-nightly

Александр@VAIO-FZ13 MINGW64 ~/bzip2-rs (master)
$ cargo build
   Compiling winapi v0.2.4
   Compiling libc v0.1.10
   Compiling winapi-build v0.1.1
   Compiling advapi32-sys v0.1.2
   Compiling gcc v0.3.17
   Compiling bzip2-sys v0.1.3 (file:///C:/Users/%D0%90%D0%BB%D0%B5%D0%BA%D1%81%D0%B0%D0%BD%D0%B4%D1%80/bzip2-rs)

build script output was not valid utf-8

When I place bzip2-rs in C:\, the build works.

The bzip2-sys crate in crates.io seems not consist with the git version, please fix

This is the build.rs in crates.io with version 0.1.6.

extern crate cc;

use std::env;

fn main() {
    let mut cfg = cc::Build::new();
    cfg.warnings(false);

    if env::var("TARGET").unwrap().contains("windows") {
        cfg.define("_WIN32", None);
        cfg.define("BZ_EXPORT", None);
    }

    cfg.include("bzip2-1.0.6")
       .define("BZ_NO_STDIO", None)
       .file("bzip2-1.0.6/blocksort.c")
       .file("bzip2-1.0.6/huffman.c")
       .file("bzip2-1.0.6/crctable.c")
       .file("bzip2-1.0.6/randtable.c")
       .file("bzip2-1.0.6/compress.c")
       .file("bzip2-1.0.6/decompress.c")
       .file("bzip2-1.0.6/bzlib.c")
       .compile("libbz2.a");
}

because of this, the header bzlib.h does not exist in output dir.

link rust-bio/rust-htslib#115 here.

bzip2-sys: features=["static"]?

Would it be possible to have static feature like libz-sys does?:

https://github.com/rust-lang/libz-sys/blob/master/Cargo.toml#L35

/cc alexcrichton/xz2-rs#62

BzCompressor fails on empty buffer?

The Read implementation of BzCompressor can return an error when the underlying buffer return zero bytes: https://github.com/alexcrichton/bzip2-rs/blob/master/src/reader.rs#L67.

I believe this is now incorrect behaviour, as EOF is now signalled through a zero-byte read instead of an error. That's what I get from the docs, anyway: https://doc.rust-lang.org/std/io/trait.Read.html#tymethod.read

Infinite loop writing an empty buffer in write::BzDecoder

This minimal example is never returning with the current master (3032f37).

extern crate bzip2;

use bzip2::write::BzDecoder;
use std::io::prelude::*;

fn main() {
    let mut d = BzDecoder::new(Vec::new());
    d.write(b"").unwrap();
    d.finish().unwrap();
}

With a similar test case cargo test was stuck:

    #[test]
    fn write_empty() {
        let d = BzDecoder::new(Vec::new());
        let mut c = BzEncoder::new(d, ::Compression::default());
        c.write(b"").unwrap();
        let data = c.finish().unwrap().finish().unwrap();
        assert_eq!(&data[..], b"");
    }

Diving into the simpler example traceback the problem seems related to try_finish in write.rs, where we are writing an empty buffer forever:

    pub fn try_finish(&mut self) -> io::Result<()> {
        while !self.done {
            self.write(&[])?;
        }
        self.dump()
    }

Maybe it is sufficient to check if we received some input? Like:

    pub fn try_finish(&mut self) -> io::Result<()> {
        // If nothing entered, there is no need to loop
        while !self.done && self.total_in() != 0 {
            self.write(&[])?;
        }
        self.dump()
    }

That done cargo test works with the added test too.

Let me know if you'd like a Pull Request, it's ready on my disk.

Shouldnt bzip2-sys have the bzip2 license?

Right now, Cargo.toml of the bzip2-sys crate claims MIT/Apache-2.0 but it contains the entire source code of the bzip2 library.

Not all data is flushed when `.into_inner()` if `.flush()` wasn't called first

It looks like when using a writer::BzCompressor, for certain buffer sizes, not all the output data gets correctly flushed.

I've added a failing test case to my fork here: https://github.com/joshuawarner32/bzip2-rs/blob/7628ede433bfd85059c6bb4f444bafb98c51ebda/src/writer.rs#L220 (that's just master on my fork, if you want to check it out).

I have yet to track that down to a specific problem - but the compressed size (131083 bytes, when flushing first, and 131072 bytes when not) lines up suspiciously with the buffer size in the writer (128*1024 = 131072 bytes). The test-case has been minified (the input started at 10MB), so the close alignment is probably more of a red-herring. I suspect being over the buffer size is the real problem.

I'll look into it more this evening, and maybe open a PR. I just wanted to give you a heads-up, in case you could spot the problem immediately.

Decompression stops silently on byte #900000

I got this using this file: https://database.lichess.org/atomic/lichess_db_atomic_rated_2015-01.pgn.bz2 (I checked at least one other file, and it also stopped on 900000th byte)
The following code:

use std::fs::File;
use std::io;

fn by_bzip2() -> std::io::Result<()> {
    let compressed_file = File::open("lichess_db_atomic_rated_2015-01.pgn.bz2")?;
    let mut decompressed_output = File::create("decoded_by_bzip2.pgn")?;

    let mut reader = bzip2::read::BzDecoder::new(compressed_file);
    io::copy(&mut reader, &mut decompressed_output)?;
    Ok(())
}

fn by_bzip2_rs() -> std::io::Result<()> {
    let compressed_file = File::open("lichess_db_atomic_rated_2015-01.pgn.bz2")?;
    let mut decompressed_output = File::create("decoded_by_bzip2-rs.pgn")?;

    let mut reader = bzip2_rs::DecoderReader::new(compressed_file);
    io::copy(&mut reader, &mut decompressed_output)?;
    Ok(())
}

fn main() {
    by_bzip2().expect("bzip2");
    by_bzip2_rs().expect("bzip2-rs");
}

returns 0 on exit, produces same files decoded_by_bzip2.pgn and decoded_by_bzip2-rs.pgn each of length exactly 900000 bytes, which are a prefix of a much larger file that my system decoder /usr/bin/bzip2 -dk lichess_db_atomic_rated_2015-01.pgn.bz2 produces. So the decoding in both bzip2 and bzip2-rs crates for some reason abruptly stops on this round number 900000 without printing any error message. Which looks like a bug in the crates, not in the file itself, since the file is handled correctly by Linux utilities.

Way to know how many compressed bytes were read when decompressing

This is more for user feedback than anything, but I was looking for a way to know how many bytes of the compressed source had been read, rather than the number of decompressed bytes, so that I can get accurate ETA's.

For example:

let input = File::open(input.expect("Could not get path"))?;
let size = input.metadata()?.len();
let reader = BufReader::new(input);
let mut md = MultiBzDecoder::new(reader);
let mut buffer = [0; BUFFER_LENGTH];
let mut n = md.read(&mut buffer)?;

n appears to me to be how many uncompressed bytes were read, How do I know how far along in the file I've read and how it compares to size?

Failed to run custom build command for `bzip2-sys v0.1.11+1.0.8`

Hello,
I am new to Rust,
I trying to compile the package to webassembly using
wasm-pack build --target web

I am getting the following error:

[INFO]: Checking for the Wasm target...
[INFO]: Compiling to Wasm...
   Compiling bzip2-sys v0.1.11+1.0.8
   Compiling miniz_oxide v0.4.4
   Compiling thiserror-impl v1.0.25
   Compiling regex v1.5.4
error: failed to run custom build command for `bzip2-sys v0.1.11+1.0.8`

Caused by:
  process didn't exit successfully: `/home/arashidesu/wasm-reader/target/release/build/bzip2-sys-33bc4ef2e16072e9/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=BZIP2_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_wasm32-unknown-unknown
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_wasm32_unknown_unknown
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_wasm32-unknown-unknown
  cargo:rerun-if-env-changed=PKG_CONFIG_wasm32_unknown_unknown
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_wasm32-unknown-unknown
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_wasm32_unknown_unknown
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  TARGET = Some("wasm32-unknown-unknown")
  OPT_LEVEL = Some("s")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_wasm32-unknown-unknown = None
  CC_wasm32_unknown_unknown = None
  TARGET_CC = None
  CC = None
  CFLAGS_wasm32-unknown-unknown = None
  CFLAGS_wasm32_unknown_unknown = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  running: "clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "bzip2-1.0.8" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "-o" "/home/arashidesu/wasm-reader/target/wasm32-unknown-unknown/release/build/bzip2-sys-c0980f696d0a153e/out/lib/bzip2-1.0.8/blocksort.o" "-c" "bzip2-1.0.8/blocksort.c"

  --- stderr


  error occurred: Failed to find tool. Is `clang` installed?


warning: build failed, waiting for other jobs to finish...
error: build failed
Error: Compiling your crate to WebAssembly failed
Caused by: failed to execute `cargo build`: exited with exit status: 101
  full command: "cargo" "build" "--lib" "--release" "--target" "wasm32-unknown-unknown"

I have gcc (GCC) with version 11.1.1 2021053 installed.
Thanks for your time!

Re-implement in pure Rust

Are you interested in re-implementing bzip2 in pure Rust or do you intend to keep it as a binding? (i.e. will you accept such work if anyone will be interested in it or not?) bzip2-sys implementation could be kept as a feature-gated disabled by default option.

Potential rewrite will help with the licensing issue and will help crates which target WASM.

bzip2::Compression type does not allow arbitrary values from 1-9

The bzip2::Compression type only allows the options Fastest (1), Default (6), or Best (9). There seems to be no way to specify a compression level of, say, 4.

Perhaps it would be better to make bzip2::Compression work like flate2::Compression (which does allow arbitrary numbers), since the two crates already have very similar APIs otherwise?

Bzip2-sys does not support the value 0 for blockSize100k

We are directly passing Compression::level to BZ2_bzCompressInit

bzip2-rs/src/mem.rs

Lines 123 to 126 in 016e181

 assert_eq!( 

 ffi::BZ2_bzCompressInit(&mut *raw, lvl.level() as c_int, 0, work_factor as c_int), 

 0 

 );

However, this API only supports values from 1..=9 as documented here in the sources

Parameter blockSize100k specifies the block size to be used for compression. It should be a value between 1 and 9 inclusive, and the actual block size used is 100000 x this figure. 9 gives the best compression but takes most memory.

This bug seems to have been introduced in #37, where we tried to support an API aligned with flate2::Compression. However, our version of Compression::none will always panic on the above assertion with BZ_PARAM_ERROR

bzip2-rs/src/lib.rs

Lines 100 to 102 in 016e181

 pub fn none() -> Compression { 

 Compression(0) 

 }

References

We had a panic in usage of zip over here :) zip-rs/zip-old#326

Suggestions

As the library doesnt appear to support disabling compression, I would prefer none to be deprecated, and a bounds check to be added to Compression::new. Hopefully there's a clean way to provide the equivalent of flate2's none and we can specifically branch on 0?

Doesn't seem to have a "raw" module as mentioned in documentation / code comments

Places such as this comment in the lib.rs source mention a "raw" module:

https://docs.rs/bzip2/0.4.1/src/bzip2/lib.rs.html#9

//! Access to the raw decompression/compression stream is also provided through
//! the raw module which has a much closer interface to libbz2.

But such raw module does not seem to actually exist. If it does and I can't see it because I'm a newbie, it would be good to include an example of how to use it.

Yank breaking versions with "*" deps

Some old versions of bzip2 have dependencies specified with "*" version requirement (bzip2-sys = "*").

Because these old versions of bzip2 are unusable in practice, and cause problems for the minimal-versions Cargo feature, I suggest yanking them:

cargo yank --vers 0.0.1
cargo yank --vers 0.0.2
cargo yank --vers 0.0.3

bzip2-sys on Crates.io still uses libc 0.1.x

bzip2-sys on Crates.io still uses libc ^0.1

I think you just need to bump the version and push a new package, since the version on github is already on libc ^0.2 .

	assert_eq!(
	ffi::BZ2_bzCompressInit(&mut *raw, lvl.level() as c_int, 0, work_factor as c_int),
	0
	);