The xz utils upstream git has a solution/vcxproj file for VS2017 now.

Hi,

I ran into a problem when trying to use the xz2 crate with crt-static under windows.
The minimal test case is a crate which depends on xz2 = "*", and has a .cargo/config file with:

[target.'cfg(all(target_arch="x86_64", target_os="windows"))']
rustflags = [
    "-Ctarget-cpu=x86-64",
    "-Ctarget-feature=+crt-static",
]

It will fail to compile with

error LNK2001: unresolved external symbol __imp__wassert
...
fatal error LNK1120: 1 unresolved externals

(aka /MD is passed to the compiler and not /MT as expected)

It's a subtly different case than using RUSTFLAGS (which is already tested),
I think because cc::Build looks at CARGO_CFG_TARGET_FEATURE
(which in my tests seems to be set to fxsr,mmx,sse,sse2).

I forked and played with a solution which works for me (using #[cfg(target_feature = "crt-static")]),
but I wasn't sure if the issue is here, in cc, in cargo, etc..
https://github.com/alexcrichton/xz2-rs/compare/master...ohadravid:bugfix/static-crt-build-fail-when-using-dot-cargo-config?expand=1

The only example I can find is the one on the main docs page, showing outputting to a string.

The xz format supports block level random access. The XzDecoder should be able to implement Seek to support it. I'd like this feature so that we could seek in a xz-compressed pcap file. Are you open to including the feature if I implement it?

I'm back with the same error as before but this time with a bit more information:

• First the code is still valid and what I said last time about the buffer size, does NOT fix the problem it HIDES the problem.
• From the look of it the problem is with process_vec. The online C example of XZ does not have this defect.
• I identfied a way to reproduce the bug: if you must call process_vec more than 3 times in the same Run action you crash it. The next time you call it with more data will be returned with the great "lzma data error".
• This means that if you manage to find a large enough output buffer size (with_capacity) for process_vec to use and return the entire decompressed data in exatcly 3 calls, where the first returns the data the second returns 0 and the thrid returns MemNeeded, then the software will continue on the next buffer of compressed data.
• If the exact pattern stated previously is not respected (for example the output vec is not large enough) then your software will inevitably crash at the next buffer.

Why is that? Is there a way to prevent that issue?

While I'm working on #39 I found a bit annoying that the code hasn't been formatted using rustfmt.

I can of course either

• disable rustfmt when working on this codebase and manually format new code, or
• write code with rustfmt but check in only non-stylistic changes

but neither is satisfactory. Are you happy to accept rustfmt changes? If so I'll make a PR.

I was confused by a difference in versions between the repo and the crates.io/docs.rs docs.

The repo README recommends using "0.1" and current version on master is "0.1.5". However the docs.rs README lists version "0.3" in the section on asyinc IO/Tokio.

Probably just an oversight but figured I would flag it.

I was looking at the time crate wondering why the time crate didn't give offsets, and I stumbled upon this thread on Internals. This seems like a big problem so I went looking for FFI crates where this could be a problem.

liblzma uses getenv https://github.com/kobolabs/liblzma/blob/87b7682ce4b1c849504e2b3641cebaad62aaef87/lib/getopt.c#L250, it looks like this wrapper has a data race when set_env is called in another thread.

When compressing a large stream of data after about 6 GB of output file, I got this error on Windows 10.

Assertion failed: memcmp(mf_ptr(mf) - 1, mf_ptr(mf) - matches[i].dist - 2, matches[i].len) == 0, file xz-5.2/src/liblzma/lz\lz_encoder_mf.c, line 42

https://github.com/alexcrichton/xz2-rs/blob/1a82c40d6d80171b7df328aea43b7054acd10c44/src/bufread.rs#L240C1-L241C59

I'm trying to figure out how to use XzDecoder with an async stream of bytes like you get from reqwest::Response::bytes_stream(). But I don't see how to transform the stream into something that implements BufRead. I don't really understand why BufRead is necessary in an async context either, where I would rather expect AsyncBufRead.

This is the closest I get:

    let stream = reqwest::get(url)
        .await?
        .error_for_status()?
        .bytes_stream()
        .map_err(|e| io::Error::new(io::ErrorKind::Other, e))
        .into_async_read();
    let decoder = XzDecoder::new(tokio::io::BufReader::new(stream));

But tokio::io::BufReader does not implement BufRead.

Using xz2 in a FreeBSD cross-build environment results in the following error:
https://travis-ci.org/rust-lang-nursery/rustup.rs/jobs/231443067#L1088

The build config is missing some defines:

#define TUKLIB_CPUCORES_SCHED_GETAFFINITY 1
#define TUKLIB_FAST_UNALIGNED_ACCESS 1

I got those by speaking with a LZMA dev.

Currently the first define allows use of lzma_cputhreads and also allows for lzma_physmem. The second define improves performance.

It would be great if there could be a tag, and a crates.io release, with the fix for #9, from May, please!

Cheers.

Requires 5.3.x release of xz: https://github.com/xz-mirror/xz/blob/e970c28ac3cb2e8051925f81db2fe953664c2645/NEWS#L103

         - Added threaded .xz decompressor lzma_stream_decoder_mt().
          It can use multiple threads with .xz files that have multiple
          Blocks with size information in Block Headers. The threaded
          encoder in xz has always created such files.

This MR already bumps the xz version: #108

Hi,

Recently we were trying to do a Rustup release and encountered an issue where lzma-sys 0.1.13 wouldn't build on our Appveyor mingw builders. You can find a log here: https://ci.appveyor.com/project/rust-lang-libs/rustup-rs/builds/22849436/job/d3upd7t46kmp4154

Reverting to 0.1.10 allowed our builds to proceed.

I am not necessarily in a position to bisect since I don't have a windows/mingw environment to hand, but if there's some way I can assist anyhow, please let me know and I'll do my best.

D.

Builds targeting WebAssemby using the WASI-SDK are now supported in liblzma 5.4.4.
https://github.com/tukaani-project/xz/releases/tag/v5.4.4
(I patched the build to work with the xz2 WebAssemby target).

This allows builds targeting wasm32-unknown-unknown and wasm32-wasi on xz2.

However, several changes have to be made to the current xz2.

1. Update liblzma version to 5.4.4 (currently xz2 uses liblzma 5.2.5)
2. Enable/disable multithreading-related features in xz2 (with liblzma 5.4.4, multithreading features must be disabled when targeting WebAseembly).
3. (Optional) Add arm64 filters (new filters have been added in liblzma 5.4 and need to be supported to pass tests against them).

There are several options for how to make the change, possible options are.

• 1. Add new feature flag (like xz-54) to enable liblzma 5.4 with continued support for the current liblzma 5.2
• 2. Switch liblzma version to 5.4

• 1. Add new feature flag (like threading) to enable/disable the multithreading feature
• 2. Disable multithreading related features when building against a WebAssembly target using #[cfg(target_arch = "wasm32")]

• 1. Add a macro definition in config.h to enable arm64 filters
• 2. Disable tests for related it

I will work on making xz2 buildable with WebAssembly as a target, as long as you indicate your policy for handling it!

I see that you have been doing some WebAssembly-related activities recently, so please respond if you are interested!

The following code:

use xz2::stream::{Filters, LzmaOptions, MtStreamBuilder};

fn main() {
    let dict_size = 0x40000;

    let mut opts = LzmaOptions::new_preset(6).unwrap();
    opts.dict_size(dict_size);

    let mut filters = Filters::new();

    filters.ia64();
    filters.arm();
    filters.arm_thumb();
    filters.lzma2(&opts);


    let stream = MtStreamBuilder::new()
        //.block_size(0x1000)
        .filters(filters)
        .check(xz2::stream::Check::Crc32)
        .encoder()
        .unwrap();
}

causes an invalid memory reference on some arches, including arm-unknown-linux-musleabi.

$ gdb ./target/x86_64-unknown-linux-musl/debug/xz-issue
GNU gdb (GDB) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./target/x86_64-unknown-linux-musl/debug/xz-issue...
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/wcampbell/projects/wcampbell/xz-issue/target/x86_64-unknown-linux-musl/debug/xz-issue.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) r
Starting program: /home/wcampbell/projects/wcampbell/xz-issue/target/x86_64-unknown-linux-musl/debug/xz-issue

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7f8a2f3 in lzma_mt_block_size (filters=0x7ffff7f638e0) at xz-5.2/src/liblzma/common/filter_encoder.c:237
237            if (fe->block_size != NULL) {

I pushed some code here that shows the issue: https://github.com/wcampbell0x2a/xz2-issue

Similar to how xz can produce:

xz -l data.xz 
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1     164    294.5 MiB  9,155.0 MiB  0.032  CRC64   data.xz

The decompressed size and ratio would be useful to me, is there any way of exposing this data?

let mut strm = Stream::new_easy_encoder(9, Check::Sha256).unwrap();
let rawbuf = read_n(file, to); //Read `to (u64)` bytes from `file` and store in `rawbuf (Vec<u8>)`
let mut retvec: Vec<u8> = Vec::new();
strm.process_vec(rawbuf.as_slice(), &mut retvec, Action::SyncFlush).unwrap();

In the code above, retvec contains no data even though rawbuf does.

I am working on porting libchdr to Rust, which relies on some legacy behaviour of the LZMA SDK to work. I am stuck on trying to set decoder properties. There are equivalent calls in xz, but it seems not even lzma-sys exposes headers from xz/liblzma.

I noticed a tricky case that may be related to #13: when a file contains several streams (for example if it was created with pxz, a parallel compressor).
From what I understand, to handle such a case, it is possible to do the following:

• open the file
• loop:
  • keep the current offset in the file (let call it offset_before)
  • create a XzDecoder for the current xz stream
  • loop to decompress this stream by calling read
  • if it ends with an error and if some bytes were read for this stream, then try to move to the start of the next stream by seeking in the file to offset_before + total_in() and skipping all the null bytes (optional stream padding). If there is an error but no byte was read, then it is really an error and we stop everything here.

I think it would be better if the case of the multi-stream files was transparently handled by XzDecoder.
edit: In fact, now I think it would be mandatory for non-seekable files (like Unix pipes)

No matter what I try and no matter what I do the decoder always throws "lzma data error". Sample code:

const READ_BLOCK_SIZE: usize = 8192;

fn block_based_inflate2(input: &mut dyn Read, output: &mut dyn Write) -> io::Result<()>
{
    let mut decoder = match Stream::new_stream_decoder(u32::MAX, xz::stream::CONCATENATED)
    {
        Err(e) => return Err(io::Error::new(io::ErrorKind::InvalidData, format!("[BPX] inflate error: {}", e))),
        Ok(v) => v
    };
    let mut idata: [u8; READ_BLOCK_SIZE] = [0; READ_BLOCK_SIZE];
    let mut status = xz::stream::Status::MemNeeded;
    let mut res = input.read(&mut idata)?;

    while res > 0
    {
        println!("{}", res);
        let mut odata: Vec<u8> = Vec::with_capacity(READ_BLOCK_SIZE * 2);
        match decoder.process_vec(&idata[0..res], &mut odata, xz::stream::Action::Run)
        {
            Ok(_) => (),
            Err(e) => return Err(io::Error::new(io::ErrorKind::InvalidData, format!("[BPX] inflate error: {}", e)))
        }
        output.write(&odata)?;
        res = input.read(&mut idata)?;
    }
    while status != xz::stream::Status::StreamEnd
    {
        let mut odata: Vec<u8> = Vec::with_capacity(READ_BLOCK_SIZE * 2);
        match decoder.process_vec(&idata[0..0], &mut odata, xz::stream::Action::Finish)
        {
            Ok(s) => status = s,
            Err(e) => return Err(io::Error::new(io::ErrorKind::InvalidData, format!("[BPX] inflate error: {}", e)))
        }
        output.write(&odata)?;
    }
    return Ok(());
}

fn main() -> io::Result<()>
{
    let mut out = std::fs::File::create(Path::new("./file.jpg"))?;
    let mut i = std::fs::File::open(Path::new("./file.jpg.xz"))?;
    block_based_inflate2(&mut i, &mut out)?;
    return Ok(());
}

Just compress a jpg file using xz --compress file.jpg then run the program. No matter what Jpeg you give it's unreadable!! You can however decompress it with xz.

Why is the decoder broken?

EDIT: I found a way to fix it however I found something really strange in return: if you give the same buffer size for the output data as for the input you get "lzma data error". The size of the output buffer must be size of input buffer * 2.

In my tests, no matter what size I gave it failed with "lzma data error" unless I give it size of input buffer * 2. Is there any reason for that? Looking at LZMA example it should not be needed.

Tring to open build directly shows error: Build not found.

The workflow main.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

When building for -msvc with the VC++ 2015 build tools, the build fails:

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.Cpp.Platform.targets(57,5): error MSB8020: The build tools for v120 (Platform Toolset = 'v120') cannot be found. To build using the v120 build tools, please install v120 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution". [C:\Users\diggs\Projects\rustup-rs\target\debug\build\lzma-sys-8de572d4e0ca373d\out\build\windows\liblzma.vcxproj]

The project should probably not be tied to a specific platform toolset version.

Is is possible to decompress .lz (lzip) files?

Test code

fn main() {
    let test_xz = File::open("dump.xz").unwrap();
    let mut output = Vec::new();
    xz2::read::XzDecoder::new(&test_xz).read_to_end(&mut output).unwrap();
}

Output of above program

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidInput, error: Options }'

Output of xz -l dump.xz

Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1       1    683.7 KiB  2,048.0 KiB  0.334  None    dump.xz

I am able to extract the xz file using unxz in linux but not with XzDecoder

Link to file (please extract the dump.zip to get xz file)
dump.zip

Work around is to do, e.g., for rustc:

echo 'fn main() { println!("cargo:rustc-link-lib=lzma"); }' > src/vendor/lzma-sys/build.rs

I'm very new to rust, trying to build the AVR Rust fork, and it's failing when building lzma-sys.

I think the issue is with this package; if not, could you point me in the right direction? I'm following the instructions to build the compiler and it's failing here:

$ make
...
error: could not find native static library `lzma`, perhaps an -L flag is missing?

error: Could not compile `lzma-sys`.
warning: build failed, waiting for other jobs to finish...
error: build failed
...

In OpenSUSE, the lib dir defaults to lib64. If I make a symlink, the build can proceed:

$ cd ./build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/build/lzma-sys-b5e13e4c4ffea1be/out
$ ls -l
total 4
drwxr-xr-x 1 alexh users 160 Jan  8 11:40 build
drwxr-xr-x 1 alexh users  20 Jan  8 11:40 include
drwxr-xr-x 1 alexh users  56 Jan  8 11:40 lib64
drwxr-xr-x 1 alexh users 526 Jan  8 11:40 src
$ ln -s lib64 lib
$ cd -
$ make
...
Compiling lzma-sys v0.1.8
...

xz2 cannot extract the xz file for the linux source code

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.14.1.tar.xz                 
$ sha256sum linux-5.14.1.tar.xz
005bc52a847b7a372a687d536847adb1e1fe0ee571817008632c6e1706d008c7  linux-5.14.1.tar.xz

use std::io::prelude::*;

fn main() {
    let input = std::fs::read("linux-5.14.1.tar.xz").unwrap();
    let mut output = Vec::new();
    xz::read::XzDecoder::new(&*input)
        .read_to_end(&mut output)
        .unwrap();
}

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `/home/nixon/git/nixos/target/debug/why`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "corrupt xz stream" }', why/src/main.rs:8:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

$ unxz linux-5.14.1.tar.xz
$ sha256sum linux-5.14.1.tar
f0978dd271f97e6c00c30deffbd1eb0c4d05c3b5e959a67bbfb4bd280817635d  linux-5.14.1.tar

Hello Team,

by default lzma-sys was compiled in a dynamic way on MacOS (static on Linux), any idea how to do it in a static way on MacOS? Or it is just not supported to build statically on MacOS (this happens a lot for other libraries).

Thanks,

Jianshu

Unless I'm missing something shouldn't this be "xz2" instead?

While working on reproducible builds for openSUSE, I found that our rustup package binaries vary from filesystem readdir order.

Probably from the read_dir in https://github.com/alexcrichton/xz2-rs/blob/1e7725be/lzma-sys/build.rs#L77

I found gyscos/zstd-rs#162 resolves a similar issue in another lib.

Please ensure consistent source file ordering.

The handcrafted build system should make sure the right lzma.h is included.

Hello,

I am not able to compile lzma-sys on my Windows machine. The build script fails with the following error:

error occurred: Command "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "xz-5.2/src/liblzma/api" "-I" "xz-5.2/src/liblzma/lzma" "-I" "xz-5.2/src/liblzma/lz" "-I" "xz-5.2/src/liblzma/check" "-I" "xz-5.2/src/liblzma/simple" "-I" "xz-5.2/src/liblzma/delta" "-I" "xz-5.2/src/liblzma/common" "-I" "xz-5.2/src/liblzma/rangecoder" "-I" "xz-5.2/src/common" "-I" "C:\\Users\\Elève\\Desktop\\rust\\xz2-rs\\lzma-sys" "-W4" "-DHAVE_CONFIG_H=1" "-FoC:\\Users\\Elève\\Desktop\\rust\\xz2-rs\\target\\debug\\build\\lzma-sys-84c50c16c770253c\\out\\xz-5.2/src/liblzma/common\\alone_decoder.o" "-c" "xz-5.2/src/liblzma/common\\alone_decoder.c" with args "cl.exe" did not execute successfully (status code exit code: 2).

error: failed to run custom build command for `lzma-sys v0.1.19 (C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys)`

Caused by:
  process didn't exit successfully: `C:\Users\Elève\Desktop\rust\xz2-rs\target\debug\build\lzma-sys-bedfe75a8fb1aecd\build-script-build` (exit code: 1)
  --- stdout
  cargo:rerun-if-changed=build.rs
  cargo:rerun-if-env-changed=LZMA_API_STATIC
  cargo:root=C:\Users\Elève\Desktop\rust\xz2-rs\target\debug\build\lzma-sys-84c50c16c770253c\out
  cargo:include=C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2/src/liblzma/api
  TARGET = Some("x86_64-pc-windows-msvc")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-pc-windows-msvc")
  CC_x86_64-pc-windows-msvc = None
  CC_x86_64_pc_windows_msvc = None
  HOST_CC = None
  CC = None
  CFLAGS_x86_64-pc-windows-msvc = None
  CFLAGS_x86_64_pc_windows_msvc = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("aes,fxsr,pclmulqdq,popcnt,rdrand,rdseed,sha,sse,sse2,sse3,sse4.1,sse4.2,ssse3")
  DEBUG = Some("true")
  running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "xz-5.2/src/liblzma/api" "-I" "xz-5.2/src/liblzma/lzma" "-I" "xz-5.2/src/liblzma/lz" "-I" "xz-5.2/src/liblzma/check" "-I" "xz-5.2/src/liblzma/simple" "-I" "xz-5.2/src/liblzma/delta" "-I" "xz-5.2/src/liblzma/common" "-I" "xz-5.2/src/liblzma/rangecoder" "-I" "xz-5.2/src/common" "-I" "C:\\Users\\Elève\\Desktop\\rust\\xz2-rs\\lzma-sys" "-W4" "-DHAVE_CONFIG_H=1" "-FoC:\\Users\\Elève\\Desktop\\rust\\xz2-rs\\target\\debug\\build\\lzma-sys-84c50c16c770253c\\out\\xz-5.2/src/liblzma/common\\alone_decoder.o" "-c" "xz-5.2/src/liblzma/common\\alone_decoder.c"
  alone_decoder.c
  C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\synchapi.h(1): warning C4821: Unable to determine Unicode encoding type, please save the file with signature (BOM)
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(349): error C2061: syntax error: identifier 'mythread_cond'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(349): error C2059: syntax error: ';'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(402): warning C4013: 'WaitForSingleObject' undefined; assuming extern returning int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(415): warning C4013: 'InitializeCriticalSection' undefined; assuming extern returning int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(422): warning C4013: 'DeleteCriticalSection' undefined; assuming extern returning int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(428): warning C4013: 'EnterCriticalSection' undefined; assuming extern returning int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(434): warning C4013: 'LeaveCriticalSection' undefined; assuming extern returning int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(439): error C2143: syntax error: missing ')' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(439): error C2143: syntax error: missing '{' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(439): error C2059: syntax error: ')'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(440): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(440): error C2054: expected '(' to follow 'cond'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(451): error C2143: syntax error: missing ')' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(451): error C2143: syntax error: missing '{' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(451): error C2059: syntax error: ')'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(452): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(452): error C2054: expected '(' to follow 'cond'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(461): error C2143: syntax error: missing ')' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(461): error C2143: syntax error: missing '{' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(461): error C2059: syntax error: ')'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(462): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(462): error C2054: expected '(' to follow 'cond'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): error C2143: syntax error: missing ')' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): error C2143: syntax error: missing '{' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): error C2371: 'mythread_mutex': redefinition; different basic types
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(344): note: see declaration of 'mythread_mutex'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): error C2143: syntax error: missing ';' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): warning C4218: nonstandard extension used: must specify at least a storage class or a type
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(471): error C2059: syntax error: ')'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(472): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(472): error C2054: expected '(' to follow 'mutex'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): error C2143: syntax error: missing ')' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): error C2143: syntax error: missing '{' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): error C2371: 'mythread_mutex': redefinition; different basic types
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(344): note: see declaration of 'mythread_mutex'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): error C2143: syntax error: missing ';' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): warning C4218: nonstandard extension used: must specify at least a storage class or a type
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(485): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(486): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(486): error C2371: 'mythread_condtime': redefinition; different basic types
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(362): note: see declaration of 'mythread_condtime'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(486): warning C4228: nonstandard extension used: qualifiers after comma in declarator list are ignored
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(486): error C2143: syntax error: missing ';' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(486): warning C4218: nonstandard extension used: must specify at least a storage class or a type
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(486): error C2059: syntax error: ')'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(487): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(487): error C2054: expected '(' to follow 'condtime'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(511): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(511): error C2143: syntax error: missing ')' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(511): error C2143: syntax error: missing '{' before '*'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(511): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(512): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(512): warning C4142: 'uint32_t': benign redefinition of type
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.33.31629\include\stdint.h(24): note: see declaration of 'uint32_t'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(512): error C2370: 'uint32_t': redefinition; different storage class
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.33.31629\include\stdint.h(24): note: see declaration of 'uint32_t'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(512): error C2146: syntax error: missing ';' before identifier 'timeout'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(512): warning C4218: nonstandard extension used: must specify at least a storage class or a type
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(512): error C2059: syntax error: ')'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(513): warning C4431: missing type specifier - int assumed. Note: C no longer supports default-int
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\common\mythread.h(513): error C2054: expected '(' to follow 'timeout'
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\liblzma\lz\lz_decoder.h(132): warning C4267: 'initializing': conversion from 'size_t' to 'uint32_t', possible loss of data
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\liblzma\lz\lz_decoder.h(159): warning C4267: 'initializing': conversion from 'size_t' to 'const uint32_t', possible loss of data
  C:\Users\Elève\Desktop\rust\xz2-rs\lzma-sys\xz-5.2\src\liblzma\lz\lz_decoder.h(160): warning C4267: 'initializing': conversion from 'size_t' to 'uint32_t', possible loss of data
  exit code: 2

  --- stderr


  error occurred: Command "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.33.31629\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-I" "xz-5.2/src/liblzma/api" "-I" "xz-5.2/src/liblzma/lzma" "-I" "xz-5.2/src/liblzma/lz" "-I" "xz-5.2/src/liblzma/check" "-I" "xz-5.2/src/liblzma/simple" "-I" "xz-5.2/src/liblzma/delta" "-I" "xz-5.2/src/liblzma/common" "-I" "xz-5.2/src/liblzma/rangecoder" "-I" "xz-5.2/src/common" "-I" "C:\\Users\\Elève\\Desktop\\rust\\xz2-rs\\lzma-sys" "-W4" "-DHAVE_CONFIG_H=1" "-FoC:\\Users\\Elève\\Desktop\\rust\\xz2-rs\\target\\debug\\build\\lzma-sys-84c50c16c770253c\\out\\xz-5.2/src/liblzma/common\\alone_decoder.o" "-c" "xz-5.2/src/liblzma/common\\alone_decoder.c" with args "cl.exe" did not execute successfully (status code exit code: 2).

Can anyone help me compile ?

With reference to the code snippet below:

• The codec may consume some of the input buffer, but run out of space on the output buffer and return LZMA_OK, with the next_in pointing a bit further along.
• LZMA assumes here that the caller can flush the output buffer and call again with the same input, where next_in allows the codec to skip the already-read input.
• The above is repeated until LZMA_STREAM_END (action for an encoder, return value for a decoder).
• In the code below, if the above situation occurs, the caller will never know that only some of their input buffer has been consumed; the {next, avail}_in state is never passed out. The code below inherently assumes that all of the input bytes are consumed.

As a result, sometimes the encoded LZMA stream will get corrupted; or the decoder will behave strangely.

Perhaps the simplest fix would be to return the number of input bytes consumed, so the caller can track it and call again with appropriate slices. For the process_vec variant, it might be a good idea to add a parameter that specifies an input offset that is factored into the slice that gets created when calling process_vec.

[stream.rs]
pub fn process(&mut self,
                   input: &[u8],
                   output: &mut [u8],
                   action: Action) -> Result<Status, Error> {
        self.raw.next_in = input.as_ptr();
        self.raw.avail_in = input.len();
        self.raw.next_out = output.as_mut_ptr();
        self.raw.avail_out = output.len();
        let action = action as lzma_sys::lzma_action;
        unsafe {
            cvt(lzma_sys::lzma_code(&mut self.raw, action))
        }
    }

When calling xz2::stream::Stream.process, an undersized output buffer will sometimes return Err(Program) [specifically returning error code 11 from the C call] instead of Ok(MemNeeded). This behavior is not mirrored in the bound C xz code.

The code below (compiled on rustc 1.71.0, platform using Linux kernel 6.4.3-arch1-1) shows this behavior. The compressed size is exactly 60 bytes.

use std::io::Read;

use xz2::{
    read::XzEncoder,
    stream::{Action, Check, Error, Status, Stream},
};

fn decompress(arr: &mut [u8]) -> Result<Status, Error> {
    let mut encoder = Stream::new_easy_encoder(6, Check::Crc64)?;
    let string = "One".as_bytes();

    loop {
        let res = encoder.process(string, arr, Action::Finish);
        if res != Ok(Status::Ok) {
            return res;
        }
    }
}

fn main() 
{
    for x in 0..64 {
        let mut arr = vec![0; x];
        println!("{x}: {:?}", decompress(&mut arr));
    }
}

Size zero returns Ok(MemNeeded) and every size between 1-59 returns Err(Program). By contrast, I've attached a modified version of 01_compress_easy.c from xz/doc/examples at tag 2327a46. It returns LZMA_BUF_ERROR for sizes 0-59 as expected.

The C code compiled on the same platform with GCC 13.1.1. Recreate my results with mv break_xz_utils.txt break_xz_utils.c && gcc break_xz_utils.c -llzma && ./a.out on a UNIX platform with gcc.

C code: break_xz_utils.txt
C output: c_output.txt
Rust output: rust_output.txt

There's some example bad files in the lzma testsuite, which is included in lzma_sys, presumably by coincedence. One of these tests causes the library to hang / burn CPU forever. This happens even in the simplest program I could come up with:

extern crate xz2;

use std::io;

fn main() {
    let stdin = io::stdin();
    let stdin = stdin.lock();
    let stdout = io::stdout();
    let mut stdout = stdout.lock();

    io::copy(
        &mut xz2::bufread::XzDecoder::new(stdin),
        &mut stdout).unwrap();
}

[package]
name = "xz2check"
version = "0.1.0"
authors = [""]

[dependencies]
xz2 = "0.1"

% xxd bad-0cat-alone.xz  
00000000: fd37 7a58 5a00 0001 6922 de36 0000 0000  .7zXZ...i".6....
00000010: 1cdf 4421 9042 990d 0100 0000 0001 595a  ..D!.B........YZ
00000020: 5d00 0001 00ff ffff ffff ffff ff00 83ff  ]...............
00000030: fbff ffc0 0000 00                        .......

% <bad-0cat-alone.xz cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/xz2check`

... runs forever.

Same on nightly and stable.

rustc 1.19.0-nightly (5b13bff52 2017-05-23)

I'm currently using this library for my squashfs library and i'm trying to let the user build this library statically: wcampbell0x2a/backhand@master...add-xz-gzip-features

This library works great when building without the static features. However, when using this features the library gives different results breaking my tests and library when decompressing.

The first is an example of the beginning of the buffers being sent into the library:

extracting data @ offset 6a
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, ea, fe, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 9d, 83, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, d4, 84, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, d2, a4, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 9b, d9, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, e7, 9e, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 04, c1, cd, ff, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, e1, 98, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, b8, 9b, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, ee, 9b, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, e4, be, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 87, 88, 02, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, ac, aa, 02, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 86, fd, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, b0, a2, 02, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, c6, fc, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, f2, 94, 02, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, d4, f8, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, e1, b6, 02, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, c8, 82, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 84, b7, 02, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, a2, 98, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, f6, 6d, 80, 80, 08, 21]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 97, ad, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 83, a2, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, bf, b5, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, c6, 8e, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, b1, d7, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, fe, bb, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, f6, 91, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 8e, 7b, 80, 80, 08, 21]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 85, 35, 80, 80, 08, 21]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, a5, 1a, 80, 80, 08, 21]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, dc, 37, 80, 80, 08, 21]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, d1, 25, 80, 80, 08, 21]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, f3, e0, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, a4, c4, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, d3, 29, 90, 81, 01, 21]
after
data bytes: 4a4090

extracting data @ offset 6a
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, ea, fe, 01, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 9d, 83, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, d4, 84, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, d2, a4, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, 9b, d9, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 03, c0, e7, 9e, 03, 80, 80, 08]
after
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 04, c1, cd, ff, 01, 80, 80, 08]
thread 'test_08' panicked at 'called `Result::unwrap()` on an `Err` value: StdIo(Custom { kind: InvalidInput, error: Options })', tests/test.rs:35:75
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I remind you, both examples are extracting the same file. All I see is that the following byte is different:

                                                                                           \/
before [fd, 37, 7a, 58, 5a, 00, 00, 01, 69, 22, de, 36, 04, c1, cd, ff, 01, 80, 80, 08]

Any help would be appreciated.

When reading a file with multiple streams, it is necessary to use new_multi_decoder (introduced in #31). Using XzDecoder::new on a multi-stream will result in an error like below:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "corrupt xz stream" }', src/main.rs:13:14

This message could be improved. Perhaps it should read "Corrupt xz stream, or multi-stream. Use new_multi_decoder if this is a multi-stream".

Additionally, is there a way to determine if something is a multi-stream before creating a decoder?

Would it be possible to have static feature for this lib like libz-sys does? Same idea as alexcrichton/bzip2-rs#56 ... working on a project that requires all MUSL deps to be static :-S

/cc alexcrichton/bzip2-rs#56

Hi @alexcrichton! I'm Hyeon Kim, programmer from South Korea.

5 years ago, I created xz and xz-sys crate. Or more precisely, I just allocated the names and forgot updating. At the time, Rust didn't have LLVM bindings at all. I tried to make my own bindings as needed, but because I ran out of time, I completed only xz-sys part and stopped working on xz to do something more urgent.

And five years passed, completely forgetting that I stopped working on xz. Today I looked up the LLVM bindings again as needed, and found that packages named xz2 and llvm-sys were created.

It's a shame that what I forgot has confused people. There were people who were confused about which of the two libraries to use.

Transfer ownership of the "xz" crate and rename this library to xz seems to be a good way in many ways. What do you think?

lzma-rs crate has evolved to the point where it seems to be a viable replacement for liblzma, at least for decompression.

Would you be interested in this crate having a Rust backend based on lzma-rs, akin to how flate2 provides several backends? Or is it out of scope for this crate?

It has been a few... years since xz2 0.1.6. During that time, some dependencies were updated and the static crate feature was added.

The latter is interesting to me because I inadvertently slipped a reference to a /usr/local/opt/xz/lib/liblzma.5.dylib in some macOS Rust binaries I built in GitHub Actions. I can presumably work around this by forcing the linker to use the liblzma inside the macOS SDK. But it sure is tempting to lean on a static crate feature and eliminate the possibility of the shared library dependency.

Are there any plans for a 0.1.7 release? Is there anything I could do to help with one?

Hi! I'm currently working on porting a C program using liblzma over to Rust, and I'm using this library as a substitute for raw unsafe bindings, so thank you for your work on it.

Unfortunately I've encountered two issues thus far - neither of which are major, but make the porting process more difficult than it needs to be.

The first is simply that liblzma has LZMA_VLI_UNKNOWN, but I can't find a related value or setting in xz2. I suspect this is just a lack of understanding on my part or something I've missed, but I'd appreciate some clarification as to where I can set the Variable-Length Integer setting to Unknown in this lib. I haven't seen any VLI options, but I also haven't seen any documentation on the matter.

The second is that the program I'm working with basically just initializes the LZMA options with a preset value (0-9) and after loading the preset, queries the dictionary size on said options for use in other things (since this is set within liblzma by the preset itself, the only access I have to dict_size is this way).

Would it be possible and/or reasonable to expose option values through xz2 in case of situations such as these? The library has the dict_size method already, so I suspect it shouldn't be unreasonable to expect access to a variable one can set. My only alternative as a user of the library is to create some kind of reference table mapping preset to dict size, but that's clunky and leaves room for future bugs on LZMA version changes.

If this is a simple enough change I would be open to doing a PR for it myself, but I wanted to ask first in case there's something I'm missing or a specific reason why it isn't included.

Thank you again for your work on the library and your time,
Z

You've managed to break swift applications using lama-sys from rust: software now refuses to link with a lot of "undefined symbols"! Adding lzma.tbd does not work and result in "Undefined symbol: _lzma_stream_encoder_mt"!

I am having an issue building lzma-sys on Linux for x86_64-unknown-linux-gnu using mingw32. I can build other projects that don't required xz2-rs fine, so I assumed that I was missing a library or devel but I can't seem to get it working.

error: failed to run custom build command for `lzma-sys v0.1.12`
process didn't exit successfully: `/target/release/build/lzma-sys-713ecaa47a182e9a/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-env-changed=LZMA_API_STATIC
cargo:include=/cargo/registry/src/github.com-1ecc6299db9ec823/lzma-sys-0.1.12/xz-5.2/src/liblzma/api
TARGET = Some("x86_64-pc-windows-gnu")
OPT_LEVEL = Some("3")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-pc-windows-gnu = None
CC_x86_64_pc_windows_gnu = Some("x86_64-w64-mingw32-gcc")
CFLAGS_x86_64-pc-windows-gnu = None
CFLAGS_x86_64_pc_windows_gnu = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "x86_64-w64-mingw32-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "xz-5.2/src/liblzma/api" "-I" "xz-5.2/src/liblzma/lzma" "-I" "xz-5.2/src/liblzma/lz" "-I" "xz-5.2/src/liblzma/check" "-I" "xz-5.2/src/liblzma/simple" "-I" "xz-5.2/src/liblzma/delta" "-I" "xz-5.2/src/liblzma/common" "-I" "xz-5.2/src/liblzma/rangecoder" "-I" "xz-5.2/src/common" "-I" "./" "-Wall" "-Wextra" "-std=c99" "-pthread" "-DHAVE_CONFIG_H=1" "-o" "/target/x86_64-pc-windows-gnu/release/build/lzma-sys-03e4a3909321bf38/out/xz-5.2/src/liblzma/common/block_util.o" "-c" "xz-5.2/src/liblzma/common/block_util.c"
cargo:warning=In file included from xz-5.2/src/liblzma/common/common.h:17:0,
cargo:warning=                 from xz-5.2/src/liblzma/common/block_util.c:13:
cargo:warning=xz-5.2/src/common/mythread.h:138:33: error: unknown type name 'sigset_t'
cargo:warning= mythread_sigmask(int how, const sigset_t *restrict set,
cargo:warning=                                 ^~~~~~~~
cargo:warning=xz-5.2/src/common/mythread.h:139:3: error: unknown type name 'sigset_t'
cargo:warning=   sigset_t *restrict oset)
cargo:warning=   ^~~~~~~~
cargo:warning=xz-5.2/src/common/mythread.h: In function 'mythread_create':
cargo:warning=xz-5.2/src/common/mythread.h:158:2: error: unknown type name 'sigset_t'
cargo:warning=  sigset_t old;
cargo:warning=  ^~~~~~~~
cargo:warning=xz-5.2/src/common/mythread.h:159:2: error: unknown type name 'sigset_t'
cargo:warning=  sigset_t all;
cargo:warning=  ^~~~~~~~
cargo:warning=xz-5.2/src/common/mythread.h:160:2: warning: implicit declaration of function 'sigfillset' [-Wimplicit-function-declaration]
cargo:warning=  sigfillset(&all);
cargo:warning=  ^~~~~~~~~~
cargo:warning=xz-5.2/src/common/mythread.h:162:2: warning: implicit declaration of function 'mythread_sigmask' [-Wimplicit-function-declaration]
cargo:warning=  mythread_sigmask(SIG_SETMASK, &all, &old);
cargo:warning=  ^~~~~~~~~~~~~~~~
exit code: 1
--- stderr
thread 'main' panicked at '
Internal error occurred: Command "x86_64-w64-mingw32-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "xz-5.2/src/liblzma/api" "-I" "xz-5.2/src/liblzma/lzma" "-I" "xz-5.2/src/liblzma/lz" "-I" "xz-5.2/src/liblzma/check" "-I" "xz-5.2/src/liblzma/simple" "-I" "xz-5.2/src/liblzma/delta" "-I" "xz-5.2/src/liblzma/common" "-I" "xz-5.2/src/liblzma/rangecoder" "-I" "xz-5.2/src/common" "-I" "./" "-Wall" "-Wextra" "-std=c99" "-pthread" "-DHAVE_CONFIG_H=1" "-o" "/target/x86_64-pc-windows-gnu/release/build/lzma-sys-03e4a3909321bf38/out/xz-5.2/src/liblzma/common/block_util.o" "-c" "xz-5.2/src/liblzma/common/block_util.c" with args "x86_64-w64-mingw32-gcc" did not execute successfully (status code exit code: 1).
', /cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.28/src/lib.rs:2314:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

I have a fix in pkg-config-rs rust-lang/pkg-config-rs#99
This is just to remind to update the pkg-config-rs dependency, once that PR is merged.

I've tried preset(0) but zstd:1 of btrfs is unable to compress it further which means it's quite compressed.

I'd like a zero-compression output(cat-like?) somehow while ./x.py install install happens for rust, in order to reduce CPU usage and time, since it seems impossible to completely avoid creating *.tar.xz (and *.tar.gz) of the same things (also src-tarball=false and dist-src=false in config.toml have no effect! so *-src*.tar.xz is created, and .gz one too)

I think this might be possible using filters? or, alternatively removing the packing of .tar into .tar.gz and .tar.xz in rust's src/tools/rust-installer/src/tarballer.rs which most likely requires finding and changing the unpacking of those somewhere else (haven't figured out yet where)

Once I add crate-type = ["staticlib"] as necessary to the Cargo.toml file, this crate fails to build using cargo-lipo.

The error is: Invalid configuration `aarch64-apple-ios': system `ios' not recognized

Is this something that can be fixed on the cargo or rust side, or is this an upstream issue?

$ cargo lipo
   Compiling libc v0.2.36
   Compiling lzma-sys v0.1.9 (file:///Users/brendan/git/xz2-rs/lzma-sys)
error: failed to run custom build command for `lzma-sys v0.1.9 (file:///Users/brendan/git/xz2-rs/lzma-sys)`
process didn't exit successfully: `/Users/brendan/git/xz2-rs/target/debug/build/lzma-sys-5d37b9581b88dc59/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-env-changed=LZMA_API_STATIC
cargo:rustc-link-search=/Users/brendan/git/xz2-rs/target/aarch64-apple-ios/debug/build/lzma-sys-4fda62eea5e77fca/out/lib
cargo:root=/Users/brendan/git/xz2-rs/target/aarch64-apple-ios/debug/build/lzma-sys-4fda62eea5e77fca/out
cargo:include=/Users/brendan/git/xz2-rs/target/aarch64-apple-ios/debug/build/lzma-sys-4fda62eea5e77fca/out/include
cargo:rerun-if-changed=xz-5.2.3/configure
cargo:rustc-link-lib=static=lzma
OPT_LEVEL = Some("0")
TARGET = Some("aarch64-apple-ios")
HOST = Some("x86_64-apple-darwin")
TARGET = Some("aarch64-apple-ios")
TARGET = Some("aarch64-apple-ios")
HOST = Some("x86_64-apple-darwin")
CC_aarch64-apple-ios = None
CC_aarch64_apple_ios = None
TARGET_CC = None
CC = None
HOST = Some("x86_64-apple-darwin")
CROSS_COMPILE = None
TARGET = Some("aarch64-apple-ios")
HOST = Some("x86_64-apple-darwin")
CFLAGS_aarch64-apple-ios = None
CFLAGS_aarch64_apple_ios = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("true")
TARGET = Some("aarch64-apple-ios")
Detecting iOS SDK path for iphoneos
running: "sh" "/Users/brendan/git/xz2-rs/target/aarch64-apple-ios/debug/build/lzma-sys-4fda62eea5e77fca/out/src/configure" "--prefix=/Users/brendan/git/xz2-rs/target/aarch64-apple-ios/debug/build/lzma-sys-4fda62eea5e77fca/out" "--disable-doc" "--disable-lzma-links" "--disable-lzmainfo" "--disable-lzmadec" "--disable-xz" "--disable-xzdec" "--disable-scripts" "--disable-shared" "--disable-nls" "--disable-rpath" "--enable-threads=yes" "--build=x86_64-apple-darwin" "--host=aarch64-apple-ios"

XZ Utils 5.2.3

System type:
checking build system type... x86_64-apple-darwin
checking host system type...
--- stderr
Invalid configuration `aarch64-apple-ios': system `ios' not recognized
configure: error: /bin/sh /Users/brendan/git/xz2-rs/target/aarch64-apple-ios/debug/build/lzma-sys-4fda62eea5e77fca/out/src/build-aux/config.sub aarch64-apple-ios failed
thread 'main' panicked at 'assertion failed: try_run(cmd)', lzma-sys/build.rs:182:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed
Cargo exited unsuccessfully: exit code: 101