wcampbell0x2a / backhand Goto Github PK

We currently emit 0x00, which is fine for the kernel. squashfs-tools/unsquashfs does more checking so I can't just emit the previous flags and not emit those features. Ideally we emit the same features.

Filenames can't be represented as String, because they maybe be not valid UTF8:

$ RUST_BACKTRACE=1 ~/src/backhand/target/release/unsquashfs -l non_utf8.squashfs
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 9, error_len: Some(1) }', src/dir.rs:73:41
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/panicking.rs:64:14
   2: core::result::unwrap_failed
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/result.rs:1791:5
   3: backhand::dir::DirEntry::name
   4: backhand::squashfs::Squashfs<R>::extract_dir
   5: backhand::squashfs::Squashfs<R>::into_filesystem_reader
   6: unsquashfs::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

• Use https://github.com/assert-rs/assert_cmd to assert different unsquashfs and add features

test_09 only supports testing if we can unsquash, test squashing

Node::file(), Node::block_device() and so on

Such as Filesystem::push_dir and so on

• Generate same bytes as mksquashfs
• Use CompressionOptions::Gzip -> window_size and strageties

Instead of having inodes and dirs having different fields, tie these together as read time, so that when grabbing files and writing to the filesystem these are already read and tied together.

Since we ship binaries, we should lock the depends and tell the user to do the same when installing.

Instead of:

// read
let file = File::open("file.squashfs").unwrap();
let squashfs = Squashfs::from_reader(file).unwrap();
let filesystem = squashfs.into_filesystem().unwrap();

You should be able to do the following, since you don't need the squashfs usually.

// read
let file = File::open("file.squashfs").unwrap();
let filesystem = filesystem.from_reader(file).unwrap();

Given the er605 test:

> stat squashfs-root/bin
Access: (0755/drwxr-xr-x)  Uid: ( 1000/wcampbell)   Gid: ( 1000/wcampbell)

> stat squashfs-root-new/bin
Access: (0775/drwxrwxr-x)  Uid: ( 1000/wcampbell)   Gid: ( 1000/wcampbell)

If the user wants to modify an existing squashfs file by just adding files, currently we decompress->compress unmodified files.

But maybe is possible to copy the original compressed data without needing to decompress it.

flowchart LR

A[Read] --> B(Decompress)
B --> C(Compress)
C --> D[Write]
A -->|Maybe just copy it| D

• Add github action that will be used during Release MR's that change the version of this crate

https://crates.io/crates/lzma-rs

The use of Filesystem as the following struct is really bad:

pub struct Filesystem {
    pub nodes: Vec<Node>,
}

The function into_filesystem, should just emit a Tree struct. This will allow the following:

• Save permissions for directories that aren't empty. This currently isn't possible
• Reduce memory overhead, as you needed both Filesystem and Tree to read/write/modify

I think Tree should be renamed to Filesystem.

• Default features will be all compression algorithms
• Each other compression should be behind a feature flag

For the test test_09, we parse a somewhat large squashfs image for openwrt. This should be further tested by running a sha256 of every file. Something like saving the output from unsquashfs -> find output -type f -exec sha256sum {} \; and comparing it against our own.

$ cargo build --release --bin unsquashfs --features xz-static
   Compiling backhand v0.9.1 (/home/wcampbell/projects/wcampbell/backhand)
    Finished release [optimized] target(s) in 2.73s

$ ./target/release/unsquashfs -f -d out-rust test-assets/test_openwrt_netgear_ex6100v2/bytes.squashfs
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidInput, error: Options }', src/bin/unsquashfs.rs:132:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

squashfs-tools/unsquashfs is just fine

$ unsquashfs test-assets/test_openwrt_netgear_ex6100v2/bytes.squashfs
Parallel unsquashfs: Using 8 processors
1249 inodes (1049 blocks) to write


create_inode: could not create character device squashfs-root/dev/console, because you're not superuser!
[================================================================================================================================================================================================================================================================================================================================================| ] 2297/2298  99%

created 1043 files
created 122 directories
created 205 symlinks
created 0 devices
created 0 fifos
created 0 sockets
created 0 hardlinks

Instead of just extracting a singular file, extract the total filesystem with paths.

The following patch forces the test suite to use the offsets and generate images with the same offsets as the test image.

diff --git a/tests/common/[mod.rs](http://mod.rs/) b/tests/common/[mod.rs](http://mod.rs/)
index d98f137..d912e72 100644
--- a/tests/common/[mod.rs](http://mod.rs/)
+++ b/tests/common/[mod.rs](http://mod.rs/)
@@ -4,6 +4,7 @@ use assert_cmd::prelude::*;
 use tempfile::tempdir;

 pub fn test_unsquashfs(control: &str, new: &str, control_offset: Option<u64>) {
+    println!("{}", control_offset.unwrap_or(0));
     let control_dir = tempdir().unwrap();
     Command::new("unsquashfs")
         .args([
@@ -24,6 +25,8 @@ pub fn test_unsquashfs(control: &str, new: &str, control_offset: Option<u64>) {
         .args([
             "-d",
             new_dir.path().to_str().unwrap(),
+            "-o",
+            &control_offset.unwrap_or(0).to_string(),
             // we don't run as root, avoid special file errors
             "-ignore-errors",
             "-no-exit-code",
@@ -53,6 +56,8 @@ pub fn test_unsquashfs_list(control: &str, new: &str, control_offset: Option<u64
     let output = Command::new("unsquashfs")
         .args([
             "-l",
+            "-o",
+            &control_offset.unwrap_or(0).to_string(),
             // we don't run as root, avoid special file errors
             "-ignore-errors",
             "-no-exit-code",
diff --git a/tests/[test.rs](http://test.rs/) b/tests/[test.rs](http://test.rs/)
index 50eb92a..a49c986 100644
--- a/tests/[test.rs](http://test.rs/)
+++ b/tests/[test.rs](http://test.rs/)
@@ -38,14 +38,14 @@ fn full_test(
     // convert to bytes
     info!("calling to_bytes");
     let mut output = File::create(&new_path).unwrap();
-    new_filesystem.write(&mut output).unwrap();
+    new_filesystem.write_with_offset(&mut output, offset).unwrap();
     // Test Debug is impl'ed properly on FilesystemWriter
     let _ = format!("{new_filesystem:#02x?}");

     // assert that our library can atleast read the output, use unsquashfs to really assert this
     info!("calling from_reader");
     let created_file = File::open(&new_path).unwrap();
-    let _new_filesystem = FilesystemReader::from_reader(created_file).unwrap();
+    let _new_filesystem = FilesystemReader::from_reader_with_offset(created_file, offset).unwrap();

     info!("starting unsquashfs test");
     match verify {

This makes the test suite fail. I assume something is up with the WithOffset structs.

Any ideas @rbran ?

unblob supports some non standard squashfs binaries, we should also.

https://github.com/onekey-sec/unblob/tree/main/tests/integration/filesystem/squashfs

They have a fork of sasquash for adding support https://github.com/onekey-sec/sasquatch

Reproduce:

mkdir tmp
touch tmp/file1 tmp/file2 empty
mksquashfs tmp test.squashfs
~/src/backhand/target/debug/add test.squashfs empty ./

Don't add the file empty, instead unsquashfs -l added.squashfs only outputs:

squashfs-root
squashfs-root/file1
squashfs-root/file2

Using add you can create directories as required of the new file path. This option (or subcommands?) Let you just add an empty directory. This should extend time, uid, and guid as options.

• Read NodeHeader from the file being added
• optional --mtime, --uid, and --gid for overriding details

The function squashfs.extract_file(..) supports grabbing files at both the root inode and beyond.

The correct functionality in my eyes would be to give the full path of the files you want, and for that path to be verified.