tarka / xcp Goto Github PK

View Code? Open in Web Editor NEW

726.0 5.0 24.0 824 KB

An extended `cp`

License: GNU General Public License v3.0

Rust 95.58% Shell 4.42%

sparse-files solid-state-disks linux copy-files command-line

xcp's People

Contributors

Stargazers

Watchers

xcp's Issues

0.14.0: error[E0308]: mismatched types

Build fails:

error[E0308]: mismatched types
   --> libfs/src/common.rs:193:37         
    |     
193 |     mknodat(CWD, dest, ftype, mode, dev)?;
    |     -------                         ^^^ expected `u32`, found `u64`
    |     |
    |     arguments to this function are incorrect
    |
note: function defined here
   --> /wrkdirs/usr/ports/sysutils/xcp/work/xcp-0.14.0/cargo-crates/rustix-0.38.26/src/fs/at.rs:438:8
    | 
438 | pub fn mknodat<P: path::Arg, Fd: AsFd>(
    |        ^^^^^^^
help: you can convert a `u64` to a `u32` and panic if the converted value doesn't fit
    |
193 |     mknodat(CWD, dest, ftype, mode, dev.try_into().unwrap())?;
    |                                        ++++++++++++++++++++

rust-1.73.0
FreeBSD 13.2

Bug: infinite recursion

xcp -r * some_folder/

causes infinite recursive calls as it tries to copy folder to the same folder over and over again

it seems that there's no check if the destination folder is in the source files or not

[feature request] option to respect .gitignore

Explanation of thrashing mentioned in readme.

Readme mentions, it will thrash the spinning disk drive by reading metadata and copying at the same time. Isn't Receiving multiple read/write request at a time common for HDD on multi program and multi user system. Is there something happening which increases this significantly? How many parallel request will is send for one file, so that HDD thrashing becomes an issue? Is there a way to avoid it through configuration?

Can xcp keep file uid/gid?

Can xcp keep linux file uid/gid?

[ Feature Request] Reflink support for btrfs?

I couldn't find any mention of reflink on readme. Is it supported? if yes, is it automatic?

Consider putting the operations in their own library crate

Copying file from virtual filesystems fails silently

There is a known issue with copy_file_range() that will result in files copied from virtual filesystems (e.g. /proc, /sys) to copy the incorrect number of bytes (usually 0 in the case of /proc, but possibly other sizes). LWN has a good summary here:

https://lwn.net/Articles/846403/

Detecting such filesystems/files is hard, as there is no "FS is virtual" flag, so all file sizes are inherently suspect when working this way. This issue is exaserbated in the case of xcp's block-parallel driver (parblock), which needs to know the file size up-front to allocate work on the queues.

This issue needs more investigation, and possibly some fundamental rearchitecting in the case of parallel block copy.

can not xcp to ecryptfs directory

it seems xcp does not work if ecryptfs folders are involved, non-ecryptfs folders worked well.

Please support --no-dereference

From man cp:

       -P, --no-dereference
              never follow symbolic links in SOURCE

The intended outcome is, rather than treating a symlink to a dir as a directory that should be recursively copied, the destination should itself be a symlink with the same target.

Thank you for creating this utility!

Question: Does it fallback to /bin/cp when a flag is not supported? In other words, can I alias cp to xcp?

(I would have put this question on the Discussion tab if it was enabled)

Faulty detection of sparse files?

The core of the detection of sparse files is:

xcp/src/os/linux.rs

Line 182 in ea28a4d

Ok(stat.st_blocks() < stat.st_size() / stat.st_blksize())

I may be wrong, but I think the use of st_blksize here is a mistake.

man 2 stat describes st_blksize as 'the "preferred" block size for efficient filesystem I/O.', and st_blocks as 'the number of blocks allocated to the file, in 512-byte units.' I checked with Rust code on a regular (non-sparse) file:

[src/main.rs:7] meta.st_blksize() = 4096
[src/main.rs:8] meta.st_blocks() = 1512
[src/main.rs:9] meta.st_size() = 771765
[src/main.rs:10] meta.st_size() / meta.st_blksize() = 188
[src/main.rs:11] meta.st_size() / 512 = 1507

I think these lines are the corresponding code in GNU cp. They use a constant ST_NBLOCKSIZE (from this header file) rather than anything from the stat struct.

If so, the impact is probably minor - I think st_blksize is only likely to be bigger than 512, so it should never erroneously decide that a dense file is sparse. And it can still detect very sparse files (on my system, where holes are 7/8 of the total size), where it's likely to be most important to optimise copying. But it looks like it will miss other sparse files.

Test failures on ZFS (Linux but not FreeBSD)

Hello,

I am experiencing test failures on ZFS filesystems on Linux. This is all running on amd64. All datasets have lots of free space.

Linux on ZFS

$ uname -sr
Linux 6.3.7-arch1-1

$ rustc --version
rustc 1.70.0 (90c541806 2023-05-31) (Arch Linux rust 1:1.70.0-1)

Test output

$ cargo test --release
    Finished release [optimized] target(s) in 0.05s
     Running unittests src/main.rs (target/release/deps/xcp-448f490b9f67ea08)

running 35 tests
test os::common::tests::test_extent_merge ... ok
test os::linux::tests::test_allocate_file_is_sparse ... ok
test os::common::tests::test_copy_range_uspace_large ... ok
test os::common::tests::test_copy_bytes_uspace_large ... ok
test os::linux::tests::test_copy_bytes_sparse ... ok
test os::linux::tests::test_empty_extent ... FAILED
test os::linux::tests::test_extent_fetch ... FAILED
test os::linux::tests::test_copy_range_middle ... ok
test os::linux::tests::test_extent_fetch_many ... FAILED
test os::linux::tests::test_sparse_detection ... ok
test os::linux::tests::test_sparse_copy_middle ... ok
test vendor::threadpool::test::test_active_count ... ignored
test vendor::threadpool::test::test_clone ... ignored
test vendor::threadpool::test::test_cloned_eq ... ignored
test vendor::threadpool::test::test_debug ... ignored
test vendor::threadpool::test::test_empty_pool ... ignored
test vendor::threadpool::test::test_join_wavesurfer ... ignored
test vendor::threadpool::test::test_massive_task_creation ... ignored
test vendor::threadpool::test::test_multi_join ... ignored
test vendor::threadpool::test::test_name ... ignored
test vendor::threadpool::test::test_no_fun_or_joy ... ignored
test vendor::threadpool::test::test_recovery_from_subtask_panic ... ignored
test vendor::threadpool::test::test_repeate_join ... ignored
test vendor::threadpool::test::test_send ... ignored
test vendor::threadpool::test::test_send_shared_data ... ignored
test vendor::threadpool::test::test_set_num_threads_decreasing ... ignored
test vendor::threadpool::test::test_set_num_threads_increasing ... ignored
test vendor::threadpool::test::test_should_not_panic_on_drop_if_subtasks_panic_after_drop ... ignored
test vendor::threadpool::test::test_shrink ... ignored
test vendor::threadpool::test::test_sync_shared_data ... ignored
test vendor::threadpool::test::test_works ... ignored
test vendor::threadpool::test::test_zero_tasks_panic ... ignored
test os::linux::tests::test_sparse_rust_seek ... ok
test os::linux::tests::test_lseek_data ... ok
test os::linux::tests::test_lseek_no_data ... ok

failures:

---- os::linux::tests::test_empty_extent stdout ----
Error: Operation not supported (os error 95)

---- os::linux::tests::test_extent_fetch stdout ----
Error: Operation not supported (os error 95)

---- os::linux::tests::test_extent_fetch_many stdout ----
Error: Operation not supported (os error 95)


failures:
    os::linux::tests::test_empty_extent
    os::linux::tests::test_extent_fetch
    os::linux::tests::test_extent_fetch_many

test result: FAILED. 11 passed; 3 failed; 21 ignored; 0 measured; 0 filtered out; finished in 0.01s

error: test failed, to rerun pass `--bin xcp`

FreeBSD on ZFS

FreeBSD ignores Linux-specific tests, so the whole test suite runs fine.

$ uname -a
FreeBSD bsd 13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64

$ rustc --version
rustc 1.68.2 (9eb3afe9e 2023-03-27) (built from a source tarball)

Test output

$ cargo test --release

warning: unused variable: `file`
   --> tests/util.rs:170:24
    |
170 | pub fn probably_sparse(file: &Path) -> Result<bool, Error> {
    |                        ^^^^ help: if this is intentional, prefix it with an underscore: `_file`
    |
    = note: `#[warn(unused_variables)]` on by default

warning: `xcp` (test "util") generated 1 warning
warning: `xcp` (test "common") generated 1 warning (1 duplicate)
    Finished release [optimized] target(s) in 0.06s
     Running unittests src/main.rs (target/release/deps/xcp-d14a3602a95775bb)

running 24 tests
test os::common::tests::test_copy_range_uspace_large ... ok
test os::common::tests::test_copy_bytes_uspace_large ... ok
test vendor::threadpool::test::test_active_count ... ignored
test vendor::threadpool::test::test_clone ... ignored
test vendor::threadpool::test::test_cloned_eq ... ignored
test vendor::threadpool::test::test_debug ... ignored
test vendor::threadpool::test::test_empty_pool ... ignored
test vendor::threadpool::test::test_join_wavesurfer ... ignored
test vendor::threadpool::test::test_massive_task_creation ... ignored
test vendor::threadpool::test::test_multi_join ... ignored
test vendor::threadpool::test::test_name ... ignored
test vendor::threadpool::test::test_no_fun_or_joy ... ignored
test vendor::threadpool::test::test_recovery_from_subtask_panic ... ignored
test vendor::threadpool::test::test_repeate_join ... ignored
test vendor::threadpool::test::test_send ... ignored
test vendor::threadpool::test::test_send_shared_data ... ignored
test vendor::threadpool::test::test_set_num_threads_decreasing ... ignored
test vendor::threadpool::test::test_set_num_threads_increasing ... ignored
test vendor::threadpool::test::test_should_not_panic_on_drop_if_subtasks_panic_after_drop ... ignored
test vendor::threadpool::test::test_shrink ... ignored
test vendor::threadpool::test::test_sync_shared_data ... ignored
test vendor::threadpool::test::test_works ... ignored
test vendor::threadpool::test::test_zero_tasks_panic ... ignored
test os::common::tests::test_extent_merge ... ok

test result: ok. 3 passed; 0 failed; 21 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/common.rs (target/release/deps/common-5ed6983ca092d56e)

running 58 tests
test basic_help ... ok
test copy_all_dirs::test_with_parallel_block_driver ... ok
test copy_all_dirs::test_with_parallel_file_driver ... ok
test copy_all_dirs_rel::test_with_parallel_block_driver ... ok
test copy_all_dirs_rel::test_with_parallel_file_driver ... ok
test copy_dirs_files::test_with_parallel_block_driver ... ok
test copy_dirs_files::test_with_parallel_file_driver ... ok
test copy_dirs_overwrites::test_with_parallel_block_driver ... ok
test copy_dirs_overwrites::test_with_parallel_file_driver ... ok
test copy_dirs_overwrites_no_target_dir ... ok
test copy_empty_dir::test_with_parallel_block_driver ... ok
test copy_generated_tree::test_with_parallel_block_driver ... ignored
test copy_generated_tree::test_with_parallel_file_driver ... ignored
test copy_pattern_no_glob::test_with_parallel_block_driver ... ok
test copy_empty_dir::test_with_parallel_file_driver ... ok
test copy_pattern_no_glob::test_with_parallel_file_driver ... ok
test copy_with_glob::test_with_parallel_block_driver ... ok
test dest_file_exists::test_with_parallel_block_driver ... ok
test copy_with_glob::test_with_parallel_file_driver ... ok
test dest_file_exists::test_with_parallel_file_driver ... ok
test dest_file_exists_noclobber::test_with_parallel_block_driver ... ok
test dest_file_exists_noclobber::test_with_parallel_file_driver ... ok
test dest_file_exists_overwrites::test_with_parallel_block_driver ... ok
test dest_file_exists_overwrites::test_with_parallel_file_driver ... ok
test dest_file_in_dir_exists::test_with_parallel_block_driver ... ok
test dir_copy_containing_symlinks::test_with_parallel_block_driver ... ok
test dest_file_in_dir_exists::test_with_parallel_file_driver ... ok
test dir_copy_to_nonexistent_is_rename::test_with_parallel_block_driver ... ok
test dir_copy_containing_symlinks::test_with_parallel_file_driver ... ok
test dir_copy_with_hidden_dir::test_with_parallel_block_driver ... ok
test dir_copy_to_nonexistent_is_rename::test_with_parallel_file_driver ... ok
test dir_copy_with_hidden_dir::test_with_parallel_file_driver ... ok
test dir_overwrite_with_noclobber::test_with_parallel_block_driver ... ok
test dir_overwrite_with_noclobber::test_with_parallel_file_driver ... ok
test dir_with_gitignore::test_with_parallel_block_driver ... ok
test dir_with_gitignore::test_with_parallel_file_driver ... ok
test file_copy::test_with_parallel_block_driver ... ok
test file_copy::test_with_parallel_file_driver ... ok
test file_copy_multiple::test_with_parallel_block_driver ... ok
test file_copy_no_perms::test_with_parallel_block_driver ... ok
test file_copy_multiple::test_with_parallel_file_driver ... ok
test file_copy_no_perms::test_with_parallel_file_driver ... ok
test file_copy_perms::test_with_parallel_block_driver ... ok
test file_copy_perms::test_with_parallel_file_driver ... ok
test file_copy_rel::test_with_parallel_block_driver ... ok
test glob_pattern_error::test_with_parallel_block_driver ... ok
test file_copy_rel::test_with_parallel_file_driver ... ok
test no_args ... ok
test glob_pattern_error::test_with_parallel_file_driver ... ok
test same_file_no_overwrite::test_with_parallel_block_driver ... ok
test same_file_no_overwrite::test_with_parallel_file_driver ... ok
test source_missing::test_with_parallel_block_driver ... ok
test source_missing::test_with_parallel_file_driver ... ok
test source_missing_globbed::test_with_parallel_block_driver ... ok
test source_missing_globbed::test_with_parallel_file_driver ... ok
test source_same_as_dest::test_with_parallel_file_driver ... ok
test source_same_as_dest::test_with_parallel_block_driver ... ok
test util::test_hasher ... ok

test result: ok. 56 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.05s

     Running tests/linux.rs (target/release/deps/linux-f36e08c5ed8f0175)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/util.rs (target/release/deps/util-b0c79de23c2e1bbb)

running 1 test
test test_hasher ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

FreeBSD on UFS

This is using a "classic" filesystem just for comparison's sake.

Test output

$ cargo test --release

running 24 tests
test os::common::tests::test_copy_range_uspace_large ... ok
test os::common::tests::test_copy_bytes_uspace_large ... ok
test vendor::threadpool::test::test_active_count ... ignored
test vendor::threadpool::test::test_clone ... ignored
test vendor::threadpool::test::test_cloned_eq ... ignored
test vendor::threadpool::test::test_debug ... ignored
test vendor::threadpool::test::test_empty_pool ... ignored
test vendor::threadpool::test::test_join_wavesurfer ... ignored
test os::common::tests::test_extent_merge ... ok
test vendor::threadpool::test::test_massive_task_creation ... ignored
test vendor::threadpool::test::test_multi_join ... ignored
test vendor::threadpool::test::test_name ... ignored
test vendor::threadpool::test::test_no_fun_or_joy ... ignored
test vendor::threadpool::test::test_recovery_from_subtask_panic ... ignored
test vendor::threadpool::test::test_repeate_join ... ignored
test vendor::threadpool::test::test_send ... ignored
test vendor::threadpool::test::test_send_shared_data ... ignored
test vendor::threadpool::test::test_set_num_threads_decreasing ... ignored
test vendor::threadpool::test::test_set_num_threads_increasing ... ignored
test vendor::threadpool::test::test_should_not_panic_on_drop_if_subtasks_panic_after_drop ... ignored
test vendor::threadpool::test::test_shrink ... ignored
test vendor::threadpool::test::test_sync_shared_data ... ignored
test vendor::threadpool::test::test_works ... ignored
test vendor::threadpool::test::test_zero_tasks_panic ... ignored

test result: ok. 3 passed; 0 failed; 21 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 58 tests
test basic_help ... ok
test copy_all_dirs::test_with_parallel_block_driver ... ok
test copy_all_dirs::test_with_parallel_file_driver ... ok
test copy_all_dirs_rel::test_with_parallel_block_driver ... ok
test copy_all_dirs_rel::test_with_parallel_file_driver ... ok
test copy_dirs_files::test_with_parallel_block_driver ... ok
test copy_dirs_files::test_with_parallel_file_driver ... ok
test copy_dirs_overwrites::test_with_parallel_block_driver ... ok
test copy_dirs_overwrites::test_with_parallel_file_driver ... ok
test copy_empty_dir::test_with_parallel_block_driver ... ok
test copy_dirs_overwrites_no_target_dir ... ok
test copy_generated_tree::test_with_parallel_block_driver ... ignored
test copy_generated_tree::test_with_parallel_file_driver ... ignored
test copy_pattern_no_glob::test_with_parallel_block_driver ... ok
test copy_empty_dir::test_with_parallel_file_driver ... ok
test copy_pattern_no_glob::test_with_parallel_file_driver ... ok
test copy_with_glob::test_with_parallel_block_driver ... ok
test dest_file_exists::test_with_parallel_block_driver ... ok
test copy_with_glob::test_with_parallel_file_driver ... ok
test dest_file_exists::test_with_parallel_file_driver ... ok
test dest_file_exists_noclobber::test_with_parallel_block_driver ... ok
test dest_file_exists_noclobber::test_with_parallel_file_driver ... ok
test dest_file_exists_overwrites::test_with_parallel_block_driver ... ok
test dest_file_in_dir_exists::test_with_parallel_block_driver ... ok
test dest_file_exists_overwrites::test_with_parallel_file_driver ... ok
test dest_file_in_dir_exists::test_with_parallel_file_driver ... ok
test dir_copy_containing_symlinks::test_with_parallel_block_driver ... ok
test dir_copy_containing_symlinks::test_with_parallel_file_driver ... ok
test dir_copy_to_nonexistent_is_rename::test_with_parallel_block_driver ... ok
test dir_copy_to_nonexistent_is_rename::test_with_parallel_file_driver ... ok
test dir_copy_with_hidden_dir::test_with_parallel_block_driver ... ok
test dir_copy_with_hidden_dir::test_with_parallel_file_driver ... ok
test dir_overwrite_with_noclobber::test_with_parallel_block_driver ... ok
test dir_overwrite_with_noclobber::test_with_parallel_file_driver ... ok
test dir_with_gitignore::test_with_parallel_block_driver ... ok
test file_copy::test_with_parallel_block_driver ... ok
test dir_with_gitignore::test_with_parallel_file_driver ... ok
test file_copy::test_with_parallel_file_driver ... ok
test file_copy_multiple::test_with_parallel_block_driver ... ok
test file_copy_no_perms::test_with_parallel_block_driver ... ok
test file_copy_multiple::test_with_parallel_file_driver ... ok
test file_copy_no_perms::test_with_parallel_file_driver ... ok
test file_copy_perms::test_with_parallel_block_driver ... ok
test file_copy_rel::test_with_parallel_block_driver ... ok
test file_copy_perms::test_with_parallel_file_driver ... ok
test glob_pattern_error::test_with_parallel_block_driver ... ok
test file_copy_rel::test_with_parallel_file_driver ... ok
test glob_pattern_error::test_with_parallel_file_driver ... ok
test no_args ... ok
test same_file_no_overwrite::test_with_parallel_block_driver ... ok
test same_file_no_overwrite::test_with_parallel_file_driver ... ok
test source_missing::test_with_parallel_block_driver ... ok
test source_missing::test_with_parallel_file_driver ... ok
test source_missing_globbed::test_with_parallel_block_driver ... ok
test source_missing_globbed::test_with_parallel_file_driver ... ok
test source_same_as_dest::test_with_parallel_block_driver ... ok
test source_same_as_dest::test_with_parallel_file_driver ... ok
test util::test_hasher ... ok

test result: ok. 56 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.09s


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 1 test
test test_hasher ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

If you don't have a ZFS-capable system at hand, I'll take a closer look at it later (although my Rust knowledge is quite rusty). Let's file a bug for now anyway.

Progress bar is too large

When copying, the progress bar is longer than the size of my terminal and it overflows to the next line. This is what I get when I copy a file.

$ cargo run A.mp4 B.mp4
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `target/debug/xcp A.mp4 B.mp4`
[00:00:00] [--------------------------------------------------------------------[00:00:00] 
[#############>------------------------------------------------------[00:00:00] 
[####################>-----------------------------------------------[00:00:00] 
[###########################>----------------------------------------[00:00:00] 
[########################################>---------------------------[00:00:00] 
[###############################################>--------------------[00:00:00] 
[######################################################>-------------[00:00:00] 
[#############################################################>------[00:00:00] 
[####################################################################[00:00:00] 
[####################################################################[00:00:00] 
[###############################################################################
#] 751.60MB/751.60MB (0s)

Suboptimal experience of copying a large file to a slow USD thumb drive.

If I start xcp normally, it shows as if the file were copied instantly, while in reality it is copied to cache (/proc/meminfo shows Dirty: as big as the file), with further actual copying happening outside xcp's control (i.e. progress bar).

If I start xcp in a memory-limited cgroup, it copies the file gradually (dirty bytes does not exceed the limit I set for it, I see destination file size gradually increasing), but the progress bar is stuck at 0% until the file finishes growing to its normal size. After that it starts actually copying the file (with normal progress bar). The data is written to vfat USB drive twice: first zero bytes, then actual content.

Shall xcp detect that it is copying the data to a USB drive and activate some O_DIRECT mode so that copy does not touch the cache and is easily abortable by user?

Progress bar does not advance smoothly when copying to NFS.

When copying a large file to a network share (e.g to an NFSv4 mount), it quickly fills up the progress bar, then waits for a long time at the end.

I expect xcp to show actual copy speed (i.e. slowest of the source reading and destination writing speed) and avoid overbuffering of content in memory (including in kernel caches).

Add an option to skip setting the file permissions

In some cases it's desirable to skip setting the destination file permissions, it would be nice if xcp had an option for that.

0.11.0 build fails on macOS due to unimplemented map_extents

While building on macOS with rustc 1.71.1 (eb26296b5 2023-08-03)

   Compiling xcp v0.11.0
error[E0308]: mismatched types
   --> /Users/dsully/.cargo/registry/src/index.crates.io-6f17d22bba15001f/xcp-0.11.0/src/drivers/parblock.rs:145:16
    |
145 |         if let Some(extents) = map_extents(&harc.infd)? {
    |                ^^^^^^^^^^^^^   ------------------------ this expression has type `Vec<std::ops::Range<u64>>`
    |                |
    |                expected `Vec<Range<u64>>`, found `Option<_>`
    |
    = note: expected struct `Vec<std::ops::Range<u64>>`
                 found enum `Option<_>`

For more information about this error, try `rustc --explain E0308`.
error: could not compile `xcp` (bin "xcp") due to previous error
error: failed to compile `xcp v0.11.0`, intermediate artifacts can be found at `/var/folders/q3/mk5rnvmd7rg59mgr20by9b0h0000gn/T/cargo-installZgCPLo`

This is due to map_extents in src/os/common.rs:

#[allow(dead_code)]
pub fn map_extents(_fd: &File) -> Result<Vec<Range<u64>>> {
    // FIXME: Implement for *BSD with lseek?
    Err(XcpError::UnsupportedOperation {}.into())
}

And is due to the change in e4d8cd2, and related to issue #37

failing syscalls on virtualbox share filesystem

Copying to a virtualbox filesystem leads to xcp printing an error and not copying the files.

[pid 31325] statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=20701696, ...}) = 0
[pid 31325] openat(AT_FDCWD, "/media/sf_c/temp/test.out", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4
[pid 31325] ftruncate(4, 20701696)      = 0
[pid 31325] flistxattr(3, NULL, 0)      = -1 EOPNOTSUPP (Operation not supported)

after adding --no-perms

[pid 31423] openat(AT_FDCWD, "/media/sf_c/temp/test.out", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4
[pid 31423] ftruncate(4, 20701696)      = 0
[pid 31423] statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=20701696, ...}) = 0
[pid 31423] copy_file_range(3, NULL, 4, NULL, 20701696, 0) = -1 EINVAL (Invalid argument)

kernel 5.10.11

xcp fails to copy from btrfs to exfat

Copying a file from a btrfs source to an exfat destination fails with the error:

Error: Operation not supported (os error 95)

However, both cp and rsync have no issues copying the file. rsync with -P even displays the progress correctly.

xcp: 0.9.1
rust: 1.63.0-1
kernel: 5.18.18
OS: Fedora 36
Arch: x86_64

Alas doesn't compile on win7

Compiling xcp v0.4.0
error[E0433]: failed to resolve: could not find unix in os
--> xcp-0.4.0\src\operations.rs:22:14
|
22 | use std::os::unix::fs::symlink;
| ^^^^ could not find unix in os

error[E0425]: cannot find function symlink in this scope
--> xcp-0.4.0\src\operations.rs:118:26
|
118 | let _r = symlink(&from, &to);
| ^^^^^^^ not found in this scope

error: aborting due to 2 previous errors

[Feature request] Display copy speed in progressbar

Thanks for making this project.

I read the LWN article on copy, and was wondering if someone had the courage to rewrite cp like for example what the author of ripgrep did for grep.

I'll open a few issues with my first thoughts using xcp, feel free to close/ignore them if you disagree.

It would be great to display the copy speed (in KB/s or MB/s) at the end of the progress bar.

I know that you are using the indicatif crate, and this is not built in, but that would be a valuable addition nevertheless.

copy files on macOS Ventura 13.4 and missed 50 % size

copying
21.550.875.699 Byte (22,2 GB auf dem Volume) für 103.345 Objekte
copied only
12.467.449.291 Byte (13,12 GB auf dem Volume) für 103.258 Objekte

the directory is a git repo

used the following variants:

xcp -v -r -w 3

xcp -v -r --gitignore -w 3

with file manager macos -> ok
with total commander -> ok

cargo install xcp
Updating crates.io index
Ignored package xcp v0.9.4 is already installed

test failure: linux::tests::test_reflink

Arch Linux. Building on xcp 0.16.0 on a build server. I believe file system is ext4, so the failure is probably expected.

failures:
    linux::tests::test_reflink

test result: FAILED. 20 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.13s

Is there a way to check the file system and mark it as an expected failure, allowing the test suite as a whole to be considered a success? Or maybe run the tests in an image file that uses a file system that supports the tested features?

Does not compile (Arch linux; rustc 1.81.0-nightly)

Compilation fails in the time crate:

error[E0282]: type annotations needed for `Box<_>`
  --> /home/benno/.cargo/registry/src/index.crates.io-6f17d22bba15001f/time-0.3.34/src/format_description/parse/mod.rs:83:9
   |
83 |     let items = format_items
   |         ^^^^^
...
86 |     Ok(items.into())
   |              ---- type must be known at this point
   |
help: consider giving `items` an explicit type, where the placeholders `_` are specified
   |
83 |     let items: Box<_> = format_items
   |              ++++++++

For more information about this error, try `rustc --explain E0282`.
error: could not compile `time` (lib) due to 1 previous error

Running cargo update fixes the issue

Feature request: add new option for treating target dir as file

Hello,

I found xcp useful because it respects .gitignore. I'm using it to copy a repo without build artifacts.

After the initial copy, since the target directory exists it will then copy to a subdirectory.

I'd like to have a feature that treats the target dir as a file, like -T in cp. https://stackoverflow.com/questions/23698183/how-to-force-cp-to-overwrite-directory-instead-of-creating-another-one-inside

If this feature is acceptable, I'm happy to make a PR for it.

0.12.0 File content missing when copying the same directory

#36 still persists when copying the whole directory recursively

mkdir test && cd test
echo "123" > test.txt
xcp -r ../test ../

and everything in the directory is basically wiped

[feature] remove copied files

Merry Christmas, first of all!

How would you feel about a feature to remove copied files? Alternatively (or, even better, in addition) xcp could support running an external command after one or more files have been copied.

`cargo install xcp` fails

❯ cargo install xcp          
    Updating crates.io index
  Installing xcp v0.18.0
    Updating crates.io index
error: failed to compile `xcp v0.18.0`, intermediate artifacts can be found at `/tmp/cargo-installwC4pO8`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.

Caused by:
  failed to select a version for the requirement `libxcp = "^0.1.0"`
  candidate versions found which didn't match: 0.18.0
  location searched: crates.io index
  required by package `xcp v0.18.0`
  perhaps a crate was updated and forgotten to be re-vendored?

Copying to a new file fails if source is FIFO pipe.

Issue

When copying from a FIFO socket, to a file that does not exist yet (the one you intend to create in the copy operation),
the following error is displayed:
Error: Unknown file-type: $filename
this does not happen if the copy destination file already exists.

Reproduce

The scripts provided generally have the side-effect of performing file/pipe creations and deletions to ./source and ./target.
They also can write/overwrite. I make a decent effort to undo any changes.
The following scripts will produce the error:

# FYI, this was created and tested with ZSH, but bash should play nice.
rm ./source ./target # clean workspace before starting
mkfifo ./source
xcp ./source ./target
# cleanup
rm ./source

rm ./source ./target # clean workspace before starting
mkfifo ./source
(sleep 2; echo "contents" > ./source) &
xcp ./source ./target
# cleanup
rm ./source ./target

Expected behavour

The failing script should perform the read from the FIFO socket as if it is a file, placing the contents into the target file.
This is expected for multiple reasons:

Reading from a FIFO socket is the same as reading from a file.
All* other utilities that performing reading functions treat a FIFO socket as expected.

*All with the caveat of a very very slight margin of error.

Control tests

The following scripts do NOT fail, for the purpose of demonstrating expected behavior as well as narrowing the issue for debugging.

# This does not fail due to creating "./test2" before invoking "xcp"
rm ./source ./target # clean workspace before starting
mkfifo ./source
touch target
(sleep 2; echo "contents" > ./source) &
xcp ./source ./target # This should block for 2 seconds
# cleanup
rm ./source ./target

# This does not fail because it invokes "cp" rather than "xcp"
rm ./source ./target # clean workspace before starting
mkfifo ./source
# We did not create "./target" which causes an error in "xcp"
(sleep 2; echo "contents" > ./source) &
/bin/cp ./source ./target # This should block for 2 seconds
# cleanup
rm ./source ./target

# This does not fail because it invokes "cat" rather than "xcp"
rm ./source # clean workspace before starting
mkfifo ./source
# We did not create "./target" which causes an error in "xcp"
(sleep 2; echo "contents" > ./source) &
/bin/cat ./source # This should block for 2 seconds
# cleanup
rm ./source

Verbose output

xcp --verbose ./source ./target returns:

07:32:59 [INFO] Copying source "./source" to "./target"
07:32:59 [ERROR] Unknown filetype found; this should never happen!
07:32:59 [INFO] Worker received shutdown command.
07:32:59 [INFO] Worker received shutdown command.

xcp version:

v0.9.4

Further notes

It seems tree_walker() is treating the target differently if the source is a FIFO socket, which it shouldn't.
I have not tested with other socket types.

Please release binary for multi os and cpu

Clarify differences with cp

xcp is a (partial) clone of the Unix cp command. It is not intended as a full replacement, but as a companion utility

I think you should clarify even further where xcp stands compared to coreutil's cp:

either state that the behavior of xcp is close to the one of cp, so mention what specific cp options or features are not supported for example. The idea is that users that do not use theses options can safely do alias cp=xcp in their interactive shell, and benefit from the progressbar and speedup.
or
state that any form of compatibility with cp is not a goal, or even an inspiration

File content is gone after copying it to itself

Repro:

Create a file f1 with some arbitrary content.
do xcp f1 f1
The file content is now garbage

How do I get my file back. I am sorry I shouldn't have done that 😭

fsync support

Do you plan to support fsync ? It could be nice feature.

unknown overlay, reflink test failure

xcp 0.21.1, AUR package. I believe it is a docker container. I didn't set it up.

Please provide guidance to obtain information needed to add support to the test script (eg, commands to run prior to the check).

==> Starting check()...
WARNING: unknown filesystem overlay, some tests might fail
found filesystem overlay, using flags use_linux
...
failures:

---- linux::tests::test_reflink stdout ----
thread 'linux::tests::test_reflink' panicked at libfs/src/linux.rs:320:9:
assertion failed: worked
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    linux::tests::test_reflink

Some tests are failing on Linux

---- os::linux::tests::test_empty_extent stdout ----
Error: Operation not supported (os error 95)
thread 'os::linux::tests::test_empty_extent' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', <::std::macros::panic macros>:5:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- os::linux::tests::test_extent_fetch_many stdout ----
Error: Operation not supported (os error 95)
thread 'os::linux::tests::test_extent_fetch_many' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', <::std::macros::panic macros>:5:6

---- os::linux::tests::test_extent_fetch stdout ----
Error: Operation not supported (os error 95)
thread 'os::linux::tests::test_extent_fetch' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', <::std::macros::panic macros>:5:6

Support brew installization?

Just hope to support brew.

Building fails on Termux

Hello,

I tried to build xcp 0.9.2 on Termux by cargo install, but it failed to compile.

The command I typed:

$ cargo install --locked xcp

Versions:

rustc 1.64.0
cargo 1.64.0
Termux Variables:
TERMUX_API_VERSION=0.50.1
TERMUX_APK_RELEASE=F_DROID
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP_PID=9719
TERMUX_IS_DEBUGGABLE_BUILD=0
TERMUX_MAIN_PACKAGE_FORMAT=debian
TERMUX_VERSION=0.118.0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://mirrors.pku.edu.cn/termux/termux-main/ stable main
Updatable packages:
All packages up to date
termux-tools version:
1.29.3
Android version:
12
Kernel build information:
Linux localhost 4.14.212-Velvet-Cyprus_v2.1 #1 SMP PREEMPT Tue Sep 13 07:06:30 UTC 2022 aarch64 Android
Device manufacturer:
Xiaomi
Device model:
Redmi Note 9S

And the output:

error[E0433]: failed to resolve: could not find `linux` in `os`
  --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:22:14
   |                                                                                    22 | use std::os::linux::fs::MetadataExt;
   |              ^^^^^ could not find `linux` in `os`                                  
error[E0599]: no method named `st_blocks` found for struct `std::fs::Metadata` in the current scope
   --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:182:13
    |                                                                                   182 |     Ok(stat.st_blocks() < stat.st_size() / stat.st_blksize())
    |             ^^^^^^^^^ method not found in `std::fs::Metadata`                         |
    = help: items from traits can only be used if the trait is in scope                 help: the following trait is implemented but not in scope; perhaps add a `use` for it:
    |                                                                                   17  | use std::os::android::fs::MetadataExt;
    |                                                                                   
error[E0599]: no method named `st_size` found for struct `std::fs::Metadata` in the current scope
   --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:182:32
    |                                                                                   182 |     Ok(stat.st_blocks() < stat.st_size() / stat.st_blksize())
    |                                ^^^^^^^ method not found in `std::fs::Metadata`        |
    = help: items from traits can only be used if the trait is in scope                 help: the following trait is implemented but not in scope; perhaps add a `use` for it:
    |                                                                                   17  | use std::os::android::fs::MetadataExt;
    |                                                                                   
error[E0599]: no method named `st_blksize` found for struct `std::fs::Metadata` in the current scope
   --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:182:49
    |                                                                                   182 |     Ok(stat.st_blocks() < stat.st_size() / stat.st_blksize())
    |                                                 ^^^^^^^^^^ method not found in `std::fs::Metadata`
    |                                                                                       = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope; perhaps add a `use` for it:      |
17  | use std::os::android::fs::MetadataExt;                                                |
                                                                                        error[E0308]: mismatched types
    --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:285:41
     |                                                                                  285  |             libc::ioctl(fd.as_raw_fd(), FS_IOC_FIEMAP, req_ptr)
     |             -----------                 ^^^^^^^^^^^^^ expected `i32`, found `u64`     |             |
     |             arguments to this function are incorrect                                  |
note: function defined here                                                                 --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.134/src/unix/linux_like/android/mod.rs:2824:12                         |
2824 |     pub fn ioctl(fd: ::c_int, request: ::c_int, ...) -> ::c_int;                      |            ^^^^^
help: you can convert a `u64` to an `i32` and panic if the converted value doesn't fit       |
285  |             libc::ioctl(fd.as_raw_fd(), FS_IOC_FIEMAP.try_into().unwrap(), req_ptr)
     |                                                      ++++++++++++++++++++                                                                                                Some errors have detailed explanations: E0308, E0433, E0599.                            For more information about an error, try `rustc --explain E0308`.
error: could not compile `xcp` due to 5 previous errors

tarka / xcp Goto Github PK

xcp's People

Contributors

Stargazers

Watchers

Forkers

xcp's Issues

Linux on ZFS

FreeBSD on ZFS

FreeBSD on UFS

Issue

Reproduce

Expected behavour

Control tests

Verbose output

xcp version:

Further notes

Recommend Projects

Recommend Topics

Recommend Org