darakian / ddh Goto Github PK
View Code? Open in Web Editor NEWA fast duplicate file finder
License: Other
A fast duplicate file finder
License: Other
Steps to reproduce:
Open a prompt with administrative (or current user) access
Navigate to build directory
Run .\ddh "c:\users\myuser\desktop"
Output:
0 Total files (with duplicates): 0 Kilobytes
0 Total files (without duplicates): 0 Kilobytes
0 Single instance files: 0 Kilobytes
0 Shared instance files: 0 Kilobytes (0 instances)
However, the program is not completely broken, as if I run a .\ddh .\
The output is:
168 Total files (with duplicates): 41476 Kilobytes
164 Total files (without duplicates): 30693 Kilobytes
160 Single instance files: 19911 Kilobytes
4 Shared instance files: 10782 Kilobytes (8 instances)
When using windows binary release from this commit:
8c64d23
It does not handle directories with spaces in their names and crashes.
i.e.
ddh.exe .\ ----- Runs fine
ddh.exe ".\space in name" ------ Causes the error below
thread 'main' panicked at 'Error canonicalizing input: Os { code: 123, kind: Other, message: "The filename, directory name, or volume label syntax is incorrect." }', libcore\result.rs:945:5
stack backtrace:
0:
1:
2:
3:
4:
5:
6:
7:
8:
9:
10: BaseThreadInitThunk
11: RtlUserThreadStart
Tried this on my home directory with ~1.5 million files.
/usr/bin/time -v target/release/ddh ~
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
thread '<unnamed>' panicked at 'Error reading Symlink Metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/lib.rs:285:25
...
There seems to be no way to turn off symlink following, so I guess it might have escaped from my home dir and tried to scan the whole filesystem. But anyways, the CPU and memory usage was extremely high. System time usage is quite unusal assuming this was I/O bound workload:
^CCommand terminated by signal 2
Command being timed: "target/release/ddh /home/pkolaczk"
User time (seconds): 50.08
System time (seconds): 983.47
Percent of CPU this job got: 776%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:13.10
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 18724728
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 4688422
Voluntary context switches: 27301
Involuntary context switches: 150856
Swaps: 0
File system inputs: 103592
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
ddh
consumed 18 GBs, while other programs like
fdupes, jdupes, rmfind, fclones need about 150-500 MB.
Steps to reproduce:
Compile with cargo build
Run setx RUST_BACKTRACE 1
Open new shell (variable does set in current shell)
Run ddh c:\users\myuser
Output: thread '<unknown>' has overflowed its stack
I got a panic from a crater run which included this crate (in case you are not aware, crater is a tool that runs rustc against a set number of locked crates to check for regressions). It was with 0.10.9 version so i'm not sure if it still holds true in the 0.11.0 or not. Also, it could be caused by one of the dependencies. I tried tracking it down but I couldn't find anything.
The main point of the crater run was to find UB according to this advisory: rustsec/advisory-db#298.
Edit: there is a small possibility that this could be due to the file permissions being denied in crater runs
The crater output is:
[INFO] [stdout] failures:
[INFO] [stdout]
[INFO] [stdout] ---- src/lib.rs - deduplicate_dirs (line 216) stdout ----
[INFO] [stdout] Test executable failed (terminated by signal).
[INFO] [stdout]
[INFO] [stdout] stderr:
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] stack backtrace:
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] thread '<unnamed>' panicked at 'attempted to leave type `internal::Local` uninitialized, which is invalid', /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661:5
[INFO] [stdout] 0: 0x55caa32b01c4 - backtrace::backtrace::libunwind::trace::h61666fa4b9834a45
[INFO] [stdout] at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
[INFO] [stdout] 1: 0x55caa32b01c4 - backtrace::backtrace::trace_unsynchronized::h01d3725fb5da8653
[INFO] [stdout] at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
[INFO] [stdout] 2: 0x55caa32b01c4 - std::sys_common::backtrace::_print_fmt::h5c7c4751bc296f90
[INFO] [stdout] at src/libstd/sys_common/backtrace.rs:78
[INFO] [stdout] 3: 0x55caa32b01c4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7729313ad2d729c7
[INFO] [stdout] at src/libstd/sys_common/backtrace.rs:59
[INFO] [stdout] 4: 0x55caa32cf9bc - core::fmt::write::hb0049f6c12bdaffc
[INFO] [stdout] at src/libcore/fmt/mod.rs:1069
[INFO] [stdout] 5: 0x55caa32ae113 - std::io::Write::write_fmt::hfb486fa6ccb90f14
[INFO] [stdout] at src/libstd/io/mod.rs:1504
[INFO] [stdout] 6: 0x55caa32b29a5 - std::sys_common::backtrace::_print::ha087cac92e84a414
[INFO] [stdout] at src/libstd/sys_common/backtrace.rs:62
[INFO] [stdout] 7: 0x55caa32b29a5 - std::sys_common::backtrace::print::h6b013d3bfe6cbcb8
[INFO] [stdout] at src/libstd/sys_common/backtrace.rs:49
[INFO] [stdout] 8: 0x55caa32b29a5 - std::panicking::default_hook::{{closure}}::h016d7de02def2736
[INFO] [stdout] at src/libstd/panicking.rs:198
[INFO] [stdout] 9: 0x55caa32b26e2 - std::panicking::default_hook::h9386f7e484840463
[INFO] [stdout] at src/libstd/panicking.rs:218
[INFO] [stdout] 10: 0x55caa32b3002 - std::panicking::rust_panic_with_hook::h3846eb1b2e5a9312
[INFO] [stdout] at src/libstd/panicking.rs:511
[INFO] [stdout] 11: 0x55caa32b2beb - rust_begin_unwind
[INFO] [stdout] at src/libstd/panicking.rs:419
[INFO] [stdout] 12: 0x55caa32ceda1 - core::panicking::panic_fmt::hc5953a7331f9805c
[INFO] [stdout] at src/libcore/panicking.rs:111
[INFO] [stdout] 13: 0x55caa32ceced - core::panicking::panic::hbfb03ddd2cbb509e
[INFO] [stdout] at src/libcore/panicking.rs:54
[INFO] [stdout] 14: 0x55caa3297489 - core::mem::uninitialized::h869ad29ea76dd030
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/mem/mod.rs:661
[INFO] [stdout] 15: 0x55caa3297489 - <crossbeam_epoch::internal::Local as crossbeam_epoch::sync::list::IsElement<crossbeam_epoch::internal::Local>>::entry_of::hdb4452d7a6e0a610
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.7.1/src/internal.rs:493
[INFO] [stdout] 16: 0x55caa329e823 - crossbeam_epoch::sync::list::List<T,C>::insert::h3ac4d85186c907f0
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.7.1/src/sync/list.rs:176
[INFO] [stdout] 17: 0x55caa3296454 - crossbeam_epoch::internal::Local::register::h91b5c90d84206ca9
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.7.1/src/internal.rs:291
[INFO] [stdout] 18: 0x55caa32a0a5e - crossbeam_epoch::collector::Collector::register::h5691db836e05a7d5
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.7.1/src/collector.rs:39
[INFO] [stdout] 19: 0x55caa329a7b4 - crossbeam_epoch::default::HANDLE::__init::h05b698916945b7d4
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.7.1/src/default.rs:17
[INFO] [stdout] 20: 0x55caa3299ec9 - core::ops::function::FnOnce::call_once::hf43b3b136e2a1b46
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/ops/function.rs:232
[INFO] [stdout] 21: 0x55caa32a053c - std::thread::local::lazy::LazyKeyInner<T>::initialize::h72653fcc8deb826a
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/thread/local.rs:289
[INFO] [stdout] 22: 0x55caa329ced2 - std::thread::local::fast::Key<T>::try_initialize::h30683c7ab15cafc2
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/thread/local.rs:425
[INFO] [stdout] 23: 0x55caa329d027 - std::thread::local::fast::Key<T>::get::hd10d89894258aff9
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/thread/local.rs:410
[INFO] [stdout] 24: 0x55caa329a7ed - crossbeam_epoch::default::HANDLE::__getit::h962c871568622af7
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/thread/local.rs:175
[INFO] [stdout] 25: 0x55caa32749d5 - std::thread::local::LocalKey<T>::try_with::h3f261a7c4f47ce5a
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/thread/local.rs:262
[INFO] [stdout] 26: 0x55caa3283a53 - crossbeam_epoch::default::with_handle::h269db34920d39f6c
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.7.1/src/default.rs:42
[INFO] [stdout] 27: 0x55caa3283d37 - crossbeam_epoch::default::is_pinned::hb6e44e79801d0481
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.7.1/src/default.rs:29
[INFO] [stdout] 28: 0x55caa328b6d2 - crossbeam_deque::Stealer<T>::steal::h9ae6be401072c7cd
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.6.3/src/lib.rs:660
[INFO] [stdout] 29: 0x55caa326c2fa - rayon_core::registry::WorkerThread::steal::{{closure}}::hca21cf343bc94b68
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:764
[INFO] [stdout] 30: 0x55caa32694cb - core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut::h59bd862209ac9487
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/ops/function.rs:274
[INFO] [stdout] 31: 0x55caa3287921 - core::iter::traits::iterator::Iterator::find_map::check::{{closure}}::h6647419eb8e863d9
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/iter/traits/iterator.rs:2239
[INFO] [stdout] 32: 0x55caa327fd65 - core::iter::adapters::filter_try_fold::{{closure}}::h0e27e4a4e21c5718
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/iter/adapters/mod.rs:935
[INFO] [stdout] 33: 0x55caa327effc - core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut::h5b9d2db37f7cb14f
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/ops/function.rs:274
[INFO] [stdout] 34: 0x55caa3287ed4 - core::iter::traits::iterator::Iterator::try_fold::h9c4820e0b9b6a2ec
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/iter/traits/iterator.rs:1878
[INFO] [stdout] 35: 0x55caa327e3c3 - <core::iter::adapters::chain::Chain<A,B> as core::iter::traits::iterator::Iterator>::try_fold::h80243e00b6de1e21
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/iter/adapters/chain.rs:83
[INFO] [stdout] 36: 0x55caa327e2a6 - <core::iter::adapters::Filter<I,P> as core::iter::traits::iterator::Iterator>::try_fold::ha3e1ae2a61dda215
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/iter/adapters/mod.rs:984
[INFO] [stdout] 37: 0x55caa327fb01 - core::iter::traits::iterator::Iterator::find_map::he70084dfab691d9a
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/iter/traits/iterator.rs:2245
[INFO] [stdout] 38: 0x55caa327e316 - <core::iter::adapters::FilterMap<I,F> as core::iter::traits::iterator::Iterator>::next::hdc570465366b354c
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/iter/adapters/mod.rs:1084
[INFO] [stdout] 39: 0x55caa326c228 - rayon_core::registry::WorkerThread::steal::h9bd36c493aa21796
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:758
[INFO] [stdout] 40: 0x55caa326bf52 - rayon_core::registry::WorkerThread::wait_until_cold::{{closure}}::h590a1fcd9de02876
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:714
[INFO] [stdout] 41: 0x55caa327916f - core::option::Option<T>::or_else::hf431f9e43aa9978c
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/option.rs:766
[INFO] [stdout] 42: 0x55caa326bc00 - rayon_core::registry::WorkerThread::wait_until_cold::hd72b2229314009a4
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:712
[INFO] [stdout] 43: 0x55caa326b5f6 - rayon_core::registry::WorkerThread::wait_until::h608ca86b940e41fd
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:692
[INFO] [stdout] 44: 0x55caa326c87d - rayon_core::registry::main_loop::hf39058cfb43bdbf9
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:812
[INFO] [stdout] 45: 0x55caa3269720 - rayon_core::registry::ThreadBuilder::run::ha145b7a27b5851e2
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:58
[INFO] [stdout] 46: 0x55caa3269c21 - <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}::ha5a4cee564920880
[INFO] [stdout] at /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.5.0/src/registry.rs:103
[INFO] [stdout] 47: 0x55caa3283e90 - std::sys_common::backtrace::__rust_begin_short_backtrace::hffc2c09a197264dd
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/sys_common/backtrace.rs:130
[INFO] [stdout] 48: 0x55caa3285171 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::hfdfb97e3734c02e3
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/thread/mod.rs:475
[INFO] [stdout] 49: 0x55caa32838b3 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h27a0583c4b616fd0
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/panic.rs:318
[INFO] [stdout] 50: 0x55caa326e384 - std::panicking::try::do_call::h8929afe6053c86ae
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/panicking.rs:331
[INFO] [stdout] 51: 0x55caa326f6dd - __rust_try
[INFO] [stdout] 52: 0x55caa326e174 - std::panicking::try::h3fadfd4c27ff95a2
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/panicking.rs:274
[INFO] [stdout] 53: 0x55caa3284323 - std::panic::catch_unwind::h422593581e096c2f
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/panic.rs:394
[INFO] [stdout] 54: 0x55caa3284f61 - std::thread::Builder::spawn_unchecked::{{closure}}::hf49384bceeda1e85
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libstd/thread/mod.rs:474
[INFO] [stdout] 55: 0x55caa3274fbf - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6ecc5d6bbd1cf8dc
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/libcore/ops/function.rs:232
[INFO] [stdout] 56: 0x55caa32a9f9f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h7deaeeba480c6ce2
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/liballoc/boxed.rs:1008
[INFO] [stdout] 57: 0x55caa32b53f3 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h968b1dfb87756523
[INFO] [stdout] at /rustc/41997647ba6a77908f6ef64401414feb00bccf16/src/liballoc/boxed.rs:1008
[INFO] [stdout] 58: 0x55caa32b53f3 - std::sys::unix::thread::Thread::new::thread_start::h2399addeb6370595
[INFO] [stdout] at src/libstd/sys/unix/thread.rs:87
[INFO] [stdout] 59: 0x7f326ac7a6db - start_thread
[INFO] [stdout] 60: 0x7f326a78b88f - __clone
[INFO] [stdout] 61: 0x0 - <unknown>
[INFO] [stdout] Rayon: detected unexpected panic; aborting
```
local directory with 930 jpeg files with 2 duplicate jpeg files
ubuntu 20.04 no GUI :
rdfind real: 0m00.657s user: 0m00.007s sys: 0m00.114s
ddh real: 0m00.007s user: 0m00.002s sys: 0m00.004s
fdupes real: 0m32.932s user: 0m00.074s sys: 0m05.685s
Windows 11 pro:
ddh total: 0m00.011s (started from a .cmd file)
powershell total: 0m00.012s (incl. grouping & sorting)
DupeGuru coulnd't the duplicate files
Overall i would say nice job ๐๐.
PS: Ssome of the rdfind options would be nice (shameless I know ๐)
LGPL-3.0 is deprecated, consider use LGPL-3.0-only
or LGPL-3.0-or-later
. (you can just add license header to the source file or in the README), thanks!
Line 7 in e4c574f
can ddh ignore known directories such as node_modules, .mypy_cache, CMakefile Release/Debug directories,etc
or just respect .gitignore by default(or not by default).
This is mostly informational, I really like ddh but needed to write a tool to do something with the duplicate files it found. So I wrote ddh-move which takes ddh output on standard input and moves the duplicate files to a specified location.
If you like it, I would be very happy if you mentioned it in your readme! If not, please let me know what you would like to see improved!
Relevant code is currently on line 152 in the hash_and_update function
assert!(input.file_paths.iter().next().expect("Error reading path from struct").is_file());
Some windows system files fail this assertion and throw a panic. Interestingly the variable input comes from data populated in the traverse_and_spawn function here
else if current_path.is_file() && !(current_path.symlink_metadata().expect("Error getting Symlink Metadata").file_type().is_symlink()){
sender.send(Fileinfo::new(0, current_path.metadata().expect("Error with current path length").len(), /*fs::canonicalize(*/current_path.to_path_buf()/*).expect("Error canonicalizing path in struct creation.")*/)).expect("Error sending new fileinfo");
If anyone with a windows system would like to look into this the help would be appreciated.
I built ddh from master branch on Ubuntu 16.04 with rustc 1.33.0-nightly (ec194646f 2019-01-02). Used 'cargo build --release' to compile.
If I run ddh from the target/release directory with the following command in the target/release directory:
./ddh ~/Downloads/ -f json -o ~/Downloads/dupes.json
The dupes.json file is created in the current working directory (target/release) but not at the location specified (~/Downloads/dupes.json).
Despite this minor issue, I really appreciate this tool! Thank you for your hard work ๐
I may have to dig deeper into this issue, however, upon running multiple ddh commands against my desktop, I receive different result. I do not believe any files are change on my desktop.
1st run:
2112 Total files (with duplicates): 586116 Kilobytes
2030 Total files (without duplicates): 548212 Kilobytes
1983 Single instance files: 523078 Kilobytes
47 Shared instance files: 25133 Kilobytes (129 instances)
2nd run:
2112 Total files (with duplicates): 586116 Kilobytes
2031 Total files (without duplicates): 549007 Kilobytes
1984 Single instance files: 523873 Kilobytes
47 Shared instance files: 25133 Kilobytes (128 instances)
3rd run:
2112 Total files (with duplicates): 586116 Kilobytes
2030 Total files (without duplicates): 548192 Kilobytes
1982 Single instance files: 522243 Kilobytes
48 Shared instance files: 25948 Kilobytes (130 instances)
Thanks for this software, has been a game changer coming from a robust but awfully slow (Python) duplicate checker with a much less sophistocated algorithm.
Would you be able to compile a new Windows release (and for that matter, Linux and MacOS)? Was a bit of a pain to have to install Rust along with the 4.6 gb of C++ build tools, as I don't code these languages locally.
ddh --help
reports:
USAGE:
ddh [OPTIONS] <Directories>...
but it should be ddh <Directories>... [OPTIONS]
.
Sadly I don't know clap and how to fix it.
When using ddh as a library, I would like an indication of progression I can show a user on a GUI.
Something like nr of files found, nr of files processed, nr of files skipped, nr of duplicates found.
Given the parallel nature of ddh, rough estimations would be enough.
This would allow users to dedup pictures even if they have different file formats, resolutions or quality. Usually one wants to keep the largest file.
im_hash delivers the rust functionality.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.