kwarf / rotsniff Goto Github PK

Hi,

I'm looking for a hash verification tool for my photos/videos collection, and came across rotsniff.

I've already had a look at some other projects, including https://github.com/laktak/chkbit-py - which is written in Python and uses md5. On my folder (400+ gbytes, 50k+ files) it finished building a database in 7 minutes (using 6 workers = num of my physical cores; with a single worker, it needed 30 minutes).

As far as I understand, blake2 is faster than md5, and Rust is faster than Python.
However, rotsniff actually took much, much longer on my data: 52 minutes
The command I'm running is just rotsniff append .

And that is the question: is it supposed to take that much longer?
My uninformed gut feeling is that something is wrong :)

My observations:

• rotsniff starts with a rather high CPU usage, so I assume it is multi-threaded by default; looks like 12 threads (= number of virtual cores) are created
• SSD read speeds are much lower than for the python tool; speed seems to be capped at around 150-160 mbytes/sec (while python tool definitely had 500+ mbytes/sec peaks). While read speed is low, SSD activity (number of requests) is very high. I wonder if there is a too small reading buffer somewhere? (This is further suggested by threads spending lots of time in a waiting state...)
• very high RAM usage: up to 30GB VIRT and 15 GB RES (based on a random htop observation for a minute or so); it does decrease at times, but seems to be in multi-gigabyte (4-6+ or so?) RES range at all times.