Hi
Is there a way to ignore certain directories? I'm trying to skip all .something dirs (and theirs content), but fnfilter switch seems to apply only to files (or using it wrong). Even though, dirs technically are also files... :P

I get the following error when I try to run the integrity check on a file that includes an emoji in the name. This happens even after I renamed the file, I suspect it's baked in the database now. Here is the error:

Traceback (most recent call last):
  File "/volume1/system/scorch/scorch.py", line 640, in inst_check
    newfi = get_fileinfo(filepath)
  File "/volume1/system/scorch/scorch.py", line 404, in get_fileinfo
    st = os.lstat(filepath)
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f917' in position 56: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/volume1/system/scorch/scorch.py", line 1647, in <module>
    main()
  File "/volume1/system/scorch/scorch.py", line 1626, in main
    rv = rv | func(opts,directory,db,dbremove)
  File "/volume1/system/scorch/scorch.py", line 707, in inst_check
    print_filepath(filepath,actions,total,opts.quote)
  File "/volume1/system/scorch/scorch.py", line 432, in print_filepath
    print(s,end=end)
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f917' in position 66: ordinal not in range(128)

Hi @trapexit

I find your tool useful to ensure that photos/music don't corrupted over time / while copying on other devices specifically for setups without RAID.

I developed a small wrapper to send scorch's output as email (and made the configuration a bit more readable)
https://gitlab.com/daufinsyd/robespierre
I wanted to let you know :)

Thank you for your great software btw !

How about a function for scorch where the hash database is checked for files that are missing - i.e. a hash is in the databasem but the corresponding file is not found in the expected path.

The use case for such a function could be if you decide to live without a RAID but want to know which files are gone for good after a disk fails faster than you can move the files off the platters.

Hi
I do some test
You have a bug
def is_readonly(st):
return not (fi.mode & (stat.S_IWUSR | stat.S_IWGRP | stat.S_IWOTH))

may be
def is_readonly(fi):

Hi,
It would be great if the tool would support portability of db.

In my case I have data that I backup to at least 2 drives and I want to verify that all drives contain the same data.
In my case i have them mounted here /mnt/y and /mnt/v
I want to check both drives using the same database but create only hashes from one of them.

So maybe just another argument that would replace /mnt/y with /mnt/v/ prefix in memory would be sufficient.

I will try to look into the code and create pull request if I find time.

Thanks,
Petr

Hello!
I have found scorch while researching data integrity tools available for Linux and one of the users mentioned your script.

The script works beautifully and works as expected and thank you for creating it - great job!

Would it be possible to adjust it so it would only print information when there is actual data integrity problem ("FAILED")?

When used with crontab this would allow to setup cron job and cron by default sends e-mail to the user when there is any output on STDOUT (ie. root).

Right now the script also shows information about "CHANGED" hashes and defeats the purpose without additional scripting.

What do you think?

if a file has 0 bytes (difficult but i have them...)
you must check it because log of 0 is error

def humansize(nbytes):
suffixes = ['B','KB','MB','GB','TB','PB','ZB']
rank = 0
if(nbytes > 0):
rank = int(math.log(nbytes,1024))

Is it ok to support a new (fast) hashing function that is not in python's hashlib?

BLAKE3 looks really interesting.

Initial implementation would be something along the lines of https://github.com/aa956/scorch/tree/blake3-hash - try to import blake3 and use if it's installed.

Hi,

When running a checksum that takes a few hours (ex: a dir with many large files), if the machine crashes or scorch exits before it completes, all results are lost.

It would be nice if scorch could periodically write-to-disk_ to prevent losing many hours of work. My suggestion would be to write every five minutes or every N entries (perhaps configurable?). Thanks!

Several free software photography tools like darktable store all the metadata, and the changes they make in the image in an xmp file with the same name. Original file is never touched.
Would it be possible to give scorch a feature where the checksum information is stored in these same xmp files instead of the database file?
The main benefits of this, is that other apps like darktable or digikam that will also be able to access and use the scorch information from the xmp file. Scorch could even be integrated and launched from these apps directly.
Most photographs don't store checksums of their photos, but just like everybody else , they suffer the consequences when it happens. This way, as soon as data corruption occurs it can be detected and the photographer can delete the corrupt file and restore a backup of just that file.

Sorry if this is the wrong spot for a feature request, but I use MergerFS also (awesome and flawless), and have been using scorch to check for corruption (also awesome). But due to the 4 8TB HDDs i have 85% full, it takes days for scorch to complete. I was wondering if it would be possible to somehow integrate with mergerfs and parallel check all the drives I have under my /mnt . Or if there is a way to better do that already that doesnt miss out on potential corruption when I use mergerfs.balance. Thanks for the awesome software!

Thanks for the great software. Scorch is fantastic!

I've been trying Scorch on a system where I have lots of legitimate changes on the filesystem that I'm running it on, but also want to be able to detect bitrot on files that are mixed in that don't change.

I'm scripting the use of Scorch and deciding what to do based on the exit code I receive back from Scorch.

Is there anyway to distinguish between files that have legitimately changed on the file system (I'm happy to ignore these), vs bitrot that I do care about? At the moment both seem to result in a exit code of 4 from Scorch and it makes it difficult for me to distinguish between the two use cases.

I realise that the log messages do show the difference, but for follow on processing I'd really like the distinction to come from the exit codes.

Would it be possible to add an additional exit code that is just for file changes without the metadata changing?

My current workflow is (feel free to let me know if I'm using Scorch wrong):

scorch -D 'size,mtime' -d ./scorch.db check+update /data
scorch -D 'size,mtime' -d ./scorch.db append /data
scorch -D 'size,mtime' -d ./scorch.db cleanup /data

I have tried running scorch update then scorch check but in the intervening period the files can change so I still get errors.

If you didn't like the idea of a separate error code for file corruption, rather than file changes and corruption, would it be possible to add new command scorch update+check which updates the metadata first and then does the hash check?

Hi @trapexit,

I've been using Scorch on a RaspberryPi 3 today and noticed that it exists after a couple of hours (exit code 0) without having processed all the files.

This is caused by the use of sys.maxsize for a number of the default parameter values in Scorch. On the RaspberryPi, due to its CPU architecture, Python3's sys.maxsize == 2147483647, which I believe is causing Scorch to exit.

I know that I can change these options on the command line, but I was also wondering if it might make sense to change the default values instead? Perhaps to 2**64/2?? Which is the default of sys.maxsize on most x64 systems.

Have a great weekend!