Hey,

can you add Pushbullet to this script?

I'd like the ability to print the entire diff log to the email / console output so I can see all new / modified / deleted etc files in the email and log.

Does this still function with Snapraid 10.xx? I would like to know before setting up in case it could mess up my setup somehow. Thanks so much.

Hello all, this might be a noob question but ever since gmail removed their less secure app access, I haven't been able to get any success or failure emails to send.

Is there a way to add in the app key that google has you generate to get these success/error emails to send?

Please help, thank you!

This is along similar lines to adding smartmon support, just with less output handling. It would be helpful to add an option to allow for arbitrary commands to be run before and/or after snapraid-runner. For example, I need to stop an instance of Plex Media Server before running a snapraid sync, and then start Plex Media Server again afterwards. This is as dumb as running a "docker stop plex" before running the snapraid processes and a "docker start plex" after.

Without built-in support for this, I need to wrap my snapraid-runner.py inside of a snapraid-runner-runner.sh, which isn't terribly elegant but works.

The main issue I can see is how to handle the output from these commands. I'm honestly not too sure what the best way to go on that would be, it's been a while since I messed around with Python subprocess handling.

The log file writes fine when the python script is run from the command line, but it does not run when run as a scheduled cron job. I get an email with the log output, so it must be storing somewhere, but I'm not sure where.

Anyone else have this issue?

I'm getting the following syntax error when running the script:

File "/opt/snapraid-runner/snapraid-runner.py", line 42
def snapraid_command(command, args={}, *, allow_statuscodes=[]):

I'm not programmer by a long shot, but I'll be happy to provide any info. I have not altered the python file in any way.
Thanks

I am using this docker image, https://github.com/xagaba/snapraid which has snapraid runner and snapraid bundled together. I started experiencing failures with the snapraid runner script. Initially it had failed because by deletethreshold of 500 was hit, but it was intentional. I did not end up correcting it right away, and it failed consecutive nights. But now the errors is different, and the snapraid diff actually exits quite early into the process.

/config # /usr/bin/python3 /app/snapraid-runner/snapraid-runner.py -c /config/snapraid-runner.conf
2022-03-29 00:16:16,672 [INFO  ] ============================================================
2022-03-29 00:16:16,692 [INFO  ] Run started
2022-03-29 00:16:16,692 [INFO  ] ============================================================
2022-03-29 00:16:16,693 [INFO  ] Running diff...
2022-03-29 00:16:16,697 [OUTPUT] Loading state from /mnt/disk1/snapraid.content...
2022-03-29 00:16:26,886 [OUTERR] WARNING! With 5 disks it's recommended to use two parity levels.
2022-03-29 00:16:26,886 [OUTPUT] Comparing..
2022-03-29 00:16:26,888 [OUTPUT]
..add xyz
..add xyz
..add xyz
..add xyz
2022-03-29 00:12:17,793 [ERROR ] Run failed due to unexpected exception:
Traceback (most recent call last):
  File "/app/snapraid-runner/snapraid-runner.py", line 235, in main
    run()
  File "/app/snapraid-runner/snapraid-runner.py", line 262, in run
    diff_out = snapraid_command("diff", allow_statuscodes=[2])
  File "/app/snapraid-runner/snapraid-runner.py", line 65, in snapraid_command
    raise subprocess.CalledProcessError(ret, "snapraid " + command)
subprocess.CalledProcessError: Command 'snapraid diff' died with <Signals.SIGSEGV: 11>.
2022-03-29 00:12:18,792 [ERROR ] Run failed

snapraid-runner.conf

[snapraid]
; path to the snapraid executable (e.g. /bin/snapraid)
executable = /usr/bin/snapraid
; path to the snapraid config to be used
config = /config/snapraid.conf
; abort operation if there are more deletes than this, set to -1 to disable
deletethreshold = 8600
; if you want touch to be ran each time
touch = false

[logging]
; logfile to write to, leave empty to disable
file = snapraid.log
; maximum logfile size in KiB, leave empty for infinite
maxsize = 10000

[scrub]
; set to true to run scrub after sync
enabled = false
percentage = 12
older-than = 30

Scripts is functioning properly, except for dispatching email. I got the following error when i ran the script manually.

2016-04-10 04:08:33,672 [ERROR ] Failed to send email Traceback (most recent call last): File "/opt/snapraid-runner/snapraid-runner.py", line 107, in finish send_email(is_success) File "/opt/snapraid-runner/snapraid-runner.py", line 92, in send_email server = smtplib.SMTP_SSL(**smtp) File "/usr/lib/python2.7/smtplib.py", line 792, in __init__ SMTP.__init__(self, host, port, local_hostname, timeout) File "/usr/lib/python2.7/smtplib.py", line 256, in __init__ (code, msg) = self.connect(host, port) File "/usr/lib/python2.7/smtplib.py", line 316, in connect self.sock = self._get_socket(host, port, self.timeout) File "/usr/lib/python2.7/smtplib.py", line 797, in _get_socket new_socket = socket.create_connection((host, port), timeout) File "/usr/lib/python2.7/socket.py", line 553, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): gaierror: [Errno -2] Name or service not known

Would it be possible to pass on Scrub option like : scrub -p new

Been running well for a long time - tx. I had to recently change my email password and now gmail is being blocked. The error from Gmail is... "Google will continue to block sign-in attempts from the app you're using because it has known security problems or is out of date. You can continue to use this app by allowing access to less secure apps, but this may leave your account vulnerable." I am using SSL and port 465 and don't understand why it was working fine before.

Would it make sense to auto run sudo snapraid -e fix after a scrub detects an error? It could be implemented easily, I just wondered if there's reasoning against it being automatic? I seem to get an error almost every other sync, so would be nice if it automatically corrected them.

At the moment we ignore the return value of snapraid diff, because it used to be useless. This is not the case anymore, so we should check for errors.

I'm getting this error when I try to use snapraid runner. I'm guessing its probably down to me rather than the script. Maybe a problematic character in a file name?

I'm using windows 10 pro, latest version of python installed.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "D:\Utilities\Python\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "D:\Utilities\Python\lib\threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Utilities\Snapraid Runner\snapraid-runner.py", line 34, in tee_thread
    for line in iter(infile.readline, ""):
  File "D:\Utilities\Python\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 890: character maps to <undefined>

in setup_logger() where the max_log_size is computed, should it not rather be max(config["logging"]["maxsize"],0) * 1024 instead of min(...,0) as we want to have the bigger number and not always "0"?

Would be really nice to have env vars support in .conf file or any other way to manage secrets there 👍

Hi, my emails from the runner were not delivered because of mail server complaining about the missing Date: header. I fixed this locally by importing email.utils and adding the header to the message using msg["Date"] = formatdate(), but I am in no way a python programmer so there might be a simpler/better solution. Anyway, might be something to consider in the future update.

Hello,

I had this error message with " host = smtp.gmail.com" :

2015-04-27 21:47:08,792 [ERROR ] Failed to send email
Traceback (most recent call last):
File "C:\snapraid-movies1\runner\snapraid-runner.py", line 108, in finish
send_email(is_success)
File "C:\snapraid-movies1\runner\snapraid-runner.py", line 93, in send_email
server = smtplib.SMTP_SSL('smtp.gmail.com', 587)
File "C:\Program Files (x86)\Python\lib\smtplib.py", line 792, in init
SMTP.init(self, host, port, local_hostname, timeout)
File "C:\Program Files (x86)\Python\lib\smtplib.py", line 256, in init
(code, msg) = self.connect(host, port)
File "C:\Program Files (x86)\Python\lib\smtplib.py", line 316, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "C:\Program Files (x86)\Python\lib\smtplib.py", line 798, in _get_socket
new_socket = ssl.wrap_socket(new_socket, self.keyfile, self.certfile)
File "C:\Program Files (x86)\Python\lib\ssl.py", line 891, in wrap_socket
ciphers=ciphers)
File "C:\Program Files (x86)\Python\lib\ssl.py", line 566, in init
self.do_handshake()
File "C:\Program Files (x86)\Python\lib\ssl.py", line 788, in do_handshake
self._sslobj.do_handshake()
SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:581)

adding "server.starttls()" and some ehlo to :

if config["smtp"]["user"]:
    server.ehlo()
    server.starttls()
    server.ehlo()
    server.login(config["smtp"]["user"], config["smtp"]["password"])
server.sendmail(
    config["email"]["from"],
    [config["email"]["to"]],
    msg.as_string())
server.quit()

solved my issue :)

Getting an error when I try and run.

Command to run:

sudo python3 /opt/snapraid-runner/snapraid-runner.py -c /opt/snapraid-runner/snapraid-runner.conf

Error:

2020-08-04 14:41:42,612 [INFO  ] ============================================================
2020-08-04 14:41:42,612 [INFO  ] Run started
2020-08-04 14:41:42,613 [INFO  ] ============================================================
2020-08-04 14:41:42,613 [INFO  ] Running touch...
2020-08-04 14:41:42,613 [ERROR ] Run failed due to unexpected exception:
Traceback (most recent call last):
 File "/opt/snapraid-runner/snapraid-runner.py", line 233, in main
   run()
 File "/opt/snapraid-runner/snapraid-runner.py", line 256, in run
   snapraid_command("touch")
 File "/opt/snapraid-runner/snapraid-runner.py", line 58, in snapraid_command
   errors="replace"
TypeError: __init__() got an unexpected keyword argument 'encoding'

Versions:
Ubuntu: 16.04
Python: 3.5.2
Snapraid: 11.5
Snapraid Runner: Current

@Chronial

So the support for the "plan" switch for scrub is still under "Unreleased" in the changelogs. But plan is already in your snapraid-runner.conf.example file. Does that mean that this plan parameter will still work with the current release of your py script (0.5)?

Hello Chronial, this is not an issue, more like a discussion, but I wanted you to be aware that I've taken your snapraid-runner and extended it to use snapraid-btrfs, and have aptly named it snapraid-btrfs-runner. It serves the same purpose as snapraid-runner and has some additional features relevant to btrfs snapshot usage with snapraid. I had originally forked snapraid-runner early last year and put in what amounted to a half-hearted effort. I've now created a new repo based on a recent development commit here.

Just an FYI to you that I'll be closely following development for snapraid-runner as the majority of the Python code is directly from this source. Thanks a lot for creating and maintaining this tool and hopefully we'll continue to touch base!

Hi,
my snapraid-runner instance is not writing to log.
My /opt/snapraid-runner/snapraid-runner.conf:

[logging]
; logfile to write to, leave empty to disable
file = /home/mikulik86/snapraid.log
; maximum logfile size in KiB, leave empty for infinite
maxsize = 50000

What am I doing wrong?

How to add timestamp in log file "2015-04-27_Snapraid.log"

1- add this line in # Global variables in snapraid-runner.py
dateTimeStamp = time.strftime('%Y-%m-%d') #in the format YYYY-MM-DD

2- add a file path in the config file :
[logging]
; log path to write to
filepath = C:\snapraid\logs
; logfile name to write to, leave empty to disable
file = _Snapraid-runner.log
; maximum logfile size in KiB, leave empty for infinite
maxsize = 5000

3- change this section :
if config["logging"]["file"]:
max_log_size = min(config["logging"]["maxsize"], 0) * 1024
file_logger = logging.handlers.RotatingFileHandler(
config["logging"]["filepath"] + dateTimeStamp + config["logging"]["file"],
maxBytes=max_log_size,
backupCount=9)
file_logger.setFormatter(log_format)
root_logger.addHandler(file_logger)

2021-02-22 05:00:05,418 [OUTPUT] update "PoolPart.f864afd4-776e-4e9d-8df8-42c18f34f8eb/Music/AURORA/All My Demons Greeting Me as a Friend (2016)/02 - Conqueror.flac"
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "C:\snapraid\Scripts\snapraid-runner.py", line 34, in tee_thread
    for line in iter(infile.readline, ""):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 3447: character maps to <undefined>

This is not confined to any particular file and occurs on every run attempt.

System info:

Edition	Windows 10 Pro
Version	20H2
Installed on	‎24-‎Oct-‎20
OS build	19042.804
Experience	Windows Feature Experience Pack 120.2212.551.0
Python 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)] on win32

Seeing the commit 4cb7e69, I thought I'd test using --plan new. It turns out that Snapraid will error out if --older-than is specified with a non-numerical --plan. I'm suggesting a fix that simply ignores that config item if a user specifies a scrub plan new, bad, or full.

Snippet from config:

[scrub]
; set to true to run scrub after sync
enabled = true
plan = new
older-than = 20

Current output:

2021-10-29 23:37:06,879 [INFO  ] Running scrub...
2021-10-29 23:37:06,882 [OUTPUT] Self test...
2021-10-29 23:37:07,091 [OUTPUT] Loading state from /var/snapraid.content...
2021-10-29 23:37:07,095 [OUTPUT] Using 1 MiB of memory for the file-system.
2021-10-29 23:37:07,095 [OUTERR] You can specify -o, --older-than only with a numeric percentage.
2021-10-29 23:37:07,096 [OUTPUT] Initializing...
2021-10-29 23:37:07,413 [ERROR ] Command 'snapraid scrub' returned non-zero exit status 1.
2021-10-29 23:37:16,216 [ERROR ] Run failed

My proposed fix in fmoledina@62e0357.

https://github.com/fmoledina/snapraid-runner/blob/62e03579cf7dabf42799b82e30a9941071a41f2f/snapraid-runner.py#L291-L311

Output after fix:

2021-10-29 23:40:21,619 [INFO  ] Running scrub...
2021-10-29 23:40:21,619 [WARNIN] Ignoring 'older-than' config item with scrub plan 'new'
2021-10-29 23:40:21,622 [OUTPUT] Self test...
2021-10-29 23:40:21,871 [OUTPUT] Loading state from /var/snapraid.content...
2021-10-29 23:40:21,877 [OUTPUT] Using 1 MiB of memory for the file-system.
2021-10-29 23:40:21,877 [OUTPUT] Initializing...
2021-10-29 23:40:21,878 [OUTPUT] Scrubbing...
2021-10-29 23:40:21,996 [OUTPUT] Using 24 MiB of memory for 32 cached blocks.
2021-10-29 23:40:22,645 [OUTPUT] 
2021-10-29 23:40:22,645 [OUTPUT]      d1 96% | **********************************************************
2021-10-29 23:40:22,645 [OUTPUT]  parity  0% |
2021-10-29 23:40:22,646 [OUTPUT]    raid  2% | *
2021-10-29 23:40:22,646 [OUTPUT]    hash  0% |
2021-10-29 23:40:22,646 [OUTPUT]   sched  0% |
2021-10-29 23:40:22,646 [OUTPUT]    misc  0% |
2021-10-29 23:40:22,646 [OUTPUT]             |______________________________________________________________
2021-10-29 23:40:22,646 [OUTPUT]                            wait time (total, less is better)
2021-10-29 23:40:22,647 [OUTPUT] 
2021-10-29 23:40:22,647 [OUTPUT] Everything OK

If this looks good to you, I'll submit a PR for fmoledina@62e0357.

Thanks!

17 pull requests not closed or merged and no activity or releases ? Does this need forking ? Just curious

Hi Christian, I am not sure how to ask you for help in github so hopefully this message is ok.

I am a newbie to Python. I have managed to get your script working and it is great. The only thing I can't get working is the email. What parts of the script needs to be updated with my address, server, PW etc?

I would really appreciate if you can point me in the right direction.

2020-04-17 09:00:57,969 [INFO ] ============================================================
2020-04-17 09:00:57,970 [INFO ] Run started
2020-04-17 09:00:57,970 [INFO ] ============================================================
2020-04-17 09:00:57,971 [INFO ] Running diff...
2020-04-17 09:00:58,015 [OUTPUT] Loading state from D:/snapraid.content...
2020-04-17 09:01:07,945 [OUTPUT] Comparing...
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Soft\Python38-32\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Soft\Python38-32\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "F:\SnapRAID\snapraid-runner.py", line 34, in tee_thread
for line in iter(infile.readline, ""):
UnicodeDecodeError: 'gbk' codec can't decode byte 0x92 in position 1096: illegal multibyte sequence

have been using Snapraid (using the Snapraid-Runner scripts) for a few months and everything works fine. My wife has been moving lots of files around and today I noticed that it is not deleting because the script has a delete threshold of 80. I updated the snapriad-runner.conf file to deletethreshold = 6000 and it died when i tried to run it. This is the message I get

2022-03-10 15:18:01,140 [INFO ] ============================================================

2022-03-10 15:18:01,141 [INFO ] Run started

2022-03-10 15:18:01,141 [INFO ] ============================================================

2022-03-10 15:18:01,142 [INFO ] Running diff...

2022-03-10 15:18:01,161 [OUTERR] Invalid command '{\rtf1\ansi\deff0\nouicompat{\fonttbl{\f0\fnil\fcharset0' in 'C:/Snapraid/snapraid.conf' at line 1

2022-03-10 15:18:01,462 [ERROR ] Run failed due to unexpected exception:

Traceback (most recent call last:)

File "c:\Snapraid\snapraid-runner-0.5-daily\snapraid-runner.py", line 233, in main

run()

File "c:\Snapraid\snapraid-runner-0.5-daily\snapraid-runner.py", line 260, in run

diff_out = snapraid_command("diff", allow_statuscodes=[2])

File "c:\Snapraid\snapraid-runner-0.5-daily\snapraid-runner.py", line 72, in snapraid_command

raise subprocess.CalledProcessError(ret, "snapraid " + command)

subprocess.CalledProcessError: Command 'snapraid diff' returned non-zero exit status 1.

I have tried changing the threshold to -1 which should turn it off completely but I get the same message. I am pretty sure I am saving the file wrong but not sure what I am doing (I can tell in notepad that it is saving as UTF-8), if that matters.

This is a great script!

Is there any way I can use more arguments of snapraid?
Like --force-zero and --pre-hash.

Allow snapraid runner to run the smart command during its run, so the output of the smart command can be included in the status emails.

This was a feature request by fubduck. I might implement it at some point if I find time. Pull requests are welcome.

You seem to be passing a --percentage option to Snapraid for scrub. Oddly enough, It seems to actually be working, even though that option is not documented. I think to be sure the script keeps working as intended, the option should be switched to --plan [percent #].

Also, consider running scrub with the --quiet option, otherwise the output log is huge since there are 100 lines of scrub output like this: "[OUTPUT] 81%, 350886 MB, 127 MB/s, 487 block/s, CPU 16%, 0:06 ETA"

We have two PRs (#26, #46) that want to run specific non-snapraid functionality before / after the snapraid run itself. I consider these specific things clearly out of scope for snapraid-runner, but having a way to run a custom command within snapraid-runner's logging/email umbrella seems a rather sensible thing to have.

This could probably be solved by having two "pre-run-command", "post-run-command" config options.

Hi I am trying to figure out how to run this script on a cron job,

this is what I got so far

cron entry:

06 15 * * * /usr/bin/python3 /opt/snapraid-runner-0.4/snapraid-runner.py >> ~/cron.log 2>&1

what I get on the logs is this:

snapraid-runner configuration file not found
usage: snapraid-runner.py [-h] [-c CONFIG] [--no-scrub]

optional arguments:
-h, --help show this help message and exit
-c CONFIG, --conf CONFIG
Configuration file (default: snapraid-runner.conf)
--no-scrub Do not scrub (overrides config)
~
I guess I have to add the configuration file somehow.

Thanks for the amazing script! :)

They just say none.

The PRs show that there is a clear demand for other notification services beside email. We have Telegram, Slack, Pushbullet, Apprise and Discord. I do not want to maintain those and consider them out of scope for snapraid-runner. It might be a good idea to add a simple, configurable hook-in point with well-defined API so others can easily provide plugins for their favorite notification service.

On my fork reed-jones/snapraid-runner I have modified the script to add python3 support. I haven't tested the email functions yet (and I haven't changed them), but everything python2 should still work the exact same. I would be happy to PR the changes back if @Chronial is interested.