sinnwerkstatt / runrestic Goto Github PK

View Code? Open in Web Editor NEW

113.0 113.0 30.0 456 KB

A wrapper script for Restic backup software that inits, creates, prunes and checks backups

License: GNU General Public License v3.0

Python 98.38% Dockerfile 1.62%

automation backups prometheus-metrics restic restic-backups

runrestic's People

Contributors

Stargazers

Watchers

runrestic's Issues

runrestic hangs when restic unexpectedly exits

# cat /etc/runrestic/home.toml
name = "home directories"
repositories = [ "rclone:test",]

[execution]
parallel = true
retry_count = 10
retry_backoff = "1:00 exponential"

[environment]
RESTIC_PASSWORD_FILE = "/etc/secrets/restic.password"
RCLONE_CONFIG = "/etc/secrets/rclone.conf"

[backup]
sources = [ "/home",]
exclude_patterns = [ "/home/_sysupgrade",]

[prune]
keep-last = 2
group-by = "host,paths"

[check]
checks = [ "check-unused", "read-data",]

[metrics.prometheus]
path = "/tmp/runrestic_home.prom"

# time /usr/local/virtualenvs/runrestic/bin/runrestic -c /etc/runrestic/home.toml -l debug -- --tag home
Parsing configuration file: /etc/runrestic/home.toml
[Environment] RESTIC_PASSWORD_FILE=/etc/secrets/restic.password
[Environment] RCLONE_CONFIG=/etc/secrets/rclone.conf
Spawning "['restic', '-r', 'rclone:test', 'backup', '--tag', 'home', '--exclude', '/home/_sysupgrade', '/home']"
Spawning "['restic', '-r', 'rclone:test', 'forget', '--tag', 'home', '--keep-last', '2', '--group-by', 'host,paths']"
Spawning "['restic', '-r', 'rclone:test', 'prune', '--tag', 'home']"

and just hangs there until I ^C out of it. Meanwhile

# RESTIC_PASSWORD_FILE=/etc/secrets/restic.password RCLONE_CONFIG=/etc/secrets/rclone.conf restic -r rclone:test prune --tag home ; echo $?
unknown flag: --tag
1

Arch Linux package builds, but errors when run

If I'm lucky, you'll recognize exactly what I'm doing wrong. Otherwise, I'll continue to dig into what is going on... probably something silly on my part. ;)

0 ✓ fryfrog@apollo ~ $ runrestic
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2451, in resolve
    return functools.reduce(getattr, self.attrs, module)
AttributeError: module 'runrestic.runrestic.runrestic' has no attribute 'main'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/runrestic", line 11, in <module>
    load_entry_point('runrestic==0.5.0', 'console_scripts', 'runrestic')()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2453, in resolve
    raise ImportError(str(exc))
ImportError: module 'runrestic.runrestic.runrestic' has no attribute 'main'

Somewhere the return code is still called "rc"

Hey @palto42 ,

I think you missed a rc somewhere - do you have time to check real quick?

Aug 16 02:51:34 bkp1 runrestic[1530969]:       "total_file_count": 250361,
Aug 16 02:51:34 bkp1 runrestic[1530969]:       "total_size_bytes": 1324666550,
Aug 16 02:51:34 bkp1 runrestic[1530969]:       "duration_seconds": 7.828546524047852,
Aug 16 02:51:34 bkp1 runrestic[1530969]:       "rc": 0
Aug 16 02:51:34 bkp1 runrestic[1530969]:     }
Aug 16 02:51:34 bkp1 runrestic[1530969]:   },
Aug 16 02:51:34 bkp1 runrestic[1530969]:   "last_run": 1660611094.014,
Aug 16 02:51:34 bkp1 runrestic[1530969]:   "total_duration_seconds": 56.1098952293396
Aug 16 02:51:34 bkp1 runrestic[1530969]: }
Aug 16 02:51:34 bkp1 runrestic[1530969]: Traceback (most recent call last):
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/bin/runrestic", line 8, in <module>
Aug 16 02:51:34 bkp1 runrestic[1530969]:     sys.exit(runrestic())
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/runrestic/runrestic.py", line 81, in runrestic
Aug 16 02:51:34 bkp1 runrestic[1530969]:     result.append(runner.run())
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/restic/runner.py", line 73, in run
Aug 16 02:51:34 bkp1 runrestic[1530969]:     write_metrics(self.metrics, self.config)
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/metrics/__init__.py", line 12, in write_metrics
Aug 16 02:51:34 bkp1 runrestic[1530969]:     file.writelines("".join(lines))
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/metrics/prometheus.py", line 205, in generate_lines
Aug 16 02:51:34 bkp1 runrestic[1530969]:     yield backup_metrics(metrics["backup"], name)
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/metrics/prometheus.py", line 222, in backup_metrics
Aug 16 02:51:34 bkp1 runrestic[1530969]:     retval += _restic_pre_hooks.format(name=name, **mtrx)
Aug 16 02:51:34 bkp1 runrestic[1530969]: KeyError: 'rc'

cheers

Broken backup output parsing with Restic 0.14.0

I guess Restic output parsing broken again after Restic update to 0.14.0. In this case after backup operation.

Before something similar hapened with prune as #51

This happens with Restic 0.14.0 and RunRestic 0.5.26.1 from pip.

$ restic version
restic 0.14.0 compiled with go1.19 on linux/arm64
$ runrestic -v
runrestic
0.5.26.1

Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 8, in <module>
    sys.exit(runrestic())
  File "/usr/local/lib/python3.8/dist-packages/runrestic/runrestic/runrestic.py", line 81, in runrestic
    result.append(runner.run())
  File "/usr/local/lib/python3.8/dist-packages/runrestic/restic/runner.py", line 56, in run
    self.backup()
  File "/usr/local/lib/python3.8/dist-packages/runrestic/restic/runner.py", line 143, in backup
    metrics[redact_password(repo, self.pw_replacement)] = parse_backup(
  File "/usr/local/lib/python3.8/dist-packages/runrestic/restic/output_parsing.py", line 19, in parse_backup
    added_to_the_repo = re.findall(
IndexError: list index out of range

Of course, this is fixed by downgrading Restic to 0.13.1: https://github.com/restic/restic/releases/tag/v0.13.1

So not an urgent issue

support for backup from stdin

https://restic.readthedocs.io/en/latest/040_backup.html#reading-data-from-stdin

`runrestic snapshots` should not just fall back to defaults actions

the https://github.com/sinnwerkstatt/runrestic/blob/master/runrestic/runrestic/configuration.py#L24 fucks with unknown arguments.

when somebody accidentally does runrestic snapshots it falls back to default actions - not good

invite to collaborate here.

hey @palto42 I send you an invite to this repo. If you like, some help is more than welcome here :)

support for healthchecks and similar services

borgmatic supports e.g. healthchecks, see https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/

I tried to archive something similar with pre/post_hooks in runrestic and curl, but had no luck as I was unable to get the errorcode.

Partial duplication of Backupninja?

Some of the core features of this project appear to be duplicating Backupninja. We are very close to adding support for Restic in the merge request Add restic support.

Would it be possible to:

Test Backupninja with Restic support, and help move that initiative forward, and also
Keep this module for Restic-specific things that can't logically be added to Backupninja?

Cache handling

Optimized cache management is important to keep cloud storage costs under control.

I would like to use --cache-dir for all operations. Having this built-in makes sure I don't forget it for any command.

The other options like --with-cache/--no-cache can be specified via -- --option if you don't want to add official support.

Furthermore, behavior different from restic default should be documented:

runrestic/runrestic/restic/tools.py

Lines 89 to 92 in 7622739

 if os.geteuid() == 0: # pragma: no cover; if user is root, we just use system cache 

 os.environ["XDG_CACHE_HOME"] = "/var/cache" 

 elif not (os.environ.get("HOME") or os.environ.get("XDG_CACHE_HOME")): 

 os.environ["XDG_CACHE_HOME"] = "/var/cache"

runrestic exit with status code 0 when parsing fails

This makes, for example, systemd thinks everything is okay, when it is not.

# systemctl  status runrestic
● runrestic.service - runrestic backup
bash-4.2# systemctl status runrestic.service
● runrestic.service - runrestic backup
   Loaded: loaded (/etc/systemd/system/runrestic.service; static; vendor preset: disabled)
   Active: inactive (dead) since Tue 2020-03-10 12:42:04 CET; 2h 5min ago
  Process: 17443 ExecStart=/usr/local/bin/runrestic (code=exited, status=0/SUCCESS)
 Main PID: 17443 (code=exited, status=0/SUCCESS)

Mar 10 12:42:04 myhostname systemd[1]: Starting runrestic backup...
Mar 10 12:42:04 myhostname runrestic[17443]: Problem parsing /etc/runrestic.toml: invalid literal for int() with base 0: '4...r 360)
Mar 10 12:42:04 myhostname systemd[1]: Started runrestic backup.
Hint: Some lines were ellipsized, use -l to show in full.

Broken prune output parsing with Restic 0.13.0

Just updated to Restic 0.13.0 and faced a traceback in the prune stage, which is likely caused by changed output

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/runner.py", line 204, in prune
    process_infos
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/output_parsing.py", line 64, in parse_prune
    )[0]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 10, in <module>
    sys.exit(runrestic())
  File "/usr/local/lib/python3.7/dist-packages/runrestic/runrestic/runrestic.py", line 75, in runrestic
    runner.run()
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/runner.py", line 57, in run
    self.prune()
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/runner.py", line 210, in prune
    ] = parse_new_prune(process_infos)
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/output_parsing.py", line 105, in parse_new_prune
    )[0]
IndexError: list index out of range

Support Python 3.10 - EOL of Python 3.6

Hi @andreasnuesslein,
What do you think of a general version update for this module since Python 3.6 is end of life already and latest Python is meanwhile 3.10?

The main changes in my view would be:

Change min supported version to 3.8 (or 3.7?)
Update the dependencies (poetry.lock)
Change versions tested in Travis to 3.8, 3.9 and 3.10

I'm not expecting any direct impact on the code, it would be mainly to catch-up with Python release cycle.
Since my main laptop uses Python 3.10 I've already using an update version of runrestic, including updated dependencies via Poetry.

config file command line argument

It will be good if runrestic will have command line argument for set config file path. a.g.: runrestic -c /mypath/my-restic.conf
It will be useful when have 2 different repositories.

Ansible role

This is not an issue but might be relevant for anyone who is looking for a (run)restic ansible role. Thank you for this amazing restic wrapper!

I created an ansible role which install restic + runrestic and configures runrestic to create the backups, similar to what the borgbase ansible role is for borgmatic.

GitHub: https://github.com/outwire/ansible-role-restic
Ansible Galaxy: https://galaxy.ansible.com/outwire/restic

Fix the setup.py in pypi

The setup.py that comes out of your pypi package uses a handful of setuptools things, but has from distutils.core import setup which means they don't work. Maybe something about poetry change could fix that?

jsonschema 3.2.0 does not satisfy "jsonschema==3.0.*,>=3.0.0"

Shouldn't a jsonschema version of 3.2.0 satisfy the >=3.0.0 portion of this from requirements.txt?

community/python-jsonschema 3.2.0-1 (93.4 KiB 582.6 KiB) (Installed)
    An implementation of JSON Schema validation for Python

0 ✓ fryfrog@apollo ~/aur/runrestic $[master] runrestic
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (jsonschema 3.2.0 (/usr/lib/python3.8/site-packages), Requirement.parse('jsonschema==3.0.*,>=3.0.0'), {'runrestic'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/runrestic", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3252, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonschema==3.0.*,>=3.0.0' distribution was not found and is required by runrestic

Support Docker

Hi,
Thank you for the wonderful work.
I am trying to run this in docker container. But unable to do so.

Can anyone can help please?

add RESTIC_PASSWORD_COMMAND

Hello!

First of all, thank you for this program.

I'm using keyring to store password for restic repo https://pypi.org/project/keyring/

in restic, it's used as RESTIC_PASSWORD_COMMAND env var

but the problem is, in runrestic json schema command is not defined as a valid value

https://github.com/sinnwerkstatt/runrestic/blob/6c10fedc1814e760e8ec41c5d167463d3f9a16e1/runrestic/runrestic/schema.json#L46C1-L49C8

so, I'm forced to specify at least this RESTIC_PASSWORD_FILE=""

AttributeError: 'ResticRepository' object has no attribute 'log'

hi all, everything working fine, just getting this error after backup:

Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/runrestic/commands/runrestic.py", line 127, in main
    run_configuration(config, args)
  File "/usr/local/lib/python3.7/site-packages/runrestic/commands/runrestic.py", line 96, in run_configuration
    logs['repositories'][repository] = repo.log
AttributeError: 'ResticRepository' object has no attribute 'log'

any ideas on that? Thanks

Runrestic on Windows

Hi!
After a few test I can conclude that runrestic is not runnable on Windows.

Is it possible to make it work on it?
It will be possible on the future? Is it planed?

Thank you in advance

add --update-restic flag

Use native bindings instead of string parsing

Use gopy to create the bindings to restic/cmd/cmd_backup.go#runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Terminal, args []string). The same should be done for the other commands of course as well.

Structs and string arrays should work out of the box, for termstatus.Terminal.New(wr io.Writer, errWriter io.Writer, disableStatus bool) one needs to implement io.Writer interface which only needs a simple Write(p []byte) (n int, err error) which should call the python logger callback.

Expose logging to user

I see you use logging library already. It would be nice, if the user could add additional logging.Formatter or logging.Handler (e.g. FileHandler with custom path and level, SMTPHandler).

Config file could consist of

import (or can this be determined based from handlerType automatically?)
handlerType, handlerArgs for constructor
level
format string

TOML config file in XDG_CONFIG_HOME is being parsed as JSON

When trying to run runrestic 0.5.23 the follow error comes up:

$ runrestic
Traceback (most recent call last):
  File "/home/connzen/.local/bin/runrestic", line 8, in <module>
    sys.exit(runrestic())
  File "/home/connzen/.local/pipx/venvs/runrestic/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 65, in runrestic
    parsed_cfg = parse_configuration(c)
  File "/home/connzen/.local/pipx/venvs/runrestic/lib/python3.8/site-packages/runrestic/runrestic/configuration.py", line 131, in parse_configuration
    else json.load(file)
  File "/usr/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I suspect the if conditional in

runrestic/runrestic/runrestic/configuration.py

Line 130 in 41715c7

if str(config_filename).endswith(".toml")

prevents the config file from being parsed as TOML. Sadly, neither .json nor .toml work because the config file's name in XDG_CONFIG_HOME is expected to have no suffix.

add time measurements to metrics

turn pre_hook and post_hook into lists

it should be possible to define multiple pre/post_hooks

Bump version on PyPi

Hi,
The latest version on PyPi is still 0.5.23 and doesn't include fix #54 for restic >= 0.13.0.
Can it be bumped?

Recursion error

I'm getting a lot of similar errors in different circumstances. They all look more or less like this:

Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/util.py", line 201, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 726, in _terminate_pool
    p.join()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 47, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 1 more time]
... snip ...
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 15 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Importantly, I have set

[execution]
parallel = false

I'm running 0.5.2 from the Arch AUR. What else do you need?

Do you have any idea what the problem could be?

Get realtime output from `restic`

show the restic process while running instead of just printing the result in the end.

pre_backup_hooks happen per repo. that's stupid

Use return code from restic

When using runrestic within a cron job it would be helpful to look at the return code to notify the admin if the backup failed. But Runrestic always return zero.

[12:03:53 root@host] runrestic -l debug
Parsing configuration file: /etc/runrestic.toml
[Environment] RESTIC_PASSWORD=**********
Spawning "['restic', '-r', 'rest:xxx', 'backup', '/data', '/root']"
{'current_try': 1, 'tries_total': 1, 'output': [(1, 'Fatal: wrong password or no key found\n')], 'time': 0.5132162570953369}
Spawning "['restic', '-r', 'rest:xxx', 'forget', '--keep-last', '3', '--keep-hourly', '5', '--keep-daily', '7', '--keep-weekly', '8', '--keep-monthly', '6', '--keep-yearly', '11']"
[(1, 'Fatal: wrong password or no key found\n')]
Spawning "['restic', '-r', 'rest:xxx', 'prune']"
[(1, 'Fatal: wrong password or no key found\n')]
Spawning "['restic', '-r', 'rest:xxx', 'check']"
[(1, 'using temporary cache in /tmp/restic-check-cache-545556970\nFatal: wrong password or no key found\n')]
{
"errors": 4,
"backup": {
"rest:xxx": {
"rc": 1
}
},
"forget": {
"rest:xxx": {
"rc": 1
}
},
"prune": {
"rest:xxx": {
"rc": 1
}
},
"check": {
"rest:xxx": {
"errors": 0,
"errors_data": 0,
"errors_snapshots": 0,
"read_data": 0,
"check_unused": 0,
"duration_seconds": 0.5457494258880615,
"rc": 1
}
},
"last_run": 1628157841.395445,
"total_duration_seconds": 2.183987855911255
}
[12:04:01 root@host] echo $?
0

runrestic shell

--help text is old

Error with large number of files

Hello

I have a Ubuntu 14.04.5 LTS system with a large number of files, doing a backup to OVH swift open stack, the backup ends well but when runrestic is computing the files added to the repo, runrestic crashes, this is the output of the process:

Files:           0 new,    10 changed, 421764 unmodified
Dirs:            0 new,     5 changed,     0 unmodified
Added:      2.698 MiB

processed 421774 files, 16.520 GiB in 6:22
snapshot fd661ba0 saved

Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/runrestic/commands/runrestic.py", line 131, in main
    run_configuration(config, args)
  File "/usr/local/lib/python3.4/dist-packages/runrestic/commands/runrestic.py", line 88, in run_configuration
    rcs += repo.backup(config.get('backup'))
  File "/usr/local/lib/python3.4/dist-packages/runrestic/restic/__init__.py", line 68, in backup
    self.log['restic_backup'] = parse_backup(output)
  File "/usr/local/lib/python3.4/dist-packages/runrestic/restic/output_parser.py", line 9, in parse_backup
    added_to_the_repo = re.findall('Added to the repo:\s+(-?[0-9.]+ [a-zA-Z]*B)', output)[0]
IndexError: list index out of range

I think the problem is the large number of files but I don't know for sure. I tried to run runrestic in a Ubuntu 14.04.5 LTS with a few files and works fine but I tried in the other system and fails.

Greetings.

Support (arbitrary) restic options

There doesn't seem to be a way to use arbitrary restic options, like --one-file-system or --no-lock, especially the former is very useful to me.

I suggest to either add a list, like CMD_ARGS to the config file, or add support the specific options like ONE-FILE-SYSTEM=True.

Github release for 0.5.4?

Hi, I see you guys have released a new version on pypi, could you also do a release here? I use it for my Arch Linux AUR package because it has your systemd sample files in it. :)

Thanks!

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Local vs Remote Repos

I've been using Runrestic for the incredibly long time of almost 72hrs now. (Quite liking it!) I do quite a lot of work without a connection, so restic + runrestic has been me slowly building up a time-machine like backup strategy. Time Machine backs up locally to a hidden partition, then copies over backups to the network storage when you're back within the network. With this background in mind, does it make sense to try to expand runrestic to have the concept of "backup" repos vs "copy-to" repos? prune and check would still need to be done against these copy-to repos, but backing up would always happen to a local destination first, with a second phase of copying out to these other repos if reachable, next.

Include restic logs

I tried runrestic and it seems to work in principle, but I don't get any log messages from restic itself, even if I specify log level "debug".
Is this intended behaviour or an issue with my installation?

I installed runrestic in Python 3.9 virtual environment and got no errors or warnings, for "backup" I only get the "spawning" message and no further putput.

Example output for "stats" command, which at least provides the result output:

$ sudo /opt/runrestic/bin/runrestic stats -l debug
Parsing configuration file: /etc/runrestic/user_backup.toml
[Environment] RESTIC_PASSWORD=**********
Spawning "['restic', '-r', 'rclone:qnap:my_pc/restic/users', 'stats', '-q', '--json']"
{
  "errors": 0,
  "stats": {
    "rclone:qnap:my_pc/restic/users": {
      "total_file_count": 3636345,
      "total_size_bytes": 52699188029,
      "duration_seconds": 156.62893509864807,
      "rc": 0
    }
  },
  "last_run": 1636301210.36535,
  "total_duration_seconds": 156.64174032211304
}

Can we allow jsonschema >= 4?

or is there anything that blocks the upgrade?

runrestic fails to parse valid `keep-within` values

runrestic -n
Problem parsing /etc/runrestic.toml: invalid literal for int() with base 0: '48h' (line 19 column 1 char 360)

restic assumes the parameter keep-within should be integer, which is not the case.

It's a duration, like 3w2d8h

--keep-within duration keep all snapshots which have been made within the duration of the latest snapshot. duration needs to be a number of years, months, days, and hours, e.g. 2y5m7d3h will keep all snapshots made in the two years, five months, seven days, and three hours before the latest snapshot.

Add "includes" in config file

Allow including other files.

Modify parse_configuration() to get a base config as parameter:

-def parse_configuration(config_filename: str) -> Dict[str, Any]:
+def parse_configuration(config_filename: str, config_base: Dict[str, Any]) -> Dict[str, Any]:
     logger.debug(f"Parsing configuration file: {config_filename}")
     with open(config_filename) as file:
         config = toml.load(file)
 
-    config = deep_update(config_base, dict(config))
+    config = deep_update(CONFIG_DEFAULTS, dict(config))
 
     if "name" not in config:
         config["name"] = os.path.basename(config_filename)
 
     jsonschema.validate(instance=config, schema=SCHEMA)
     return config

Replace

runrestic/runrestic/runrestic/runrestic.py

Line 65 in 7622739

parsed_cfg = parse_configuration(c)

with

parsed_cfg = parse_configuration(c, CONFIG_DEFAULTS)
for include in reversed(config['includes']):
    parsed_cfg = parse_configuration(include, parsed_cfg)

Support of S3 backend

Hi 👋,

I'm trying to setup runrestic to push my backup in S3.
By reading the sample config file, it's not clear to me in you support all environments variables or only RESTIC_PASSWORD and RESTIC_PASSWORD_FILE` ?

The schema.json file into your code let me think it's not supported by I prefer to ask.
If it's not supported, do you plan to do so ?

drop fastjsonschema for jsonschema

I'm not a Python pro, but it seems to be feasible and simple to drop fastjsonschema for jsonschema to simplify packaging (jsonschema is mostly already available as dist package).

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

	if os.geteuid() == 0: # pragma: no cover; if user is root, we just use system cache
	os.environ["XDG_CACHE_HOME"] = "/var/cache"
	elif not (os.environ.get("HOME") or os.environ.get("XDG_CACHE_HOME")):
	os.environ["XDG_CACHE_HOME"] = "/var/cache"

sinnwerkstatt / runrestic Goto Github PK

runrestic's People

Contributors

Stargazers

Watchers

Forkers

runrestic's Issues

Recommend Projects

Recommend Topics

Recommend Org