yoshtec / snapbtrex Goto Github PK

Came across a bad error handling: pv was not installed and it took me some time to find out, why the snapbtrex was not working. This should be fixed.

Hello!

Everything is fine with just a small request.
I want to add a postfix to subvolume names in date format.

For example:

root@NAS-j1900:[~]# ls -1 /mnt/sdb1/BACKUP/VPS/lxd-old/alpine-gps/
20210606-000406

To:

root@NAS-j1900:[~]# ls -1 /mnt/sdb1/BACKUP/VPS/lxd-old/alpine-gps/
20210606-000406-daily

that is, I want to append the "-daily" postfix to the name.
I would model daily weekly and monthly cron jobs like this.
Where to rewrite the script?

Thanks!
Regards

Hello

I only have one question. The Script is working properly.

At 23:45 the script starts every day.
crontab: 45 23 * * *

Yet he is 2 hours earlier in the name of snapshots.
20210601-214547: EXEC >-> sudo btrfs subvolume snapshot -r /mnt/sda3/containers/alpine-lighttpd /mnt/sda3/autosnap/alpine-lighttpd/20210601-214547

23:45 VS 21:45

How can i set the exact time on behalf of the snapshot?

Thank you very much in advance for your help!

Regards

Hello,

The script has not started since the last Ubuntu update.
It has worked perfectly so far. I updated it a few days ago and it has happened ever since.

[11:55:09][root@vmd58232:~]$ /usr/local/sbin/snapbtrex.py --snap / --path /mnt/sda2/autosnap/daily --target-backups 10 --verbose --remote-host 192.168.10.9 --remote-dir /mnt/sdb2/BACKUP/VPS-rootfs/daily --remote-keep 15
Local > snapshotting path=/ to newdir=/mnt/sda2/autosnap/daily/20220428-115545
EXEC >-> sudo btrfs subvolume snapshot -r / /mnt/sda2/autosnap/daily/20220428-115545
OUTPUT > Create a readonly snapshot of '/' in '/mnt/sda2/autosnap/daily/20220428-115545'

Local > sync filesystem /mnt/sda2/autosnap/daily
EXEC >-> sudo btrfs filesystem sync /mnt/sda2/autosnap/daily
Local > done sync filesystem /mnt/sda2/autosnap/daily
Local > done snapshotting
Remote > list remote files host=192.168.10.9, dir=/mnt/sdb2/BACKUP/VPS-rootfs/daily
EXEC >-> ssh -p 22 192.168.10.9 ls -1 /mnt/sdb2/BACKUP/VPS-rootfs/daily
OUTPUT >

20220402-234501
20220404-234501
20220406-234502
20220408-234502
20220410-234502
20220412-234501
20220413-234501
20220414-234501
20220415-234501
20220416-234502
20220417-234501
20220418-234501
20220419-234502
20220420-234502
20220421-234501

Traceback (most recent call last):
File "/usr/local/sbin/snapbtrex.py", line 1072, in
sys.exit(main(sys.argv))
File "/usr/local/sbin/snapbtrex.py", line 1026, in main
transfer(
File "/usr/local/sbin/snapbtrex.py", line 548, in transfer
targetsnaps = set(operations.listremote_dir(target_host, target_dir, ssh_port))
File "/usr/local/sbin/snapbtrex.py", line 275, in listremote_dir
return [
File "/usr/local/sbin/snapbtrex.py", line 276, in
d for d in self.check_call(args, dry_safe=True).splitlines() if timef(d)
File "/usr/local/sbin/snapbtrex.py", line 134, in timef
v = math.exp(_timestamp(x) / TIME_SCALE)
File "/usr/local/sbin/snapbtrex.py", line 149, in _timestamp
return time.mktime(time.strptime(os.path.split(x)[1], DATE_FORMAT))
File "/usr/lib/python3.8/_strptime.py", line 562, in _strptime_time
tt = _strptime(data_string, format)[0]
File "/usr/lib/python3.8/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '' does not match format '%Y%m%d-%H%M%S'
[11:55:45][root@vmd58232:~]$

[11:19:32][root@vmd58232:~]$ python3 --version
Python 3.8.10

Regards!

For local backups on another filesystem I'd like to use the --sync-target and --sync-keep options. However, on the receiving filesystem the snapshots pile up and there is never a clean up. With --sync-keep 7 I would expect to see 7 or less snapshots only. The clean up on the sending side is working well.

/usr/local/bin/snapbtrex.py \
--target-backups 1 --keep-backups 1 --keep-only-latest \
--snap / --path /mnt/btrfs/xps_system/snapshots/root/ \
--sync-target /mnt/btrfs/sa190919/backup/system/root/ --sync-keep 7 \
-v

Verbose output:

Local  > snapshotting path=/ to newdir=/mnt/btrfs/xps_system/snapshots/root/20190929-213433
EXEC  >-> sudo btrfs subvolume snapshot -r / /mnt/btrfs/xps_system/snapshots/root/20190929-213433
OUTPUT > Create a readonly snapshot of '/' in '/mnt/btrfs/xps_system/snapshots/root/20190929-213433'

Local  > sync filesystem '/mnt/btrfs/xps_system/snapshots/root/'
EXEC  >-> sudo btrfs filesystem sync /mnt/btrfs/xps_system/snapshots/root/
Local  > done sync filesystem '/mnt/btrfs/xps_system/snapshots/root/'
Local  > done snapshotting 
Local  > Sync: last possible parent = 20190929-213303
Local  > transfer: parent=20190929-213303 snap=20190929-213433
Local  > send snapshot from 20190929-213433 with parent 20190929-213303 to path=/mnt/btrfs/sa190919/backup/system/root/
EXEC  >-> sudo btrfs send -v -p /mnt/btrfs/xps_system/snapshots/root/20190929-213303 /mnt/btrfs/xps_system/snapshots/root/20190929-213433 | pv -brtf |  sudo btrfs receive -v /mnt/btrfs/sa190919/backup/system/root/
OUTPUT > At snapshot 20190929-213433

STDERR > At subvol /mnt/btrfs/xps_system/snapshots/root/20190929-213433
receiving snapshot 20190929-213433 uuid=4c3e1037-d4fb-c547-834e-ba6de5b5d389, ctransid=289810 parent_uuid=eef97a2c-5cf6-6f4a-a291-c16119cbd8b4, parent_ctransid=289805
BTRFS_IOC_SEND returned 0
joining genl thread
BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=4c3e1037-d4fb-c547-834e-ba6de5b5d389, stransid=289810
4,50MiB 0:00:00 [ 130MiB/s]

Local  > Parameters for cleandir: keep_backups=1, target_freespace=None, target_backups=1, max_age=None, keep_latest=True 
Local  > will delete backup: '20190929-213433'
Local  > remove snapshot '20190929-213303'
EXEC  >-> sudo btrfs subvolume delete /mnt/btrfs/xps_system/snapshots/root/20190929-213303
OUTPUT > Delete subvolume (no-commit): '/mnt/btrfs/xps_system/snapshots/root/20190929-213303'

Local  > done remove snapshot '20190929-213303'
Local  > Reached number of backups to keep: 1 

I am not sure if the output to STDERR indicates an error: except for this issue everything works fine. Or maybe the problem is that --sync-keep is stored the same as for --remote-keep but gets evaluated only for remote transfers (which would be my guess for why this happens).

Hi,
This is a great utility, my favourite btrfs snapshotting system, but I've hit a problem.
I often take my machine off my home network, which is where the remote backup destination is.
The script handles this without problem, but while disconnected it can delete local snapshots that stay on the remote system.
When you come back to the home network the script finds the last parent on the remote, and tries to send based on that parent. However, this snapshot has been deleted from the local system, so the script fails.

Local  > snapshotting path=/ to newdir=/.snapshots/20161119-182002
EXEC   > sudo btrfs subvolume snapshot -r / /.snapshots/20161119-182002
OUTPUT > Create a readonly snapshot of '/' in '/.snapshots/20161119-182002'

Local  > sync filesystem '/.snapshots'
EXEC   > sudo btrfs filesystem sync /.snapshots
Local  > done sync filesystem '/.snapshots'
Local  > done snapshotting 
Remote > list remote files host=192.168.1.3, dir=/btrfs_backups
EXEC   > ssh -p 22 192.168.1.3 ls -1 /btrfs_backups
OUTPUT > 20161102-215502
20161102-220353
20161102-220416
[snip]
20161110-110004

Remote > last possible parent = 20161110-110004
Remote > transfer: parent=20161110-110004 snap=20161110-120011
Remote > send snapshot from 20161110-120011 with parent 20161110-110004 to host 192.168.1.3 path=/btrfs_backups
EXEC   > sudo btrfs send -v -p /.snapshots/20161110-110004 /.snapshots/20161110-120011 | pv -brtf | ssh -p 22 192.168.1.3 'sudo btrfs receive -v /btrfs_backups '
OUTPUT > At snapshot 20161110-120011

STDERR > At subvol /.snapshots/20161110-120011
receiving snapshot 20161110-120011 uuid=fc611a41-f847-8946-9e06-71644893ed98, ctransid=1336 parent_uuid=1c3306a7-50de-6549-ba0f-b41e33b9b715, parent_ctransid=1221
ERROR: cannot find parent subvolume
1.98MiB 0:00:00 [3.16MiB/s]
ERROR: failed to dump stream: Broken pipe

It looks like when deleting a local snapshot the script should make sure it doesn't still exist on the remote, or find some way to fall back to an earlier parent.

I followed the instructions to install the script. I am able to execute the script with sudo as my regular user but it won't work when logged in as snapbtr. The output is

[snapbtr@rob-thinkpad ~]$ sudo /bin/snapbtrex.py --path /mnt/.snapshots/ --snap /home
Traceback (most recent call last):
  File "/usr/bin/snapbtrex.py", line 922, in <module>
    sys.exit(main(sys.argv))
  File "/usr/bin/snapbtrex.py", line 892, in main
    operations.snap(path=pa.snap)
  File "/usr/bin/snapbtrex.py", line 313, in snap
    self.check_call(args)
  File "/usr/bin/snapbtrex.py", line 268, in check_call
    shell=shell)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied

I tried creating a snapshot using "btrfs subvolume create", which works fine, however "btrfs subvolume delete" doesn't, it also throws "Operation not permitted". Maybe this has something to do with it?

Hi Support,

Thanks for the script, but I have a little trouble with it.

I'm trying to replicate LXC containers to remote Host.

I get the following error:

[16:20:12][root@vmd58232:~]$ /usr/local/sbin/snapbtrex.py --snap /mnt/sda3/containers/alpine-znc --path /mnt/sda3/autosnap/alpine-znc/ --target-backups 5 --verbose --remote-host 192.168.10.8 --remote-dir /mnt/vps-lxd-snapshots/alpine-znc/ --remote-keep 15
Local  > snapshotting path=/mnt/sda3/containers/alpine-znc to newdir=/mnt/sda3/autosnap/alpine-znc/20210204-152013
EXEC  >-> sudo btrfs subvolume snapshot -r /mnt/sda3/containers/alpine-znc /mnt/sda3/autosnap/alpine-znc/20210204-152013
OUTPUT > Create a readonly snapshot of '/mnt/sda3/containers/alpine-znc' in '/mnt/sda3/autosnap/alpine-znc/20210204-152013'

Local  > sync filesystem /mnt/sda3/autosnap/alpine-znc/
EXEC  >-> sudo btrfs filesystem sync /mnt/sda3/autosnap/alpine-znc/
Local  > done sync filesystem /mnt/sda3/autosnap/alpine-znc/
Local  > done snapshotting
Remote > list remote files host=192.168.10.8, dir=/mnt/vps-lxd-snapshots/alpine-znc/
EXEC  >-> ssh -p 22 192.168.10.8 ls -1 /mnt/vps-lxd-snapshots/alpine-znc/
OUTPUT > 20210204-151328

Traceback (most recent call last):
  File "/usr/local/sbin/snapbtrex.py", line 1067, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/sbin/snapbtrex.py", line 1021, in main
    transfer(
  File "/usr/local/sbin/snapbtrex.py", line 543, in transfer
    targetsnaps = set(operations.listremote_dir(target_host, target_dir, ssh_port))
  File "/usr/local/sbin/snapbtrex.py", line 270, in listremote_dir
    return [
  File "/usr/local/sbin/snapbtrex.py", line 271, in <listcomp>
    d for d in self.check_call(args, dry_safe=True).splitlines() if timef(d)
  File "/usr/local/sbin/snapbtrex.py", line 134, in timef
    v = math.exp(_timestamp(x) / TIME_SCALE)
  File "/usr/local/sbin/snapbtrex.py", line 149, in _timestamp
    return time.mktime(time.strptime(os.path.split(x)[1], DATE_FORMAT))
  File "/usr/lib/python3.8/_strptime.py", line 562, in _strptime_time
    tt = _strptime(data_string, format)[0]
  File "/usr/lib/python3.8/_strptime.py", line 317, in _strptime
    raise TypeError(msg.format(index, type(arg)))
TypeError: strptime() argument 0 must be str, not <class 'bytes'>

  }
}

[17:05:01][root@vmd58232:~]$ btrfs fi sh
Label: none  uuid: ada7cc9c-81ed-4416-96dc-2b8c98e2ddec
	Total devices 1 FS bytes used 16.88GiB
	devid    1 size 69.85GiB used 23.02GiB path /dev/sda2

Label: none  uuid: 929266a2-f629-457c-8715-7540f9e10299
	Total devices 1 FS bytes used 33.92GiB
	devid    1 size 321.31GiB used 48.02GiB path /dev/sda3

[17:05:01][root@vmd58232:~]$ python
python            python2           python2.7         python3           python3.8         python3.8-config  python3-config    

[17:04:54][root@vmd58232:~]$  lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

Thank you very much!