Feature request: parse-able output (JSON?) about btrfs-backup HOT 9 CLOSED

Hmmm actually, just double checked this... .outstanding_transfers never gets updated so my remote backups are not staying synced once connections return :(

from btrfs-backup.

Hi,

You use ssh:... as the destination, right? What does the .outstanding_transfers file contain after all backups have been synced? This file just contains the snapshot/destination pairs of transfers that have been started but not completed successfully, so that they can be removed and re-transferred at the next run. After a snapshot has been transferred successfully to a remote destination, the "lock" for that particular snapshot/destination pair is removed from the .outstanding_transfers file.

Please note that the file only works reliably when you specify the exact same destination string at each and every invocation of btrfs-backup.

Best regards
Robert

from btrfs-backup.

As an indication of a failed snapshot transfer...

From btrfs-backup/__main__.py, line 91:

logging.error("Error during btrfs send / receive")

This message goes to stderr (fd 2) and can be checked for in your monitoring script.

from btrfs-backup.

And to retry failed transfers independent of the normal backup procedure, you could run

btrfs-backup [your retention options] --no-snapshot --locked-dests <SOURCE>

regularly. It will only transfer existing snapshots that are not successfully sent yet.

from btrfs-backup.

I've never seen the file contain anything, despite the destination being offline for > 6hrs (we do hourly snapshots for 24hrs locally and keep 30 days on the remote end). Below is the command that is ran every hour. -- So maybe doing local + remote causes an oddity here?

btrfs-backup -P --num-snapshots 24 --num-backups 744 --snapshot-folder ../snapshots/lib --ssh-sudo /home/project/lib ssh://backups/media/backups/project/snapshots/lib

In this case, backups is a hostname defined in the ssh-config to keep the ssh process simple.

from btrfs-backup.

Ok, can you please check whether the file contains something while the transfer is in progress?

from btrfs-backup.

I think the issue here is that .outstanding_transfers is only meant for cases in which the connection drops during the transfer itself. In your case, the preparation of the endpoint already fails and it doesn't get to the point where it picks snapshots to transfer. This is by design and can't be changed easily at the moment. An option would be to create a lock for each snapshot as soon as the destination fails to initialize...

BTW: 700+ snapshots seems way much for btrfs... don't you get performance issues with such an amount of snapshots? In your case, I'd prefer something like borg or restic for the remote backups. Maybe you want to have a look into https://github.com/efficiosoft/backwrap, which makes using these tools really handy together with btrfs snapshots.

from btrfs-backup.

I'm currently adding the keeping of local snapshots to backwrap, it should be available later today.

EDIT: Done. :-)

from btrfs-backup.

.outstanding_transfers does get updated during transfers.

up to ~500 for 2 different snapshots and so far no noticeable negative side effects -- these are almost never accessed directly on the remote system. Generally the 24 hourly backups are sufficient for our needs.

Will checkout backwrap and the other tools.

from btrfs-backup.