Support scp backed storage about docker-volume-backup HOT 8 CLOSED

Stats about the upload would be like how long it took (separate from the rest of the process) and how many bytes were sent if that can be parsed from the output. It's moot for me since I'm not using the built in backends anyway.

If there will ever be a v3 it might be worth considering, but as long as this image stays a v2, it's probably too complex to support.

Makes sense, and my little hack is functional enough for now.

from docker-volume-backup.

A better idea might be supporting rclone and having users mount an rclone.conf file. Instant support for a ton of backends. Personally, I sync my backups to two cloud locations using rclone and it would be awesome to mount the script to this container and run it using docker-volume-backup.exec-post instead of adding it to my crontab. (It would also be cool to get a shoutrrr binary, which my script also uses)

Hmm maybe I should learn just enough go to convert my script and go run it post backup.

from docker-volume-backup.

Nope, go was too much trouble. A couple hiccups but I hooked up my shell script that syncs my latest backup using rclone to be triggered once a backup job has run.

Add to the backup container itself:

    labels:
      - docker-volume-backup.exec-post=/bin/sh -c '/sync/setup.sh'
    volumes:
      - ./config/sync_backup:/sync:ro

And the contents of ./config/sync_backup:
32 bit rclone executable
32 bit shoutrrr executable
rclone.conf, generated by running rclone config --config ./config/sync_backup/rclone.conf
any other files required by rclone (key files for remotes)

setup.sh:

#!/bin/sh
apk add --update openssh-client at
atd
# need to run the sync after the backups have been rotated and the symlink updated
echo /sync/sync_backup.sh | at now +10 minutes

sync_backup.sh:

#!/bin/sh

SHOUTRRR="/sync/shoutrrr send -u $NOTIFICATION_URLS"
RCLONE="/sync/rclone copy --config /sync/rclone.conf -vL /archive/backup-latest.tar.gz"

$SHOUTRRR -t "Backed up volume sizes" -m "`du -sm /backup/* | sort -rg | sed 's#/backup/##' | sed 's#_vol##' | sed 's#\([0-9]\+\)#\>

max_retry=5

counter=0
START_TIME=`date +%s`
echo "Uploading to Google Drive"
until
   $RCLONE gdrive:
do
   sleep 1
   [ "$counter" -eq "$max_retry" ] && echo "Failed!" && break
   echo "Trying again. Try #$counter"
   counter=$((counter+1))
   $SHOUTRRR -t "Backup sync" -m "google drive upload failed on iteration $counter/$max_retry"
done
END_TIME=`date +%s`
$SHOUTRRR -t "Backup sync" -m "google drive upload took $((($END_TIME-$START_TIME)/60)) min"

counter=0
START_TIME=`date +%s`
echo "Uploading to VPS"
until
   $RCLONE vps:backup
do
   sleep 1
   [ "$counter" -eq "$max_retry" ] && echo "Failed!" && break
   echo "Trying again. Try #$counter"
   counter=$((counter+1))
   $SHOUTRRR -t "Backup sync" -m "vps upload failed on iteration $counter/$max_retry"
done
END_TIME=`date +%s`
$SHOUTRRR -t "Backup sync" -m "vps upload took $((($END_TIME-$START_TIME)/60)) min"

from docker-volume-backup.

Thanks for sharing this. Considering that a small image size is a key design goal of this image, it would be ideal if we could be using something here that is available as a native Go package. Installing other software in the image and then calling through should probably not happen.

Then again people who want to do something like this could also use the image as a base for their own image, triggering pre/post scripts. I'll need to think about this a little.

from docker-volume-backup.

Yeah it's definitely a hack. Rclone is written in go and can probably replace all the other backends that are currently available. I tried importing it as a package but apparently it only supports being built as an executable, but maybe someone who knows go can figure it out.

from docker-volume-backup.

Seems using rclone from within Go code is possible but not officially supported (i.e. no API stability and such): https://forum.rclone.org/t/rclone-as-a-golang-module-or-other-way-of-using-it-in-code/17172

from docker-volume-backup.

Did an experiment with adding rclone. Installing via apk or copying it in from the rclone/rclone image both grow the compressed size from 14 MB to 25 MB on my machine. Some of that is offset by removing the minio-go and gowebdav modules. I don't know how go works but if importing a module just downloads the whole repo, minio-go is about 7 MB compressed, and gowebdav is about 160 KB.

It would also be nice to have a feature to send shoutrrr notifications with stats on backup uploads.

Incidentally, changing --update to --no-cache reduces the compressed size from 14 MB to 12 MB.

from docker-volume-backup.

Incidentally, changing --update to --no-cache reduces the compressed size from 14 MB to 12 MB.

Oh this is a nice find, I am surprised the install didn't use --no-cache yet. I changed this in 5ec2b2c and will include it in the next release. Thanks for mentioning.

It would also be nice to have a feature to send shoutrrr notifications with stats on backup uploads.

The image as is using shoutrrr to send notifications already https://github.com/offen/docker-volume-backup#send-email-notifications-on-failed-backup-runs - is there anything you would like to do, but cannot do using that feature? You can also include stats about the backup in the notifications https://github.com/offen/docker-volume-backup/blob/main/docs/NOTIFICATION-TEMPLATES.md

About replacing literally everything with rclone I am a little skeptical as this might make sense but is a very big undertaking that brings a lot of compatibility issues when adding it to the existing image. If there will ever be a v3 it might be worth considering, but as long as this image stays a v2, it's probably too complex to support.

from docker-volume-backup.