Rotate Backups about docker-volume-backup HOT 11 OPEN

One more point related to this came to my mind:

(PRE|POST)_COMMAND are not really telling names revealing what will be done.
Can we rename them to e.g. (PRE|POST)_ARCHIVE_COMMAND? I.e., "commands that are executed before and after the backup has been moved to /archive".

I will give an update soon. :)

from docker-volume-backup.

Oh yeah, makes total sense; being able to run pre/post commands shouldn't depend on whether you're backing up locally or remote, they can be useful for both. 👍

from docker-volume-backup.

Hi!

I do kinda like the "batteries included" attitude of just including a rotation utility in the image itself. There's a few caveats, though:

1. As you said, it bloats the image. Personally I wouldn't lose a lot of sleep over it, as most of the time you pull an image like this once, and then it sticks either on the host cache or some other cache (e.g. your cloud provider's) along the way. But I also know some people are quite put off by huge images.
2. You still can include any such tool by using this image as a base in your own Dockerfile. But we don't need to make that choice of tool (and bloat) for users.
3. Even if we include such an utility into the image, it still won't work for scp targets (right?), only local targets.

Given all of the above, I quite like the (PRE|POST)_(|SCP_)COMMAND approach. It's slightly less "batteries included", but by far the most flexible.

from docker-volume-backup.

Hy @jareware,

1. Sounds good. This is also my thinking.
2. Yes, I can integrate it in my fork of the project. I have to make two things clear: First is that I do not prefer to release my fork of the project. Second is that I did not find out if you are open to integrate it into your project (the root project so to say). This is a minor communication issue. Please give me a unequivocally "Yes, please work in this.!" or a "No, I do not want this here." :)
3. If we integrate rotate-backups directly into the image and if we upload a backup by means of scp, I do believe that we could mount the target directory on the remote host by means of sshfs and execute rotate-backups there. Not 100% sure, but I think it is worth a try.

Can you please provide a little bit more information on your idea of PRE_COMMAND and POST_COMMAND?
I understood that these commands are executed before and after the backup has been copied to /archive. To execute rotate-backups (like POST_COMMAND: rotate-backups --daily 7 /archive), it has to be integrated in the image. Is that something you that covers your idea?

Regards,
Jan

from docker-volume-backup.

I did not find out if you are open to integrate it into your project

Sorry, maybe I didn't quite catch that.

I would say that "no", I don't think we should integrate rotate-backups to the main project. The added size and complexity doesn't seem worth it right now. We definitely want to support backup rotation, but for now, I would just add options that allow calling an external script/container do to that. People can then use something simple (like rm) or fancy (like rotate-backups) as they please.

Can you please provide a little bit more information on your idea

Yes, I would hope it worked exactly as (PRE|POST)_SCP_COMMAND, except the command would be executed locally (i.e. within the backup container) instead.

If you do something simple (like rm), you don't need to do anything extra, just come up with a glob pattern or age restriction that works the way you like.

If you want something fancy (like rotate-backups), and you've mounted the Docker socket, you can docker run any image/command that does it for you. For a really crappy example:

docker run --rm -v /your/backup/location:/archive python:3-alpine bash -c 'pip install rotate-backups && rotate-backups ...'

But of course this could be packaged into a dedicated image:

docker run --rm -v /your/backup/location:/archive jan-brinkmann/rotate-backups

Assuming your SCP_HOST also has Docker available, you can use the exact same command over there.

What do you think?

from docker-volume-backup.

Oh, and looks like there's also some solutions out there already, e.g. https://github.com/Glideh/docker-cron-rotate-backups

from docker-volume-backup.

Hi,

thank you for point out that we are able to start arbitrary Docker containers.

Indeed, this is a possibility to run rotate-backups on /archive without integrating it into the backup image.
I already have created a merge request for PRE_COMMAND and POST_COMMAND and a simple example where all backups older than seven days are deleted.

The docker-cron-rotate-backups you mentioned is imho no suitable solution for POST_COMMAND because is creates a cron job. What we need is a container that executes rotate-backups once and immediatelly. I have played a little bit around here: https://github.com/jan-brinkmann/docker-rotate-backups. I think I will add an example soon.

Regards,
Jan

from docker-volume-backup.

#46 was exactly what I had in mind! 🙇

Also https://github.com/jan-brinkmann/docker-rotate-backups looks really neat. Let's add an example to the README in this repo on how to use that?

from docker-volume-backup.

Thank you for merging!

I will add an example the next days.

from docker-volume-backup.

Allright, let me know and let's cut a release after!

from docker-volume-backup.

Recently, I found out that rotate-backups can be applied on remote directories. I have implemented that feature in the external container here: https://github.com/jan-brinkmann/docker-rotate-backups#remote-directories

We can use this feature when we upload backups by means of SCP. If we would do this, rotate-backups does not have to be installed on the remote host where you transfer the backup to. Currently, rotate-backups must be installed. Thus, we can get rid of this restriction.

In order to apply rotate-backups with the external container docker-rotate-backups, we need a command that is executed in the docker-volume-backup container. Currently, we have POST_SCP_COMMAND which is executed on the remote host, and we have POST_COMMAND which is executed only if the backup is transferred to /archive.

I propose to move POST_COMMAND below the block that ends here:

Does this make sense to you?

from docker-volume-backup.