Comments (10)
Something like https://github.com/fsnotify/fsnotify could trigger a config reload.
from docker-volume-backup.
The reason I'm avoiding multiple sidecars is the logistics in scheduling the backups as I don't think it's too good an idea to have multiple sidecars backing up at the same time
from docker-volume-backup.
I personally run a setup where we have a host running 7 applications that are getting backed up by 7 distinct container running on 7 distinct schedules (all of them are daily so it's pretty arbitrary), which is probably the reason things are working as they are right now.
Your proposal however makes sense. I think it could be implemented in a way that if you set BACKUP_SOURCES=/backup/app1,/backup/app2
you'd get two archive files (one per item in the given list) instead of one. The default behavior would still be to backup all of /backup
.
BACKUP_SOURCES
is already there:
docker-volume-backup/cmd/backup/main.go
Line 90 in ad7ec58
file
that is handled a list of files instead.
Would this API make sense to you?
The downside in this scenario would be that this means all apps are down while the archives are being created and you have no more granularity in starting / stopping containers while their data is being backed up. This is the reason I ended up using the solution as described in the beginning btw: one of my volumes is so big that it takes ~2 minutes to create the archive and I didn't want all of my services being down during that time.
from docker-volume-backup.
Yeah you have a point there, some of my backups are in the 10s of Gs so that would take a while. And as you say the rest of my "daily" group would be down while this was going on.
What I am really looking for is a way to:
Take app1 down
Backup App1
Bring app1 up
Take app2 down
Backup App2
Bring app2 up
All from the same starting schedule eg. at 1AM
Not sure if it's possible to combine your suggestion with multiple matching labels?
eg. BACKUP_SOURCES=/backup/app1,/backup/app2
and docker-volume-backup.stop-during-backup=app1,app2
But maybe that is a bit too frail
from docker-volume-backup.
But otherwise for smaller not so important containers your suggestion makes sense and I could definitely use that functionality.
I run 12 "production" containers and 15-20 test/containers that are not required to be online at night so they could all be off for 8 hours if that's what it would take to back them up.
from docker-volume-backup.
Considering you can already run multiple containers in parallel in case you need granularity on stopping I think coming up with a rather brittle/confusing API that conflates backup sources and stop labels isn't really a good option here.
Adding support for multiple sources is a good idea though which I'd like to add as soon as I find the time to do so.
If you (or someone else) would like to work on this, let me know and I'm happy to help with getting this merged.
from docker-volume-backup.
I was thinking about this further and there is another API challenge to solve when implementing this:
Backing up multiple sources into multiple archives means we also need to define a way of specifying multiple target filenames. Right now consumers set BACKUP_FILENAME="backup-%Y-%m-%dT%H-%M-%S.tar.gz"
but this won't work when there are multiple files being saved.
I can see two options right now:
Making BACKUP_FILENAME
a list of templates
This list would simply map to the sources defined in BACKUP_SOURCES
. This is relatively easy to implement but might be complicated to configure as consumers now need to keep those two configuration values in sync. This also raises the question what the script should be doing when it encounters lists of different lengths (probably bail).
Interpolating the source into the BACKUP_FILENAME
BACKUP_FILENAME could start interpolating a special token of sth like {SOURCE}
so that consumers would define BACKUP_FILENAME="backup-{SOURCE}-%Y-%m-%dT%H-%M-%S.tar.gz"
and the token would be replaced with a (cleaned and filename safe version) of the source. Open questions would be:
- can we guarantee that those source names are unique when making the safe to use in a filename
- what happens when
{SOURCE}
is not given in the filename
from docker-volume-backup.
I think this again makes the complexity a bit too high, I quite like how simple it is to use this image.
What about a config file mounted to the container for "advanced" stuff like this, then potentially this issue and other could more easily be handled without necessarily compromising the simplicity of the environment args.
Either way I got it solved by spreading out my backup schedule a bit, and actually running some at the same time by choosing different target drives to backup up to (then the write performance is fine)
from docker-volume-backup.
After moving this around in the back of my head for a little longer, I think this is how it could work:
- consumers can mount an arbitrary number of configuration files into the container's
/etc/backup.d
directory (e.g./etc/backup.d/10app.env
and/etc/backup.d/20database.env
) - the entrypoint script checks whether that directory exists
- in case yes, it creates a crontab entry for each of the files and tells the
backup
binary to read configuration values from the file in addition to the defined environment variables - in case no, a cron will be created from environment variables (i.e. the current behavior)
Open questions:
- this means that when configuration files change, the container will need to be restarted. Is that common practice or possibly unexpected behavior?
from docker-volume-backup.
This is now possible as of v2.14.0
. While I already had it working I did not implement inotify
for file changes as it would require adding the openrc
package to the container, adding slightly more than 2MB to the image size. In case this turns out to be a problem / requested feature for users, it can always be added in a later step.
from docker-volume-backup.
Related Issues (20)
- "Page not found" error when using with balena HOT 9
- Hard to understand error message during exec HOT 3
- archive-pre is triggered on each container backup instead of just container with exec-label HOT 3
- Health Check HOT 6
- Notifications Delay HOT 1
- Replace Log line feeds with HTML <br> tag HOT 3
- Backup the entire content without directory HOT 4
- Backup to subfolders HOT 2
- Resolve date in AWS_S3_PATH HOT 1
- Can't get stop during backup working HOT 2
- Remove local backup after upload to S3 HOT 3
- Invalid memory address crash when running a backup HOT 16
- Fail to configure cron schedule HOT 3
- Docker swarm error HOT 5
- "swarm exec" support HOT 1
- "input/output error" when copying file to NFS mount HOT 4
- Azure storage: Support access tier HOT 2
- schedule the docker-volume-backup? HOT 1
- [Feature] Automated backups of all named volumes HOT 6
- [FR] upgrade `shoutrrr` to v0.8.0 to support custom fields in calls to Generic Webhooks HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-volume-backup.