Comments (9)
When using GPG_PASSPHRASE the result of the tar command could (probably) directly be piped to gpg
Since this is not a bash script, it's not that easy, but it could still be done without having to resort to using the intermediate file and instead chain the tarWriter and the OpenPgpWriter. Maybe it would even be possible to upload the file in a streaming manner for some backends and remove the need for an intermediate file entirely.
This is a little complicated though (mostly because the code is written with that mechanism in mind), so it's much more than a one-line change.
If anyone wants to pick this up, I am happy to review and merge in PRs. Else I might be able to look into this at some point, but I cannot really make any estimates.
A side note question: the artifacts do get deleted properly after a backup run, right? So the storage footprint is only an issue while the backup is running, correct?
from docker-volume-backup.
Thanks for the reply. Chaining the upload would've been my next question 👍
Maybe I can find some time to start developing this feature.
from docker-volume-backup.
On a very high level this could work like this (from the top of my head, so it might be wrong in certain details):
(*script).file
becomes anio.ReadWriter
(?) instead of a string pointing to a file- methods on
*script
can now operate on this writer instead of reading / writing from the file - in case the storage backend requires to work off a file, the ReadWriter can be flushed to an intermediate file, in case the backend supports streaming, it can be piped through
In case you decide to start working on this feel free to ask questions any time and also don't shy away from improving the pipeline if you find things that are rather odd right now.
from docker-volume-backup.
I looked into how this could implemented a little and found a tricky detail hidden: if a streaming pipeline would be used that possibly does create tar archive -> possibly encrypt -> upload
this would also mean that users who want to stop their containers during backup might see a considerable increase in downtime as this now means we could see backpressure or similar. I think it's possible to work around this mostly (emit an event once the tar writer stopped writing or similar), but this makes the refactoring needed even bigger.
I'll try to think about how the entire script orchestration could be refactored so it can accomodate for such requirements.
from docker-volume-backup.
Maybe different modes, which the user can select, would make sense here:
- Stream mode: Archive + Encrypt + Upload all at once as a pipeline (if the storage backend supports it)
Hybrid mode: Archive + Encrypt -> Upload (Maybe speedup by using more CPU resources by archiving and encrypting simultaneously)- Sequential mode: Classic Archive -> Encrypt -> Upload (like now; default?)
- Adaptive mode: Archive + Encrypt + Upload all at once but also build up an adjustable buffer of Archive+Encrypt (either a set amount or free storage space minus x%) so the container can be started again sooner, as the upload can use the remaining buffer
Adaptive mode would solve the backpressure issue. For example with Backblaze B2 I have usually 250 MBit/s Upload (~30 MB/s) and using zstd as compressing method (magnitudes faster on my maschines) plus multicore encryption... I'd assume that IO would usually be often faster than the network... but that's just a sample size of one (me). Makes sense to have the backpressure issue in mind imo.
Alternatively to the suggested modes, only adaptive and sequential could be fine too as stream mode
is just adaptive mode
with the buffer set to zero or very low.
from docker-volume-backup.
One thing that just occured to me is that implementing archive-encrypt-upload in a streaming fashion would be a breaking change as it would mean the command lifecycle would be chaning, i.e. when streaming, there is no more possibility of running pre-archive
commands or similar. It would just allow for pre
and post
commands. That's probably ok, but I wanted to leave it here as I just thought of it.
from docker-volume-backup.
True, there would only be a start and a finish hook - no matter if using streaming
or buffered streaming
... unless the user-defined buffer is quite large, then it would make sense to restart the container early and trigger a post-archive
hook.
The buffer size
value could be:
0
to essentially disable the buffer ->streaming
-1
for automatic size: amount of available space minus some offset ->buffered streaming
with optimal downtime- Otherwise
xM
/xG
or simply an integer in MB/GB ->buffered streaming
with manual space usage limit
I'd definitely keep the classic sequential
processing around in the form of the automatically sized stream buffer (-1
buffer size) to optionally keep downtime of containers as low as possible without risking a storage issue.
Having all "modes" in basically one logic (but with different buffer sizes) would make the code base cleaner and configuration easier, since we wouldn't need two entirely different approaches (sequential vs streaming).
The default option should perhaps be the buffered streaming
with a buffer size of -1
(automatic), since a long downtime is bad, but a full disk with potential crash/freeze/data-loss is worse IMO. That could be the best of both worlds: restart containers as soon as possible, but don't use more space than available.
I'd need to do some testing whether df
or similar go functions inside the container report sensible values (for me in amd64 Win11 desktop and arm64 Ubuntu server, df
did)... only then, the automatic buffer size is feasible.
from docker-volume-backup.
I was thinking one could have some sort of "event" that signals the end of the archive stream which would then trigger restart while bytes are still munged further down the stream, see #95 (comment)
Not sure how realistic that is though.
In any case I would argue optimizing for as little downtime as possible is more important than optimizing for disk space. Disk space is cheap, service downtime is not.
from docker-volume-backup.
An event of some sort after the archiving stage is done definitely makes sense.
That's usually the case yes. Maybe we can omit the buffer size
entirely and just have buffer=true/false
(working title) which would be automatic buffer size (-> use all available space minus some offset) for low downtime or no buffer (false) for low space usage.
That being said, I don't know if there is any benefit of using buffer=false
, since the occupied space with the automatic buffer size is freed after the backup anyway... so why not just archive ASAP with the available buffer and get the containers running again quickly. Would mean less complexity in the implementation too.
If that's what you essentially meant, then I fully agree :)
from docker-volume-backup.
Related Issues (20)
- Hard to understand error message during exec HOT 3
- archive-pre is triggered on each container backup instead of just container with exec-label HOT 3
- Health Check HOT 6
- Notifications Delay HOT 1
- Replace Log line feeds with HTML <br> tag HOT 3
- Backup the entire content without directory HOT 4
- Backup to subfolders HOT 2
- Resolve date in AWS_S3_PATH HOT 1
- Can't get stop during backup working HOT 2
- Remove local backup after upload to S3 HOT 3
- Invalid memory address crash when running a backup HOT 16
- Fail to configure cron schedule HOT 3
- Docker swarm error HOT 5
- "swarm exec" support HOT 1
- "input/output error" when copying file to NFS mount HOT 4
- Azure storage: Support access tier HOT 2
- schedule the docker-volume-backup? HOT 1
- [Feature] Automated backups of all named volumes HOT 6
- [FR] upgrade `shoutrrr` to v0.8.0 to support custom fields in calls to Generic Webhooks HOT 3
- [Feature] Allow BACKUP_COMPRESSION to be disabled HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-volume-backup.