tasket / wyng-backup Goto Github PK
View Code? Open in Web Editor NEWFast backups for logical volumes & disk images
License: GNU General Public License v3.0
Fast backups for logical volumes & disk images
License: GNU General Public License v3.0
Backup and restore additional metadata that the user or host system associates with each volume. For example, VM or container configurations.
This could require additional packaging / un-packaging steps before or after the send
or receive
operation runs.
receive
.blake2b
hashing (faster than sha256)Also explore whether referencing a zstd dictionary for each volume is appropriate.
If feasible/easy to implement, it'd be nice to keep comments in sparsebak.ini when the file is rewritten (eg. after using add
/delete
commands).
If it isn't, maybe add a static top comment that user comments will be deleted when the file gets rewritten ?
And/or create a backup .ini file ?
Right now, I do something like this which is buggy cause it introduces root volumes for VMs that are not TemplateVMs:
qvm-volume list --quiet|grep -v volatile|awk -F "/" {'print $2'}|awk -F " " {'print $1'}|while read line; do sudo ~/sparsebak/sparsebak.py add $line; done
Many systems are installed with fixed (non-thin) lvm volumes and documentation suggests thin snapshots can be created for fixed volumes. If so, it may be possible to use the thin lvm tools to track deltas for fixed volumes the same way as in an all-thin environment.
The challenges for implementation are:
Barring the above, it may be helpful to reference documentation showing the best practices for converting fixed-lvm systems over to thin pools, btrfs, etc.
Implement a variation of verify
that receives and verifies every manifest entry and chunk for a session.
Selection:
--all
option could signify verification of all sessions.--session
could be used to specify one session or a range (could include --all-before
).A bug from a recent update is causing data chunks to be sent with incorrect path names.
Will have a fix soon with instructions for correcting any incorrect paths sent to an archive.
Change print
output to python logging module and offer different log levels such as quiet and debug.
A good testing regime could include the following:
Changing program_name
in the code could allow the test data set to be created under a different base folder name without interfering with real-life backup data.
Currently some command-specific options such as --save-to
are not checked for appropriate-ness at startup. This is partly due to the simplified processing of commands/options, which also limits where options can be placed on the command line.
Re-structure option parsing using sub-parsers
Add more conditional checking of options
Confine more option processing to main section
Implement a command to initialize an archive without having to edit an ini file.
But takes the whole flat space of Qubes volume.
Is that normal?
This is a feature request. Hacked my way around it with this to recover 200GB of unneeded backups (last backup session not being pruneable)
sudo ~/sparsebak/sparsebak.py list |sort|tail -n +7 | while read appvm; do echo $appvm:; sudo ~/sparsebak/sparsebak.py list $appvm|tail -n -4|head -n +1|awk -F " " '{print $(NF-1)}'| while read last_deleteable_backup; do echo $last_deleteable_backup|sudo ~/sparsebak/sparsebak.py -u prune $appvm --all-before --session $last_deleteable_backup; done; done
Is that even possible?
Following wyng logic, I understand that it would be as simple of asking it to add root, monitor it and send it to backup?
Retrieving individual files from backed-up volumes.
A read-only FUSE type filesystem that can mount any volume-session in the archive as a complete volume should be attainable without a ton of effort. This is because the archive format is similar to Apple's sparsebundles.
For restoring volumes, support a mode where an existing volume on the source isn't wiped before data is received. Instead, truncate/lvresize it then write only those incoming chunks which are different.
This results in less consumption of disk space because more than just zeros are treated sparsely... useful if a user is reverting a volume to an earlier time.
Implement a rename function to change a volume name in the archive.
Reinstalled Qubes 4.0.2RC1 from scratch. Previously done a native QubesOS backup to restore needed VMs. Redeployed sparsebak.
Was expecting to be able to use sparsebak arch-init that would re-add all archived volumes present in backup subdir and be able to call send receive and prune. It doesn't seem so.
Couple of comments:
Let me know if I need to be clearer :)
The monitor
command outputs text to stdout so on non-Qubes systems an email will likely be sent by the cron daemon each time the job is run (or /var/spool/... will silently be filled if the mta is not properly configured).
Maybe it's enough to redirect stdout to /dev/null
but I couldn't find a way to trigger an error with monitor
to test if errors are still shown (ie. output to stderr).
EDIT: alternatively monitor
could not output anything at all (except warnings/errors of course) and there could be a --verbose
option
Remove unnecessary burden on dest filesystem: zero-length files representing zero-filled chunks. The manifest entries for these chunks are sufficient.
sparsebak was recording info for zero-length chunk files in the wrong order (the opposite order used for non-zero files). This was fixed in commit 13e6e30.
The manifest files are not yet used by any sparsebak function, so you can ignore this if you won't keep your current archive set for long. But you can fix the problem in existing archives for compatibility with future verify, restore operations with the following command:
cd sparsebak/group%pool
find . -name manifest | xargs -I++ sed -E -i 's/^(x[0-9a-f]{16}) 0/0 \1/' ++
Note that manifests are stored on both the admin host (where sparsebak.py is run, such as dom0) and the destination storage system so its recommended to run the command in both places.
Archive metadata
Destination system status & access
Local storage model supporting different types
To support situations where the user is trying to restore after a system wipe, or restore to an alternate system, provide a receive
option such as --from
to allow quick specification of the archive location.
Implement handling of conditions when archives should be accessed as read-only:
Lock archive (in addition to in_process) so no writes can occur during receive, verify, etc.
Identify the home system in archive.ini. This is the only system that can write by default. Provide a function to re-home the archive to a new system.
Check metadata for '-tmp' when starting a send, then find which chunks were sent and continue sending from that point.
Associated changes:
class Volume
initial detectionprepare_snapshots()
with special condition for monitor_only modemonitor_send()
and send_volume()
Changes that may improve throughput, especially for send:
When sparsebak starts any operations on the destination, it should check that the metadata on the source and destination are in sync.
There is an untrusted data handling aspect to this, and it may be best to wait until encryption is integrated and can verify the destination manifests before adding this check.
When receiving an older session, automatically resync the volume by digesting all the later manifests to the volume's deltamap. This will cause all the chunk addresses backed up since the selected session to be backed up again. (Currently the user receives a notice advising to use --remap diff
after receiving the older session but this step can be saved.) Doing a differential save (always comparing blocks before writing) could be of value here.
When user has recovered archive metadata (e.g. after recovering or reinstalling their OS) using --from arch-init
, intelligently re-check metadata to determine if its in sync with LVs and do a full remap only if absolutely necessary.
Needs a volume delete
command that goes beyond the current purge-metata
option.
A special case exists for when the user wants to remove the most recent backup session (instead of removing the whole volume history). Prune will currently not allow this as its not a good fit for the pruning concept, and deletion doesn't require merging while still being a little tricky.
General steps for deleting last session:
Sorry if this is duplicate. Just wanted to make sure an issue was tracking missing feature talked about here
Test basic functions for bugs in non-Qubes environments (where the ssh and internal modes apply).
Features that enable and improve management of very large volumes:
Current volume address size is limited to 64 bits. Increase perhaps to 80 bits (into zetabyte range). This is mainly a function of increasing the hexadecimal places in the chunk filenames.
Graduated chunk sizing would set automatic address boundaries for each chunk size, for example:
(128kB is the current chunk size for all addresses.)
There are a number of possibilities including a multiprocess/multithread send, stream transport with less overhead than tar, etc., and addressing Python techniques and/or alternate language.
Scratchpad, for later expansion into separate issues:
/sparsebak/sparsebak.ini
:
[var]
vgname = qubes_dom0
poolname = pool00
destvm = qubes://work
destmountpoint = /home/user/mntenc-spbak
destdir = backups
[volumes]
vm-cftest-private = enable
Notes:
cftest
is a standaloneVM that I use for booting compact flash images from embedded routers. I don't think I've ever written anything to the private volume - that could very well be the issue here.mntenc-spbak
is an ext4 fs on a luks volume on a file from an nfs shareInitial backup of the VM works:
sudo /usr/local/bin/sparsebak.py send
:
Configured Volumes:
vm-cftest-privateStarting backup session S_20181211-211105
Preparing snapshots...
Initial snapshot created for vm-cftest-private
Current snapshot created: vm-cftest-private.tockScanning volume metadata...
Processing Volume : vm-cftest-private
Backing up to VM work
100% 2047 x000000007ffe0000 DATA
161 bytes sent.
Rotating snapshots for vm-cftest-privateDone.
The backup is then properly shown:
sudo /usr/local/bin/sparsebak.py list vm-cftest-private
Configured Volumes:
vm-cftest-privateSessions for volume vm-cftest-private :
20181211-161151Done.
... as are files in the work
VM:
ls -1 mntenc-spbak/backups/sparsebak/qubes_dom0%pool00/vm-cftest-private/S_20181211-161151/
000000000
000000001
000000002
000000003
000000004
000000005
000000006
000000007
info
manifest
But then, subsequent backups fail:
sudo /usr/local/bin/sparsebak.py send
:
Configured Volumes:
vm-cftest-privateStarting backup session S_20181211-211316
Preparing snapshots...
Delta map not finalized for vm-cftest-private ...recovering.
Current snapshot created: vm-cftest-private.tockScanning volume metadata...
Acquiring LVM delta info.Processing Volume : vm-cftest-private
Updating block change map: file = /tmp/sparsebak/delta.vm-cftest-private
bkchunksize = 131072
dblocksize = 512
bs = 512
Traceback (most recent call last):
File "/usr//local/bin/sparsebak.py", line 1016, in
monitor_send(options.volumes, monitor_only=False)
File "/usr//local/bin/sparsebak.py", line 551, in monitor_send
= update_delta_digest()
File "/usr//local/bin/sparsebak.py", line 314, in update_delta_digest
raise ValueError("Chunk size error")
ValueError: Chunk size error
EDIT: the issue is reproducible after "resetting" everything (rm /sparsebak/... + lvremove the tick/tock volumes in dom0, and rm mntenc-bak/... in the dest VM)
EDIT2: forgot to add that sparsebak.py monitor
gives the same error (only difference is the action - line 1013, monitor_send(monitory_only=True
)
Update: There are now 4 different dedup methods. See #8 (comment) for initial details.
Looking for ideas about a possible deduplication feature.
One simple idea that I've already tried manually with find | uniq | ln
is to find the duplicate hashes in the manifests and link their files together on the destination, thus saving disk space. This unsophisticated approach has low CPU & IO overhead but the prospective space savings is lackluster.
It is also possible to retain and correlate more detailed metadata from thin_dump
which indicates where source volumes share blocks. However, I've seen complaints about the CPU power needed to process these large xml files (that is not to say they couldn't be cleverly pre-processed to reduce overhead).
More advanced dedup techniques include sliding hash window comparisons. At this point the dedup is actively doing new forms of compression and its not clear this is worth the trade-offs for most users. At the very least it appears beyond what Python can do efficiently.
Detecting when a bkchunk is updated only at its start or end may save significant space (at least when lvm chunks are small). This would require special routines using extra bandwidth in merge and receive functions.
Integrate an encryption layer that can also be used to verify metadata and data from the destination archive.
Looking for examples and discussion on applied cryptography techniques from best practices to implementations in various tools including qvm-backup, restic, Time Machine, etc.
Wyng's threat model appears to be most similar to an encrypted database: A mass of data that is updated and curated periodically. Attackers gaining access to the entire volume ciphertext, possibly on successive occasions may be assumed.
Encryption scheme should be robust and have low interactivity and complexity as well as high isolation potential.
Isolation would be in the form of a Qubes-like environment where the Admin VM (e.g. Domain 0) running the backup process is blocked from direct network access, and encryption/decryption is performed only there. Wyng should be able to encrypt effectively in such an isolated environment.
Compatibility with Admin isolation also extends to how any guest containers/VMs are handled: Encryption and integrity verification cannot rely on the guest environments or their OS templates.
LUKS or VeraCrypt on a loop device (which can be isolated) with backing in a remote/shared image file. For example: cryptsetup -> losetup -> sshfs. This solution is readily available but imposes a performance penalty of ~20% on a VM-isolated configuration. It also requires painstaking user setup in a Linux-specific environment; difficult to integrate; poor choice for remote/cloud.
Encfs - A FUSE file-encrypting layer may improve performance over a setup based on a loop device. It may also be simpler to setup or even integrate. Advantage: automatic filename (but not size or sequence) obfuscation. Drawback: issue with hardlinks in some encryption modes.
CryFS - Another FUSE layer with built-in support for network transports. Complete file metadata obfuscation. Claims superior resistance to attack. Unknowns: Hardlink support, transport isolation potential.
Direct crypto library/AES utilization - Uses no external layers, but requires painstaking attention to detail and review by a cryptographer if possible. This option may be a natural choice, given the simplicity of the archive chunk format; any issues around the implementation security should have direct analogues to a wide field of other implementations and their use cases. See initial comments on AES modes.
Some encrypted backup tool that can accept a stream of named chunks with very low interactivity between the front end and back end (e.g. a 'push' model).
(After some deliberation and using Wyng with external encryption layers, this issue will be primarily concerned with an integrated solution similar to item 4.)
Wyng keeps volume data and metadata as separate files, and the metadata validates the volume data.
See Issue #79 for specifics on metadata, which is expected to use separate encryption keys.
Following a core tenet of cryptography that the application must be understood thoroughly before making specific decisions, a substantial familiarity with Wyng is required to make sense of this issue (ye have been warned...).
Its suggested that making some incremental backups with Wyng and looking at the metadata under '/var/lib/wyng.backup' is a good starting point. In the source code, the classes under ArchiveSet()
are instructive in addition to merge_manifests()
and the places where its used.
So, I've made two backups for my test VM but sparsebak.py verify
verifies only the last one despite using the --session ...
option.
So, I have sessions "20181212-151804" and "20181212-152201" for the VM, as shown by sparsebak.py list vm-cftestbkp-private
:
Configured Volumes:
vm-cftestbkp-privateSessions for volume vm-cftestbkp-private :
20181212-151804
20181212-152201Done.
But trying to verify session "20181212-151804" isn't possible:
sudo /usr//local/bin/sparsebak.py --session 20181212-151804 verify vm-cftestbkp-private
(or sudo /usr//local/bin/sparsebak.py verify vm-cftestbkp-private --session 20181212-151804
):
Configured Volumes:
vm-cftestbkp-privateReading manifests
Receiving volume vm-cftestbkp-private S_20181212-152201
x000000007ffe0000 OK
Received bytes : 2147352576Done.
EDIT: using the --session ...
option with 'receive' works.
Looking for a new, permanent name for what is currently called sparsebak.
Some current candidates I'm tossing around...
delta-v
parq
exidi
distar
sparsebak.py send
100% 500Mb
terminated untar process!
Is this okay?
Although ssh
with Linux shell is currently supported, this is not commonly offered by large 'cloud' storage services.
Some protocols that have been already suggested:
sftp
amazon S3
swift
webDAV
...or using FUSE to access one of the above or other storage type such as @cryfs .
I have a "base" VM image and a few writable snapshots of it that have been modified .. sparsebak is cool but it would be really good if it could detect that a newly added volume has its origin in a thin volume that is already archived.. is this technically possible? I'm a little hazy on how the underlying thinpool chunk mapping system works but it seems like it ought to be.
Wondering if it could ultimately evolve into a system for distributing base VM images and deltas around the place :)
Moving away of sparsebak to wymg, attempting to prune old sessions (#46) resulted in never ending merging (no activity on disk) which I stopped (ctrl-C), and then following backups tried to resume the operation(which constantly failed: no disk activity) and now I get this error when trying to start new AppVMs.
Any guidelines to follow when I will reboot my machine to fix pool?
Currently wyng may refuse to run on a remote system if that system previously ran wyng as root (in source mode). This is due to different permissions on the /tmp/wyng directory.
Actions:
$ sudo /sparsebak/sparsebak.py send --save-to sys-usb:/run/media/user/USBdrive/sparsebak
I was attempting to "send" all my VM's private sections to a drive mounted on sys-usb, and I seem to have run into a problem with one particular AppVM. When it got to my vm-vault-private it printed:
Delta map not finalized for vm-vault-private ...recovering.
and a few seconds later this exception:
Processing Volume : vm-vault-private
Updating block change map: Traceback (most recent call last):
File "/sparsebak/sparsebak.py", line 1008, in
monitor_send(options.volumes, monitor_only=False)
File "/sparsebak/sparsebak.py", line 543, in monitor_send
= update_delta_digest()
File "/sparsebak/sparsebak.py", line 307, in update_delta_digest
assert bkchunksize >= (bsdblocksize) and bkchunksize % (bsdblocksize) == 0
AssertionError
All the other AppVM's appeared to be processed/snapshot'ed Ok, but since there was this one fatal exception the backup obviously did not complete. I'm not sure what the complaint is with the blocksize because the AppVM runs fine and with no errors that I can see in the log file.
Hello, it's me again.
After attempting approximative migration from sparsebak to wyng, I got my qubesos installation foobar. Got smart enough to clone my disk to another one prior to mess with it even more and was able to use qubesos native backup to restore daily used VMs on fresh installation.
Now, I still have my sparsebak backups of used and unused AppVMs volumes on internal encrypted additional m2 drive, mounting the luks encrypted container. All backups are there, including the one that got stalled on the prune attempt, under /backups/sparsebak
I would love to know which steps are needed to move things around, do arch-init properly and continue monitor and send operations effectively, while trying to salvage data from the backup that caused it and locked the LVM and caused me to reinstall.
I just cloned my daily used disk to another disk and am ready to test instructions and report back any issue in a migration procedure away from sparsebak to wyng.
Thanks!
An anomaly can crop up in the archive if the computer's local time switches to a value that is less than the most recent session.
Possible ways to address the situation:
In 'new4' branch:
Failure to receive to an existing logical volume because lvresize
returns an error if the volume is already the same size:
Traceback (most recent call last):
File "./sparsebak.py", line 1495, in
save_path=options.saveto)
File "./sparsebak.py", line 1210, in receive_volume
"-f", save_path])
File "/usr/lib64/python3.5/subprocess.py", line 316, in check_output
**kwargs).stdout
File "/usr/lib64/python3.5/subprocess.py", line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['lvresize', '-L',
'2147483648b', '-f', '/dev/qubes_dom0/vm-xyxyxy']' returned
non-zero exit
status 5
Reported by @taradiddles
For v0.2 implement basic recovery mode, especially for delete, send and merge
.
This isn't the same as issue #15, which is a full-featured resume for the send operation. Initially, recovering send will merely delete the prior attempt.
Let's suppose that source host is down and we want to restore volume data. How this can be done ? Host is different.
I have tried this:
sparsebak arch-init --source=dummy/nothing --destination=internal:/path/to/backup
but
sparseback receive my-precious-volume --save-to=internal:./myfile.img
fails with
Destination not ready to receive commands.
I tried:
sparsebak add my-precious-volume
but that does not help
In 'new4' branch, monitor
doesn't work when processing a volume that has been configured but not yet sent:
Preparing snapshots...
Traceback (most recent call last):
File "./sparsebak.py", line 1465, in
monitor_send(datavols, monitor_only=True)
File "./sparsebak.py", line 839, in monitor_send
= prepare_snapshots(volumes if len(volumes) >0 else datavols)
File "./sparsebak.py", line 525, in prepare_snapshots
+snap1vol+" is missing!")
RuntimeError: ERROR: Map and snapshots in inconsistent state,
vm-printserver-private.tick is missing!
Reported by @taradiddles
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.