Comments (7)
I'm not saying it wouldn't make sense, but doing it in a good way would likely be hard.
Ideally, when adding a new set of files, you'd like to end up with a similar result than if you had recompressed the whole file system. But that would mean having to insert the files in between other files that are already in the file system in order to be able to make good use of the available redundancy. I'm not saying this can't be done, just that it's probably too much work to get done any time soon.
What I'd probably do is something like:
- create an initial DwarFS image
- create a writable overlay and install new builds to this overlay
- once per week/month or so, build a new DwarFS image directly from the overlay so it includes both old and new builds
I don't know if that would would help in your case, but it's something that could probably be automated quite easily.
from dwarfs.
Yeah, I didn't imagine it was a five-min fix. If we do end up using DwarFS in the near future we'd very likely do something pretty similar to what you suggest. However, we already have an automated system with zstd archives of single recent commits and lrzip archives of bundles of older commits. The appeal of DwarFS (assuming it has a compression ratio similar enough) is that we could rip out all the code we have to handle decompressing the different versions when wanting to use a specific commit and just run something at a known path in the filesystem.
Thanks for the quick response, and I wish we'd known about DwarFS back in 2016 when we were first creating our system!
from dwarfs.
Just to clarify the last comment, the reason lrzip
was chosen is that zstd
didn't have long-range mode back then. Now it does and I even have some code to do the transition from lrzip to zstd only, but dwarfs looks so cool… :)
FWIW here's our journey: Raku/whateverable#23
from dwarfs.
Just a quick update: I'm planning to add support for "snapshots" (or whatever the feature will ultimately be called), which would definitely address this issue; in fact, it will go a lot further. You'll be able to not only mount/extract the latest update, but also all previous updates. Each update would only store the changes relative to the previous state, which would e.g. allow you to use DwarFS for incremental backups. There's no timeline, so don't hold your breath, but it'll hopefully happen before v1.0.0. :)
from dwarfs.
Thanks for the quick response, and I wish we'd known about DwarFS back in 2016 when we were first creating our system!
Well, in 2016 it was still sitting on my laptop, I sadly didn't have the time (and energy) to publish it back then.
from dwarfs.
Just a quick update: I'm planning to add support for "snapshots" (or whatever the feature will ultimately be called), which would definitely address this issue; in fact, it will go a lot further. You'll be able to not only mount/extract the latest update, but also all previous updates. Each update would only store the changes relative to the previous state, which would e.g. allow you to use DwarFS for incremental backups. There's no timeline, so don't hold your breath, but it'll hopefully happen before v1.0.0. :)
If this gets implemented, what's the intention with regards to writeability? Obviously incremental backups would require providing both a source directory and existing DwarFS image, but would mounting an modifying an existing one be a consideration, too? Would that become the default behavior if that were the case or would images remain read-only by default?
from dwarfs.
If this gets implemented, what's the intention with regards to writeability? Obviously incremental backups would require providing both a source directory and existing DwarFS image, but would mounting an modifying an existing one be a consideration, too? Would that become the default behavior if that were the case or would images remain read-only by default?
I have no plans for making the file system writable at this point.
from dwarfs.
Related Issues (20)
- cannot enter subdirs of overlayed dwarfs HOT 10
- Error while building HOT 2
- Fuse Passthrough
- Are the prebuilt binaries affected by xz/liblzma backdoor? HOT 6
- [Feature request] Allow providing dwarfs with a dedup library HOT 4
- official debian package, allow building without git? HOT 4
- mkdwarfs always crashes with SIGABRT HOT 13
- [MacOS] DwarFS mount not seen through Finder HOT 5
- [Core Dump] Signal 7 (SIGBUS) (code: nonexistent physical address) on making archive of currently running OS (possibly bad use case) HOT 2
- Homebrew formula HOT 26
- some problem on the README.md files. Please Check up and Fix. HOT 1
- read scalability issues with large archives HOT 9
- Unexpected exception: `inode has no file (any)` HOT 8
- [Feature Request] Mounting multiple archives to the same path HOT 12
- Segfault when using the mold linker HOT 2
- Cannot build v0.9.9 on Ubuntu 22.04 HOT 5
- Vendor fbthrift & folly using vcpkg HOT 2
- exception thrown in worker thread: class dwarfs::runtime_error: lzma_stream_encoder HOT 2
- [Feature Request] Provide non-generic packaging CI for major linux distributions HOT 2
- Document memory behaviour and give tips for dealing with many files HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dwarfs.