Comments (17)
This is not hard to implement, but I need to think about the best API for this a little further (this could either be Windows-focused or expose tar options). I'll think about this for a little while. When I come up with something that feels solid, I'll circle back here.
from docker-volume-backup.
How do you untar the archive?
from docker-volume-backup.
Any input on this @danielmbrown603? To expand on my question: Pax Header files are expected to be in the archive in certain cases (Go will selectively pick a header format based on the current requirements for writing file info) but they should not be part of the untared content.
This is what package tar
says:
// If the format is unspecified when Writer.WriteHeader is called,
// then it uses the first format (in the order of USTAR, PAX, GNU)
// capable of encoding this Header (see Format).
Format Format
which makes me think the Synology files cannot be written to USTAR
header and are using PAX
format instead, but for whatever reason these will be interpreted as actual files when you untar the archive. To me this sounds as if there's a mismatch between the way Go writes tar headers and the way your untar mechanism treats these headers. Do you use plain Linux/MacOS tar
to do this or are you using something else? Could you provide an archive that lets me reproduce the behavior?
from docker-volume-backup.
@m90 Sorry, busy week.
I just started using this with synology, so I haven't untar'd anything yet.
I'm using what comes with the container and synology.
I'll message you a link to download the tar.
from docker-volume-backup.
The file you sent to me does not contain any header files when I inspect in on my machine:
➜ tar -xvf san2-2022-04-03T12-58-46.tar.gz
tar: Entferne führende „/“ von Elementnamen
/backup
/backup/offen
/backup/offen/notifications
/backup/offen/notifications/notifications.tmpl
➜ tree backup
backup
└── offen
└── notifications
└── notifications.tmpl
so I haven't untar'd anything yet
How do you know the archive contains such extraneous files then?
from docker-volume-backup.
Interesting. I opened up the file with 7zip, that is how I saw the PaxHeaders.
Hmm
from docker-volume-backup.
Seems this is a limitation of 7zip and it doesn't know about PAX headers so it treats them as files. Is there another common way of extracting tar files on Windows? Does it produce the same output?
It would definitely be possible to add a UstarHeadersOnly
config option or similar that makes sure the archive does not use Pax Headers ever, but then I am not entirely sure if a. it's worth it and b. it has other unintended consequences.
from docker-volume-backup.
Good insight. Ok, so I replicated the exact backup but from Ubuntu 20.04.4 LTS running my other docker containers. I also opened it the exact same method with 7zip and listed the contents as shown below. What do you think is going on? Although its a new Synology running the latest software, maybe Synology lags behind in some of the linux packages?
from docker-volume-backup.
I'm not entirely sure when that happens exactly, but it seems Go will selectively choose which header format to use:
// If the format is unspecified when Writer.WriteHeader is called,
// then it uses the first format (in the order of USTAR, PAX, GNU)
// capable of encoding this Header (see Format).
Format Format
Something must make the Synology setup choose the PAX format when your Ubuntu setup can use USTAR format for the same content. The code choosing this lives here https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/archive/tar/common.go;l=333-412;drc=refs%2Ftags%2Fgo1.18 but right now I don't really have a good explanation for the behavior you are seeing.
from docker-volume-backup.
Knowing where those files are coming from, are you able to use the archives on Windows without having to wrangle the header files or should we look into adding a "WindowsCompatibleArchive" (or a better name) option @danielmbrown603
from docker-volume-backup.
That could definitely work but is there some way to add an exclusion option? Although different but with rclone, I have a section that I can exclude folders. This exclusion is regardless of the actual path. It's something like this.
--exclude *fuse_hidden*
--exclude *_HIDDEN
--exclude .recycle**
--exclude .Recycle.Bin/**
--exclude *.backup~*
--exclude *.partial~*
from docker-volume-backup.
The only way to exclude files is currently the one described in #42, globbing or similar is not supported at the moment.
from docker-volume-backup.
That's right, ok.
Do you want to try WindowsCompatibleArchive? I'm more than happy to test.
from docker-volume-backup.
@danielmbrown603 When you find the time, could you try building an image from the header-format branch:
git clone [email protected]:offen/docker-volume-backup.git
cd docker-volume-backup
git checkout header-format
docker build . -t offen/docker-volume-backup:header-format
and then try running a backup using the offen/docker-volume-backup:header-format
image while setting TAR_ARCHIVE_HEADER_FORMAT="USTAR"
and see if it would still contain the header files when unzipped using 7zip?
from docker-volume-backup.
I get this error
docker@docker:~/docker$ git clone [email protected]:offen/docker-volume-backup.git
Cloning into 'docker-volume-backup'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
from docker-volume-backup.
See https://docs.github.com/en/authentication/troubleshooting-ssh/error-permission-denied-publickey#make-sure-you-have-a-key-that-is-being-used or you could also use a https URL to clone the repo if you don't have git set up with an SSH key
from docker-volume-backup.
Closing this because it's stale and I also start being doubtful this tool should try to work around odd behavior in 7zip at the cost of a highly specific API that possibly eats expected metadata of backups when used incorrectly.
If anyone runs into this problem (unexpected PaxHeaders
files) as well, I'd recommend using tar.exe
instead of 7zip. It can handle the archives created by this image as expected.
from docker-volume-backup.
Related Issues (20)
- Hard to understand error message during exec HOT 3
- archive-pre is triggered on each container backup instead of just container with exec-label HOT 3
- Health Check HOT 6
- Notifications Delay HOT 1
- Replace Log line feeds with HTML <br> tag HOT 3
- Backup the entire content without directory HOT 4
- Backup to subfolders HOT 2
- Resolve date in AWS_S3_PATH HOT 1
- Can't get stop during backup working HOT 2
- Remove local backup after upload to S3 HOT 3
- Invalid memory address crash when running a backup HOT 16
- Fail to configure cron schedule HOT 3
- Docker swarm error HOT 5
- "swarm exec" support HOT 1
- "input/output error" when copying file to NFS mount HOT 4
- Azure storage: Support access tier HOT 2
- schedule the docker-volume-backup? HOT 1
- [Feature] Automated backups of all named volumes HOT 6
- [FR] upgrade `shoutrrr` to v0.8.0 to support custom fields in calls to Generic Webhooks HOT 3
- [Feature] Allow BACKUP_COMPRESSION to be disabled HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-volume-backup.