PaxHeaders.0 throughout my tar file about docker-volume-backup HOT 17 CLOSED

This is not hard to implement, but I need to think about the best API for this a little further (this could either be Windows-focused or expose tar options). I'll think about this for a little while. When I come up with something that feels solid, I'll circle back here.

from docker-volume-backup.

How do you untar the archive?

from docker-volume-backup.

Any input on this @danielmbrown603? To expand on my question: Pax Header files are expected to be in the archive in certain cases (Go will selectively pick a header format based on the current requirements for writing file info) but they should not be part of the untared content.

This is what package tar says:

	// If the format is unspecified when Writer.WriteHeader is called,
	// then it uses the first format (in the order of USTAR, PAX, GNU)
	// capable of encoding this Header (see Format).
	Format Format

which makes me think the Synology files cannot be written to USTAR header and are using PAX format instead, but for whatever reason these will be interpreted as actual files when you untar the archive. To me this sounds as if there's a mismatch between the way Go writes tar headers and the way your untar mechanism treats these headers. Do you use plain Linux/MacOS tar to do this or are you using something else? Could you provide an archive that lets me reproduce the behavior?

from docker-volume-backup.

@m90 Sorry, busy week.
I just started using this with synology, so I haven't untar'd anything yet.
I'm using what comes with the container and synology.

I'll message you a link to download the tar.

from docker-volume-backup.

The file you sent to me does not contain any header files when I inspect in on my machine:

➜  tar -xvf san2-2022-04-03T12-58-46.tar.gz 
tar: Entferne führende „/“ von Elementnamen
/backup
/backup/offen
/backup/offen/notifications
/backup/offen/notifications/notifications.tmpl
➜  tree backup 
backup
└── offen
    └── notifications
        └── notifications.tmpl

so I haven't untar'd anything yet

How do you know the archive contains such extraneous files then?

from docker-volume-backup.

Interesting. I opened up the file with 7zip, that is how I saw the PaxHeaders.

Hmm

from docker-volume-backup.

Seems this is a limitation of 7zip and it doesn't know about PAX headers so it treats them as files. Is there another common way of extracting tar files on Windows? Does it produce the same output?

It would definitely be possible to add a UstarHeadersOnly config option or similar that makes sure the archive does not use Pax Headers ever, but then I am not entirely sure if a. it's worth it and b. it has other unintended consequences.

from docker-volume-backup.

Good insight. Ok, so I replicated the exact backup but from Ubuntu 20.04.4 LTS running my other docker containers. I also opened it the exact same method with 7zip and listed the contents as shown below. What do you think is going on? Although its a new Synology running the latest software, maybe Synology lags behind in some of the linux packages?

from docker-volume-backup.

I'm not entirely sure when that happens exactly, but it seems Go will selectively choose which header format to use:

	// If the format is unspecified when Writer.WriteHeader is called,
	// then it uses the first format (in the order of USTAR, PAX, GNU)
	// capable of encoding this Header (see Format).
	Format Format

Something must make the Synology setup choose the PAX format when your Ubuntu setup can use USTAR format for the same content. The code choosing this lives here https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/archive/tar/common.go;l=333-412;drc=refs%2Ftags%2Fgo1.18 but right now I don't really have a good explanation for the behavior you are seeing.

from docker-volume-backup.

Knowing where those files are coming from, are you able to use the archives on Windows without having to wrangle the header files or should we look into adding a "WindowsCompatibleArchive" (or a better name) option @danielmbrown603

from docker-volume-backup.

That could definitely work but is there some way to add an exclusion option? Although different but with rclone, I have a section that I can exclude folders. This exclusion is regardless of the actual path. It's something like this.

--exclude *fuse_hidden*
--exclude *_HIDDEN
--exclude .recycle**
--exclude .Recycle.Bin/**
--exclude *.backup~*
--exclude *.partial~*

from docker-volume-backup.

The only way to exclude files is currently the one described in #42, globbing or similar is not supported at the moment.

from docker-volume-backup.

That's right, ok.

Do you want to try WindowsCompatibleArchive? I'm more than happy to test.

from docker-volume-backup.

@danielmbrown603 When you find the time, could you try building an image from the header-format branch:

git clone [email protected]:offen/docker-volume-backup.git
cd docker-volume-backup
git checkout header-format
docker build . -t offen/docker-volume-backup:header-format

and then try running a backup using the offen/docker-volume-backup:header-format image while setting TAR_ARCHIVE_HEADER_FORMAT="USTAR" and see if it would still contain the header files when unzipped using 7zip?

from docker-volume-backup.

I get this error

docker@docker:~/docker$ git clone [email protected]:offen/docker-volume-backup.git
Cloning into 'docker-volume-backup'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

from docker-volume-backup.

See https://docs.github.com/en/authentication/troubleshooting-ssh/error-permission-denied-publickey#make-sure-you-have-a-key-that-is-being-used or you could also use a https URL to clone the repo if you don't have git set up with an SSH key

from docker-volume-backup.

Closing this because it's stale and I also start being doubtful this tool should try to work around odd behavior in 7zip at the cost of a highly specific API that possibly eats expected metadata of backups when used incorrectly.

If anyone runs into this problem (unexpected PaxHeaders files) as well, I'd recommend using tar.exe instead of 7zip. It can handle the archives created by this image as expected.

from docker-volume-backup.