Hello, first – thanks for the updated tool, was always hoping someone could improv

Thanks! I'll have a look over the weekend. <div class="email-quote

0 byte files after running afsctool (1.7/1.6.9 with LZVN & zlib support) about afsctool HOT 19 CLOSED

rjvb commented on July 21, 2024

0 byte files after running afsctool (1.7/1.6.9 with LZVN & zlib support)

from afsctool.

Comments (19)

RJVB commented on July 21, 2024 2

I think I have found the issue. While re-organising code recently I moved the file truncation before a certain number of condition tests that assumed the original file was still intact. And the problem files here were probably all examples where compression INCREASED the size, something afsctool refuses to do. Stupidly there was no warning, nor an attempt to restore the file. I'm finishing up a few changes that should prevent this from happening in the future, I think there were a few other (even more) corner cases where failure could lead to an empty file.

from afsctool.

RJVB commented on July 21, 2024 1

And I think the culprit is simply this commit: #bbd7c341 . Before that the file was truncated after writing the extended attributes (= the compressed resource fork and/or the decmpfs xattr). The goal was to avoid free space fragmentation when working with large files ... that overshot its mark a bit for some users...

from afsctool.

ahelpingchip commented on July 21, 2024 1

Thank you for this! I've been running into the same issue and was just about to put together a sample test case, and arrived at the issues page to find it already resolved 🎉

I can confirm that in my case (mostly PNGs and assets used in apps), the problematic images are no longer truncated to zero-byte files.

from afsctool.

RJVB commented on July 21, 2024

I've heard about this issue but only really in a recent report here where in the end the problem was not caused by my afsctool version. From what you're reporting here it would seem that there is indeed an issue. Do you get it with both LZVN and (all levels of) ZIP compression? Does it also affect files that were already compressed (it shouldn't so if it does there's another bug)? You are not getting any messages about failed operations, not even with -vvv, do you? Since the original afsctool (built from source I presume?) is unaffected the only explanation I can see is that my code somehow fails to set the compression attributes. If that happens you indeed end up with an empty file, exactly as when you mount a disk with compressed files on a Mac OS X < 10.6 version. Can you please create a dmg from a folder that contains a few files that compress OK and a few that lead to empty files, taking care to preserve all attributes these files may have? The surest way would be to create a writable dmg, mount it, and then copy the sample folder with `ditto --rsrc $folder /Volumes/afsctoolSample/`, unmount the dmg and upload it here. That way I can try to reproduce your issue and hopefully understand what happens. The other thing you could do is to checkout the very first commit from my repo which should be the original v34 version with just a new build system and see if it's issue free. If so you can work your way back up the commit log, a priori with the help of `git bisect`.

from afsctool.

subzerofun commented on July 21, 2024

Thanks for the quick response! I've created four test folders and copied them via ditto to an dmg. You can download it here: https://we.tl/t-YZQlmMfcGr

I get the empty files with both the LZVN and ZIP option – the compression level does not seem to change the outcome (tried 5 & 9).

In the PNG and JPEG folders (ZLIB & LZVN) there are a few 0 byte files (Samsung*.png). When using the brew afsctool these files compress without losing their content (folder test-afsctool164).

I've also copied all terminal outputs from building the LZVN libraries and afsctool files as well as the output (with 'vvv') of afsctool as txt files (plus some screenshots). Hope you can solve this issue – i have no idea what goes wrong during compression.

Have compiled the first version (v34) and when i try to compress the test folder i get the error "No compressable files in folder"... Strange. I've doublechecked the files, they are all uncompressed. File permissons are OK. Tried running it with "sudo", but get the same error. Will go up the ladder and see if later versions work.

from afsctool.

RJVB commented on July 21, 2024

Thanks! I'll have a look over the weekend.

Have compiled the first version (v34) and when i try to compress the test folder i get the error "No compressable files in folder"... Strange.

Without having looked at the folder contents I can only say that this may not be strange; afsctool never tried to compress files that can reasonably be expected not to compress, by looking at file types. Png files will most likely be skipped, for instance. So "not compressable" is really to be taken in the broad sense of "I (probably) cannot make this any smaller". PS: yikes, yes, there's a problem with compressing png files! Judging from an older MacPorts version I still had installed commit #28ec92ecb65660da0d591e036c44592db9921e23 should still be safe.

from afsctool.

subzerofun commented on July 21, 2024

With v34 i get this error message for every folder i select – i wonder if something simply did go wrong during the build process. But it doesn't matter – i am right now at commit 331893d (oct 2018) and both ZIP and LZVN compression work without producing empty files! Will try out the next few versions to narrow down the breaking change.

This issue is not only limited to png and jpg files, it also happened with a >20 MB PSD file. And i was so smart to compress two crucial folders on my system: /usr/local/Cellar (all the homebrew files) and the "site-packages" folders of my python2.7 and 3.6 installs. And after that i had to clean up hundreds of 0 byte .py, .rb and other source files... At least now i have some commands ready to reinstall all my brew and pip packages for the next catastrophe :-).

from afsctool.

RJVB commented on July 21, 2024

Will try out the next few versions to narrow down the breaking change.

Thanks! Remember `git bisect`. Never used it myself but it's designed to pinpoint the commit that introduced an error. Fortunately afsctool is small and should build quickly (invoking make in the build dir. after a checkout should be enough).

This issue is not only limited to png and jpg files, it also happened with a >20 MB PSD file.

PSD is PhotoShop? My hunch would be that it happens to any file judge to be incompressable, I must have overlooked an error path (or introduced one) that leaves the files empty.

And i was so smart to compress two crucial folders on my system: /usr/local/Cellar (all the homebrew files) and the "site-packages" folders of my

Thing is, I may have done similar things without yet having discovered it. I've been known to run afsctool on /Applications ... So thanks for catching this!! The hunt will be on first thing after I finish breakfast :)

from afsctool.

gingerbeardman commented on July 21, 2024

Are the files really zero? What happens if you decompress them? What happens if you load/run them?

The only reason I ask is that in Clusters case the data was still there?

from afsctool.

RJVB commented on July 21, 2024

No, they appear to be really empty, and lack the special xattrs too (so the data isn't even in the resource fork either). I'm really confused that I could have missed this :-/

from afsctool.

gingerbeardman commented on July 21, 2024

I've not seen it, or found any compressed apps that won't work because of it. Strange.

from afsctool.

RJVB commented on July 21, 2024

Let's keep the issue open until we have confirmation this was all...

from afsctool.

gingerbeardman commented on July 21, 2024

What commit did things go bad in? I guess I could be using my own build from before that date?

from afsctool.

gingerbeardman commented on July 21, 2024

On closer inspection a couple of apps I compressed did suffer from this (missing icons, so easy to spot!)

from afsctool.

gingerbeardman commented on July 21, 2024

For those needing to repair/restore from backup, you can find zero-byte files easily within a directory:

find ./ -type f -size 0

from afsctool.

RJVB commented on July 21, 2024

Also, please, if you find such files please check if they are image files or any kind of content that is already compressed. If they are, they probably ended up truncated because of the bug I just fixed (hopefully), because most likely the HFS compression INCREASED their size. If they are not that kind of file, please verify if they are now compressed OK (on a copy of course!) Thanks!

from afsctool.

gingerbeardman commented on July 21, 2024

I restored Acorn.app and Transmit.app by downloading fresh apps, and compressing now they are fine (icon does not become 0-bytes)

I only really compress apps, which can be easily recovered or restored, so not much harm was done here for me.

Thanks for fixing quickly.

from afsctool.

RJVB commented on July 21, 2024

Thanks for the heads-up!!

from afsctool.

RJVB commented on July 21, 2024

Seems safe to close this now.

from afsctool.

0 byte files after running afsctool (1.7/1.6.9 with LZVN & zlib support) about afsctool HOT 19 CLOSED

Comments (19)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent