Comments (6)
RJVB
commented on August 22, 2024
1
Clearly I should have though of and tested this long ago, so thanks for stumbling across this issue! I'm testing a fix as we speak, which will queue each file only once for compression.
What must be happening here is a classic race condition. A thread accessing a file-being-compressed just a bit later than the other thread(s) reads an empty file, and writes the compressed 0 bytes to the file, overwriting the actual contents. Which should trigger certain protections too and maybe even not produce 0-byte files (because zip will be adding its header).
My existing exclusive lock protection can also cause a deadlock if you specify a directory with lots of files twice; I'm not certain why but it no longer happens with the fix.
Debugging this to figure out exactly why you got empty files without warning (or I deadlocks) won't be trivial and hopefully unnecessary with the queue filter.
Serial processing was never concerned because the algorithm will reject files that are already compressed (or just recompress them as if they weren't compressed at all).
from afsctool.
biziclop
commented on August 22, 2024
1
Non-MAS applications and their Application Support folder, especially those ridiculously huge gigabyte eating apps which contain at least 3 Chromium engine just for fun, email programs' databases which store each message in a separate file (mine went down from 14GB to 9GB), various downloads, PlayOnMac's wine prefixes, other things in Library that not expected to change often, stupid WordPress and whatever installations with many-many well-compressable source files, various sh*t in /usr/local, …
There is always something.
I wonder when will it fail catastrophically. :)
from afsctool.
RJVB
commented on August 22, 2024
I don't think it ever occurred to me to reject duplicates from the list of files to process (but then again, maybe I did!).
Either way and a priori, afsctool shouldn't be able to process the same file multiple times at the same time because it opens them in exclusive mode. That's intended to be a protection against processing files that are currently open by someone else (imperfect because there is no guarantee that that someone else also used exclusive mode). That said, I don't actually know if O_EXLOCK gives any protection for opening a file multiple times in the same process!
Regardless of all that, you chose to compress a directory that might have open files in it, or where processes could do things with files while you are too. The fontconfig cache files are examples of that.
Afsctool tries very hard to be safe and "transparent" for directories and files that are "offline" (unused) when they're being compressed, but isn't designed for any other kind of use. That's basically impossible anyway (OS X has no sure and above all no efficient way of telling is a file is opened by someone else; this is probably also why ejecting a disk that has in-use files on it is so cumbersome and can really rev up the fans).
from afsctool.
biziclop
commented on August 22, 2024
Maybe file locking only locks the main data fork, but not the other forks/xattrs/etc?
Anyway, I made a simple test, text files filled with aaa…, but even in the most simple case it zaps even a single file specified twice:
afsctool -cvvv -J1 ~/test/0/0.txt ~/test/0/0.txt
but it also "works" with the entire directory:
afsctool -cvvv -J1 ~/test ~/test
If I set any thread option (-[jJ][123…]) it zeroes, otherwise it doesn't.
I tried it on my main fusion drive, a pendrive, inside a dmg on and external ntfs hd, they all produced the phenomenon.
Here is the test zip:
test.tar.gz
from afsctool.
biziclop
commented on August 22, 2024
Thank you for looking into it!
Anyway, it's f*g amazing, It already freed 15GB space! Why didn't Apple enabled it by default… Maybe they want us to buy the larger built-in soldered-on storage. :)
from afsctool.
gingerbeardman
commented on August 22, 2024
Thank you for looking into it!
Anyway, it's f*g amazing, It already freed 15GB space! Why didn't Apple enabled it by default… Maybe they want us to buy the larger built-in soldered-on storage. :)
I think it's easier said than done. It should only be used on files that are (mostly?) read-only, so apps are a good fit. Apps from the MAS and System files from OS install already have it applied. Where else would you recommend it being applied?
from afsctool.
Related Issues (20)
- Is this afsctool the clear successor to the original? Should the homebrew afsctool formula point here instead? HOT 4
- Does your afsctool support APFS? HOT 2
- Unable to build LZVN on apple m1 HOT 15
- building for macOS-x86_64 but attempting to link with file built for macOS-arm64 HOT 8
- About maintenance of the fork HOT 3
- Is this identical to 'brew install afsctool' or is that the original version? HOT 4
- Failed to build on M1 - Monterey 12.0.1 with Xcode 13.1 HOT 11
- Process stuck waiting HOT 12
- Are we ready for a new release? HOT 3
- Unable to build when the path to Xcode contains spaces HOT 3
- Build errors with GCC 5 HOT 9
- Recovering missing/deleted files HOT 4
- Time Machine risks? HOT 3
- Incorrect license HOT 4
- -t and -i options usage HOT 10
- afsctool breaks binaries, sometimes (HFS+) HOT 9
- Compress, decompress, compress results in lots of mismatch errors HOT 19
- compressing breaks app (workaround: restart mac) HOT 25
- Cannot build HOT 11
- Build problems with macOS Sonoma HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from afsctool.