Will this tool work on High Sierra with APFS volumes? Are there any

I'll update you in a couple of weeks. <div class="email-fragm

Compression and APFS about afsctool HOT 22 OPEN

rjvb commented on July 21, 2024 1

Compression and APFS

from afsctool.

Comments (22)

RJVB commented on July 21, 2024 2

No, you're both right. The crux is in Matt's earlier report: ``` $ afsctool -cvvv -T ZLIB mage /Users/matt/Downloads/2019-06-13/mage: file has a compressed chunk that's larger than the original chunk; -L to compress /Users/matt/Downloads/2019-06-13/mage: Unable to compress file. ``` I didn't add such a test to the LZVN compression algorithm (simply because I don't know if it's needed), so the file will indeed compress fine with LZVN. BTW: HFS compression works on a chunk basis, which keeps memory requirements for things like searching low. The default ZLIB compressor will refuse to compress an entire file if one of those chunks becomes larger after compression. There is no indication anywhere that this has to be the case but I'm keeping it as default for backwards compatibility. If you remove the verbose options from the command above you will get the "already compressed" error message. Which is in error. I'll be pushing a fix.

from afsctool.

RJVB commented on July 21, 2024 1

On Tuesday September 26 2017 08:05:20 Szabolcs Horvát wrote: Will this tool work on High Sierra with APFS volumes?

If it works with APFS it ought to work wherever you take your computer, even high up in the Himalaya ;) Seriously, check the other issues and pull requests as well as the commit history. IIRC someone tested it with APFS and got it to work - it was only a question of detecting APFS as a compatible filesystem. I cannot recall having made any changes to the code that does the actual compression and rewriting.

Are there any known problems with converting a HFS+ volume containing compressed files (compressed using `afsctool`) tp APFS? Is it safe to upgrade to High Sierra if we have HFS+ compressed file?

All I can say is that the original author seems to have done his homework; files compressed with afsctool behave exactly as files compressed by Apple ... as far as I've been able to tell in the few years that I'm using this now. As to the migration to APFS: I think it should not be significantly less safe with than without compression. Apple use compression too (and well they should!). Try first on a thumb drive, and whatever you do, make a full backup first (in addition to any TimeMachine backups you may have). The best way I found to upgrade between major Mac OS upgrades is this: - clone your current (internal) boot volume to an external (using Carbon Copy Cloner) - boot once from the external to make sure it works, boot back into your internal disk (this also gives you a clean session) - run the upgrade application and point it to the external - test-drive the result from the external When pleased, everything works etc etc, and while booted from the external: - format your internal (just the partition holding the OS if you have more than 1 partition) - clone back the external onto your internal disk - don't forget to clone the recovery partition too (there are other tools for that, though).

from afsctool.

szhorvat commented on July 21, 2024 1

Reporting back after High Sierra upgrade:

afsctool appears to work fine with APFS

Compressed files were maintained as such during HFS+ -> APFS conversion.

from afsctool.

DanielSmedegaardBuus commented on July 21, 2024 1

Wait, what? I was under the impression that compression hasn't yet been implemented in APFS? Are you saying that you can use afcstool on APFS and it actually compresses files? As in, it doesn't just say it does, it actually does?

from afsctool.

RJVB commented on July 21, 2024 1

I'll update you in a couple of weeks.

I just pushed a minor upgrade which makes the code a bit more legible (cf. comparable to libarchive, by borrowing some of their constants). Doing ```

afsctool -v /path/to/file

``` will now also print out the compression type (and claimed uncompressed file size) if those data are available from the compression header. For the file giving you errors: ```

afsctool -v /System/Library/CoreServices/CoreTypes.bundle/Contents/Library/AppExceptions.bundle/Exceptions.plist

/System/Library/CoreServices/CoreTypes.bundle/Contents/Library/AppExceptions.bundle/Exceptions.plist: File is HFS+/APFS compressed. Compression type: decmpfs xattr (3) File content type: com.apple.property-list File size (uncompressed data fork; reported size by Mac OS 10.6+ Finder): 5285 bytes / 5 KB (kilobytes, base-10) File size (compressed data fork): 3068 bytes / 3 KiB Compression savings: 41.9% Number of extended attributes: 0 Total size of extended attribute data: 0 bytes Approximate overhead of extended attributes: 268 bytes Approximate total file size (compressed data fork + EA + EA overhead + file overhead): 3584 bytes / 4 KiB Uncompressed file size reported in compressed header: 5285 bytes ``` The take-home message here is that afsctool (and libarchive) only implement 2 different compression types. Type 3 is for small files that compress to less than 3082 bytes and take up only a single disk block: here the compressed data is stored directly in the extended attribute (xattr). All other files have type 4, which means the compressed data is stored in the resource fork. Afsctool does a lot of checking, and should not leave "compressed" files around that end up having another compression type than those 2.

from afsctool.

lpar commented on July 21, 2024 1

A Mac executable built using go build. Looks like a failure to compress is being reported incorrectly:

meta@updog % afsctool -v mage 
/Users/meta/Programming/Go/bin/mage:
File is not HFS+/APFS compressed.
File content type: public.unix-executable
File data fork size (reported size by Mac OS X Finder): 4298904 bytes / 4.3 MB (megabytes, base-10)
Number of extended attributes: 0
Total size of extended attribute data: 0 bytes
Approximate total file size (data fork + resource fork + EA + EA overhead + file overhead): 4300800 bytes / 4.1 MiB
meta@updog % afsctool -c mage 
Unable to compress file (already compressed).
meta@updog %

from afsctool.

lpar commented on July 21, 2024 1

Everything built and installed with no problems, thanks.

from afsctool.

RJVB commented on July 21, 2024

Great to know, thanks!

from afsctool.

szhorvat commented on July 21, 2024

The Finder reports smaller sizes. Also, compression is preserved during conversion to APFS>

But there are issues. I have some apps compressed and when running them from a terminal, I get messages like this (but everything appears to work fine):

/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Common/ChunkCompression.cpp:50: Error: unsupported compressor 8
/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Libraries/CompressData/CompressData.c:353: Error: Unknown compression scheme encountered for file '/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist'
/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Common/ChunkCompression.cpp:50: Error: unsupported compressor 8
/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Libraries/CompressData/CompressData.c:353: Error: Unknown compression scheme encountered for file '/System/Library/CoreServices/CoreTypes.bundle/Contents/Library/AppExceptions.bundle/Exceptions.plist'

@RJVB I haven't forgotten about this but didn't yet find the time to look at it.

from afsctool.

RJVB commented on July 21, 2024

I've seen some mention of this on another issue, but understood it as an indication that APFS supports additional compression types. So do you get systematic warnings about the compression type after compressing with afsctool, but you can still read the files without problem? What happens when you transfer compressed files preserving their compression, or when you convert a HFS volume with compressed files? Is there a difference when the files concerned are compressed by afsctool or came from Apple (for instance, the iTunes installer creates a compressed app bundle, AFAIK)? I don't have much time to work on this right now (nor a system that supports APFS) but I can try to figure out if there's a way to print the compression types, and if there are symbolic representations (constants) somewhere in the header files.

from afsctool.

szhorvat commented on July 21, 2024

@RJVB Don't bother with this now. I still have to investigate when the warnings appear, I just haven't had the time yet. I mentioned it in response to Daniel's comment.

I mainly use afsctool to compress Mathematica. It's 6-8 GB and I need several versions installed. When I start the GUI part of Mathematica from a terminal, I sometimes see these messages. When I start its "kernel" part (the command line program that actually does computations) I never see them. It might have something to do with Mathematica specifically.

I use the program daily and I have not yet noticed any tangible problems.

I'll update you in a couple of weeks.

from afsctool.

RJVB commented on July 21, 2024

Okidoki, but getting a handle on valid and unsupported compression types could be useful anyway. I was going to suggest you figure out what files cause the errors, but from the looks of it the ones you copied here concern a system file (Exceptions.plist). It would be very weird if that error goes away when you run an uncompressed copy of Mathematica (or the message reports the wrong file). That file is compressed for me, but contains entries like AppNapOverrides for applications that I doubt were prerecorded there by Apple (Adium, but also Handbrake...). Maybe Mathematica causes an exception to be recorded in there which somehow leads to the error, independently of the fact you used afsctool?

from afsctool.

Henrietta1989 commented on July 21, 2024

Anybody tested in Mojave?

from afsctool.

maximumsomething commented on July 21, 2024

@Henrietta1989 it works just as expected in Mojave.

from afsctool.

lpar commented on July 21, 2024

Confirmed, seems to work on test files. Go binaries show up as already compressed, which I'm not entirely convinced by, but maybe the linker is that smart?

from afsctool.

RJVB commented on July 21, 2024

Go binaries show up as already compressed, which I'm not entirely convinced by, but maybe the linker is that smart?

What kind of binaries, created how? The (go) linker *could* indeed apply compression by default (under certain conditions).

from afsctool.

RJVB commented on July 21, 2024

A Mac executable built using `go build`. Looks like a failure to compress is being reported incorrectly:

Can you point me to an example of such a file, or attach one here?

from afsctool.

lpar commented on July 21, 2024

Well, I had to zip it, but here you are.
mage.zip

from afsctool.

gingerbeardman commented on July 21, 2024

This compresses for me, with LZVN and the latest afsctool (it seems you're using that)

$ afsctool -cvvv -T LZVN mage
/Users/matt/Downloads/2019-06-13/mage:
Compression type: LZVN in resource fork (8)
File content type: public.unix-executable
File resource fork size: 2346302 bytes
File DECMPFS attribute size: 16 bytes
File size (uncompressed; reported size by Mac OS 10.6+ Finder): 4298904 bytes / 4.3 MB (megabytes, base-10)
File size (compressed): 2347008 bytes / 2.2 MiB
Compression savings: 45.4%
Number of extended attributes: 1
Total size of extended attribute data: 19 bytes
Uncompressed file size reported in compressed header: 4298904 bytes

With ZLIB I get this:

$ afsctool -cvvv -T ZLIB mage
/Users/matt/Downloads/2019-06-13/mage: file has a compressed chunk that's larger than the original chunk; -L to compress
/Users/matt/Downloads/2019-06-13/mage:
Unable to compress file.
File content type: public.unix-executable
File data fork size (reported size by Mac OS X Finder): 4298904 bytes / 4.3 MB (megabytes, base-10)
Number of extended attributes: 1
Total size of extended attribute data: 19 bytes
Approximate total file size (data fork + resource fork + EA + EA overhead + file overhead): 4300819 bytes / 4.1 MiB

I'm on 10.13.6 with an APFS filesystem.

How about you @lpar ?

from afsctool.

lpar commented on July 21, 2024

I unzipped the zip file and tried compressing the unpacked copy and it still wouldn't compress:

% afsctool -c mage
Unable to compress file (already compressed).
% afsctool --help
afsctool 1.6.9. (1.6.8.6-132-gc437635)

OS is 10.14.5 with an APFS filesystem.

Trying to use LZVN I get Sorry, LZVN compression has not been enabled in this build. I just used the README instructions to build it.

from afsctool.

lpar commented on July 21, 2024

With ZLIB, same results as you, so maybe the root problem is that the build doesn't have LZVN enabled.

from afsctool.

gingerbeardman commented on July 21, 2024

Thanks @RJVB

@lpar let us know if you have a problem building with LZVN support which requires cmake+make+install of https://github.com/RJVB/LZVN before cmake+make+install of this afsctool.

from afsctool.

Compression and APFS about afsctool HOT 22 OPEN

Comments (22)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent