dyorgio / apfs-clone-checker Goto Github PK

Testing whether clone_checker works correctly within a controlled environment

• APFS Clone Checker - Version: 1.0.0.0
• macOS Big Sur 11.2.1

$ cd ~/Documents/Tests/copying-on-APFS
$ ls -l
-rw-r--r--@ 1 myUserName  staff  71632724 Feb 18 11:10 1-original.mov
-rw-r--r--@ 1 myUserName  staff  71632724 Feb 18 11:10 2-copy-on-same-APFS-volume.mov
-rw-r--r--@ 1 myUserName  staff  71632724 Feb 18 11:10 3-copy-via-external-APFS-volume.mov
# All files have a link count number of 1.
# This prooves that the duplication/copy operations were not achieved via hardlinking.

$ sha1sum *
sameCheckSum  1-original.mov
sameCheckSum  2-copy-on-same-APFS-volume.mov
sameCheckSum  3-copy-via-external-APFS-volume.mov
# Checksum tool shows that all files indeed had the same content.

$ clone_checker 1-original.mov 2-copy-on-same-APFS-volume.mov
1 # Indeed a copy-on-write clone!
$ clone_checker 1-original.mov 3-copy-via-external-APFS-volume.mov 
0 # Not as copy-on-write clone as it had travelled through another APFS volume.
  # Which was even on another physical device to be absolutely sure.

Conclusion: clone_checker seems reliable to me from these tests!

Nice tool but can't handle sitution when clone involved in place compressed files, then it displays:

"Cannot convert logical to physical offset. 45 Operation not supported"

error message

Video files tend to be large, how is iMovie dealing with them?

• iMovie has had self-contained libraries and projects for ease-of-use and robustness since its inception in 1999.
  • On those elder iMovie versions running macOS on a HFS+ file system this often meant a huge waste of file storage when you had the file both somewhere in your folder structure and imported it into iMovie. Because this was a real copy taking up space twice.
  • Project efforts like dedup-imovie-library show this.
• As I had gained trust in apfs-clone-checker's reliability in #1 I now wanted to check:
  • Do contemporary versions of iMovie on macOS with APFS make good use of it's copy-on-write feature?
  • Answer: Yes! Details below:

Versions

iMovie 10.2.2
macOS 11.2.1 Big Sur
Internal SSD formatted as APFS

Investigation

# Added 1-original.mov from Finder via drag-n-drop to "iMovie Library" into event "General"
# Disk I/O shows only few KBs. Not the 72MB the file takes.
# A first good sign. Either copy-on-write or linking took place.

$ clone_checker ~/Documents/Tests/copying-on-APFS/1-original.mov \
~/Movies/iMovie\ Library.imovielibrary/General/Original\ Media/1-original.mov 
1 # Nice! 1 means it is a copy-on-write clone. Consumes no extra storage except its metadata overhead!

# Within iMovie I added a movie from "LIBRARIES > Photos"
# to "LIBRARIES > iMovie Library > "Family", a so called "event".
# Think of it like a category or folder for organizing clips.

$ clone_checker ~/Pictures/Fotos\ Library.photoslibrary/originals/C/very-long-UID.mov \
~/Movies/iMovie\ Library.imovielibrary/Family/Original\ Media/very-long-UID.mov 
1  # Nice! Again a real clone!

If you're interested in using this to deduplicate files on your disk, you might want to check out https://github.com/jbruchon/jdupes. It supports creating clonefiles with duplicates via the -B/--dedupe flag. Note that the MAN page incorrectly says this depends on btrfs, but it supports both btrfs and APFS since around v1.17.

This is a great app.
One interesting way to expand it is to give you unique size of a folder.
So for example, if I have a folder with a file and its clone. I modify one byte within that clone (e.g. with dd), the total size of that folder on disk is the size of one file + one block. However, regardless of any editing, Finder reports double size, though Storage Manager correctly reports free space.
If this app could be extended to do that too, it'd be fantastic.
Cheers

P.S. I'm on Catalina, not sure about newer macOS releases.