Lots of metadata errors when trying to export with `--exiftool --keyword-template "{label}"` about osxphotos HOT 7 CLOSED

Another option is to export sidecars using --sidecar XMP as that preserves the metadata but doesn't run exiftool. You can always use exiftool later to apply the sidecars to the images or use another digital asset manager that reads the sidecars.

from osxphotos.

From the data you sent, the PNG file is really a JPEG. It looks like it may have been adjusted by Google Photos. I've seen this in the past with images touched by an external app having the wrong UTI.

    `originalFilename': 'IMG_5953.PNG',
    'filename': '16C6E421-064D-4B7A-B372-07EB1CC49721.png',
    'adjustmentFormatID': 'com.google.photos.editing.filtering.nondestructive',
    'UTI': 'public.jpeg',

For the HEIC image, Photos thinks this is a HEIC:

    'originalFilename': 'IMG_8411.HEIC',
    'filename': 'FB13365A-AB09-4B10-A14A-68873207523D.heic',
    'UTI': 'public.heic',
    'hasAdjustments': 0,
    'adjustmentFormatID': None,

And I can see no indication an external app touched this image so that's a mystery.

I have seen other cases where the UTI is wrong in Photos (always associated with an external app). There's an issue (#382) to add an option to fix these upon export. One of these days I'll get around to implementing that; it's not a trivial change to the export code because I'll need to maintain the mapping between "reported" and "actual" extension for every photo exported and will need to figure out a way to verify the actual UTI (ideally without calling exiftool).

from osxphotos.

These are probably actual errors where the photo in the database has the wrong UTI (uniform type identifier) associated with it. I've seen this happen before, particularly when external apps are used to edit photos. If the photo type is wrong, exiftool will not corrupt the image but also likely won't export the metadata to the image. But just to make sure there's not a bug in osxphotos, please run the following commands and send me the output to [email protected]

osxphotos debug-dump --dump photos --uuid 16C6E421-064D-4B7A-B372-07EB1CC49721 --uuid FB13365A-AB09-4B10-A14A-68873207523D > debug.txt

from osxphotos.

For some further output, here's the result of that command listed in my Reproduce section with a fresh library:

Exporting 8636 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 8636 photos, exported: 4301, updated: 2, skipped: 0, updated EXIF data: 4301, missing: 8714, error: 114

So looks like 114 files had errors of some variety. Not sure why 8000 reported as missing, maybe some downloads failed.

from osxphotos.

@gabeklavans for one of your errors "atom is too large for rewriting
(3969 MB)" it looks the file is too big 4GB so you may have to use exiftool option. -api LargeFileSupport=1. Y providing it to osxphotos via --exiftool-option. Do osxphotos export -h for more info.

See: exiftool option largeFileSupport

As @RhetTbull points out the UTI info may be incorrect. But please note not all image types support EXIF and XMP data in the same way. PNG for instance does not. I normally convert then into JPG with sips command and try to move as much metadata over as exiftool can.

I also do it for TIF files.

Not the smartest code I've written ;) but I do that with the following bash functions:

# Command which convert .PNG files into .JPG and copies EXIF fields ...

exifpng2jpg () 
{ 
    for a in "$@";
    do
        fname=`basename "$a" .png`;
        xtension=png;
        echo 1st - ${fname}.${xtension};
        if [ \( -f "${fname}".png -o -f "${fname}".PNG \) -a ! -f "${fname}".jpg ]; then
            echo ok - ${fname}.${xtension};
            sips -s format jpeg -s formatOptions 90 """${fname}".${xtension}"" --out """${fname}".jpg"";
            exiftool -overwrite_original_in_place -TagsFromFile "${fname}".${xtension} "-all:all>all:all" "${fname}".jpg;
        else
            fname=`basename "$a" .PNG`;
            xtension=PNG;
            echo 2nd - ${fname}.${xtension};
            if [ \( -f "${fname}".png -o -f "${fname}".PNG \) -a ! -f "${fname}".jpg ]; then
                echo ok - ${fname}.${xtension};
                sips -s format jpeg -s formatOptions 90 """${fname}".${xtension}"" --out """${fname}".jpg"";
                exiftool -overwrite_original_in_place -TagsFromFile "${fname}".${xtension} "-all:all>all:all" "${fname}".jpg;
            fi;
        fi;
    done
}
export -f exifpng2jpg

# Command which convert .TIF files into .JPG and copies EXIF fields ... Uses ModifyDate as the source for DateTimeOriginal, File    ModifyDate, CreateDate fields

exiftif2jpg () 
{ 
    for a in "$@";
    do
        fname=`basename "$a" .tif`;
        dname=`dirname "$a" `;
        if [ -f "${dname}/${fname}".tif -a ! -f "${dname}/${fname}".jpg ]; then
            echo ok - ${fname};
            sips -s format jpeg -s formatOptions 100 """${dname}/${fname}".tif"" --out """${dname}/${fname}".jpg"";
            exiftool -overwrite_original_in_place -TagsFromFile "${dname}/${fname}".tif "-all:all>all:all" "${dname}/${fname}".jpg;
            echo Performing... ""-DateTimeOriginal\<FileModifyDate"" ""-CreateDate\<FileModifyDate"" on "${dname}/${fname}".jpg;
            exiftool -overwrite_original_in_place """-DateTimeOriginal<ModifyDate""" """-CreateDate<ModifyDate""" """-FileModifyDate<ModifyDate""" -P -v "${dname}/${fname}".jpg;
        fi;
    done
}
export -f exiftif2jpg

from osxphotos.

@gabeklavans for one of your errors "atom is too large for rewriting
(3969 MB)" it looks the file is too big 4GB so you may have to use exiftool option. -api LargeFileSupport=1. Y providing it to osxphotos via --exiftool-option. Do osxphotos export -h for more info.

Thanks for the tip on this. However, it doesn't seem to work, in my case. When I add the following to my command: --exiftool-option "-api LargeFileSupport=1" I still get errors like:

❌️  Error exporting photo (24C469FA-6FD6-42F2-91EC-B3E5B32A1AF7: IMG_4593(1).MOV) as /Users/gabe/Pictures/osxphotos-gabe/IMG_4593(1).MOV: Error: 'etTy' atom is too large for rewriting
(742 MB) - /var/folders/c6/gjhzls_n4gb8twdhljq6bhs00000gn/T/osxphotos_export_6an33s1q/24C469FA-6FD6-42F2-91EC-B3E5B32A1AF7_exiftool.mov

I will probably still keep exif on as it works for most of my exports. I just saw a bunch of errors and worried that it was messing up my export. It may be messing with the ability to download missing files, as when I ran without the exiftool option it seemed to successfully download way more photos, but maybe now that they're actually downloaded I can continue to use the exiftool option with minimal worry.

Man, I really wish Apple would just provide a proper export tool...

from osxphotos.

Another option is to export sidecars using --sidecar XMP as that preserves the metadata but doesn't run exiftool. You can always use exiftool later to apply the sidecars to the images or use another digital asset manager that reads the sidecars.

Yeah, I considered that. A pretty neat option, but I was striving for maximum convenience, at maybe the cost of some lost metadata. As long as I can see all the photos, I think I'm happy.

Anyway, it sounds like all my questions have been answered, or have been given paths forward, and that it's not a problem with the application. Thanks for the help!

from osxphotos.