Regression in lossy nonphotographic images about libjxl HOT 7 OPEN

Just a quick follow-up: I’m comparing version 0.10.2 with the latest master version (commit 45e688c) to see how things have changed. I retested the images ultraman.png and tf2_wallpaper.png, both at a distance of 3.

Here are the results:

It seems that a small amount of chroma banding is now gone. Additionally, the chroma noise issues have been resolved as well.

Overall, it’s a significant improvement. While there is still some excessive blurring compared to version 0.8.2, the quality has definitely improved.

from libjxl.

# JPEG XL encoder v0.8.0 adfc39bc [AVX3,AVX2,SSE4,SSSE3,Unknown]
Read 1920x1070 image, 720686 bytes, 196.2 MP/s
Encoding [VarDCT, d1.000, effort: 7], 
Compressed to 86933 bytes (0.339 bpp).
1920 x 1070, 3.63 MP/s [3.63, 3.63], 1 reps, 16 threads.
# JPEG XL encoder v0.9.2 41b8cda [AVX2,SSE4,SSE2]
Encoding [VarDCT, d1.000, effort: 7]
Compressed to 110.8 kB (0.432 bpp).
1920 x 1070, 4.922 MP/s [4.92, 4.92], 1 reps, 16 threads.
# JPEG XL encoder v0.10.0 19bcd82 [AVX2,SSE4,SSE2]
Encoding [VarDCT, d1.000, effort: 7]
Compressed to 135.8 kB (0.529 bpp).
1920 x 1070, 4.440 MP/s [4.44, 4.44], 1 reps, 16 threads.
# JPEG XL encoder v0.10.2 e148959 [AVX2,SSE4,SSE2]
Encoding [VarDCT, d1.000, effort: 7]
Compressed to 135.8 kB (0.529 bpp).
1920 x 1070,  4.516 MP/s [4.52, 4.52], , 1 reps, 16 threads.

not sure if there's undocumented change in definitions of distance, effort etc. but i do also find the compression ratio regression across versions from 0.8.0 to 0.10.2 on almost all -d [>0] -e settings.

just tried this particular image from random website (k i admit it's the firm i'm working at) though, found this issue related, so i guess maybe not worth creating a new issue.

maybe just another piece of sample data. btw interestingly, on this image, the lossless encoding has a big jump on compression ratio between -e 7 and -e 8

from libjxl.

I did a little testing and the the encoding defaults have changed between 0.9 and 0.10. For lossy images, the alpha channel was originally compressed with the same distance as the color channels. Now, alpha is lossless by default, which drastically inflates file sizes for all RBGA images. If this was an intentional change, I don't see it documented anywhere.

from libjxl.

Follow Up:
I did more testing for 0.9.2 and the git master (4da4b9a) versions. Keep in mind I used the same testing methodology from before. I'm using the image metric tools provided in 0.10.2 and I replaced ssimulacra2 and butteraugli scores in the tables below. Here is what I found:

Artwork:
0.9.2

_ultraman.png
Encoding      kPixels    Bytes          BPP  E MP/s  D MP/s     Max norm  SSIMULACRA2   PSNR        pnorm       BPP*pnorm   QABPP   Bugs
----------------------------------------------------------------------------------------------------------------------------------------
jxl:d0.776       1387   313942    1.8095526   2.122  87.061   1.05535292  88.18518366  44.43   0.456649  0.826350682147   1.910    0
jxl:d1.56        1387   198478    1.1440215   2.198  92.191   1.98494279  82.74740828  41.04   0.749756  0.857736472899   2.271    0
jxl:d3.068       1387   123160    0.7098907   2.205  93.007   3.65560531  72.99446351  37.41   1.213732  0.861617436951   2.595    0
Aggregate:       1387   197247    1.1369262   2.175  90.715   1.97106804  81.08107204  40.86   0.746240  0.848420035114   2.241    0

git-master

C:\Users\Fabio\Pictures\_ultraman.png
Encoding      kPixels    Bytes          BPP  E MP/s  D MP/s     Max norm  SSIMULACRA2   PSNR        pnorm       BPP*pnorm   QABPP   Bugs
----------------------------------------------------------------------------------------------------------------------------------------
jxl:d0.751       1387   313919    1.8094201   2.543  67.087   1.16311717  88.02712453  44.91   0.460241  0.835016182014   2.094    0
jxl:d1.482       1387   198438    1.1437909   2.743  63.108   1.97989273  82.68202392  41.37   0.738425  0.848231468962   2.265    0
jxl:d3.02        1387   122897    0.7083748   2.781  55.470   4.41225004  72.99714138  37.92   1.239742  0.879129429439   3.126    0
Aggregate:       1387   197088    1.1360123   2.687  61.697   2.16223236  80.87302586  41.30   0.751688  0.853926904503   2.456    0

Photograph:
0.9.2

donald-michael-chambers-x2d-xcd55v-4.png
Encoding      kPixels    Bytes          BPP  E MP/s  D MP/s     Max norm  SSIMULACRA2   PSNR        pnorm       BPP*pnorm   QABPP   Bugs
----------------------------------------------------------------------------------------------------------------------------------------
jxl:d0.769       6370   620713    0.7795449   2.435 132.488   1.20697951  88.41863950  48.71   0.426740  0.332718142497   0.942    0
jxl:d1.62        6370   342599    0.4302653   2.560 150.726   2.20098257  83.80748849  45.25   0.690876  0.297259921485   0.947    0
jxl:d3.98        6370   169074    0.2129945   2.497 137.780   3.49073076  74.86919107  41.74   1.131144  0.240927722578   0.744    0
Aggregate:       6370   330394    0.4149368   2.497 140.125   2.10211930  82.15074307  45.14   0.693507  0.287761842558   0.872    0

git master

C:\Users\Fabio\Desktop\test\donald-michael-chambers-x2d-xcd55v-4.png
Encoding      kPixels    Bytes          BPP  E MP/s  D MP/s     Max norm  SSIMULACRA2   PSNR        pnorm       BPP*pnorm   QABPP   Bugs
----------------------------------------------------------------------------------------------------------------------------------------
jxl:d0.748       6370   620959    0.7798538   4.594  76.309   1.19779276  88.24129648  48.26   0.441826  0.344559445998   0.934    0
jxl:d1.47        6370   343129    0.4309310   5.098  82.650   2.14825725  83.21586599  44.79   0.733236  0.315971614256   0.926    0
jxl:d3.0         6370   167612    0.2105016   3.413  98.681   3.79526829  73.36415435  41.32   1.237906  0.260574322859   0.799    0
Aggregate:       6370   329313    0.4135793   4.307  85.379   2.13752235  81.36121861  44.70   0.737432  0.304986967964   0.884    0

Compilation of all results so far:
Artwork:

Photo:

Conclusion
Libjxl 0.8.2 consistently scores higher Butterguli metrics across all quality ranges. For higher quality images, the newer versions of libjxl score the same or higher for pnorm and ssimulacra2. This probably means the gap in quality for high quality nonphotographic images is probably within margin of error. For medium quality images the difference in butteraguli/ssimulacra2 scores increase in favor of 0.8.2. Pnorm and dssim scores compare equally. So the regression for medium quality images is subjective at best. For low quality images there is a complete sweep in favor of 0.8.2. The exception is pnorm which decreases instead. This is where I think a regression is certainly possible.

For the photo it's a similar story. 0.8.2 scores higher Butterguli metrics across all quality ranges. Oddly enough, for high quality tests 0.8.2 performs better under all metrics. For medium quality images, all metrics score in favor of 0.8.2 when compared to 0.10.2 and the git master branch. The exception is 0.9.2 where it keeps up with 0.8.2 in dssim/ssim2. Finally, low quality images is where things start to get interesting. Comparing 0.8.2 to 0.10.2/git master, all metrics point in favor of 0.8.2. However 0.9.2 beats 0.8.2 in every metric by a large margin! (except butteraguli) I wonder if the quality improvement is noticeable under the scrutiny of the human eye (it could be a fluke). Anyways, it doesn't really matter since the difference is close enough where there likely isn't a regression present here.

To wrap it all up, the two different types of images shows us a possible regression in two different qualities. For lossy nonphotographic images, the regression seemed to me most apparent for low quality images around distances 2.5-3.0. For lossy photographic images, metrics favor higher qualities at distances 0.75-1.00 (probably just a fluke however). It seems like the git-master branch has improvements (#3531) (#3529) which bumps it up a notch over 0.10.2 but it still has a ways to go to match 0.8.2. Tweaking and refining of the next version is always happening, like (#3535) and more to come. Therefore the regression might slowly disappear over time. At the end of the day it's just algorithms and these results mean nothing when confronted with the human eye. Thanks for reading!

Signing off,
A very tired galaxy

from libjxl.

Thank you for doing this!

I always make the quality-related decisions only by my eyes and more or less ignore the metrics.

I'd know how to make the metrics much better but images to look worse.

The process of eye-balling the quality may lead metrics to agree or disagree. I strongly emphasize on worst case behaviour and try to find mitigations for worst case rather than balanced performance -- since I anticipate that each worst case behaviour will cause practitioners to increase the quality settings and leading to broad increase in bytes for all content.

Do you see -- with your own eyes -- degradation between 0.8 and 0.10, or any other ?

from libjxl.

Thanks for the input!

I just use metrics as a sanity check to make sure I'm not just seeing things.

If you want to focus on the worst case scenario, then lets look at images where .avif excels.

For example the _ultraman image.

Settings used are -d 3 for 0.10 and -d 2.6 for 0.8.

There is more noise in the cross hatching around the eye and forehead that shouldn't be there, so I think 0.8 looks better and more natural than 0.10.

Zooming in, there is a weird grey area where there should be color, its very subtle but I realized that it was present in all images I compressed with 0.10. Obviously its much less visible in photos, but we are talking about the worst case scenario. I saw (#3520) and I tried it with the master branch (4da4b9a) but I got the same result.

Here is another image illustrating similar regressions: tf2_wallpaper

Settings used are -d 3 for 0.10 and -d 2.45 for 0.8.

His face, cap, and gun all have noticeably more noise in 0.10. The strap covering his shirt is possibly the worst offender. There is more chroma noise around his neck. And the gun barrels underneath him are blurrier in 0.10.2. Same thing with the wrap around his hand, the lines are somewhat blurred out.

Here is where the smooth, low-contrast gradients get blurred and noise is added in 0.10. It's even more apparent when zoomed in.

The photo I included in my benchmark shows that 0.10 is visually better that 0.8 in my opinion, despite the metrics favoring 0.8. And it clearly outperforms .avif since this is the kind of image jpegxl was tuned for The skin texture is sharper in the .avif but, there is a very clear banding/smoothing on shaded parts of the face, especially the cheeks. So jpegxl aims for total image consistency rather than focusing on points of interest (which is better imo).

I understand that jpegxl is tuned for general use cases, but the additional noise/blurring leads to worse performance in nonphotographic images like the two examples I've shown. Where small changes to sharpness and smooth gradients are very noticeable. So if tuning the algorithms to benefit these kind of images hurts performance in other photos, then maybe something like, --tune-artwork similar to x264 could be implemented.

from libjxl.

My main issue with recent versions of libjxl is the excessive blurring, which is most apparent in nonphotographic images. According to my testing, this happened between versions 0.8 and 0.9, and it still isn't fixed in master.

from libjxl.