feat(server): thumbnail and preview in JPEG XL format

[develar]

Why

• I use Lightroom. Since edited RAW cannot be rendered (Adobe does not provide a command-line tool; please correct me if I am wrong), photos must be exported. Lightroom supports JPEG XL but not WebP.
• JPEG XL is supported on all my viewer devices, so one modern and great format for all — thumbnail, preview, and "original" (Immich does not yet support RAW+exported pair, so "original" is quoted :))
• In the future, we can use a specific custom solution to generate thumbnails as fast as possible because JPEG XL format supports it (progressive load or embedded). At least, we can try to measure the performance.

Machine Learning

ML uses Python, and both currently used libs do not support JPEG XL.

Pillow doens't support JPEG XL yet, so, I use jxlpy. I decided not to use pillow-jpegxl-plugin as it requires Rust (a more complicated Docker file).
We need libjxl, and as the version in Debian 12 (bookworm) is outdated (jxlpy uses newer API), we build it from sources (clone by tag v0.10.2).

OpenCV also does not support JPEG XL. To overcome this restriction, we re-encode JPEG XL into PNG (no compression, effort: 0 to reduce CPU usage).

Tests

Here I need your help:

• Unit tests: It looks like we need to rename GetAssetThumbnailFormatEnum.WEBP to Small and GetAssetThumbnailFormatEnum.JPEG to GetAssetThumbnailFormatEnum.JPEG.LARGE. The current usage of getThumbnailPath API is somewhat unusual — WEBP is requested when we want a small thumbnail, and JPEG when we want a large thumbnail. I decided not to change all existing usages in Svelte files and await your decision. For example, src="/API/asset/thumbnail/b4196cb9-f386-49c9-be31-888374beffa8?format=WEBP" is used now, and format=WEBP actually means "get me small thumbnail."
• E2E tests: from what I see, current tests do not check custom configuration, say, in asset.e2e-spec.ts we do have tests variable where we list files in various formats, and then we use it for test should upload and generate a thumbnail for ${input}, but I do not see how do I can specify here a custom configuration and how do we currently test "jpeg vs webp" thumbnails (looks like, we don't test it?).

Manual testing

As ML is affected, I checked Smart Search and Facial Recognition. I cannot check Smart Search quality, as it currently looks like it doesn't work correctly in any case — search by "cat" on demo immich instance, and the result will be not only cats, but also fish, goat, etc. :)

[benmccann]

I wonder if it'd be better to target avif than jpeg xl. Avif is pretty widely supported by browsers, I believe it's supported by Lightroom, and is generally smaller than webp. I think Immich should be moving towards avif by default. The main issue this far has been a lack of support in flutter, for the mobile app, but it looks like there's now a library available for that: flutter/flutter#61229 (comment)

[develar]

I wonder if it'd be better to target avif than jpeg xl

Bit depth — 12 vs 32 bit. Yes, 12 is quite enough. One consideration here — Sony ARW 14-bit, though hdr rec 2020 uses 10 or 12 bits in any case (it is a default color space for export).

Well, I stumbled upon the same question when I started to evaluate Immich and was forced to export images.

https://tonisagrista.com/blog/2023/jpegxl-vs-avif/

JXL offers lossless recompression of JPEG images. This is important for compatibility, as you can re-encode JPEG images into JXL for a 30% reduction in file size for free. AVIF has no such feature.

In my photo collection, there are images in JPEG. Encoding it into JPEG XL may potentially be performed with less chance of quality degradation.

Also, there's no way to do full progressive in AVIF.

Particularly compared to the AVIF file format, an open-source format that is ideal for storing photos or animated image sequences, the JPEG XL codec has a faster encoding speed. (The format can run at an encoding speed of 50 MP/s and a decoding speed of 132 MP/s.)

Encoding speed is promised to be faster than AVIF. Well, I do not have numbers for encoding speed, as just in five minutes, I stopped an export task in AVIF format and decided that JPEG XL is a superior modern format :)

[develar]

I checked AVIF vs JPEG XL in Lightroom —

(7008x4672, 50MB Lossless Compressed RAW)

image 1 (forest and river): 26.1 MB vs 8.6MB. encoding: 5s vs 1s
image 2 (sky, river, bridge): 12.4 MB vs 2.9MB. encoding: 4s vs 1s.

Therefore, for me, AVIF is not acceptable. Since Lightroom is not open-source, I can't explain the significant difference, but it might be related to how Adobe prioritizes formats, also indicated by the fact that JXL is listed before AVIF in the combo box.

[alextran1502]

Before we put in more work on this PR, I just want to let you know that we will not support formats that are not widely supported by most browsers because it will create edge cases in supporting the community.

[develar]

Galaxy 24 uses JPEG XL as Expert RAW.

Additionally, storage capacity has been reduced while maintaining image quality by providing JPEG XL format.

Samsung Joins Apple and Adobe in Supporting JPEG XL

https://r2.community.samsung.com/t5/CamCyclopedia/JPEG-XL-Image-Codec/ba-p/15356525

[mertalev]

I have to agree that AVIF makes more sense right now. It's supported by major browsers and there's a convenient package for it in Flutter.

Granted, it's not like every video codec we have is supported by every client either. But this being just for Safari right now makes it a poor fit for something that's meant to enable sharing with others.

For the size comparison you made, I think it should be noted that the results will depend heavily on the settings chosen. There may be cases where JPEG XL is better (or it might even be better in general), but I really doubt the difference is that big.

[develar]

Okay. I can always have a private fork, but I want to make it as small as possible.

Pillow does not support AVIF, as well. pillow-avif-plugin can be used as a solution (same as for JPEG XL).
We still have to build libavif from sources, as libjxl — Debian 12 has ancient 0.11 version.

Hmm... but in this light, it needs to be clarified what benefits AVIF brings compared to WebP. I have to check the overall size difference of thumbnails / preview.

[mertalev]

AVIF should on average be about 20% smaller than WebP at the same quality. It also has the advantage of supporting higher bit depths and HDR where WebP is stuck at 8-bit output.

[develar]

AVIF is not supported correctly by Safari — https://www.reddit.com/r/AV1/comments/186eb8w/hdr_avif_files_look_very_different_in_chrome_and/

Safari renders JPEG XL correctly — exactly as Chrome renders AVIF. But the same photo in AVIF is not rendered correctly by Safari.

In this light, I will close this PR for now.

top-left: safari jxl
bottom-left: chrome avif

right: safari avif (darker image, take a look on the roof)

[benmccann]

I have to imagine that Safari's support for HDR avif images will improve over time. Given that they still display, even if not optimally, it might be worth pursuing still. The real comparison should be done to webp, which doesn't support HDR at all and is what users will see today. Meanwhile, avif should hopefully improve display in the mobile app and prototyping it will let us flush out all the edge cases and let us know if there's any additional support needed in any upstream libraries that we need to request.

[develar]

The real comparison should be done to webp

I checked — WebP renders correctly; I don't see much difference (I'm using Apple Studio Display — P3 wide color gamut, 10-bit depth).

That's why I closed the PR — if we don't want to add JPEG XL because it can lead to confusion for users (although Immich is a self-hosted solution and the target audience is known in advance), then adding AVIF also leads to high maintenance as the rendering is not the same (it's even worse, because not working at all is better than displaying a wrong photo).

will improve over time

So, wait and see — or JPEG XL will be supported by Google Chrome, or AVIF support will be fixed in Safari :)