Ideas for the TurboJPEG 4 API

[dcommander]

Referring to #517, the TurboJPEG 3 API overhaul addressed all of the outstanding TurboJPEG API issues and implemented the most requested features from the user community. For the most common workflows, it shouldn't be necessary to use the libjpeg API anymore. However, I had to scale back my ambitions somewhat due to lack of time and funding. Even with the scaled-back ambitions, it still took me 170 hours to complete the overhaul, of which only about 95 were funded ahead of time. (Fortunately, the community has stepped up to close that funding gap-- and the rest of the funding gap for the libjpeg-turbo 3.0 release-- retroactively.) I wanted to create this issue to record my current thoughts regarding potential future improvements to the TurboJPEG API.

1. Referring to A new industry-standard JPEG compression/decompression API #313, there is some demand for the ability to directly access quantized coefficients. This can sort of already be accomplished using a custom transform filter, but there is room for improvement to that interface. (Suggestions needed.)

2. Extend the transformation function to allow requantization, i.e. changing the JPEG quality without fully decompressing/recompressing the image. This code already exists thanks to the jpegtran drop feature. It would just need to be exposed.

3. Stateful management of both packed-pixel and JPEG buffers. I envision that the API state will have "buffer slots" for:

   • an 8-bit, 12-bit, or 16-bit packed-pixel buffer
   • a YUV buffer (either unified or separate planes)
   • a JPEG buffer

   Then the compress, decompress, encode, decode, and packed-pixel image I/O functions could simply use or set those buffer slots, and it would no longer be necessary to use a particular function for a particular data precision. The packed-pixel image load function would load from a file into the packed-pixel buffer slot (accepting an argument to specify the target data precision or use the data precision of the source file.) The packed-pixel image save function would save from the packed-pixel buffer slot into a file. The compress function would compress from the YUV buffer or the packed-pixel buffer, depending on which was set, and it would set the data precision automatically based on the data precision of the packed-pixel buffer. Referring to TurboJPEG 3 API overhaul #517 (comment), there are some issues to work out vis-a-vis ensuring that buffer pointers don't become orphaned. (Suggestions welcome.) In conjunction with this, it would make sense to manage the pitch/row alignment, row order, pixel format, and compression/encoding/decompression/decoding subregion as part of the packed-pixel and YUV buffer state. It would also probably make sense to provide a way for the C API to optionally allocate destination buffers, as the Java API does. Buffers might need to be wrapped in new buffer objects. It might be nice to also make the awkward tj*Offset[] and tjPixelSize[] arrays stateful, perhaps by adding new API parameters that return a value based on the pixel format of the current packed-pixel buffer. It might make sense to replace the stateless buffer size functions with "set image" functions that are designed to be called twice, once with a NULL argument to return the required buffer size and again with the actual buffer.

4. Referring to A new industry-standard JPEG compression/decompression API #313, it should be possible to expose direct-to-disk compression/decompression, by allowing a JPEG file rather than a JPEG buffer to be specified and deferring the loading and saving of packed-pixel images until the packed-pixel buffer is first needed (i.e. within the body of the compress, decompress, encode, or decode function.) That would potentially benefit memory-constrained applications. However, direct-to-disk compression/decompression is a lot less relevant than it was 30 years ago and thus is probably not a high priority. (These days, disks are mostly just a slower form of memory.)

5. The transformation function could potentially be modified so that it uses TurboJPEG API instances to store API parameters and buffers for the source image and each transformed image. New API parameters could hold the transform operation and options. TJXOPT_PROGRESSIVE, TJXOPT_OPTIMIZE, and TJXOPT_ARITHMETIC could be replaced with TJPARAM_PROGRESSIVE, TJPARAM_OPTIMIZE, and TJPARAM_ARITHMETIC for a particular transform instance. TJXOPT_GRAY could be replaced with TJPARAM_COLORSPACE=TJCS_GRAY for a particular transform instance. TJXOPT_CROP could be replaced with a cropping region (set with tj3SetCroppingRegion()) for a particular transform instance. Requantization could be specified using the existing TJPARAM_QUALITY parameter for a particular transform instance.

6. Full support for arbitrary and per-component subsampling, using new parameters that either extend or replace the existing TJPARAM_SUBSAMP parameter (perhaps TJPARAM_HSUBSAMP0...TJPARAM_HSUBSAMP10 and TJPARAM_VSUBSAMP0...TJPARAM_VSUBSAMP10.) Ideally we could also make the MCU block size stateful as well, replacing the awkward tjMCUWidth[] and tjMCUHeight() arrays.

7. Support for per-component quantization, using new parameters that either extend or replace the existing TJPARAM_QUALITY parameter (perhaps TJPARAM_QUALITY0...TJPARAM_QUALITY10.)

8. The ability to specify caller-supplied alloc/free functions (would need to also extend the libjpeg API to accommodate this.)

9. Support for any additional libjpeg API features that may be desirable. (If the feature involves simply setting or getting an integer value, then it could be accommodated with a simple extension to the existing TurboJPEG 3 API.)

Open question:
Does it even make sense to spend much more energy on this at the libjpeg-turbo level, or does it make more sense for the open source community to maintain a higher-level API that can handle multiple types of images, including JPEG-1?

Obviously a lot of these are "grand visions", and some of them aren't even within the scope of what I can realistically provide as an independent open source developer. This is partly a note to the next generation.