Investigate ASTC RDO

[solidpixel]

There is some interest in rate distortion optimization to improve compressibility of ASTC textures in application package bundles for download.

Approaches

The first option would be an in-situ RDO, where the output is still valid ASTC data. This is transparent to the application, which is a major advantage, but harder to pull off (especially given the packed variable-length scrambled BISE encoding used by ASTC).

The second option would be a repack RDO, where we transcode into a light-weight packaging format. This is not transparent - the application must unpack at load time - but gives opportunity to systematically target inefficiency in the encoding. One particular strength of this approach is that we can use data tables that are built into the encoder/decoder rather than wasting bits based on dynamic tables built into the compressed stream.

In both cases it can be an advantage to add an RDO mode to the compressor, where it knows that its output will be post-processed by an RDO compressor and so tries to constrain the raw encodings to values that are more amenable to RDO compression.

Inefficiencies we can target

Static inefficiencies are those that are invariant of the texture being compressed. Dynamic inefficiencies are those that may be relevant for some types of data.

• The block mode encoding is statically inefficient for small block sizes (e.g. most decimation modes are invalid for 4x4).
• The block mode encoding is statically inefficient because the encoder will now never encode 2 weight planes for 2 or more partitions.
• The partition hash is statically inefficient, particularly for small block sizes (e.g. duplicate partitions, seeds that don't generate all N partitions, partitions that are not used because they are low value). This is probably an area for compressor restriction in an RDO mode to avoid using low value partitions.
• The CEM modes are dynamically inefficient (e.g. never use HDR in an LDR texture, never use RGBA in a RGB texture).

Other random ideas

• Does it make sense to force "same CEM class" and/or "same CEM" to reduce the need for multiple CEM fields.
• Does using blue-contract reduce compressibility during stream compression or reusability during repacking?
• Does using BISE scrambling reduce compressibility during stream compression or reusability during repacking?

Links to other projects

• Add RDO support to the ASTC encoder. KhronosGroup/KTX-Software#588

[YunHsiao]

Here's one possible way to do it:
4.7.0...YunHsiao:astc-encoder:rdo

Still got some rough edges to be ironed out but feel free to work on top of it.