DDS parsing and exporting

[heinezen]

Required skills: Python, Cython

Difficulty: Medium

DDS files are used to store terrain textures in the Definitive Editions of AoE1 and AoE2. We will need to convert them to PNG so that they can be used by openage. pillow supports DDS, but their parser is written in Python. For maximum speedup, we should implement A parser written in Cython with Python glue code to attach it to the converter.

Renderdoc has a C++ implementation of a DDS parser that we could use as an inspiration

Further Reading

• https://web.archive.org/web/20170802060935/http://oss.sgi.com/projects/ogl-sample/registry/EXT/texture_compression_s3tc.txt

[sandsmark]

I guess you don't need a fully fledged DDS converter, iirc. they use one of the simpler encodings.

[TobiasFP]

Another source of inspiration could be:
https://code.google.com/archive/p/gimp-dds/source#

[TobiasFP]

@heinezen Do you know if there is any reason we cannot simply use wand?
Do we want to avoid more dependencies?

I tested wand with this small script, using a file from DEII:
prerequisities: pip3 install wand
It seams to even use cython in the backbone of it, so it should be performant enough.

from wand import image

def main():
    with image.Image(filename='input/002_50730.DDS') as img:
        with img.convert('png') as converted:
            converted.save(filename='output/002_50730.png')
if __name__ == "__main__":
    # execute only if run as a script
    main()

[heinezen]

@TobiasFP It require imagemagick, does it? If so, I'd rather not use wand because it introduces a pretty heavy dependency for something that is essentially only required when the converter runs (which is ideally only used once during the setup phase). We would also have to bundle it for the Windows builds.

So good idea because it would simplify it for us, but imagemagick has to much overhead imo :D

[TobiasFP]

Yeah, that was what i thought too.
Pillow also support dds, which is written in python, but i don't think it is very optimised/not using cython, so it might be too slow.
But would that be less of a problem since it is all python?

If not, the dds importer would be of tremendous help writing the parser (and listed under a cc0 license).

https://github.com/python-pillow/Pillow

[heinezen]

I see that pillow extended their DDS support which is nice :) Although it is probably still slow.

I've added a link to pillow's code to the initial post. We could use their implementation, but then we miss out on PNG optimization and speed. I would still prefer a fast Cython implementation.

[TobiasFP]

Do we have an estimate when the dds2png library is necessary to push on?
If we need it quite soon, i suggest we implement pillow for now (Since it would take so little time), and get back to the proper dds2png library when it becomes important improve speed.

If you think this is the wrong way to go, I will gladly begin on the cython implementation, but it will take quite a while to finish, as I do not have much spare time.

I see that pillow extended their DDS support which is nice :) Although it is probably still slow.

I've added a link to pillow's code to the initial post. We could use their implementation, but then we miss out on PNG optimization and speed. I would still prefer a fast Cython implementation.

[heinezen]

@TobiasFP It is not strictly necessary, but using pillow is better than no support. We can implement this issue using pillow and leave it open until the Cython implementation is ready :)