Streaming shader preprocessor cache

[Fletterio]

Description

The current cache holds all resources on memory at all times. This could be problematic if, for example, the cache is required to hold many shaders

A streaming shader preprocessor cache is proposed: stored as an LZMA CArchive that would hold:

• An LZMA-compressed header IFile that holds the JSON describing each of the Cache's container entries
• One LZMA-compressed virtual IFile per entry described above, holding that entry's shader code

And then load shaders on-demand when compilation is requested

Solution proposal

I think the best would be to load the whole JSON at cache creation time, and then load each shader's byte code onto memory on-demand (so only when compilation for that shader is requested and not earlier)

Additional context

Since each entry contains ALL of the plaintext for ALL its dependencies, that could also start to pile up, so an option is to also have each entry's dependencies stored in an LZMA-compressed file. We would then only load an entry's dependencies onto memory when those need to be compared when compilation is requested. This could additionally come with a Cache method to drop all loaded dependencies from memory if the cache starts taking up too much space in memory (not counting shader code). Or if it's fast enough we could simply make it more "pure-streaming": load dependencies, compare, immediately drop

As it stands right now, shared dependencies are repeated once for each shader. That could also be problematic, so an optimization would be to handle dependencies in another manner so they're unique and shared by entries. This also holds for the current Cache implementation

[devshgraphicsprogramming]

We need to dedup the contents of files, and keep the dependencies LZMA compressed before we approach on-demand streaming from a file