Generate optimized code for shaders using a new language

[Dawoodoz]

To both allow previewing models with third party shaders in the model editor and get high performance in the final release of an application, a portable shader language is needed for vertex and texture shaders. The generated code should be so readable that one can modify it by hand in C++, but the option of generating more optimized instrinsic functions for C/C++ as well would make the language reusable outside of the framework.

Texel shaders would process regions of an image, similar to the Halide language, but with built-in functions for sampling light. This would be like a subset of the media machine, but optimized for performance by allowing floating-point operations in virtualization for compound instructions and generated code for full speed. Shading to texture is the technique used on GPUs for subsurface scattering to make skin look more realistic by blurring the shadows after calculating colored glow from the light 's direction.

Pixel shaders could be referred to using names in the materials and then fetch a pre-compiled pixel shader from the graphics engine. A collection of generic built-in pixel shaders can also be added to the core renderer, to extend what can be done with interpreted vertex shaders.

Vertex shaders should work pretty much like on a GPU, but with certain limitations in control flow to make it faster on a CPU.

Interpreted mode for quick prototyping in the 3D editor:

• Interpreted with large planar buffers for inteemediate values.
• Intrinsic compound functions for common combinations of virtual instructions can be applied because no clamping is performed implicitly when floating-point operations are used.
• Work with planar data formats to get zero overhead when mixing channels from different attributes.

Compiled mode:

• Calling SIMD.h from automatically generated code.
• Work with packed data formats and process one SIMD vector at a time.
• The generated C++ code can be saved with the other code, so that one does not depend on getting the shader compiler to work, just the SIMD.h hardware abstraction layer. This makes sure that basic maintenance does not require knowledge about compilers or assembler.

Calling the shader generation should be easy with both external build systems and the library's own build system.

The language syntax should abstract away both vector length and how the data is stored, so that it does not matter if one uses a planar or packed vertex structure.

Conditional if statements should not be allowed, because that would not be data parallel for vectorization. Masking operations should be used instead.

Models
For best integration with the Model API, the old format will become the default vertex structure and a new type of pointer similar to SafePointer will contain a padded power-of-two element stride, so that it can access packed, planar and semi-planar data by changing the element stride in the pointer and automatically get the correct element from the [] operand by pre-multiplying the element size with the stride and getting the base two logarithm for bit shifting. If requesting a part's vertex color from a model, you get a pointer to the first FVector4D with a stride to the next element, which will be the same for all vertices in the same packing. One packing is for the final render, so that you don't need to read the other attributes used for generating light. Rarely used data is packed together further back in the allocation. Separate vertex buffers could potentially be used for different types of animation. In the 3D model editor, one can use the same type of pointer for accessing planar buffers from separate allocations, which are currently used to save memory by only cloning the attributes that changed for planar immutable undo history.

[Dawoodoz]

To keep it simple, one could start by assigning vertex, pixel and texel shaders using thread-safe lambda functions. It would however still benefit from having customizable vertex structures, because they can easily be loaded from the model. Then the 3D model editor can get released with good performance before having a shader compiler, by allowing a difference in visual appearance. Just need a few pre-compiled shaders in a generic graphics engine. The code for these lambda functions can then later be transpiled from a shader language and bundled into the specialized graphics engines, without any limitations from scripting.

Vertex shaders would need more than one function to get a good integration. Before transforming individual vertices, transform the bounding shape and let culling and occlusion decide if vertices should be transformed for occlusion shapes and visible geometry.