EXT_accessor_additional_types

[spnda]

Addresses #2216.

This PR adds a new extension, EXT_accessor_additional_types. This extension simply defines more valid accessor types, equivalent to GL_INT, GL_UNSIGNED_INT64_ARB, GL_HALF_FLOAT, and GL_DOUBLE, and doesn't allow their usage for meshes, animations, morph targets, ...

However, this extension does also allow the use of half-precision floating point numbers for UV coordinates, as this is commonly used nowadays for reducing vertex memory cost, and is quite simple to implement. And this avoids the possible quantization errors that would otherwise occur with KHR_mesh_quantization.

Open questions:

• Could this become a KHR extension, similar to KHR_mesh_quantization? Not sure on the procedure there.
• Should the allowance of half-precision floats for TEXCOORD attributes be put into another question, something like KHR_mesh_quantization_2?
• Should this also add a signed 64-bit integer? I couldn't find an OpenGL type for that, which is why I did not add it. For completeness, perhaps it could make sense to also define it. Though this would require adding a enum constant which does not align with OpenGL.

I'd really like to see this properly considered by the glTF team, especially since half floats are quite important in the modern graphics world for various reasons. Also, assets using double-precision positions have existed for a while, and it is already supported by a lot of glTF tooling.

[javagl]

As mentioned in the linked issue: I always thought that the omission of (U)INT64 and DOUBLE was a bit unfortunate (but probably justified by the GL world at this time).
(BTW: There now seems to be GL_INT64 (0x140E) sometimes used as a shortcut for GL_INT64_NV....)

I'm a bit skeptical about HALF_FLOAT. While I see the point about its use in shaders, it's usually not a "built-in" type in most programming languages, and requires special libraries. But this may be a weak counterargument, considering its possible benefits and broader usage in the graphics world.

Also, the question about which categories of accessors may contain which data types may be difficult to sort out, and I wonder whether it's possible to define this, once and for all, right from the beginning, or how to handle the case that some constraints should be relaxed here in the future...

But beyond "It would nice to have this", I don't have strong opinions (or frankly: just not enough background knowledge) about any of these specific points.

---

Could this become a KHR extension, similar to KHR_mesh_quantization? Not sure on the procedure there.

There are some details about these procedures that are still waiting to be described more clearly. But roughly:

• The KHR_ prefix does no longer have a "special meaning", beyond the fact that the extension was originally proposed from within the Khronos working group
• An extension that is proposed from outside, but supposed to be vendor-independent can have the EXT_ prefix
• After a certain (formal) voting procedure, an extension can become "ratified", but this is independent of the prefix

^{The exact meaning of being "ratified" is a bit beyond the scope of this comment (and again: my knowledge). But it may be worth mentioning that this does not mean that the name of the extension would then be changed (from EXT_ to KHR_), because that would be a breaking change. There already are some EXT_ extensions that are ratified, and this one could eventually fall into this category}

[spnda]

The exact meaning of being "ratified" is a bit beyond the scope of this comment (and again: my knowledge). But it may be worth mentioning that this does not mean that the name of the extension would then be changed (from EXT_ to KHR_), because that would be a breaking change. There already are some EXT_ extensions that are ratified, and this one could eventually fall into this category

Ah, interesting. I did not notice that. I thought it'd be the same concept as I know from Vulkan extensions, where ratified extensions get the KHR prefix. Then I guess it can stay as-is.

I'm a bit skeptical about HALF_FLOAT. While I see the point about its use in shaders, it's usually not a "built-in" type in most programming languages, and requires special libraries. But this may be a weak counterargument, considering its possible benefits and broader usage in the graphics world.

But, it is a built-in in nearly every GPU programming language, albeit through an extension or newer language version. GLSL, HLSL, Metal, all have it. A lot of GPUs also have dedicated fp16 ALUs, aside from all of them supporting fp16 storage. Yes, x86 or arm CPUs don't support fp16 natively but we're not rendering on the CPU. So, yeah, very weak counterargument.

[lexaknyazev]

The formats included in this proposal have very different use cases and platform support so the functionality should be split into several smaller extensions along such lines.

Allowing more component types for accessors is not enough to make them usable by vertex attributes as the latter are further restricted by the spec. For example, already-supported unsigned 32-bit integer accessors may be used only for indices.

Now to the format-specific comments.

Double-precision floats

These could safely work in all cases where the spec allows single-precision floats.

Proper support would require having the following extension spec language:

• Allowing the double-precision floating-point enum for the component type property
• Requiring the normalized flag to be false for such accessors
• Allowing such accessors to be used in all cases where single-precision floating-point accessors are supported

This could be one well defined extension. Given that GPU hardware support for 64-bit floats is far from universal, the extension should suggest fallback behavior such as converting data to single-precision on load and warning that such conversion is lossy.

Half-precision floats

Although half-precision floats are well supported in graphics hardware and may provide noticeable computation performance improvements, their benefits for data storage are questionable.

Due to the way floating-point encoding works, the precision distribution is non-uniform and is generally worse than of normalized shorts. Specifically, half-precision floats have only 15361 unique values in the [0.0, 1.0] range (out of which only 1025 are in the [0.5, 1.0] range) compared to 65536 provided by unsigned shorts. On top of that, only values in the [0.0, 0.03125] range are better represented with half-precision floats meaning that the remaining ~97% of the normalized range would be better with unsigned shorts. If the original data is in the [-1.0, 1.0] range, half-precision benefits are limited to the [-0.0625, 0.0625] range compared to signed shorts.

All that said, there are reasonable use cases for half-precision floating-point attributes such as porting pre-existing data from other formats, supplying data for custom attributes, or having a unique model that has been confirmed to yield fewer artifacts when half-precision floats are used, e.g., with non-trivial texture tiling.

Proper support would require having the following extension spec language:

• Allowing the half-precision floating-point enum for the component type property
• Requiring the normalized flag to be false for such accessors
• Allowing such accessors to be used in all cases where single-precision floating-point accessors are supported except animation data (to avoid CPU conversions)
• Thoroughly explaining pros and cons as well as describing the workflow

This could be another well defined extension.

Integers

glTF accessors currently have no distinction between pure and scaled integer attribute types, e.g., an unsigned non-normalized byte attribute accessor provides floating-point values in [0.0, 255.0] range instead of the 0x00 ... 0xFF integers. This is fine for the base spec because scaling 8- and 16-bit integer values to single-precision floats is lossless but will cause issues with 32-bit integers.

Proper support for 32-bit integer attributes would require having the following extension spec language:

• Allowing signed int enum for the component type property
• Requiring the normalized flag to be false for such accessors
• Allowing unsigned 32-bit accessors for JOINTS_ attributes
• Allowing signed and unsigned 32-bit accessors for custom attributes that contain pure integer data

This could be one more well defined extension.

64-bit Integers

GPU hardware support for 64-bit integers is very limited, e.g., not available on Apple and most Android devices, therefore including them would make the extension less adoptable. Besides, use cases for 64-bit per-vertex data seem to be quite exotic (GUIDs? Bitsets?) and could be covered by interleaving two 32-bit integer attributes anyway (or defining a separate extension).

[lilleyse]

+1 for this extension.

For EXT_structural_metadata we would have liked to use accessors to store arbitrary typed data but couldn't due to the lack of INT, INT64, and DOUBLE component types. There's a chance we would update our extension to use the extension(s) proposed here.

[lilleyse]

The extension may also need to specify alignment rules for 64-bit types, e.g. for use with Float64Array

• accessor.byteOffset + bufferView.byteOffset + GLB binary chunk offset would need to be aligned to 8-bytes

[javagl]

Although ^ this is implicitly covered by the wording in the spec that says

The offset of ... MUST be a multiple of the size of the accessor’s component type.

it should definitely be emphasized in the extension specification. I could imagine that some/many/most glTF libraries are taking the "lazy" way and say "always align everything to 4 bytes", and these may have to be updated accordingly to handle this extension.

[wallabyway]

Yes to INT, INT64 and DOUBLE component types !

[lexaknyazev]

Partially moving the proposal forward starting with double-precision floats as a Khronos extension in #2397. Other types are still being considered.

[spnda]

Ah, interesting. I had my last final exam yesterday so I was kinda distracted by studying. I was going to split up this PR into 3 extensions, one float16, one float64, and one for integers, as you had suggested. Though if you're doing that "officially" now that'd also be great. I'm thinking perhaps I could repurpose this PR for the float16 extension first, since that is what I was originally going for myself.

[lexaknyazev]

@spnda
Could you please elaborate on specific examples that showcase half-float storage benefits compared to 16-bit normalized?

[lexaknyazev]

FWIW, half-precision floats seem to have much better support in general-purpose systems than before:

• JS (draft): https://tc39.es/proposal-float16array/
• Java (since JDK 20): Float.float16ToFloat
• .NET (since v5): https://learn.microsoft.com/en-us/dotnet/api/system.half
• C++ (since C++23): std::float16_t, see "Optional extended floating-point types" spec section