Building WebGL libraries that work with each other in the same webgl context.

[trusktr]

Description of the problem

It's difficult and complex these days to mix WebGL libraries together, to draw things using different libraries in the same webgl context. (examples of difficulties: mrdoob/three.js#8147, pixijs/pixijs#3230, pixijs/pixijs#3345, pixijs/pixijs#1366, pixijs/pixijs#298, jonobr1/two.js#233)

For example, suppose we would like to render Three, Babylon, and Pixi objects into the same WebGL context, in the same 3D space.

It is currently very difficult to do this because each WebGL library manages state of the context in their own ways, and these private internals often change and break solutions that people come up with because there's no standard way to do it.

Pixi.js v4 goes through efforts to make Pixi compatible with Three.js, so that it can render in a Three scene, but this is obviously fragile.

Solution

Enter Regl to the party.

Maybe if the foundations for each library (Three, Babylon, Pixi, Two, etc) were built on Regl, we'd have a common way of rendering to a single context.

Regl makes an abstraction just on top of raw WebGL for managing WebGL state. It doesn't render for you all the things that Three.js can, it only provides the minimal foundation for working with raw WebGL in a stateful way that is easy to manage.

It seems that libraries like Three, Babylon, PlayCanvas, Pixi, Two, etc, could benefit from using a standardized way for managing WebGL state, which would make it easy to combine renderings from any of these libraries into the same WebGL context.

---

What are your thoughts on refactoring the foundation of Luma.gl to use Regl for managing WebGL state? Does Regl offer enough flexibility for Luma.gl to do what it needs to do on top of Regl?

[ibgreen]

@trusktr Thanks for engaging! Cross-WebGL framework interoperability would certainly be a holy grail, and we'd be interested in participating in a "cross framework maintainers" type of discussion around this. It is a big topic, with lots of technical details and opinions, so I'll just start off responding to some of your ideas.

For example, suppose we would like to render Three, Babylon, and Pixi objects into the same WebGL context, in the same 3D space.

• We have considered making luma.gl and deck.gl compatible with THREE.js many times (mainly to increase the appeal of our frameworks to people already invested in THREE.js).
• In fact, luma.gl was designed from the start so that it can accept contexts created by other frameworks, and through its WebGL state management it has "zero" impact on that context's WebGL state (see below).
• But the concern is always that even though we have already solved those basic problems, there will still be a long tail of other issues (synchronizing cameras, different shader stacks etc etc) that will cause complications and support and maintenance efforts that go beyond what we can justify. But maybe we are overly pessimistic here.

What are your thoughts on refactoring the foundation of Luma.gl to use Regl for managing WebGL state?

It would not add anything new for us:

• luma.gl already includes a comprehensive system for management of global WebGL state. It intercepts all state-related WebGLRenderingContext methods on the prototype level and tracks all changes to global state and has the capability to snapshot and reset the context to known states.
• In fact this state management system is completely separated from the rest of the code in the src/webgl-utils folder and we have even offered to help another open source WebGL project adopt it (TBD).

Does Regl offer enough flexibility for Luma.gl to do what it needs to do on top of Regl?

It is not a direction we are pursuing. I can give some pointers to how we think:

• luma.gl is designed to be a set of "thin" class wrappers on top of the WebGL API, on top of which higher level abstractions can be built.
• regl combines similar basic WebGL wrappers with a higher level functional framework. We build our functional WebGL framework (deck.gl) on top of luma.gl. We like the clean separation and would only use half of regl.
• Our main reason for maintaining luma.gl is to enable us to focus on efficiently writing shaders, help us move quickly into WebGL2, GPGPU compute and high performance rendering using the very latest extensions. Controlling the code base and being able to move quickly with it is something we value.
• As mentioned above, I am not convinced that even if all the frameworks you mention were refactored to use regl as a base, it would solve enough compatibility problems to make interoperability practical as many concepts are controlled on higher level in the frameworks.

Again, as I said initially, happy to engage in a bigger discussion, especially if it is multilateral.

Finally, a quick question, is regl even alive anymore? When I checked, the last commit was from April 21st...