Building WebGL libraries that work with each other in the same webgl context.

[trusktr]

Description of the problem

It's difficult and complex these days to mix WebGL libraries together, to draw things using different libraries in the same webgl context. (examples of difficulties: #8147, pixijs/pixijs#3230, pixijs/pixijs#3345, pixijs/pixijs#1366, pixijs/pixijs#298, jonobr1/two.js#233)

For example, suppose we would like to render Three, Babylon, and Pixi objects into the same WebGL context, in the same 3D space.

It is currently very difficult to do this because each WebGL library manages state of the context in their own ways, and these private internals often change and break solutions that people come up with because there's no standard way to do it.

Pixi.js v4 goes through efforts to make Pixi compatible with Three.js, so that it can render in a Three scene, but this is obviously fragile.

Solution

Enter Regl to the party.

Maybe if the foundations for each library (Three, Babylon, Pixi, Two, etc) were built on Regl, we'd have a common way of rendering to a single context.

Regl makes an abstraction just on top of raw WebGL for managing WebGL state. It doesn't render for you all the things that Three.js can, it only provides the minimal foundation for working with raw WebGL in a stateful way that is easy to manage.

It seems that libraries like Three, Babylon, PlayCanvas, Pixi, Two, etc, could benefit from using a standardized way for managing WebGL state, which would make it easy to combine renderings from any of these libraries into the same WebGL context.

---

What are your thoughts on refactoring the foundation of Three.js to use Regl for managing WebGL state? Does Regl offer enough flexibility for Three.js to do what it needs to do on top of Regl?

[mrdoob]

I'm more curios about the use case of mixing Three, Babylon, Pixi and/or Two...

[trusktr]

@mrdoob, because, for example, imagine using pixi-svg to draw WebGL vector graphics, then moving those graphics around in 3D space with something like Object3D. It'd be nice to have a way to do that easily. I suppose currently I can take a canvas texture, and send it into Three.js, but that seems like more overhead. Composing things like this in Regl, in a single webgl context, seems really practical.

[mrdoob]

Wouldn't it be better to use CSS3D for that?

[trusktr]

Not really because

1. CSS3D is super buggy (I can provide plenty of examples in multiple browsers)
2. It is slow (conversion of numbers to strings to numbers when applying matrices via style="transform: matrix3d()" attributes).
3. It is difficult to use, especially now that Chrome adopted transform flattening (I can provide plenty of broken examples that used to work before flattening was introduced, and workarounds can be horrendous)
4. We can not make real time shadows on DOM elements (no shaders in CSS 3D).
5. We can not apply curvature to DOM elements (no shaders in CSS 3D).
6. We can not apply lighting, shininess, or reflections to DOM elements (no shaders in CSS 3D).
7. There's TONs we can't do because we can write shaders for DOM-based stuff.

[mrdoob]

That is true.

[mrdoob]

For that specific case, I think I would port pixi-svg to three.

Basically, getting everyone to refactor their engines to use regl is quite a big task. Maybe it's something that could be done for WebGL2Renderer though, but I would have to study regl first.

[fernandojsg]

Just my 2cts: I really like regl but I'm not sure we want to include such big dependency on three.js when:

• The project is marked as "completed": regl status regl-project/regl#423
• Many people seem to have problems understanding its core (Improve comments in core.js and util/codegen.js regl-project/regl#200) so I believe we can't afford adding an important dependency without really understanding what is going on there and how to change it and evolve it as we need.
• It looks like WebGL2.0 is not making it to regl (Investigate WebGL 2.0 support regl-project/regl#378)

[mrdoob]

Agreed!

However, when this situation changes, then I think that the best way forward would be via a separate fork (call it regl2) that supports only WebGL 2.0.

If they do regl2 in time, we'll consider it 😊

[trusktr]

For that specific case, I think I would port pixi-svg to three.

Porting everything into everything is doable... and time consuming.

Moreso than actually using regl, the main concept is maybe having something like regl, if not actually regl, who's goal is to coordinate with library authors to build composable libraries.

I like regl's concept, but I also see ways it can be used that would entirely defeat this very concept.

It's good food for thought.

[ivanpopelyshev]

Pixi core dev here.

I'm making hybrid renderer on typescript that can handle different stage trees inside each other. But its based on pixi-v5 and not the regl2. And that's possible only because I have deep understanding of both pixi and threejs architecture.

@mrdoob senpai !!!