Creating HUD over 3d Scene (WebGLRenderer) ?

[remoe]

Hi

does anyone know what's the correct way to create a HUD layer (2d sprites, UI) over a 3d scene with THREE.js ? I want emulate a mouse-cursor and render some images over scene. My idea is creating an ortho-camera and using of sprites. but i don't know how i can layer this renderings over the main rendering. On the examples i found some "postprocessing", but i don't think this is the way to go.

Thanks
Remo

[jbaicoianu]

Use HTML

[erisdev]

+1 for HTML. You're developing for a platform designed for UI; make use of it. :)

[Loupax]

Actually I don't see why you SHOULDN'T use HTML! +1

[mrdoob]

Yep, unless it needs to be integrated in the 3d scene I would go with HTML on top of the canvas.

[remoe]

Thanks.
mrdoob, yes i want to integrate in the 3d scene. I want to use WebGL for some 2d graphics. But i didn't see a working sample to do this with THREE.js.

[Loupax]

When you say "intergrate in the 3D scene" you mean you don't want the UI as an overlay on top of the 3D, righ?

[remoe]

Loupax, I've changed the title a littlebit :) I mean on top but with WebGL rendered. i mean something like that:

http://www.smokycogs.com/blog/o3d-tutorial-4-the-beginnings-of-a-hud/
http://code.google.com/p/o3d/source/browse/trunk/samples_webgl/o3d-webgl-samples/hud-2d-overlay.html?spec=svn218&r=218

[fabricasapiens]

I would simply use HTML for the GUI, so simply < div>'s etc on top of the < canvas> element...

If you want the GUI to be rendered with Three.js on top of your actual scene, I would use two canvas elements on top of each other, one for the game/scene/whatever, and the top one for the gui. You would need to ensure that the top one is transparent between the GUI elements, but I don't know if that is possible with Three.js...

[alteredq]

Yes, for the most cases HTML is the way to go, it's much simpler.

If you do want to get fancy and have your HUD rendered by WebGL (e.g. for animated 3D or some shader effects), you could do something like this:

http://mrdoob.github.com/three.js/examples/webgl_rtt.html

(there is probably some way how to do it without FBO just playing with depth writing / reading / masking, at least for 2d overlays)

[remoe]

alteredq, thank you for this link. I currently haven't found a way with "playing". But i analyse the code on WebGLRenderer and found out that i can use something like that:

var sprite = new THREE.Sprite( { map: tex, useScreenCoordinates: true } );
scene.addChild(sprite);

This works.

But i'm still interested about the way with dual rendering and blending this.

[jaamo]

I was working on a WebVR project and DOM solution wasn't an option. I ended up doing the HUD with pure Three.js. I created a separate scene and rendered that on top of my original scene.

You can see the live example here:
http://codepen.io/jaamo/pen/MaOGZV

And blog post about the solution here:
http://www.evermade.fi/pure-three-js-hud/

[trinketmage]

I am actually working on a HUD tool for webVR.
It's based on the StereoEffect.js that you can find in three.js examples.
It renders the two part of the first scene, then the two part of the HUD scene on overlay all with StereoEffectCamera that are PerspectiveCamera.

There is a demo with a crosshair and interaction with some elements in the scene.
https://github.com/trinketmage/three.js-vr-hud/

I am currently working on a "label" feature within the frustum of the HUD scene, i would appreciate any tips or any help.
Thanks

[cecilemuller]

If someone is still looking for how to render an Ortho scene on top of a perspective one (so you can align objects to top/left/right/bottom borders or have sizes in pixels), I had an example there.

[stefek99]

@cecilemuller - your project does not allow me to open issues so I'm commenting here. Does it work? http://jsbin.com/sisutopede/edit?html,js,output (I replaced let const with var) I can only see a blue sphere, I don't seem to see any UI elements?

@trinketmage - thank you for creating fantastic demo / plugin and taking time to explain issue 1, right now I'm trying to figure issue 2: trinketmage/three.js-vr-hud#2 - it works flawlessly when I'm using keyboard, the moment I start using my mobile device / Chrome Dev Tools and simulate orientation it is jumping...

• video: https://www.youtube.com/watch?v=fO_yJ1bcSCM (23s)
• live: https://stefek99.github.io/three.js-vr-hud/example/

---

On a sidenote - enough of copy & paste - at some point I should be able to write my own code given all the knowledge in this thread...

[cecilemuller]

@stefek99 It should be showing colored rectangles on top of the blue sphere; just tried the jsbin in chrome and firefox, here's a screenshot:

Although yes, there are some es6 features in the code, ,so you can use the Babel REPL to convert to es5 if it's not running.

[sylwesterdigital]

It's 2020, what is the best way to create 2D HUD overlay inside ThreeJS without using hybrid or weird methods? Ideally to have access to something like PixiJS inside ThreeJS?

[WestLangley]

@sylwesterdigital You can use the three.js forum if you need help.

[trinketmage]

After years of mooing around, I guess that there is no absolute ways to do hud in WebGL, it totally depends on project, but my thought are:

• If you don't need specific effects on HUD, go for HTML, there is a "compositing cost" but as long as you don't have too much overlay (if i use 4 times/ 5 times sizes of screen overlays it get laggy on Chrome with Mac Book Pro Retina mid 2015), example of a HUD in html : demo waysmaze.
• if you need it to be in the gl for whatever reason, I would put it directly a Group that each frame would be place in front of the camera rendered on top of everything; Calculate the extents of the group from the camera at a specific z distance on resize to place the item on top/right/bottom/left; This ways give flexibility to make use of the perspective (if you need 3D elements in HUD) or not and i guess it should be the lowest cost for performance (please correct me if i'm wrong).
• The more "logical" way @sylwesterdigital, in my opinion, would be to do a second render (wether Orthographic or Perspective) on top of the scene as a texture inside the same scene; But the huge downside is that it is the less performance wise way cause a second renderer of size of screen multiplied by the pixel ratio, or, if HUD renderer is smaller you lose quality.
• Pixi.js on top of Three.js is a bad solution @sylwesterdigital, cause I guess you would have 2 WebGL instances and I guess it is heavy ?
• Good old canvas on top of three.js should work also, one of my amazing coworkers @Namide taught me a huge developer lesson: "how would they optimize it for videogame ?", in Unity, common ways to handle HUD is to CanvasRenderer on top of everything.

So in conclusion, in my opinion, the HUD visual components should depend on what cost of performance you can afford.

[wilson0x4d]

Seems like using a custom shader that remapped depth would be an option for platforms that don't have a DOM, like React Native. Same shader could also be made to perform an orthographic projection, so you don't need to create a secondary camera nor GL context, but you would need to, uhh, understand the maths.

That said.. it's 2021 and the truth is we have platforms which are "HTML-like" but do not allow for a DOM-based compositing solution as many assumed should be possible. Some platforms do not offer a Canvas element at all and the GL Context comes from something else entirely, so doing an overlay effect is a non-trivial effort. We experience similar struggles with examples that depend on DOM elements and/or mixed levels of evergreen technologies (like Blobs & Buffers.) React Native is an excellent example of this. Luckily, three.js is open source and anything that doesn't port well can be rewritten to work correctly cross-platform/cross-toolkit, and generally the offenses tend to only come from /examples/ in my experience ... still, a small part of me wishes WebGL-proper allowed a gl context to be created against a UInt8Array, or Buffer, or Blob, or similar.. (ie. as a Kronos-dictated specification for implementers), because then we could feed the thing into a fullscreen quad shader (wrapped as a DataTexture, or equivalent), which could radically change conversations like this one. Unfortunately, the specification assumes there is a Canvas element backing every GL context -- which turns out to be not true, and debatably unnecessary (internally GL really just needs a contiguous block of memory to serve as a framebuffer, after all, one could argue that presentation of a buffer is all that a Canvas provides wrt WebGL.)

Anyway just offering up one more idea to the next person stumbling upon this thread, this was the first result for me on Google. Good Luck! 😃