Add access to WebXR depth texture + add example

[cabanier]

Some authors want to use the depth texture to add an effect using a custom shader.
This diff enhances the WebXR code to allow access to the texture and add a sample.

Thanks to @hybridherbst !

This contribution is funded by Meta

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

Filesize dev	Filesize PR	Diff
676.2 kB (167.9 kB)	676.3 kB (167.9 kB)	+98 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

Filesize dev	Filesize PR	Diff
456.8 kB (110.7 kB)	456.9 kB (110.8 kB)	+98 B

[cabanier]

@Mugen87 is there a way to change an existing shader? I'd like to reuse MeshLambertMaterial by adding my helper functions and inserting the calculation of the edge pixels.

[gkjohnson]

@Mugen87 is there a way to change an existing shader? I'd like to reuse MeshLambertMaterial by adding my helper functions and inserting the calculation of the edge pixels.

You can use onBeforeCompile and modify the existing shader with find and replace:

material.onBeforeCompile = shader => {
  
  shader.uniforms.depthTex = { value: /* texture here */ };
  shader.fragmentShader = shader.fragmentShader
    .replace( /#include<color_fragment>/g, match => /* glsl */`
      
      // your code here
      ${ match }

    ` );

};

This should also address #27664

[hybridherbst]

If you want it globally included you can also replace things in ShaderChunks.someSpecificChunk, then it doesn't need to be modified in every single material.
What I'm not sure in that second approach is what the right way is to set some uniforms either for all objects or per object; @gkjohnson do you have a recommendation there?

[gkjohnson]

If you want it globally included you can also replace things in ShaderChunks.someSpecificChunk, then it doesn't need to be modified in every single material.

Generally I'd discourage modifying the built in shaders in the examples. It works but it's brittle. And of course having multiple libraries or components modifying built in materials can easily lead to overwritten changes and breakage.

What I'm not sure in that second approach is what the right way is to set some uniforms either for all objects or per object;

Global uniforms aren't supported so you'll have to set it per material. UBOs are now in core but the current implementation doesn't make this case much more ergonomic. See #16922.

Encapsulating the above in a class can make this easier to manage at an application level but I don't think it's necessary for the example:

class CustomMaterial extends MeshStandardMaterial {
  set depthTex( v ) { 
    this._uniforms.depthTex.value = v;
  }
  constructor( ...args ) {
    super( ...args );
    this.onBeforeCompile = shader => {
      // ...
      this._uniforms = shader.uniforms;
    };
  }

}

[hybridherbst]

having multiple libraries or components modifying built in materials

The same is true for Material.onBeforeCompile which isn't an event listener but a single assigned method, or am I understanding that wrong?

I'm just not sure how someone could take that sample and extend it; a common flow would be "this is a cool sample, now I want to load a glTF and keep the materials in that and apply that fancy occlusion." – how would someone do that? Need to be super careful then to not "miss" any material to modify it, can't use other onBeforeCompile callbacks, etc.

[gkjohnson]

The same is true for Material.onBeforeCompile which isn't an event listener but a single assigned method, or am I understanding that wrong?

You can design for this case into your app or design your wrapper class to take an onBeforeCompile function and call it if it's set (ie adding a getter / setter for onBeforeCompile to your class)

I'm just not sure how someone could take that sample and extend it; a common flow would be "this is a cool sample, now I want to load a glTF and keep the materials in that and apply that fancy occlusion."

This is always the case if you want to use a different material on a loaded model. My understanding is that onBeforeCompile is the recommended way to modify built in materials but don't get me wrong - I think it's more difficult than it should be right now (see my issues on override materials, global uniforms, etc). I'd like to see improvements but I think discussions around how to adjust the ergonomics should happen in a separate thread and not block this PR.

[cabanier]

@Mugen87 it doesn't look like I have access to the internal shaders to extend them. What do you think of this change?
Alternatively, I can just make the changes to the webxr manager class and leave it up to authors to use the depth sensing texture.

[Mugen87]

I would leave the PR in its current state then 👍 .

[odysseyjason]

Is it possible to query the depth texture per frame from the render-loop, to for instance bounce objects off of walls sensed through the depth.api? (the webxr cubes example currently has the cubes getting occluded behind walls, would it be possible to query the depth data to see if they should bounce off of them instead?)

I have looked at the code from the webxr cubes example and WebXRDepthSensing.js and WebXRManager.js but could not find where it was being defined if this depth data was GPU optimized texture or CPU optimized or understand where one could interject real time interaction possibilities with the depth data.

Any example that could show interaction with the Depth api texture data would be most helpful! Also, a CPU optimized depth texture example would be great!

[cabanier]

Any example that could show interaction with the Depth api texture data would be most helpful! Also, a CPU optimized depth texture example would be great!

If you look at this PR, the getDepthTexture call returns you the depth texture. You can use it in your shader just like a regular WebGL texture.