Platforms: VR/XR issues (reversed rendering in URP etc.)

[Aupuma]

Hi, congrats for the effort on the project!
I've been testing it with a VR headset, and I was able to see splats using OpenXR in Multi-Pass mode (Single pass instanced does not work). However, I tried to switch to URP to get some performance benefits but the rendering is reversed, the Left image renders on the right eye and viceversa. Any ideas on how to fix it?

[aras-p]

Since I know nothing about either URP nor VR, and I don't have any VR devices around, then no idea. If someone figures it out, let me know!

[andybak]

I might take a look at this at some stage as I am particularly interested in Gaussian Splats for VR (seems like the perfect use-case)

In the meantime there is also: https://github.com/clarte53/GaussianSplattingVRViewerUnity

[unitycoder]

Those debug modes seem to render betterr in URP+VR(multipass), compared to splats.

so i'm guessing its something to do with RenderGaussianSplats.shader,
float4 centerClipPos = view.pos; // what is view.pos about?

[gaetanthiesson]

Seems like there is an issue with the camera projection matrix on URP with stereo rendering not exactly sure but seems like the matrix return at this line is wrong or doesn't take into account both eyes.

Managed to fix it but using Unity built-in transformations matrices but the result is not exactly the same, looks like the splat scale is slightly smaller but at least it fixed the rendering issue. Here is how SplatUtilities.compute looks like for me now.

Added include for Unity built-in stuff:
#include "UnityCG.cginc"

And changed those 2 lines:
line 81 float4 centerClipPos = mul(UNITY_MATRIX_VP, float4(centerWorldPos, 1));
and 95 float3 cov2d = CalcCovariance2D(splat.pos, cov3d0, cov3d1, _MatrixMV, UNITY_MATRIX_P, _VecScreenParams);

[changruizhu96]

Hi, congrats for the effort on the project! I've been testing it with a VR headset, and I was able to see splats using OpenXR in Multi-Pass mode (Single pass instanced does not work). However, I tried to switch to URP to get some performance benefits but the rendering is reversed, the Left image renders on the right eye and viceversa. Any ideas on how to fix it?

Hi, could you please share some hints about how to test it with openXR?

[Mr-Anderson]

I have been chasing what I think could be a related issue with #52. In my case, I am trying to get rendering working on a LookingGlass light field display. The issue I believe I found is that if the render target is set to something custom for a camera the render target will be changed to BuiltinRenderTextureType.CameraTarget by GaussianSplatRenderer ignoring the custom target.

Would someone who has this issue be willing to test after commenting out line 190 in GaussiannSplatRendere.cs. This will cause SortAndRenderSplats to render directly to the original render target. This hack skips the compositing step of rendering though so the image will be washed out.
https://github.com/aras-p/UnityGaussianSplatting/blob/fded9c09d041e862502acc57d1bbc4c084ae4e6f/package/Runtime/GaussianSplatRenderer.cs#L190C31-L190C31

[ChiefBreakeverything]

It's a real long shot but could this theoretically work on stand-alone VR platforms like Quest? It works amazing in the Editor, but several parts of the Compute Shaders fall over when you change the platform to Android.

I wonder if they're minor changes? I had a look at the source but it's black magic to me! It would be life-changing to have splats in stand-alone VR.

[ptc-lexvandersluijs]

For those looking for a VR implementation with the Built-In Render Pipeline (BIRP), I have just put the code for my project online here: https://github.com/ptc-lexvandersluijs/Unity3DGS_VR . This is made with multi-pass rendering, Unity 2023.1.14f and tested with OpenXR / HTC Vive.

[bmegli]

The hack to make URP VR left eye left, right eye right mentioned in:

• Platforms: VR/XR issues (reversed rendering in URP etc.) #17 (comment)
  • thank you @gaetanthiesson for original hack

With current 08a270a codebase is more like this:

• around line 197
• around line 233
• and include at the beginning

diff --git a/package/Shaders/SplatUtilities.compute b/package/Shaders/SplatUtilities.compute
index a23a282..718d63d 100644
--- a/package/Shaders/SplatUtilities.compute
+++ b/package/Shaders/SplatUtilities.compute
@@ -30,6 +30,7 @@

 #include "DeviceRadixSort.hlsl"
 #include "GaussianSplatting.hlsl"
+#include "UnityCG.cginc"

 float4x4 _MatrixObjectToWorld;
 float4x4 _MatrixWorldToObject;
@@ -193,7 +194,8 @@ void CSCalcViewData (uint3 id : SV_DispatchThreadID)
     SplatViewData view = (SplatViewData)0;

     float3 centerWorldPos = mul(_MatrixObjectToWorld, float4(splat.pos,1)).xyz;
-    float4 centerClipPos = mul(_MatrixVP, float4(centerWorldPos, 1));
+    //float4 centerClipPos = mul(_MatrixVP, float4(centerWorldPos, 1));
+    float4 centerClipPos = mul(UNITY_MATRIX_VP, float4(centerWorldPos, 1));
     half opacityScale = _SplatOpacityScale;
     float splatScale = _SplatScale;

@@ -229,7 +231,8 @@ void CSCalcViewData (uint3 id : SV_DispatchThreadID)
         float splatScale2 = splatScale * splatScale;
         cov3d0 *= splatScale2;
         cov3d1 *= splatScale2;
-        float3 cov2d = CalcCovariance2D(splat.pos, cov3d0, cov3d1, _MatrixMV, _MatrixP, _VecScreenParams);
+        float3 cov2d = CalcCovariance2D(splat.pos, cov3d0, cov3d1, _MatrixMV, UNITY_MATRIX_P, _VecScreenParams);
+        //float3 cov2d = CalcCovariance2D(splat.pos, cov3d0, cov3d1, _MatrixMV, _MatrixP, _VecScreenParams);

         DecomposeCovariance(cov2d, view.axis1, view.axis2);

---

Another thing that seems necessary is setting Unity editor game view resolution to match VR single eye resolution.

---

Reversing eyes is new funny way to get headache.