Compute depth as the distance along the ray?

[jb-ye]

Currently the depth is calculated as p_view.z, while it makes sense to me that we can possibly calculate it as the distance of p_view from camera origin.

Why does it make difference? when sorting gaussians that has footprints on the same tile, their center actually sit at different rays, using |p_view| rather than p_view.z could make more sense to me: because current approach could cause some unstable order of sorted gaussian, if one perturb the camera orientation by a little bit, the p_view.z can change greatly and their actual order may be affected due to radix quantization; however the |p_view| remain the same, so does their quantized values.

It would be nice to add such feature in gsplat and one can switch between these two modes.

[maturk]

@jb-ye I think if this is done then the gradients in the project gaussians would have to be modified. Currently the gradients of the linear approximation Jacobian are wrt to the z component of the gaussian mean (in camera coordinates).

[jb-ye]

Yeah, I also realized that this could be a non-trivial implementation change. It would be better for someone who is already familiar with the code to work on it.

[vye16]

Yeah this is a good suggestion! We need the z-depth for various Jacobians, but for the sort itself, we can also compute the ray length and use that instead -- no gradients flow through that direction. Thoughts @liruilong940607 @brentyi?

[liruilong940607]

I think if they would like depth supervision they would need gradients to flow back through the ray depths, which can be a bit annoying to implement.

Another way that is much easier to implement, is that after we render the z depth image, we can convert the per-pixel z depth to ray depth using the intrinsic matrix in torch. This make it differentiable but might be slightly less efficient than write it in cuda.