visual improvement?

[JonaTh4nX]

Hi there ! 🙂

Here is one of my capture using Splatfacto training method with 30K ite .

In the image on the left, this is a capture in Unity showcasing the rendering quality achieved through the plugin developed by Aras-P , which is nearly perfect regarding the visual quality.

On the image to the right, this is a capture from the Mark Kellogg's viewer ( and it look roughly the same using the viewer of Antimatter).

Take a look at the wall on the right, the sky, the shadow of the roof on the white wall to the left, or the car's bodywork... Why is there such a visual difference? Is there any room for improvement on this?

[mkkellogg]

Others have had similar issues and I'm not exactly sure what the root cause is. Would you be able to share your capture so I can do some debugging on my end? Also, have you tried viewing your capture in both/either PlayCanvas' SuperSplat editor or Polycam's viewer? I'd be interested to see how it looks in those viewers.

[JonaTh4nX]

Regarding the visual I provided, it seems to come from the training method. I'm working on it ....

Nevertheless, when we look more closely, there are visual differences between the viewers. Unity offers by far the most beautiful visual rendering, while Polycam is not far behind, and from what I have observed so far, it seems to be the web viewer that comes closest to the visual quality of Unity. Yours remains very competitive because it is much smoother in terms of navigation and has many other advantages! Do you think your viewer can reach the visual quality of Polycam?

Here are some screenshots to illustrate all this.

[Manuel3DGS]

Hello all,

I've noticed that we have a similar issue. With the SIBR local viewer the result is AWESOME but when we use your web viewer, the quality seems to be bellow.

I've see this thread: graphdeco-inria/gaussian-splatting#455

It's seems to be related to the spherical harmonics and the renderer/diff-gaussian-rasterization

@mkkellogg have you the possibility to implement the support for different degree of spherical harmonics ? With the SIBR local viewer, it support sh_degree = 3. It can have a big impact on the quality.

[mkkellogg]

So the discrepancy seems to be mostly due to the lack of support for spherical harmonics in my renderer. When I load up the models in Aras Pranckevičius' Unity-based viewer (https://github.com/aras-p/UnityGaussianSplatting), and edit the code to disable the spherical harmonics support, the output looks much, much closer to mine.

It looks like I need to prioritize support for spherical harmonics :) The big reason I have avoided it is because of the large memory requirement, but maybe some sort of minimal-loss compression scheme could make it feasible.

[Manuel3DGS]

OMG, yes, if you can prioritize that it could be simply huge 🤩.

[JulienMDST]

Hi,

Very interesting thread. @mkkellogg, I sent you an email about it, it might be in your spam folder.

Looking forward to hearing from you!

Have a nice day.

Julien

[JulienMDST]

Hello @mkkellogg ,

I hope you're well. Did you well received my email ?

Have a great day 😊.

Julien

[lxzbg]

Hi @mkkellogg Based on your work, I simply implemented a first-order SH that supports the ply model. There is a certain quality improvement, but there are still black fog artifacts. Is this because it is not a third-order SH? Could you help me check the correctness of my implementation?

[mkkellogg]

@lxzbg I have also been working on my own implementation for 1st degree spherical harmonics. I tried to mimic Aras Pranckevičius' implementation: https://github.com/aras-p/UnityGaussianSplatting. To verify if yours is correct, I would load his implementation in Unity and force it to only use 1st degree SH by commenting out this block: https://github.com/aras-p/UnityGaussianSplatting/blob/0265bd094d67685fa692914c9b9c2f0ce3ba8e98/package/Shaders/GaussianSplatting.hlsl#L154
Then compare the result to yours.

[mkkellogg]

So it looks like some of the folks involved with the original gaussian splatting research at INRIA have developed a way to significantly reduce the memory footprint of splat scenes and spherical harmonics: https://repo-sam.inria.fr/fungraph/reduced_3dgs/. As soon as they release their source code I will attempt to port it to my viewer.

[Manuel3DGS]

Great to read that Mark. Top !

[mkkellogg]

I've been working to support 1st degree spherical harmonics as they are formatted in the original .ply format and the result is a mild increase in visual quality. I have a PR up that work here: #216. I also have a WIP branch to support 2nd degree spherical harmonics here: https://github.com/mkkellogg/GaussianSplats3D/tree/spherical_harmonics_2nd_degree. The visual quality changes are more significant in that branch, so if anyone wants to try it out, let me know how it works for you :)

Note: The .ply files currently take a long time to load when incorporating spherical harmonics. Also, you need to pass sphericalHarmonicsDegree: <desired spherical harmonics degree> to the viewer to enable it.