Is the geometry of high quality?

[isxulu]

Firstly, thank you very much for your excellent work! When I tried your code, I found that the quality of the rendering in some scenes of the hypernerf dataset by 4d-gs was not good. In these monocular scenes, is there a need to add some additional supervision to improve the geometry? Also, could it be possible that the final trained deformation field is overfitting to rgb? This is just my guess. Have you conducted any related experiments? Thanks for your reply!

[guanjunwu]

Hi, thanks for your nice problem!
I encountered the same problem, I've tried to supervise the training process with depth and it seems to improve rendering quality. However, overfitting to training images is an existing problem. Maybe adding the disturbance in time mentioned in deformable 3DGS will ease the issue (I guess).
Besides, I'm also working hard to train a better 4DGS to fit any other dataset. If you have any ideas to improve 4DGS, please feel free to contact to me.
Best,
Guanjun

[isxulu]

Thank you for your information!
Yes, I also believe that adding depth or optical flow supervision can enhance geometry. However, I recently tried to use optical flow supervision, and without adding this additional loss, I supervised the rendering of flow only with the existing loss. I found a significant discrepancy between the flow map rendered from the Gaussian deformed through the deformation field and the flow map estimated by some excellent estimators. I have previously added optical flow supervision in static GS work, which can significantly improve the quality of geometry. Therefore, I think the deformation block of 4dgs is not robust enough.

[guanjunwu]

Thanks for your insight!
I'm not sure about is deformation block(hexplane) of 4DGS or if the deformation field is not robust enough. Many deformation methods were used in NeRF and proved that the deformation field can work and even can be supervised by optical flow.
Do you think the Gaussian deformation field can solve the problem?

[azzarelli]

There's other 4D GS models that use deformation and it's not a great solution (honestly this 4D-GS seems like a better approach then most), plus introduces new issues that hex planes methods avoids. For example with the banana scene, a deformable model might find it challenging to model the appearance and geometry of the banana slices after they have been cut as they might not be present in the canonical field (as part of a canonical+deformable model). Also for sparse scenes (where geometry is going to suffer the most), I have seen some methods periodically use D-SSIM loss to improve the multi-view aspect ( for 6DoF). They use it every 1000 epochs or so as D-SSIM can be a bottle neck. Methods like Sparse-Controlled GS (for 4-D) also use "control" points, essentially the same way hierarchical voxel geometry works, just with point clouds. Seems there are some regularizers for these that can deal with rigid motions better compared to unconstrained solutions