Viewmat gradient backward to optimize camera position

[pierremerriaux-leddartech]

Hi @vye16 and @kerrj
In order to re activate camera pose opt in nerfstudio, I cherry pick the gradient backward proposed in branch : https://github.com/nerfstudio-project/gsplat/tree/vickie/camera-grads

My tests were not really good.
Did you get any plan to continue with this feature ? Anyway I could help on it ? Any idea how we could test properly.
thanks

[pierremerriaux-leddartech]

Hi few more experiments on my side:
I used camera poses directly got from pandaset:
we got this reconstruction:

I added gaussian noise on trainset camera poses:

• translations : sigma .2m (in real world coordinate, before rescale in -1 1 cube)
• rotation : sigma 1degree

of course the reconstruction is very bad like that:

When I activate camera pose optimization I get this one:

It difficult to compare metric because even if I didn't add noise to eval set, the "world" seems to be not reconstructed exactly at the same position. So eval metrics are badly similar for noisy and optimized poses. But we could have a look at train psnr (noisy_notInEval2, is the optimize one):

By carefully manually review images between no noise and optimized poses version, optimized is a little less quality. But in another hand I now that in my in case, even with no noise, when I render on shift trajectory the quality decrease.

And at the end, camera optimization don't help me for multiview cameras (front, left, right....), at it was in case nerffacto nerfstudio-project/nerfstudio#2863