Re-using Dust3r output with new images

[relh]

Hi all!

I have a setup where I run dust3r on a few images, then I want to add images to this and run it again.

I've been following the issues #54 , #30 , #17 . And using this ModularPointCloudOptimizer: 4a414b6

I'm wondering if there's anything else that can be preset in a scenario like this. Basically, re-using existing depth_maps with _set_depthmap (doesn't seem to save time, maybe I need to disable grad on the set depth_map?).

I haven't been able to get the normal PointCloudOptimizer with compute_global_alignment(init='known_poses' to work either. Each time I try I get an error about the requires_grad when running scene.preset_principal_point.

I'm mostly just opening this issue in-case you guys can think of a way to incorporate new images into an existing Dust3r scene efficiently, as this is my use case.

Thanks so much for making such an incredible project! Looking forward to Mast3r too :).

[relh]

Also it's interesting that going from 2 images with the PairViewer and 3 images with the ModularPointCloudOptimizer the scale of the reconstruction often changes quite dramatically! Easy to fix if we track scale and initial pose but just something I noticed~

[hturki]

Re: "Each time I try I get an error about the requires_grad when running scene.preset_principal_point"

Unintuitively, I think that you need to initialize global_aligner with the "optimize_pp=True" option beforehand

[relh]

This is really helpful! I've gotten further than before.

> /home/relh/Code/???????????/dust3r/dust3r/cloud_opt/init_im_poses.py(61)init_from_known_poses()
     60         assert known_poses_msk[n]
---> 61         _, i_j, scale = best_depthmaps[n]
     62         depth = self.pred_i[i_j][:, :, 2]

ipdb> print(n)
0
ipdb> best_depthmaps
{1: (4.287663459777832, '1_0', tensor(0.7287, device='cuda:0')), 2: (2.592036485671997, '2_1', tensor(0., device='cuda:0'))}

I've got here and if I get further will update!

[ljjTYJR]

This is really helpful! I've gotten further than before.

> /home/relh/Code/???????????/dust3r/dust3r/cloud_opt/init_im_poses.py(61)init_from_known_poses()
     60         assert known_poses_msk[n]
---> 61         _, i_j, scale = best_depthmaps[n]
     62         depth = self.pred_i[i_j][:, :, 2]

ipdb> print(n)
0
ipdb> best_depthmaps
{1: (4.287663459777832, '1_0', tensor(0.7287, device='cuda:0')), 2: (2.592036485671997, '2_1', tensor(0., device='cuda:0'))}

I've got here and if I get further will update!

For registering the new image, I ran into the problem that the estimated scale is sensitive to the noise. I guess it is due to the procrustes problem is not robust, do you have some idea on it?