point_triangulator step always stuck after I use known camera poses as priori info

[harveyyang66]

Thank you for this amazing work, I have read the manual and related issues, but really cannot solve the following problems:

My hardware setup is a static DSLR camera with a rotational stage to place the object, the stage will turn 360° and the camera will shoot every 6°, so 60 images are taken in total. I use a high-accurate checkerboard to calibrate the intrinsic parameters and 60 extrinsic matrixes (camera poses at each shot) with Matlab, visualization plot out:

(determine the fixed position of the camera in the world coordinate system for each shot so that the reconstructed point cloud is unscaled and coincides with the size of the actual object)

Image shooting step: place the object on the stage and 60 pictures are taken in the same way (the camera position at each shot is the same as the calibration step). e.g. :

I prepared the cameras.txt, images.txt and points3D.txt(empty) shown below:
cameras.txt:

1 OPENCV 5184 3456 11745.66649476915 11641.56983513971 2613.174925744359 1247.096050748680 0.016983831269729 1.817648990204491 -0.003573730692592 0.001283209998103

images.txt:

1 0.6843303295923687 -0.34397605925228447 0.30679080105226014 0.5650235467867123 14.451284 0.225208 274.136317 1 Image_001.jpg

2 0.7128195774528081 -0.35881940969407977 0.2886778877978043 0.5289630951879444 11.931327 -0.807915 275.884581 1 Image_002.jpg

......

59 0.6207574002780797 -0.30929353385717473 0.3416717221655159 0.6342385122169001 18.933814 2.7759 270.37 1 Image_059.jpg

60 0.6531213516644514 -0.32696297473017233 0.3249778918712275 0.6007639300109507 16.823402 1.430973 272.224882 1 Image_060.jpg

if I convert this three .txt files into .bin and make it visable in GUI, it will shows like below, all cameras are in the correct position and their FoV are overlapping:

I can successfully run the 'feature_extractor' and 'exhaustive_matcher' like this:

colmap feature_extractor --database_path database.db --image_path images --ImageReader.camera_model OPENCV
==============================================================================
Feature extraction
==============================================================================
18384 feature_extraction.cc:254] Processed file [1/60]
18384 feature_extraction.cc:257]   Name:            Image_001.jpg
18384 feature_extraction.cc:283]   Dimensions:      5184 x 3456
18384 feature_extraction.cc:286]   Camera:          #1 - OPENCV
18384 feature_extraction.cc:289]   Focal Length:    11952.49px (Prior)
18384 feature_extraction.cc:302]   Features:        9051
18384 feature_extraction.cc:254] Processed file [2/60]
18384 feature_extraction.cc:257]   Name:            Image_002.jpg
18384 feature_extraction.cc:283]   Dimensions:      5184 x 3456
18384 feature_extraction.cc:286]   Camera:          #1 - OPENCV
18384 feature_extraction.cc:289]   Focal Length:    11952.49px (Prior)
18384 feature_extraction.cc:302]   Features:        9045
......

18384 feature_extraction.cc:254] Processed file [59/60]
18384 feature_extraction.cc:257]   Name:            Image_059.jpg
18384 feature_extraction.cc:283]   Dimensions:      5184 x 3456
18384 feature_extraction.cc:286]   Camera:          #1 - OPENCV
18384 feature_extraction.cc:289]   Focal Length:    11952.49px (Prior)
18384 feature_extraction.cc:302]   Features:        9298
18384 feature_extraction.cc:254] Processed file [60/60]
18384 feature_extraction.cc:257]   Name:            Image_060.jpg
18384 feature_extraction.cc:283]   Dimensions:      5184 x 3456
18384 feature_extraction.cc:286]   Camera:          #1 - OPENCV
18384 feature_extraction.cc:289]   Focal Length:    11952.49px (Prior)
18384 feature_extraction.cc:302]   Features:        9055
13188 timer.cc:91] Elapsed time: 0.198 [minutes]


colmap exhaustive_matcher --database database.db
==============================================================================
Exhaustive feature matching
==============================================================================
25796 feature_matching.cc:231] Matching block [1/2, 1/2]
25796 feature_matching.cc:46]  in 17.096s
25796 feature_matching.cc:231] Matching block [1/2, 2/2]
25796 feature_matching.cc:46]  in 0.748s
25796 feature_matching.cc:231] Matching block [2/2, 1/2]
25796 feature_matching.cc:46]  in 6.445s
25796 feature_matching.cc:231] Matching block [2/2, 2/2]
25796 feature_matching.cc:46]  in 0.712s
25796 timer.cc:91] Elapsed time: 0.418 [minutes]

but when I came to point_triangulator, the error occurs:

colmap point_triangulator --database_path database.db --image_path images --input_path created/sparse --output_path triangulated/sparse
==============================================================================
Loading model
==============================================================================
Loading database
==============================================================================
18668 database_cache.cc:54] Loading cameras...
18668 database_cache.cc:64]  1 in 0.000s
18668 database_cache.cc:72] Loading matches...
18668 database_cache.cc:78]  402 in 0.004s
18668 database_cache.cc:94] Loading images...
18668 database_cache.cc:143]  60 in 0.017s (connected 60)
18668 database_cache.cc:154] Building correspondence graph...
18668 database_cache.cc:190]  in 0.040s (ignored 0)

I've tried several times and it always stuck at here, don't know why this happening. I have checked image/camera ID in GUI and they all correct, and modified the camera model and params with my calibrated data:

May I ask if there is any other step I forget?
your kindly help will be highly appreciated, thanks again 🙏

[jytime]

My wild guess is probably the images are too huge. 5000x3500 may be too huge, which provides too many keypoints to colmap and hence it takes too much time/out of memory.

Has your run reached this function, or not?

colmap/src/colmap/controllers/incremental_mapper.cc

Line 540 in 187a4e1

void IncrementalMapperController::TriangulateReconstruction(

[harveyyang66]

My wild guess is probably the images are too huge. 5000x3500 may be too huge, which provides too many keypoints to colmap and hence it takes too much time/out of memory.

Has your run reached this function, or not?

colmap/src/colmap/controllers/incremental_mapper.cc

Line 540 in 187a4e1

void IncrementalMapperController::TriangulateReconstruction(

Hi @jytime , thank you very much for the reply, I didn't reach this step. Honestly, I didn't even reach the bundle adjustment step.

So I used to make a mistake (not relevant to the issues raised above), I fed Colmap with a set of wrong camera poses, the translation is correct but the rotation is wrong, like below:

as you can see, the cameras are parallel in this case because of wrong quaternion. But the weird thing is this one can pass through the point_triangulator step, but only 40 points in the generated sparse cloud, and the following patch_match_stereo step always shows like:

14668 patch_match.cc:383] Ignoring reference image Image_004.jpg, because it has no source images.
14668 patch_match.cc:383] Ignoring reference image Image_013.jpg, because it has no source images.
14668 patch_match.cc:383] Ignoring reference image Image_016.jpg, because it has no source images.

if the wrong parallel camera poses can somehow pass through the point_triangulator step, I don't understand why the correct camera pose is stuck.

For the image resolution problem, I used 5184 x 3456 for reconstruction in a previous test (without using any calibrated camera intrinsic and extrinsic info to rewrite cameras.txt/images.txt). The un-scale-calibrated sparse cloud and related uncalibrated camera position look perfect:

the only problem is that this cloud is uncalibrated and the size won't match the real object. But if this test can successfully pass through all processes, I think it probably means the image resolution is acceptable?

[jytime]

Hey given this it may be not due to the image size. You can use LOG(INFO) to print the logs inside COLMAP, so that you can know which function gets you stuck.