New Executable for Control Point Registration on the Command Line

[edanvoye]

This PR adds a new executable to run Control Point Registration on the command line.

I have simply moved the existing code from ui_openMVG_control_points_registration, which performs the same operation, but from user input.

Having a command line utility for the control point operation is very useful if an exterior tool is providing the control point position in the images, and their corresponding world position.

Here is an example Python code to load an existing SfM file (in json format), read all the images, detect Aruco markers with opencv, then align the scene with this new exe, using desired world position for the markers.

# Sample code: Detect Aruco Markers and add them to the json SfM scene file
# Author: Etienne Danvoye

def align_from_aruco(sfm_filename):

    # Generate Aruco Board
    aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50)

    # These are our desired positions for the markers
    marker_info = {       
    
        # Markers from new Aruco calibration panel
        # These are the desired world position of the markers
        'aruco_0':  (0.0, 4.0, 0.0),
        'aruco_1':  (1.5, 4.0, 0.0),
        'aruco_2':  (3.0, 4.0, 0.0),
        'aruco_3':  (4.5, 4.0, 0.0),
        'aruco_4':  (6.0, 4.0, 0.0),

        'aruco_5':  (0.0, 2.5, 0.0),
        'aruco_6':  (1.5, 2.5, 0.0),
        'aruco_7':  (3.0, 2.5, 0.0),
        'aruco_8':  (4.5, 2.5, 0.0),
        'aruco_9':  (6.0, 2.5, 0.0),

        'aruco_10': (0.0, 1.0, 0.0),
        'aruco_11': (1.5, 1.0, 0.0),
        'aruco_12': (3.0, 1.0, 0.0),
        'aruco_13': (4.5 ,1.0, 0.0),
        'aruco_14': (6.0, 1.0, 0.0),
    }

    # sfm_filename needs to be a SfM file in json format
    if not '.json' in sfm_filename:
        raise Exception("First convert SfM file to json")

    # Load json SFM Data
    with open(sfm_filename, 'r') as f:
        sfm_data = json.load(f)

    # Clear existing control points
    sfm_data['control_points'] = []

    marker_index = 0
    marker_dict = {}

    # Detect markers
    root_path = sfm_data['root_path']
    for view in sfm_data['views']:
        filename = view['value']['ptr_wrapper']['data']['filename']
        full_path = os.path.join(root_path, filename)

        print 'Detecting markers in %s' % filename

        # Read image file
        img = cv2.imread(full_path, cv2.IMREAD_UNCHANGED)
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

        # detect aruco markers in image
        corners,ids,rejected = cv2.aruco.detectMarkers(gray, aruco_dict)

        for corner,marker_id in zip(corners, ids):

            marker_name = 'aruco_%d' % marker_id
            marker_upper_left = corner[0][0]

            if marker_name not in marker_info:
                continue

            # Create new marker
            if marker_name not in marker_dict:
                item = {}
                item['key'] = marker_index
                marker_index = marker_index + 1
                item['value'] = {}
                item['value']['X'] = list(marker_info[marker_name])
                item['value']['observations'] = []

                marker_dict[marker_name] = item
                sfm_data['control_points'].append(item)
            else:
                item = marker_dict[marker_name]

            #  Add observation to marker
            obs = {}
            obs['key'] = view['value']['ptr_wrapper']['data']['id_view']
            obs['value'] = {}
            obs['value']['id_feat'] = 0 # not used for control_points                        
            obs['value']['x'] = [float(marker_upper_left[0]),float(marker_upper_left[1])]
            item['value']['observations'].append( obs )

    # Write new json
    json_file_with_markers = os.path.splitext(sfm_filename)[0] + '_markers.json'
    with open(json_file_with_markers, 'w') as f:
        print 'Writing json file %s' % json_file_with_markers
        json_str = json.dumps(sfm_data)
        f.write(json_str)

    # Run new exe to align scene from control points
    output_file = os.path.splitext(json_file_with_markers)[0] + '_aligned.bin'
    cmd = [os.path.join(OPENMVG_BIN,'openMVG_main_ControlPointsRegistration.exe'), 
        '-i', json_file_with_markers, 
        '-o', output_file
    ]
    subprocess.check_output(cmd, stderr=subprocess.STDOUT, cwd=cwd)

[pmoulon]

Great!
Thank you for this PR!

Some month ago I was thinking about adding a binary to detect marker (abstract detection) to create landmark image observation. So then the user would have only to edit the 3d location of the marker thanks to the GUI and perform the registration.

Your script is nicely going in this way and I'm very happy to see that you can use the full flexibility of OpenMVG json scene file there!

Notes:

• I saw a typo in your script # sfm_filename needs to be a SfM file ni json format | ni|in
• Do we want use the upper_left corner of the middle of the marker?

My long term plan was to move this prototype code to a sfm_data registration module but I did not found the time to make it yet.

We can think also to add an example with a tiny dataset. It would be really useful for the community.

[edanvoye]

Yes, I am using the upper-left corner in this example, but that's totally arbitrary. The Aruco library in opencv returns all 4 corners, so it would be trivial to use the center.

I do have a 4 image set with the Aruco markers, which works well with the above python test code. But these are not high quality images in any way. It's just a set of 4 grayscale images with the markers printed on a paper. I could upload them if that is useful.

(Btw, OpenMVG will crash on grayscale images, so I triplicate the channel to fake an RGB file)

[edanvoye]

I have added a repository with sample images to explain the usage of this new exe and the above python script.

GitHub photogrammetry_example_seashell

[pmoulon]

Sidenote: OpenMVG should support grayscale image for only some part of the pipeline. We can try to fix the one where it was not working.

[edanvoye]

Yes, regarding grayscale images, I only have problems with the exporters (example OpenMVS), since most of them try to undistort the original images using an Image<openMVG::image::RGBColor>. The other parts of the pipeline read all images as grayscale anyways.

I'm not sure what the cleanest way to fix it since the Image class is templated. Probably a generic undistort function which would call ReadImageHeader, then use the appropriate template for Image.

In general it would be usefull to check the result of ReadImage to catch errors

[pmoulon]

Yes, regarding grayscale images, I only have problems with the exporters (example OpenMVS), since most of them try to undistort the original images using an ImageopenMVG::image::RGBColor. The other parts of the pipeline read all images as grayscale anyways.

Most of the time I don't even know if some MVS 3rd party tools does even support grayscale image...

In general it would be useful to check the result of ReadImage to catch errors

Yeah, for sure OpenMVG core is more robust than the exporter. But thanks to the community opensource projects feedbacks and contributions become stronger ;-)

[FlachyJoe]

Is this return really needed ? Even if THIS point cannot be used, usable data should be extracted.

[edanvoye]

I agree, the number of valid points is already checked just below : if (vec_control_points.size() < 3)

We could simply remove the return.This is also true where this code originated from: src/software/ui/SfM/control_points_registration/MainWindow.cpp

[FlachyJoe]

Hi,
I'm trying your soft and obtain an error :
terminate called after throwing an instance of 'std::out_of_range' what(): map::at Abandon
Here is the full return :

Control points observation triangulations:
 0.237131 0.0906828  0.162301  0.233294  0.309503  0.384567  0.305002  0.380132  0.456689  0.526456  0.166337  0.310198   0.38771  0.458625  0.240638
 0.215731  0.221772   0.15733 0.0903247  0.153154  0.212664   0.02411 0.0862854  0.147867  0.204376  0.282014  0.275063  0.337258  0.270393  0.340737
  1.68473    1.6827   1.72159   1.76095    1.7249   1.68921    1.8003   1.76418   1.72848   1.68012   1.64569   1.64763    1.6124   1.64927   1.60998

Control points coords:
1 0 0 0 1 2 0 1 2 3 1 2 3 3 2
2 3 2 1 1 1 0 0 0 0 3 2 2 1 3
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Found transform:
 scale: 9.65743
 rotation:
 0.727759   0.58144 -0.363724
-0.685731  0.626052 -0.371257
0.0118466  0.519603  0.854326
 translation:  4.03997  8.30749 -15.0176
CP index: 0
CP triangulation error: 7.81256 pixel(s)
CP registration error: 0.00799349 user unit(s)

CP index: 1
CP triangulation error: 8.23382 pixel(s)
CP registration error: 0.0218952 user unit(s)

CP index: 2
CP triangulation error: 7.68641 pixel(s)
CP registration error: 0.0209476 user unit(s)

CP index: 3
CP triangulation error: 9.94075 pixel(s)
CP registration error: 0.0102225 user unit(s)

CP index: 4
CP triangulation error: 10.5302 pixel(s)
CP registration error: 0.0241044 user unit(s)

CP index: 5
CP triangulation error: 7.07803 pixel(s)
CP registration error: 0.0322966 user unit(s)

CP index: 6
CP triangulation error: 5.16747 pixel(s)
CP registration error: 0.023384 user unit(s)

CP index: 7
CP triangulation error: 7.95292 pixel(s)
CP registration error: 0.0197364 user unit(s)

CP index: 8
CP triangulation error: 5.54255 pixel(s)
CP registration error: 0.0434862 user unit(s)

CP index: 9
CP triangulation error: 14.3761 pixel(s)
CP registration error: 0.0784767 user unit(s)

CP index: 10
CP triangulation error: 8.13979 pixel(s)
CP registration error: 0.016814 user unit(s)

CP index: 11
CP triangulation error: 6.45697 pixel(s)
CP registration error: 0.0257906 user unit(s)

CP index: 12
CP triangulation error: 1.70939 pixel(s)
CP registration error: 0.0230392 user unit(s)

CP index: 13
CP triangulation error: 1.90647 pixel(s)
CP registration error: 0.0142711 user unit(s)

CP index: 14
CP triangulation error: 3.97717 pixel(s)
CP registration error: 0.011258 user unit(s)

Bundle adjustment with GCP
terminate called after throwing an instance of 'std::out_of_range'
  what():  map::at
Abandon

I add the bundle adjustment comment in the code to know when it fails but I'm not able to search deeper.
I can provide the dataset and all the generated files if needed.

Regards,

[Xartos]

I'm using this code for a school project and I think I've found the issue with the "out_of_range" error.
On line 104 when it calls IsPoseAndIntrinsicDefined and it returns false, the code just goes to the next observation without removing the observation. Because the add_markers just adds all the observations without checking if pose exists the Bundle adjustment chrashes when it tries to fetch the pose for this observation.

So I've created a function that removes the observation from the sfm_data when this occurs.

void removeObservation(SfM_Data & sfm_data, const int id_pose){
  Landmarks * sfm_cps = &(sfm_data.control_points);

  // for all control points
  for (auto cp_it = sfm_cps->begin(); cp_it != sfm_cps->end(); cp_it++){
    Observations * observations = &(cp_it->second.obs);
    auto obs_it = observations->begin();

    // for all observations
    while (obs_it != observations->end()){
      if(obs_it->first == id_pose){
        std::cout << "Erasing observation id " << obs_it->first
                  << " from control point " << cp_it->first
                  << std::endl;
        obs_it = observations->erase(obs_it);
      } else {
        obs_it++;
      }
    }
  }
}

and the if statement on line 104 is changed to:

if (!sfm_data.IsPoseAndIntrinsicDefined(view)){
  removeObservation(sfm_data, view->id_pose);
  continue;
}

I'm quite new to C++ so I'm sure the function could be written better, but it works.

Anyways thanks a lot for this module!

[edanvoye]

I was able to repro this problem by having a landmark detected in one image which does not have a valid pose (this image is not connected to the others).

There is a crash in Bundle_Adjustment_Ceres because the code iterates over all control points, and tries to access their pose.

@Xartos , your fix would work because it removes that pose from all control points, but we could avoid that gymnastics by adding a check directly inside the bundle adjustment, there are already checks for valid intrinsics, there should be checks for a valid pose as well.

Specifically this code assumes that the pose is valid:

        if (cost_function)
        {
          if (!map_intrinsics.at(view->id_intrinsic).empty())
          {
            problem.AddResidualBlock(cost_function,
                                     nullptr,
                                     &map_intrinsics.at(view->id_intrinsic)[0],
                                     &map_poses.at(view->id_pose)[0],
                                     gcp_landmark_it.second.X.data());
          }
          else
          {
            problem.AddResidualBlock(cost_function,
                                     nullptr,
                                     &map_poses.at(view->id_pose)[0],
                                     gcp_landmark_it.second.X.data());
          }
        }

I suggest we add this same check right before the cost_function calculation.

		if (!sfm_data.IsPoseAndIntrinsicDefined(view))
			continue;

[edanvoye]

I also added a fix if a control point has no observations at all.

[edanvoye]

@pmoulon This pull request to create an additional executable specifically for the Control Point Registration was updated and tested for the current develop branch. Example dataset and usage instructions available at photogrammetry_example_seashell

[edanvoye]

Found an issue while testing

Thanks for testing. What you describe is not really an issue, it's a limitation. The aruco panel needs to be printed very precisely, and it also needs to be very flat, otherwise the registration will not work. If printed on paper, I recommend glueing the sheet to a rigid panel to prevent warping.

[paaweel]

Hey, any status update on this? seems like a great feature. Thanks for all the work so far!