Regarding object 3D vertices in meters (8 corners and 1 centroid) extraction and project them to 2D image plane (pixel cordinate)

[ArghyaChatterjee]

Describe the issue

I was using a code from visp. They have a script where you can generate all the images (RGB, depth, normal etc) and a 2D bounding box for yolov7 export. Now there was 1 problem that I saw in their script that the pose estimation w.r.t camera frame seems wrong (cTo) but there bounding box was correct.

Now I want to extract the 8 corners and 1 centroid in 3D space as well as the 2D projection of those 9 vertices on camera plane (9 pixel cordinates). Is there a script you suggest that is already doing it the way visp is doing it (scriptable way) so that I can just incorporate the code block ??

Minimal code example

They have a bounding box code like this:

def bounding_box_2d_from_vertices(object: bproc.types.MeshObject, K: np.ndarray, camTworld: np.ndarray) -> Tuple[List[float], float, np.ndarray]:
  '''
  Compute the 2D bounding from an object's vertices
  returns A tuple containing:
    [xmin, ymin, xmax, ymax] in pixels
    the proportion of visible vertices (that are not behind the camera, i.e., negative Z)
    The 2D points, in normalized units in camera space, in ViSP/OpenCV frame
  '''
  worldTobj = homogeneous_no_scaling(object)
  obj = object.blender_obj
  verts = np.ones(len(obj.data.vertices) * 3)
  obj.data.vertices.foreach_get("co", verts)
  points = verts.reshape(-1, 3)
  # verts = [v.co for v in obj.data.vertices]
  camTobj = camTworld @ worldTobj
  points_cam = camTobj @ np.concatenate((points, np.ones((len(points), 1))), axis=-1).T
  points_cam = convert_points_to_visp_frame((points_cam[:3] / points_cam[3, None]).T)

  visible_points = points_cam[points_cam[:, 2] > 0]
  visible_points_m_2d = visible_points[:, :2] / visible_points[:, 2, None]
  visible_points_px_2d = K @ np.concatenate((visible_points_m_2d, np.ones((len(visible_points_m_2d), 1))), axis=-1).T
  visible_points_px_2d = visible_points_px_2d.T[:, :2] / visible_points_px_2d.T[:, 2, None]

  mins = np.min(visible_points_px_2d, axis=0)
  assert len(mins) == 2
  maxes = np.max(visible_points_px_2d, axis=0)

  return [mins[0], mins[1], maxes[0], maxes[1]], len(visible_points) / len(points_cam), visible_points_m_2d

I tried to add something similar to this:

def project_bounding_box_to_image_plane(object, K, camTworld):
    """
    Project the 3D bounding box and centroid of an object onto the 2D image plane using BlenderProc.
    
    Parameters:
    - object: bproc.types.MeshObject, the object whose bounding box to project.
    - K: The camera intrinsic matrix.
    - camTworld: The camera-to-world transformation matrix.
    
    Returns:
    - List of 2D coordinates of the projected bounding box vertices and the centroid on the image plane.
    """
    # Get the 8 corners of the bounding box in world coordinates
    bounding_box_world = np.array(object.get_bound_box())

    # Calculate the centroid of the bounding box
    centroid_world = np.mean(bounding_box_world, axis=0)

    # Combine bounding box corners with the centroid
    points_world = np.vstack([bounding_box_world, centroid_world])

    # Transform points from world to camera coordinates
    worldTcam = np.linalg.inv(camTworld)
    points_world_homogeneous = np.hstack((points_world, np.ones((points_world.shape[0], 1))))
    points_camera_homogeneous = worldTcam @ points_world_homogeneous.T

    # Project points onto image plane
    points_image_homogeneous = K @ points_camera_homogeneous[:3, :]
    points_image = points_image_homogeneous[:2, :] / points_image_homogeneous[2, :]

    return points_image.T.tolist()

But it seems that both the object dimension and keypoints didn't match.

Files required to run the code

No response

Expected behavior

Here was the output:

Both the keypoint location and the dimension is incorrect.

BlenderProc version

v2.7.0

[cornerfarmer]

Hey @ArghyaChatterjee,

sorry for the late response. Since version 2.7.0 there exists now the function bproc.camera.project_points() which might be exactly what you want.