Detecting and displaying shared regions of interest.

[wamj45]

Here is my setup:

• Intel RealSense D455
• A second camera with a zoom lens. I call this the barcode camera
• Both cameras are fixed on the same horizontal and vertical planes

What I can do today:

• I can use an inference engine to detect a barcode with the barcode camera
• I have the X1, X2, Y1, Y2 values of the bounding box
• The RealSense can see the barcode but it is to far to detect, hence why I have the barcode camera 😄

What I want to achieve:

• I want to draw the bounding box of the barcode in the RealSense camera image.

What I have tried:

1. Using Aruco markers to create a Homography matrix. Then using cv2.perspectiveTransform() to get the points of the bounding box in RealSense coordinates. This kinda works but if the scene changes too much then the transformation matrix does not hold true.
2. Using OpenCV and Charuco markers I have determined the camera_matrix,and dist_coeffs of each camera. I calculated the rotation_vector and trans_vector of both cameras. This is where I am stuck. I have tried different things here like taking the dot product of the vectors and feeding that into cv2.projectPoints(). I have tried to also undistort the points then project them. I tried to to use cv2.solvPnP() but I do not fully understand what is meant by objectPoints OpenCV solvePnP docs

Does anyone have any advice, tips, anything that can point me in the right direction?
Any help is appreciated thanks!

[MartyG-RealSense]

Hi @wamj45 Instead of using a separate camera with zoom lens for barcode detection, if the D455 can see the barcode then it might be able to detect it if the RealSense SDK's RGB Sharpness option is maximized to a value of '100' to sharpen the distant barcode image. #11246 has Python scripting for setting the Sharpness value.

color_sensor.set_option(rs.option.sharpness, 100)

[wamj45]

@MartyG-RealSense
What you are saying is completely true.

However, this is not how my system is setup. The barcodes are 1-3m away from the RealSense and covered in shrink wrap so there is glare coming into play. There is no way the barcode can be detected from that distance covered in shrink wrap. That is why I have the zoom lens camera to detect that can detect the barcodes.

That is why I need to use a transformation from the zoom view coordinates to the RealSense coordinates.

[MartyG-RealSense]

Glare from reflections can be greatly negated if a thin-film linear polarization filter is placed over the front of the camera. Because most polarizing filters work so song as they are the linear type, they are inexpensive to obtain. You can search stores such as Amazon with the term linear polarization filter sheet to find examples.

The image below from Intel's optical filter guide for 400 Series cameras shows an RGB image without (left) and with (right) a linear polarization filter that depicts how glare from glass windows can be negated by a filter.

https://dev.intelrealsense.com/docs/optical-filters-for-intel-realsense-depth-cameras-d400

If you are aiming though to align a RealSense RGB image to the barcode camera's RGB image then that would be difficult as alignment usually has to have a depth image as one of the two image types that are being aligned. Also, the barcode camera's RGB view would be offset from the RealSense RGB view because the RGB sensors of the two cameras would be in different physical positions and so not seeing the scene from the exact same origin position.

[wamj45]

If you are aiming though to align a RealSense RGB image to the barcode camera's RGB image then that would be difficult as alignment usually has to have a depth image as one of the two image types that are being aligned. Also, the barcode camera's RGB view would be offset from the RealSense RGB view because the RGB sensors of the two cameras would be in different physical positions and so not seeing the scene from the exact same origin position.

This is exactly what I am trying to do. And I have done it with a Homography matrix and cv2.perspectiveTransform() but it is not robust enough for my use case.

Yes the offset will have to come into play here but I wanted to first get a more reliable and robust solution before bringing offsets.

[MartyG-RealSense]

There is an OpenCV / Python project at the link below for aligning full-HD images from two cameras.

https://github.com/cbuhl/CameraAlignment