Shallow depth information at edge of objects

[jonasdieker]

Required Info
Camera Model	D455
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	Ubuntu 22.04
Kernel Version (Linux Only)	(e.g. 5.15)
Platform	Jetson Orin AGX
SDK Version
Language	python
Segment	Robot

Issue Description

I am recording RGB and depth data from the D455 in order to build an "obstacle point cloud". The steps I take are as follows:

1. Run the RGB image through a deep learning network, which outputs a segmentation mask
2. Use segmentation mask to get pixels of "obstacle" classes
3. Use the corresponding pixels in the depth map to re-project pixels to 3D points

Of course the segmentation is not pixel perfect, so there are always some wrong points, but that can be solved with further post-processing. One thing however that I noticed is that at the edge of objects, I actually have pixels which are projected to be in front of the objects.

The point on the very right is the camera origin (some pixels which I map to zero). Directly left of that there are 3 points which are just floating mid-air and cannot be attributed to an object in the scene. When rotating the view, it can be seen that these originate from the very edge of an obstacle. What could be the cause of this?

Below is a screenshot, where the camera origin is the point in the centre:

Edit: I use the decimation post-processing filter and the spatial filter.

[MartyG-RealSense]

Hi @jonasdieker If there are floating areas of depth detail that do not correspond to areas in the real-world scene then this is indicative of a phenomenon called a repetitive pattern, where there are repeating horizontal or vertical arrangements of objects in the scene (such as floor or ceiling tiles) that can confuse the camera's depth sensing. Advice about negating repetitive patterns is provided at #6713 (comment)

The D455f model of RealSense camera that is equipped with a light-blocking filter can also help to negate repetitive patterns.

https://www.intelrealsense.com/depth-camera-d455f/

[jonasdieker]

Hi @MartyG-RealSense, first of all thank you for the prompt response.

I don't think this is the repetitive pattern issue, as there are no such repetitive structures in the scene. The scene contains a tree trunk on the right and a person's leg and arm on the left in an outdoor environment. My intuition is that the features of these objects are not similar enough for there to occur a mismatch in features.

I think what further confirms what I said above, is that if there was a repetitive pattern issue, the wrong points should have a depth similar to another object in the scene, however there is nothing at a similar depth. Please let me know if with this information, you still think that this is the problem. :)

I have a suspicion that these depths might be caused by the spatial post-processing filter. Could this be the problem?

[MartyG-RealSense]

Having 'ghost noise' on an image (depth information that should not be there) can be produced by repetitive patterns. However, there have been a couple of past cases where applying a spatial filter has caused incorrect areas of depth to appear on a pointcloud.

I note that you rotated the view of the pointcloud. This too can cause the cloud to distort, as shown with the example of a banana at #3741 that becomes elongated when its pointcloud is rotated side-on.

[jonasdieker]

Hmm, ok maybe I will try again without the spatial filter and see if this problem still occurs. I think I will also give the "high accuracy" preset a try. Could you tell me what exactly this does? My understanding is that the camera only returns a depth for points for which it has a higher confidence. How is this confidence computed and what is the threshold confidence used?

I don't think my particular issue is related to #3741, as the elongation/object spraying only happens behind the object and not in front. I also a little bit of elongation (behind the objects) and I still need to apply some post-pressing to remove those points, but that's a separate problem :)

[MartyG-RealSense]

You are correct that the High Accuracy preset applies a high confidence threshold to depth values and excludes values from the depth image that do not meet its criteria. This preset sets a high value for an Advanced Mode parameter called 'Second Peak Delta' to accomplish this. This parameter will disqualify a depth pixel if the "second best" match is too close to the best match. The default for Second Peak Delta is '325' and High Accuracy uses a value of '624'.

High Accuracy tends to over-strip detail from the depth image and make it sparse though, so I would recommend the Medium Density preset that sets Second Peak Delta to '600' and gives a good balance between accuracy and the amount of detail on the depth image.

[jonasdieker]

Thank you for the great explanation! I will give both of those a try then.

[jonasdieker]

@MartyG-RealSense Do you happen to have an example/issue, where the spatial filter caused incorrect areas of depth to appear on a pointcloud?

[MartyG-RealSense]

A spatial filter was being applied with the elongated banana case at #3741 and also in the cases at the link below.

https://support.intelrealsense.com/hc/en-us/community/posts/360050492334-Accuracy-with-sunlight-noise

https://support.intelrealsense.com/hc/en-us/community/posts/4404004402579-About-FoV-issue-of-D435i

[MartyG-RealSense]

Hi @jonasdieker Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.