Question of Alignment image and Depth Scale unit on D405

[qidihan]

Required Info
Camera Model	{ D405 }
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	Linux (Ubuntu 22.04)
Kernel Version (Linux Only)
Platform	PC
SDK Version	{ newest }
Language	{python}
Segment	{Robot }

Issue Description

Question 1: Alignment issue with 640x480 resolution
I've encountered an alignment issue when using the rs.align() function with a 640x480 resolution. I've run the example codes for "Rendering depth and color with OpenCV and Numpy" and "Align & Background Removal". I noticed that alignment fails when using 640x480, but works with 640x320 or 1280x720. Additionally, the example code for "Rendering depth and color with OpenCV and Numpy" also aligns correctly. Does this indicate that using rs.align() with only 640x480 is incorrect or unable to align?
Question 2: Depth scale and unit conversion
I'm trying to record RGB and depth images in PNG files, stored in millimeter units (mm) with uint16 format. However, I'm unsure if my code is correct. (# cv2.imwrite((depthpath+ filename), (depth_image*10).astype(np.uint16)))I've seen issues mentioning that the d405 camera's depth scale is 0.01, but my output shows:
Python

depth_sensor = profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
print("Depth Scale is: ", depth_scale) # Output: Depth Scale is: 9.999999747378752e-05

I don't fully understand this unit. Can someone explain it to me and tell me how to set up my code to output the depth image with millimeter units?

[qidihan]

Question 3: Different intrinsic parameters after alignment
I've run the example codes for "Rendering depth and color with OpenCV and Numpy" and "Align & Background Removal". After alignment, I noticed that the intrinsic parameters (intrinsics) are different between the two codes:

No alignment code:

intrinsics = frames.profile.as_video_stream_profile().intrinsics
print(intrinsics) # Output: 'cam_K': [379.8869934082031, 0.0, 312.4703369140625, 0.0, 379.8869934082031, 240.33531188964844, 0.0, 0.0, 1.0]

Alignment code:

intrinsics = frames.profile.as_video_stream_profile().intrinsics
print(intrinsics) # Output: 'cam_K': [431.27752685546875, 0.0, 313.7019958496094, 0.0, 430.9690856933594, 240.52565002441406, 0.0, 0.0, 1.0]

Why are the intrinsic parameters different after alignment? Shouldn't they be the same?

[MartyG-RealSense]

Hi @qidihan The Align & Background Removal script that you refer to, also known as align_depth2color.py, uses 640x360 for its alignment resolution by default. At #11329 a RealSense user created a specially edited version of this script for use on D405 and set the resolution to 640x480 but found that it had problems with the alignment. In the end they settled for using 640x360.

Yes, the default depth unit scale of D405 is 0.01. 9.999999747378752e-05 when input into the RealSense Viewer's depth unit scale option auto-converts it into a value of 0.0001, whilst changing the '05' on the end of the value to '02' is required in order for the Viewer to recognize it as 0.01.

It may be unnecessary to retrieve and output the depth scale, as the default scale for a particular camera will be automatically applied (0.01 for D405 and 0.001 for the rest of the 400 Series range) and will not change unless deliberately changed by user input or by a program instruction that sets the depth scale value.

Each resolution has different intrinsic fx / fy values. The default resolution of D405 for depth and color is 848x480. After checking the intrinsics of my own D405, I wonder if you are seeing the intrinsics for 848x480 when alignment is not being performed, but when the alignment script is run and resolution is set at 640x360, you are seeing the different intrinsics for 640x360 resolution.

[qidihan]

Thank you for your response. I still have some questions regarding the depth image unit.

1. I need to output my depth image with unit mm, but when I use:

depth_image = np.asanyarray(aligned_depth_frame.get_data())
cv2.imwrite((depthpath), (depth_image).astype(np.uint16))

I'm unsure what unit the output depth_image.max() value represents. Could you please clarify what unit this value is in? For example, it outputs 65536, which I guess to be equivalent to 655.36mm (how it relates to 0.01 depth scales?). However, I'm unsure how to relate this to the unit millimeter.
My main question is: can I directly output the depth image with unit millimeter by using cv2.imwrite((depthpath), (depth_image).astype(np.uint16)), or do I need to multiply the depth values by 100 to convert them to millimeters?
(I'm working on the BundleSDF project ((https://github.com/NVlabs/BundleSDF)) and need depth images for testing. Thank you for your help and clarification.)

2. Additionally, I have a question regarding the intrinsic parameters. When I use align-depth2color.py and Rendering depth and color with OpenCV and Numpy codes, the intrinsic parameters are different for the same pixel (640*360). However, the output aligned images look perfect. Could you please explain why this is the case?
   The project also needs data like:
   {"cam_K": [614.7604370117188, 0.0, 322.7101135253906, 0.0, 614.9583129882812, 233.1604766845703, 0.0, 0.0, 1.0], "depth_scale": 1.0000000474974513}
   I'm unsure what the depth_scale should be output in my case, if the depth image need unit of millimeter. The example code the author uses is (He is using the D435 camera. not d405):

Get depth profile and camera intrinsics

intrinsics_matrix = {
"cam_K": [
intrinsics.fx,
0.0,
intrinsics.ppx,
0.0,
intrinsics.fy,
intrinsics.ppy,
0.0,
0.0,
1.0,
],
"depth_scale": depth_scale * 1000,
}

[MartyG-RealSense]

0 to 65536 represents the depth as a pixel depth value. To convert this value to real-world distance, you multiply it by the depth scale of the camera.

For example, if a non-D405 camera was being used then its default scale would be 0.001, so 65535 x 0.001 = would be 65.535 (meters).

However, D405 (which is a short-range camera) uses a scale of 0.01 by default. This is centimeter scale instead of millimeter scale. So 65535 x 0.01 = 655.35 (centimeters), equivalent to 6.5535 meters. So you would multiply the centimeter result by 10 to get the distance in millimeters.

When the align_to instruction is used to perform alignment (like in align_depth2color.py), the RealSense SDK's align processing block performs automatic adjustment calculations.

[MartyG-RealSense]

Hi @qidihan Do you require further assistance with this case, please? Thanks!