Issues with depth cameras

[LKFierz]

Environment

• OS Version: Ubuntu 20.04

• Source or binary build?
  Gazebo Garden source build
  gz-rendering is on branch gz-rendering7 with the latest commit being:
  commit 451e34e
  Update examples to use metal as default graphics API on mac (Update examples to use metal as default graphics API on mac #937)

  • Use metal as default on apple, fix compile warning, remove optix
  • Add defaultGraphicsAPI utility function

  ---

  Signed-off-by: Ian Chen ichen@openrobotics.org

• If this is a GUI or sensor rendering bug, describe your GPU and rendering system. Otherwise delete this section.
  The GPU information I get when entering the sudo lshw -C display prompt is this:
  description: VGA compatible controller
  product: GP106GL [Quadro P2200]
  vendor: NVIDIA Corporation
  physical id: 0
  bus info: pci@0000:02:00.0
  version: a1
  width: 64 bits
  clock: 33MHz
  capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
  configuration: driver=nvidia latency=0
  resources: irq:86 memory:91000000-91ffffff memory:70000000-7fffffff memory:80000000-81ffffff ioport:2000(size=128) memory:92000000-9207ffff

  • Rendering plugin: [ogre2].
    • Sensor rendering error.
    • GUI rendering error.
  • EGL headless mode: Not sure what
    • Running in EGL headless mode
  • Generally, mention all circumstances that might affect rendering capabilities:
    • running a [gz-ros2](https://github.com/gazebosim/ros_gz.git -b humble) and ros2-ros bridge to transfer the messages to ros
  • Rendering system info:
    • On Linux, provide the outputs of the following commands:

      LANG=C lspci -nn | grep VGA  # might require installing pciutils
      

0000:02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2200] [10de:1c31] (rev a1)

      echo "$DISPLAY"

:1

      LANG=C glxinfo -B | grep -i '\(direct rendering\|opengl\|profile\)'  # might require installing mesa-utils package

direct rendering: Yes
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: Quadro P2200/PCIe/SSE2
OpenGL core profile version string: 4.6.0 NVIDIA 545.23.08
OpenGL core profile shading language version string: 4.60 NVIDIA
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL version string: 4.6.0 NVIDIA 545.23.08
OpenGL shading language version string: 4.60 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 545.23.08
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20

      ps aux | grep Xorg

gdm 1119 0.0 0.2 25384456 67268 tty1 Sl+ Feb13 0:00 /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/125/gdm/Xauthority -background none -noreset -keeptty -verbose 3
username 1879 1.6 0.4 25518904 139304 tty2 Sl+ Feb13 15:37 /usr/lib/xorg/Xorg vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3
username 52408 0.0 0.0 9348 2644 pts/19 S+ 09:36 0:00 grep --color=auto Xorg

      sudo env LANG=C X -version  # if you don't have root access, try to tell the version of Xorg e.g. via package manager
      ```

X.Org X Server 1.20.13
X Protocol Version 11, Revision 0
Build Operating System: linux Ubuntu
Current Operating System: Linux ramses 5.15.0-94-generic #104TILDESIGN 20.04.1-Ubuntu SMP Tue Jan 16 13:34:09 UTC 2024 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-94-generic root=UUID=41b36428-5a36-44f3-af1d-b236b2e37d7d ro quiet splash vt.handoff=7
Build Date: 29 January 2024 12:44:21PM
xorg-server 2:1.20.13-1ubuntu1~20.04.15 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.38.4
Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.

Description

• Expected behavior:

• When I create a camera sensor in the model.sdf file like this:

    <frame name="camera_stereo_reference_frame">
    <pose>0.23 0.046 -0.12 0 1.5708 0</pose> 
    </frame>
  
     <visual name="camera_visual_left">
    <visibility_flags>1</visibility_flags>
    <pose relative_to="camera_stereo_reference_frame"> 0 0 0 0 0 0 </pose>
    <transparency>0.5</transparency>
    <geometry>
      <box>
        <size>0.025 0.025 0.025</size>
      </box>
      <!-- <sphere><radius>0.005</radius></sphere> -->
    </geometry>
  </visual>
  <sensor name="camera_sensor_left" type="camera">
    <pose relative_to="camera_stereo_reference_frame"> 0 0 0 0 0 0 </pose>
    <always_on>1</always_on>
    <update_rate>45</update_rate>
    <topic>camera/image_left</topic>
  
    <camera name="left_camera">
      <camera_info_topic>left_info</camera_info_topic>
      <image>
        <width>640</width>
        <height>480</height>
        <format>R8G8B8</format>
      </image>
  
      <lens>
        <type>stereographic</type>
        <scale_to_hfov>false</scale_to_hfov> 
        <cutoff_angle>3.14</cutoff_angle>
        <env_texture_size>640</env_texture_size>
  
        <intrinsics>
          <fx>256.666691776 </fx> 
          <fy>254.19732909</fy> 
          <cx>353.781709595</cx> 
          <cy>236.287076</cy> 
          <s>0</s>
        </intrinsics>
      </lens>
  
      <distortion>
        <k1>0</k1>
        <k2>0</k2>
        <k3>0</k3>
        <p1>0</p1>
        <p2>0</p2>
        <center>0 0</center>
      </distortion>
  
      <noise>
        <type>gaussian</type>
        <mean>0.0</mean>
        <stddev>0.0</stddev>
      </noise>
  
    </camera>
  </sensor>
  

And I want to create a depth camera at the exact same position with the exact same view to compare my stereo algorithm (that I am performing on the above and a shifted but equal camera) like this:

   <visual name="depth_camera_visual">
    <pose relative_to="camera_reference_frame"> 0 0 0 0 0 0 </pose>
    <visibility_flags>0</visibility_flags>
    <geometry>
      <box>
        <size>0.03 0.03 0.03</size>
      </box>
      <sphere><radius>0.005</radius></sphere>
    </geometry>
    <transparency>0.5</transparency>
  </visual>

  <sensor name="GT" type="depth_camera">
    <pose relative_to="camera_reference_frame"> 0 0 0 0 0 0 </pose>
    <always_on>1</always_on>
    <update_rate>35</update_rate>
    <topic>camera/depth_GT</topic>
    <camera_info_topic>depth_info</camera_info_topic>

    <camera name="GT_depth_camera">
    <camera_info_topic>depth_info</camera_info_topic>
      <horizontal_fov>1.789572604</horizontal_fov>
      
      <depth_camera>
        <camera_info_topic>depth_info</camera_info_topic>
        <clip>
          <near>0.05</near>
          <far>1</far>
        </clip>
      </depth_camera>

      <image>
        <width>640</width>
        <height>480</height>
        <format>L8</format>
      </image>

      <lens>
        <type>stereographic</type>
        <scale_to_hfov>false</scale_to_hfov> 

        <cutoff_angle>3.14</cutoff_angle>
        <env_texture_size>640</env_texture_size> 
        <intrinsics>
          <fx>256.666691776 </fx>
          <fy>254.19732909</fy>
          <cx>353.781709595</cx>
          <cy>236.287076</cy>
        </intrinsics>
      </lens>

      <clip>
        <near>0.05</near>
        <far>1000</far>
      </clip>

      <distortion>
        <k1>0</k1>
        <k2>0</k2>
        <k3>0</k3>
        <p1>0</p1>
        <p2>0</p2>
        <center>0 0</center>
      </distortion>

      <noise>
        <type>gaussian</type>
        <mean>0.0</mean>
        <stddev>0.0</stddev>
      </noise>

    </camera>
  </sensor> 

I get a different coverage of the underground.
**Note ** as the intrinsic parameters of the depth map did not have an influence on the view and only the horizontal_fov tag had an effect I calculated this:

  FoV = 2 × arctan（w / 2f）

Which for a focal length fx = 256.666691776 pixels and an image resolution of 640x480 pixels (aka w = 640 pixels) yields a fov of 102.535° or 1.789572604 radians.

I am assuming that the depth cameras and normal cameras have the same model. They have the exact same pose.

• Actual behavior:
  The Images from the depth camera and the normal camera are covering different areas. I tried to analyze it using visual editors and I am quite sure it is not just a translational shift. There is a different FOV.

Steps to reproduce

1. Put the camera sensor in a model.sdf file and load them into any gazebo map with texture to verify.
2. Publish the sensor messages and look at them in rqt or any other tool

Output

I apologize for the low quality map. It is hard to see but the difference can be seen from the position of the rocks.

In case I missed anything check out my stackoverflow post about this

[iche033]

hmm looks like it could be a bug in depth camera FOV and/or the camera intrinsics published are incorrect.

gazebosim/gz-sensors#363 also looks related

[azeey]

There is code in CameraSensor.cc that handles custom intrinsic parameters: https://github.com/gazebosim/gz-sensors/blob/27239b4ac9877d87273a18010a307172899dd7a6/src/CameraSensor.cc#L319, but I don't see this in DepthCameraSensor.cc. The same is true for RgbdCameraSensor. That might explain things.