load_map_and_localize not work on non-horizontal surfaces

[tuch77731]

camera: realsense d435i
isaac_ros_visual_slam version: 2.1
jetson orin nx

I set up the camera and looked at the ceiling
base_link to camera_link tf: 0, 0, 0 , 1.57, -0.785, 0 (xyz, rpy)

parameters:
force_planar_mode: False
base_frame: base_link
input_base_frame: base_link

After I run for a while, I saved map. Next, I load_map_and_localize that Input is localize_near_point. I found some issue.

First, the coordinate of localize_near_point is base on camera_link not on base_link. For example, when robot is on (5, 0, 0), I set (5,0,0) is always False , and It's work I must set (3.5, 3.5, 0). It would be more intuitive if base_link could be set.

Second, localize_near_point is not work when camera_link is not horizontal to the initial point. It's work about -2<z<2 and rx,ry<0.1. Other situations are failures. For example, robot was on (0, 5 ,0), so that camera_link was on (0, 0, -5). The localize_near_point was always failure when I set (0, 5, 0) or set (0, 0, -5) or other value.
The other example, I localize_near_point success when my robot was on xyz(0, 0, 0) and rpy(0, 0, 0). Next i spin my robot to xyz(0,0,0) and rpy(0, 0, 1.57)(Camera_link will get some values of rx, ry) . The action localize_near_point is always failure when i use localize_near_point (0,0,0). The reason i think is I cannot specify the initial direction when I set localize_near_point.

[swapnesh-wani-nvidia]

The pose for LoadAndLocalize action is aligned with the base_link frame (x - forward, y - left, and z - up). See the tutorial here.

I don't understand how your camera and base_link is attached. If your robot is at 5 m in the Y direction in map frame how can camera be at -5 in Z axis?

For example, robot was on (0, 5 ,0), so that camera_link was on (0, 0, -5).

Also, for the LoadAndLocalize to work, if the current features from the viewing angle don't match the stored landmarks, it will fail.

[tuch77731]

I show my experiment.
Everything is fine when the camera is mounted horizontally
However, when installed at an angle, it will be unable to be LoadAndLocalize in certain positions.

thank you

[Seongoo]

I have the same issue regarding the third experiment to localize my robot using a RealSense camera on a mapped surface parallel to the xy-plane. If I set my robot at the position (0, 0, 0) and try to localize the robot facing towards the starting point, where the robot looked at when I started mapping, localization was done successfully by giving a localize_near_point (0, 0, 0). However, localization always failed by giving the localize_near_point (0, 0, 0) after only rotating my robot over ±90 degree, even though the transition did not change at all. Would adding a rough robot's orientation for localization solve such an issue?