Using lidar sensor in orbit

[LukasVindbjerg]

Hi,

I have a robot that uses a 2D lidar sensor. I have added the RTX Lidar sensor to the robot inside Isaac Sim. When I run the simulation from orbit I can see the dots created by the lidar sensors, indicating that it is working.

However, I'm not sure how to access the data generated by the lidar sensors from orbit. From the documentation it reads:
"USD-prim sensors: Available in Omniverse and require creating a USD prim for them. For instance, RTX ray tracing camera and lidar sensors."

I'm not sure if this means that I can access the lidar sensor I created in Isaac Sim from its prim path, or if I have to spawn a new one from the orbit API. Can someone clarify the workflow for accessing data from lidar sensors in orbit?

[Mayankm96]

Hi @LukasVindbjerg ,

We had planned to add support for RTX lidar but since it wasn't parallelized, we decided to remove it in favor of our custom wrap-based ray-caster. However, there are certain benefits to the RTX implementation (such as surface reflectivity and realism beyond physical meshes).

At this point, the feature is not in our immediate development plans. However, it would be great to support this feature. As a starting point, you can make the sensor class similar to how Camera works in Orbit. If you need more guidance, please do not hesitate to ask.

[Mayankm96]

Probably need to provide --offscreen_render flag to allow rendering when in headless mode.

https://isaac-orbit.github.io/orbit/source/tutorials/00_sim/launch_app.html#understanding-the-output-of-help

[LukasVindbjerg]

Thanks @Mayankm96, that almost solved the issue. However, now the lidar measurements seem to be updated only in the first frame, and then they have the same value for the rest of the training when I use --offscreen_render. E.g.:

lidar ranges: tensor([[30.0000, 30.0000, 30.0000,  ..., 30.0000, 17.6805, 30.0000],
        [17.5025, 30.0000, 17.6963,  ..., 30.0000, 17.6805, 30.0000],
        [17.5025, 30.0000, 17.6963,  ..., 30.0000, 17.6805, 30.0000],
        ...,
        [30.0000, 30.0000, 30.0000,  ..., 30.0000, 30.0000, 30.0000],
        [30.0000, 30.0000, 30.0000,  ..., 30.0000, 30.0000, 30.0000],
        [30.0000, 30.0000, 30.0000,  ..., 30.0000, 30.0000, 30.0000]],
       device='cuda:0')

It works as expected when I run on non-headless mode, e.g.:

lidar ranges: tensor([[0.6031, 0.5970, 0.5911,  ..., 1.4989, 1.5016, 1.5043],
        [0.3798, 0.3791, 0.3783,  ..., 2.9556, 2.9827, 3.0104],
        [0.3971, 0.3975, 0.3979,  ..., 1.6996, 1.6691, 1.6390],
        [0.3195, 0.3188, 0.3181,  ..., 2.9194, 2.9439, 2.9689]],
       device='cuda:0')

I'm not sure if this is due to an issue with my implementation of the lidar class. Have you encountered similar issues with the --offscreen_render before?

[LukasVindbjerg]

It works when I use the combination of --headless, --offscreen_render, and --video. However, this is quite expensive in terms of memory, so I would really like to find a solution that avoids using the --video flag.

[Mayankm96]

You need to call env.render() to allow rendering of sensor data. I noticed that we right now only call it implicitly in the step call when GUI is enabled. When you use RSL-RL or RecordVideo wrapper, they internally call the render after step so you see it as working.

https://github.com/NVIDIA-Omniverse/orbit/blob/main/source/extensions/omni.isaac.orbit/omni/isaac/orbit/envs/rl_task_env.py#L175-L177

But I'm not happy with this arrangement. Would be nice to have a general way to "call" render when a sensor needs it.

[LukasVindbjerg]

Thank you very much for your help! I changed the line to:

if self.sim.has_gui() or self.sim._offscreen_render:
    self.sim.render()

Probably not the most ideal solution - but in my case I can now simulate multiple environments with lidar, which was the goal.

While the code is far from polished I'll leave a link to my fork here in case anyone else is interested in doing something similar with lidar sensors:
https://github.com/Lukas-Malthe-MSc/orbit/tree/f1tenth-environment

[Mayankm96]

Nice! Would you like to contribute the Lidar sensor implementation to Orbit? @pascal-roth for vis.

[LukasVindbjerg]

Yes absolutely. However, right now the code has just been implemented to suit my own needs and could use a bit more work put into it. Also, I recently tried to pull the changes from the main branch into my development branch - which unfortunately seemed to break some of the Isaac Sim APIs that I use.

I'd be happy to try and fix the lidar implementation. But right now other parts of my project are a higher priority for me.

[Mayankm96]

Just for an update. We have fixed the behavior in this commit: 7ac5588