Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User misconfiguration of parameters may cause instantaneous crashs #4005

Open
GoesM opened this issue Dec 11, 2023 · 8 comments
Open

User misconfiguration of parameters may cause instantaneous crashs #4005

GoesM opened this issue Dec 11, 2023 · 8 comments
Labels
3 - Low Low Priority

Comments

@GoesM
Copy link
Contributor

GoesM commented Dec 11, 2023

Bug report

Required Info:

  • Operating System:
    • Ubuntu 22.04
  • ROS2 Version:
    • humble
  • Version or commit hash:
  • DDS implementation:
    • default

Steps to reproduce issue

launch the navigation2 within defaulted tb3_simulation_launch.py as following commands

source install/setup.bash
source /opt/ros/humble/setup.bash
export TURTLEBOT3_MODEL=waffle
export GAZEBO_MODEL_PATH=$GAZEBO_MODEL_PATH:/opt/ros/humble/share/turtlebot3_gazebo/models
ros2 launch nav2_bringup tb3_simulation_launch.py params_file:=my_nav2_params.yaml

there's only one difference between my_nav2_params.yaml and defaulted nav2_params.yaml, (we wrongly set the height of local_costmap as -20), as following:

...
...
local_costmap:
  local_costmap:
    ros__parameters:
      ...
      ...
      height: -20
...
...

and then , try to set an /initial_pose in rviz by 2D_estimate_pose or send a msg into /initial_pose in another terminal

Expected behavior

  • process should not crash / at least, it should not make the user's computer unable to continue working
  • the memory usage should not be too high

Actual behavior

  • process crashed when launching
    specifically, during the startup process of nav2, there may be computer lag, terminal crashes, or even a blue screen
    dbd3b31c83f94342cc9af850a9a897df_

  • abnormal memory usage
    we found that nav2_controller process occupies a very large amount of memory (up to 10G, sometimes even exceeding 20G, which is very abnormal), as following:
    db3537365059f4ae705cd6d092537db
    here's a normal memory usage when we use the defaulted nav2_params.yaml, which is for comparision.
    normal
    the max memory usage of whole nav2 is below 1G.

@SteveMacenski
Copy link
Member

Thanks for the report - in general its not planned for us to protect users from themselves with every parameter if they choose something impossible. We try where we can, but the codebase would balloon to 2x the size for checking every single parameter for every way that it could be misinterpreted.

I don't disagree with you that you could make the system crash on initialization if you set the width/height to a negative value. I put that in the category of "fair" since its not a crash mid-execution, non-deterministically, or during operations. That is a crash you would see immediately when you tried to bring the system up, every time, that could be debugged before proceeding. I take crashes very seriously, but for parameter misconfiguration of this nature, its not realistic today that we're going to protect users from making errors on every one of a thousand parameters.

However, that's not an unreasonable ask, so I'm going to rename the ticket in line with that and it'll stay in the tracker so that perhaps in the future when we're better resourced we can potentially have some interns do this, but no promises / commitments.

@SteveMacenski SteveMacenski changed the title user-misconfiuration may cause system memory crash User misconfiguration of parameters may cause instantaneous crashs Dec 11, 2023
@SteveMacenski
Copy link
Member

@GoesM perhaps your extra memory problems are related to your misconfigurations 😉

Ask on Robotics stack exchange with your config file changes and I'll follow up there on that topic. I don't want tickets to become > 1 topic, so I don't want to pollute this thread with that discussion. I hope that makes sense

@SteveMacenski SteveMacenski added the 3 - Low Low Priority label Dec 11, 2023
@GoesM
Copy link
Contributor Author

GoesM commented Dec 12, 2023

I've forwarded this issue to the Robotics stack exchange stack: question 106008.
Thanks for your feedback. 🙂

@Ryanf55
Copy link
Contributor

Ryanf55 commented Feb 2, 2024

Keep an eye on this thread for potential solutions to add parameter validation without ballooning the code.
https://discourse.ros.org/t/simplifying-how-to-declare-parameters-in-ros-2/33272

@SteveMacenski
Copy link
Member

SteveMacenski commented Feb 2, 2024

It is already possible with parameter descriptions. It is more a matter of time / inclination to set bounds on all N params in the stack (and make sure the dynamic parameter callbacks are respecting them too). The blockage isn’t the tech, it’s the value proposition of time.

You should see the amount of code is required per-parameter in this case. Certainly some can be abstracted into Utils, but still.

@SteveMacenski
Copy link
Member

SteveMacenski commented Feb 29, 2024

Another example #4157

& impl note: #4157 (comment)

@GoesM
Copy link
Contributor Author

GoesM commented May 13, 2024

add tickets:

@GoesM
Copy link
Contributor Author

GoesM commented May 13, 2024

[NOTICE]

I think here're two issue which needs to be focused on:

they show a result that the API get_parameter() may read a wrong value sometimes (different from the value in nav2_params.yaml )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Low Low Priority
Projects
None yet
Development

No branches or pull requests

3 participants