Migrate to AWS warehouse simulation scenario #2797

lucabonamini · 2022-02-02T20:04:16Z

Basic Info

Info	Please fill out this column
Ticket(s) this addresses
Primary OS tested on	Ubuntu
Robotic platform tested on	Personal laptop

Description of contribution in a few bullet points

Changed tests simulation scenario from turtlebot world to AWS warehouse

Description of documentation updates required from your changes

Future work that may be required in bullet points

For Maintainers:

Check that any new parameters added are updated in navigation.ros.org
Check that any significant change is added to the migration guide
Check that any new features OR changes to existing behaviors are reflected in the tuning guide
Check that any new functions have Doxygen added
Check that any new features have test coverage
Check that any new plugins is added to the plugins page
If BT Node, Additionally: add to BT's XML index of nodes for groot, BT package's readme table, and BT library lists

…e world

…house world

mergify · 2022-02-02T20:06:03Z

This pull request is in conflict. Could you fix it @lucabonamini?

mergify · 2022-02-02T20:06:04Z

@lucabonamini, please properly fill in PR template in the future. @SteveMacenski, use this instead.

Check that any new parameters added are updated in navigation.ros.org
Check that any significant change is added to the migration guide
Check that any new features OR changes to existing behaviors are reflected in the tuning guide
Check that any new functions have Doxygen added
Check that any new features have test coverage
Check that any new plugins is added to the plugins page
If BT Node, Additionally: add to BT's XML index of nodes for groot, BT package's readme table, and BT library lists

mergify · 2022-02-02T20:12:39Z

@lucabonamini, your PR has failed to build. Please check CI outputs and resolve issues.
You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

…world

…to refactor/aws

mergify · 2022-02-02T20:18:29Z

@lucabonamini, your PR has failed to build. Please check CI outputs and resolve issues.
You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

…rehouse world

…world

mergify · 2022-02-02T20:37:08Z

@lucabonamini, your PR has failed to build. Please check CI outputs and resolve issues.
You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

…world

…use world

… warehouse world

mergify · 2022-02-05T17:09:24Z

@lucabonamini, your PR has failed to build. Please check CI outputs and resolve issues.
You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

SteveMacenski · 2022-03-24T00:13:47Z

@lucabonamini what's the status here?

mergify · 2022-03-24T00:18:21Z

This pull request is in conflict. Could you fix it @lucabonamini?

nav2_system_tests/src/system/CMakeLists.txt

nav2_system_tests/src/waypoint_follower/test_case_py.launch

nav2_system_tests/src/system_failure/test_system_failure_launch.py

nav2_system_tests/src/system/test_wrong_init_pose_launch.py

nav2_system_tests/src/system/test_system_launch.py

nav2_simple_commander/launch/nav_through_poses_example_launch.py

nav2_simple_commander/launch/nav_to_pose_example_launch.py

nav2_simple_commander/launch/picking_demo_launch.py

nav2_simple_commander/launch/security_demo_launch.py

nav2_simple_commander/launch/waypoint_follower_example_launch.py

SteveMacenski · 2022-03-24T22:27:33Z

I don't suppose you have a video of a few of the jobs running / starting positions (particularly the keepout or speed zones one)? That would help to put a graphic to the changes in starting pose / test zones

lucabonamini · 2022-03-24T22:31:07Z

I don't suppose you have a video of a few of the jobs running / starting positions (particularly the keepout or speed zones one)? That would help to put a graphic to the changes in starting pose / test zones

Just tell me which photos / videos you want, I'll do them asap 😃

SteveMacenski · 2022-03-25T01:03:58Z

the keepout zone test and the default nav2_bringup launch file position would be good enough for me!

nav2_simple_commander/launch/inspection_demo_launch.py

mergify · 2022-04-04T21:36:38Z

This pull request is in conflict. Could you fix it @lucabonamini?

…ion_launch.py

AlexeyMerzlyakov · 2022-05-23T09:59:51Z

If I commit the roll-back version, could you or @SteveMacenski give it a try on your laptop?

I've checked it today locally. That is also does not work for me with the same assertion px != 0 failed issue when using latest refactor/aws branch and works fine from main one. I've started to compare what else was happened between main and development branch and found out that for the problem fails it was responsible GAZEBO_RESOURCE_PATH environment variable. Setting it to non-empty will cause the problem even on main branch:

[leha@leha-PC nav2_ws]$ export GAZEBO_RESOURCE_PATH=abracadabra
[leha@leha-PC nav2_ws]$ ros2 launch nav2_bringup tb3_simulation_launch.py headless:=False 2>&1 | tee run.log
...
[gzserver-1] gzserver: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = gazebo::rendering::Scene; typename boost::detail::sp_member_access<T>::type = gazebo::rendering::Scene*]: Assertion `px != 0' failed.
[ERROR] [gzserver-1]: process has died [pid 102349, exit code -6, cmd 'gzserver -s libgazebo_ros_init.so -s libgazebo_ros_factory.so /home/leha/nav2_ws/install/nav2_bringup/share/nav2_bringup/worlds/world_only.model'].
[gzclient-2] gzclient: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = gazebo::rendering::Camera; typename boost::detail::sp_member_access<T>::type = gazebo::rendering::Camera*]: Assertion `px != 0' failed.
[ERROR] [gzclient-2]: process has died [pid 102351, exit code -6, cmd 'gzclient'].

Could we live without this environment variable in Nav2 launch-scripts?

AlexeyMerzlyakov · 2022-05-23T10:39:37Z

I am also have one note appeared when locally launched the TB3 simulation in AWS:

Also I forgot this space is somewhat large, please increase the maximum range on the lidar from whatever it is now to like 20m so we have visibility. The toy TB3 lidar isn't long enough range for this size of open space to stay even remotely localized

The maximum laserscan range now changed from 3.5m -> to 20m. I think, we need to reflect these changes in the ROS2 params, such as the size of local costmap (width and height params) and raytrace_max_range/obstacle_max_range for laser scanner in both local and global costmaps. I would increase them by 2x-3x times comparing to original values. @SteveMacenski, did I forget something here (like laser_likelihood_max_dist for AMCL where I still doubting: according to docs it seems this value should be untouched as it is related to AMCL inflation algorithms internals and not dependent on maximum laserscan range)?

AlexeyMerzlyakov · 2022-05-23T10:56:02Z

And, the latest comment: @lucabonamini, please do a rebase of your changes over latest main branch, in order to avoid merges and "intertwined" non-linear history in main.

SteveMacenski · 2022-05-23T19:38:04Z

The max range is already set to 100 https://github.com/ros-planning/navigation2/blob/main/nav2_bringup/params/nav2_params.yaml#L17 so I think it should be OK. If you change it from 3.5->20 and you see that the robot isn't getting totally lost when moving from the starting position across the map anymore, then its fixed and I think you're fine. If its still being weird, yeah we should play with it more, but no, we should not adjust the costmaps.

We broadly need to do a cleanup of parameters and plugins in the default bringups, so we can keep that in mind, but I figure we do that once we have either (1) MPPI in the stack or (2) we swap out robots to another platform.

Do all the tests use that same nav2_bringup/worlds/waffle.model? If not, we should also update it to 20 in all the system test copies of the TB3 model

From current state of branch I get the following while just trying to load the world

[gzserver-1] gzserver: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = gazebo::rendering::Scene; typename boost::detail::sp_member_access<T>::type = gazebo::rendering::Scene*]: Assertion `px != 0' failed.
[gzclient-2] gzclient: /usr/include/boost/smart_ptr/shared_ptr.hpp:734: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = gazebo::rendering::Camera; typename boost::detail::sp_member_access<T>::type = gazebo::rendering::Camera*]: Assertion `px != 0' failed.

AlexeyMerzlyakov · 2022-05-24T11:32:00Z

From current state of branch I get the following while just trying to load the world

This is related to #2797 (comment). Currently refactor/aws branch still has GAZEBO_RESOURCE_PATH variable set, so the problem most likely to appear:

    model, plugin, media = GazeboRosPaths.get_paths()
...
    if 'GAZEBO_RESOURCE_PATH' in environ:
        media += pathsep+environ['GAZEBO_RESOURCE_PATH']
...
    env = {'GAZEBO_MODEL_PATH': model,
           'GAZEBO_PLUGIN_PATH': plugin,
           'GAZEBO_RESOURCE_PATH': media}
...
    start_gazebo_server_cmd = ExecuteProcess(
        condition=IfCondition(use_simulator),
        cmd=['gzserver', '-s', 'libgazebo_ros_init.so',
             '-s', 'libgazebo_ros_factory.so', world],
        additional_env=env,
        cwd=[aws_dir], output='screen')

    start_gazebo_client_cmd = ExecuteProcess(
        condition=IfCondition(PythonExpression(
            [use_simulator, ' and not ', headless])),
        cmd=['gzclient'],
        additional_env=env,
        cwd=[aws_dir], output='screen')

When removing this variable from environments, the problem vanished on my side.

Update: the problem is that we set env['GAZEBO_RESOURCE_PATH'] to media variable in any case, even if this variable is not in the environment. So, it is needed to be assigned inside if-statement to fix. Something like:

    if 'GAZEBO_MODEL_PATH' in environ:
        model += pathsep+environ['GAZEBO_MODEL_PATH']
        env['GAZEBO_MODEL_PATH'] = model
    if 'GAZEBO_PLUGIN_PATH' in environ:
        plugin += pathsep+environ['GAZEBO_PLUGIN_PATH']
        env['GAZEBO_PLUGIN_PATH'] = plugin
    if 'GAZEBO_RESOURCE_PATH' in environ:
        media += pathsep+environ['GAZEBO_RESOURCE_PATH']
        env['GAZEBO_RESOURCE_PATH'] = media

SteveMacenski · 2022-05-24T19:10:21Z

Sounds reasonable!

SteveMacenski · 2022-05-26T22:04:44Z

So just that change and this should be good to go?

Crazy irritating, but ROS 2 depreciates Gazebo and we have to move to Ignition either sooner (if we can't get gazebo ros plugins released) or later (if we can, but sometime this year). So I've filed a ticket aws-robotics/aws-robomaker-small-warehouse-world#22 about getting the AWS warehouse world into Ignition so that after this PR is merged, the system tests work remains the same, but we would change the gz calls to ign calls and call it a day.

What's somewhat in question though is if we should move to using a ignition provided warehouse world 1 or 2 rather than the AWS warehouse world. I rather like the AWS warehouse world and I really appreciate all the time you've put into this and want that to be reflected in the stack. But curious if you had thoughts on it and if you think one of these worlds might be better for some reason. I don't think its worth redoing all this work to move to a slightly different warehouse if we can get the AWS world in Ignition, you'd done enough and I want to get this work in.

mergify · 2022-05-27T07:08:18Z

@lucabonamini, your PR has failed to build. Please check CI outputs and resolve issues.
You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify · 2022-05-27T11:55:48Z

@lucabonamini, your PR has failed to build. Please check CI outputs and resolve issues.
You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

lucabonamini · 2022-05-27T11:55:49Z

So just that change and this should be good to go?

I pushed the change @AlexeyMerzlyakov suggested, but at the moment i can't test it locally as i am having a lot of problems with gazebo.

Crazy irritating, but ROS 2 depreciates Gazebo and we have to move to Ignition either sooner (if we can't get gazebo ros plugins released) or later (if we can, but sometime this year). So I've filed a ticket aws-robotics/aws-robomaker-small-warehouse-world#22 about getting the AWS warehouse world into Ignition so that after this PR is merged, the system tests work remains the same, but we would change the gz calls to ign calls and call it a day.

What's somewhat in question though is if we should move to using a ignition provided warehouse world 1 or 2 rather than the AWS warehouse world. I rather like the AWS warehouse world and I really appreciate all the time you've put into this and want that to be reflected in the stack. But curious if you had thoughts on it and if you think one of these worlds might be better for some reason. I don't think its worth redoing all this work to move to a slightly different warehouse if we can get the AWS world in Ignition, you'd done enough and I want to get this work in.

I've checked Ignition warehouse worlds, and they seem very similar to the one provided by AWS. Currently I cannot estimate the effort to migrate to these new worlds, but I think the best choice is closely linked to which of the worlds will be actively supported in the future.

As you said, migrate AWS world in Ignition would be the best choice right now 😄

lucabonamini · 2022-05-27T12:03:20Z

@SteveMacenski I think i will have to restart from a clean install of Ubuntu and ROS to fix the problems I'm facing.

SteveMacenski · 2022-05-27T18:20:12Z

OK! I think our plan is to stick with the AWS world and have that migrated to Ignition. I don't want to redo your work and you've done so much here, I want your hard work reflected. And as the project leader, I can just make that executive decision 😆

mergify · 2022-05-27T18:57:08Z

This pull request is in conflict. Could you fix it @lucabonamini?

mergify · 2022-06-01T14:21:29Z

@lucabonamini, your PR has failed to build. Please check CI outputs and resolve issues.
You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

lucabonamini · 2022-06-01T14:26:36Z

@SteveMacenski I've tested last changes with a fresh ROS installation, and I was able to successfully launch tb3_simulation_launch.py with no need for sourcing /usr/share/gazebo/setup.bash.

Now CI is failing due to some problem related to aws-robomaker-small-warehouse-world rolling debian.

SteveMacenski · 2022-06-01T18:58:00Z

sigh I'm not sure. It appears that it hasn't been released to 22.04 because the release bin jobs are disabled in Rolling. It also has listed dependencies on gazebo which also doesn't exist in 22.04 so that's an issue regardless

Unfortunately since you pulled in updated amcl code for 22.04, it broke my install on 20.04, I can no longer test this until I can get my work machine updated to 22.04 (which then wouldn't work because AWS warehouse doesn't exist -- so this PR is in a bit of a catch 22, I'd revert the 22.04 AMCL stuff so this still works on 20.04 for the time being).

This gazebo / 22.04 issue is kill'n me

lucabonamini · 2022-06-01T19:24:54Z

sigh I'm not sure. It appears that it hasn't been released to 22.04 because the release bin jobs are disabled in Rolling. It also has listed dependencies on gazebo which also doesn't exist in 22.04 so that's an issue regardless

Unfortunately since you pulled in updated amcl code for 22.04, it broke my install on 20.04, I can no longer test this until I can get my work machine updated to 22.04 (which then wouldn't work because AWS warehouse doesn't exist -- so this PR is in a bit of a catch 22, I'd revert the 22.04 AMCL stuff so this still works on 20.04 for the time being).

Maybe should I cherry-pick my last commit and push to main branch right before the changes in amcl? Although the issue with AWS warehouse remains.

This gazebo / 22.04 issue is kill'n me

How can I help?

SteveMacenski · 2022-06-01T23:26:14Z

Wait - gazebo in 22.04 should be working actually from discussions on another thread, I'm not sure why then the aws warehouse world isn't released / handled.

SteveMacenski · 2022-06-23T20:42:08Z

@nuclearsandwich released AWS Warehouse world for Humble/Rolling so this should turn over in the next sync!

@lucabonamini @AlexeyMerzlyakov is this ready to merge otherwise?

lucabonamini · 2022-07-07T16:39:54Z

Any updates? @SteveMacenski

SteveMacenski · 2022-07-07T19:29:26Z

Still no update yet for Rolling / Humble, so still waiting...

mergify · 2023-03-20T17:51:54Z

This pull request is in conflict. Could you fix it @lucabonamini?

mergify · 2023-10-01T22:25:56Z

This pull request is in conflict. Could you fix it @lucabonamini?

lucabonamini added 5 commits January 25, 2022 22:31

Moving nav2_bringup to AWS warehouse simulation scenario

15d4aa8

Moving nav2_simple_commander to AWS warehouse simulation scenario

bdb933b

fix(test_behavior_tree_node): changed include order to fix flake8 error

5b18241

refactor(nav2_system_tests/src/planning): moved to AWS small warehous…

641e1c5

…e world

refactor(nav2_system_tests/src/localization): moved to AWS small ware…

a60017b

…house world

Merge branch 'main' into refactor/aws

1c24b6d

lucabonamini added 2 commits February 2, 2022 21:15

refactor(nav2_system_tests/src/system): moved to AWS small warehouse …

cd1adde

…world

Merge branch 'refactor/aws' of github.com:lucabonamini/navigation2 in…

ccc295a

…to refactor/aws

lucabonamini added 2 commits February 2, 2022 21:29

refactor(nav2_system_tests/src/system_failure): moved to AWS small wa…

801a0e3

…rehouse world

refactor(nav2_system_tests/src/updown): moved to AWS small warehouse …

715b0c7

…world

lucabonamini added 3 commits February 5, 2022 17:59

refactor(nav2_system_tests/src/updown): moved to AWS small warehouse …

0482c8d

…world

refactor(nav2_system_tests/src/recoveries): moved to AWS small wareho…

53e84d9

…use world

refactor(nav2_system_tests/src/waypoint_follower): moved to AWS small…

7f634a3

… warehouse world

lucabonamini marked this pull request as ready for review February 5, 2022 17:03

Merge branch 'main' into refactor/aws

b353c01

SteveMacenski reviewed Mar 24, 2022

View reviewed changes

sathak93 reviewed Mar 26, 2022

View reviewed changes

nav2_simple_commander/launch/inspection_demo_launch.py Show resolved Hide resolved

SteveMacenski mentioned this pull request Apr 6, 2022

Replace TB3 + Sandbox world in CI for more realistic robot use-cases & world #1933

Closed

lucabonamini added 2 commits May 21, 2022 19:46

Merge branch 'main' into refactor/aws

68b3cc9

Roll back to direct execution of gzserver and gzclient in tb3_simulat…

16d7698

…ion_launch.py

Merge branch 'main' into refactor/aws

8df9acd

Change set up GAZEBO env variables

30888a5

Merge branch 'main' into refactor/aws

9f417d6

SteveMacenski mentioned this pull request Jun 1, 2022

Why are Rolling Binaries Disabled? aws-robotics/aws-robomaker-small-warehouse-world#23

Closed

SteveMacenski mentioned this pull request Jun 2, 2022

Migrate to Ignition / TB4 simulation #2997

Closed

5 tasks

Migrate to AWS warehouse simulation scenario #2797

Are you sure you want to change the base?

Migrate to AWS warehouse simulation scenario #2797

Conversation

lucabonamini commented Feb 2, 2022

Basic Info

Description of contribution in a few bullet points

Description of documentation updates required from your changes

Future work that may be required in bullet points

For Maintainers:

mergify bot commented Feb 2, 2022

mergify bot commented Feb 2, 2022

mergify bot commented Feb 2, 2022

mergify bot commented Feb 2, 2022

mergify bot commented Feb 2, 2022

mergify bot commented Feb 5, 2022

SteveMacenski commented Mar 24, 2022

mergify bot commented Mar 24, 2022

SteveMacenski commented Mar 24, 2022

lucabonamini commented Mar 24, 2022 • edited

SteveMacenski commented Mar 25, 2022

mergify bot commented Apr 4, 2022

AlexeyMerzlyakov commented May 23, 2022

AlexeyMerzlyakov commented May 23, 2022 • edited

AlexeyMerzlyakov commented May 23, 2022 • edited

SteveMacenski commented May 23, 2022 • edited

AlexeyMerzlyakov commented May 24, 2022 • edited

SteveMacenski commented May 24, 2022

SteveMacenski commented May 26, 2022 • edited

mergify bot commented May 27, 2022

mergify bot commented May 27, 2022

lucabonamini commented May 27, 2022 • edited

lucabonamini commented May 27, 2022

SteveMacenski commented May 27, 2022

mergify bot commented May 27, 2022

mergify bot commented Jun 1, 2022

lucabonamini commented Jun 1, 2022

SteveMacenski commented Jun 1, 2022 • edited

lucabonamini commented Jun 1, 2022 • edited

SteveMacenski commented Jun 1, 2022

SteveMacenski commented Jun 23, 2022 • edited

lucabonamini commented Jul 7, 2022

SteveMacenski commented Jul 7, 2022

mergify bot commented Mar 20, 2023

mergify bot commented Oct 1, 2023

lucabonamini commented Mar 24, 2022 •

edited

AlexeyMerzlyakov commented May 23, 2022 •

edited

AlexeyMerzlyakov commented May 23, 2022 •

edited

SteveMacenski commented May 23, 2022 •

edited

AlexeyMerzlyakov commented May 24, 2022 •

edited

SteveMacenski commented May 26, 2022 •

edited

lucabonamini commented May 27, 2022 •

edited

SteveMacenski commented Jun 1, 2022 •

edited

lucabonamini commented Jun 1, 2022 •

edited

SteveMacenski commented Jun 23, 2022 •

edited