ROS2 Iron pub/sub performance - Python EXTREMELY slow

[jhrncar]

Bug report

Required Info:

• Operating System:
  Ubuntu Server 22.0.4
• Installation type:
  Debian packages
• Version or commit hash:
  current Iron
• DDS implementation:
  default
• Client library (if applicable):
  • rclpy

Steps to reproduce issue

Quite easily, just follow the tutorial for simple pub/sub https://docs.ros.org/en/iron/Tutorials/Beginner-Client-Libraries/Writing-A-Simple-Py-Service-And-Client.html and change the timer to for example 0.001 = 1000 Hz and open htop. I will post the screens from htop here but TLDR: Python has 100% usage and lagging, C++ has like 5% usage

C++:

Python:

Sorry if I am reading the htop incorrectly, but at first glance, the numbers seem kinda obvious :(

I have stumpled upon issues that were already opened on this matter, for example ros2/rclpy#763 but it is 2 years old, for Foxy and closes with fixes...

#### Expected behavior
At least similar performance

#### Actual behavior
Python code is EXTREMELY slow

#### Additional information
If this is known and is not a bug, then I would like to ask if this can be put on the doc to inform new users such as myself

[fujitatomoya]

@jhrncar thanks for posting issue. there has been discussion related to rclpy (python client library) performance before, it would be really nice to have the performance improvement but i do not know if anyone is working on that.

[jhrncar]

@jhrncar thanks for posting issue. there has been discussion related to rclpy (python client library) performance before, it would be really nice to have the performance improvement but i do not know if anyone is working on that.

Hi, thanks for a quick reply. So, if I understand this correctly, this issue is well-known? If yes, could it please be possible to add this warning to the documentation? The performance is so dramatically different that I think the warning is warranted - in my use case, I wasted a good amount of time making the architecture for my project, only to discover that when I increased the refresh rate, it became unusable. I would like others to not make my mistakes

[clalancette]

If yes, could it please be possible to add this warning to the documentation?

I don't think we should add a warning to the documentation. First of all, it is not even clear where we would add it so that people would see it.

But more importantly, whether rclpy is fast enough for particular use cases is entirely application specific. We have 100s of packages that use rclpy successfully. Is it slower? No doubt. But whether that slowness is OK depends.

Finally, what I'd much rather see here is someone spend the time to find out why it is slow, and propose solutions to that. We depend on the community to find and fix problems, and this seems like something that would benefit everyone.

[jhrncar]

If yes, could it please be possible to add this warning to the documentation?

I don't think we should add a warning to the documentation. First of all, it is not even clear where we would add it so that people would see it.

But more importantly, whether rclpy is fast enough for particular use cases is entirely application specific. We have 100s of packages that use rclpy successfully. Is it slower? No doubt. But whether that slowness is OK depends.

Finally, what I'd much rather see here is someone spend the time to find out why it is slow, and propose solutions to that. We depend on the community to find and fix problems, and this seems like something that would benefit everyone.

Hello, sorry for any confusion. I am not saying a fix should not be worked on. I am just suggesting to add a disclaimer for new users to the doc that Python library is dramatically slower than its C++ counterpart - this would be a stop-gap measure while a fix is not implemented. If you are open to the idea, I can go over the documentation and find a good place for it and suggest it here.
As for the middle paragrapgh... i am sure that packages run just fine and developers optimized their applications around it. But, I also think that a refresh rate of 100Hz in terms of robotics and automation - which is the primary use case - is not unreasonable, For example, we are creating an RC car that uses UDP sockets for communication with the controller. The controller has refresh rate of 125Hz and as the RC car can get to speeds of 50 km/h (we are creating something similar to F1Tenth project https://f1tenth.org/), the controls need to be responsive and 100Hz is a number that we landed upon in tests.
Thanks for any replies btw!

[fujitatomoya]

thank you for sharing thoughts.

IMO, if i work on embedded system, i would not even use shell script, instead compile them into executables to keep the performance. (python would be just out of the option, i probably do not even consider...) anyway it depends on use case and situation (or constraints) including platform resources.

doc that Python library is dramatically slower than its C++ counterpart

i think this is really general information as language (dis)advantages? i am not inclined to note this as warning in ROS 2 documentation.

but it is true that there has been discussions about rclpy performance, so i am more interested in technical analysis, another solution and optimization ideas.

thanks,

[gavanderhoorn]

A notice letting prospective/new rclpy users know there is a known issue with performance which is being investigated seems like it would be low cost to add while saving developers like @jhrncar potentially quite some time and/or frustration.

Does it have to be an either/or here? A notice doesn't prevent anyone from working on it, would it?

[fujitatomoya]

i do not think it has to be exclusive, i am happy to review. but adding like rclpy is slower than rclcpp would generate other questions, what is the appropriate platform resource, how slower and so on? and that is gonna be dependent for the use case, i guess. besides, there are packages dependent on rclpy...

i am not sure where and how to describe this document for user. any recommendation or suggestions?

[sloretz]

@clalancette assigned to you to collect and close related issues as duplicates of this one 🧇

[RocketFramework]

I conducted testing on a ROS 2 (IRON) setup running Ubuntu Linux - Jammy (22.04) 64-bit, and my observations indicate that rclpy does not exhibit a significant slowdown. In the course of multiple tests, I did encounter a marginal increase in average load time during one rare and random instance. However, it's crucial to note that various factors may have contributed to this, including background processes, ongoing resource and memory cleanup, or the influence of other dependencies.

While it is challenging to unequivocally disprove the reported observation, it is evident that rclpy is generally not characterized by substantial slowness. The occasional variation in load times can often be attributed to the dynamic and complex nature of the system, with numerous elements influencing performance.

In conclusion, based on my testing, rclpy can be considered as performing reasonably well, and any observed fluctuations in load times are likely influenced by external factors rather than inherent issues with rclpy itself.

Recommend: Close the issue

[jhrncar]

I conducted testing on a ROS 2 (IRON) setup running Ubuntu Linux - Jammy (22.04) 64-bit, and my observations indicate that rclpy does not exhibit a significant slowdown. In the course of multiple tests, I did encounter a marginal increase in average load time during one rare and random instance. However, it's crucial to note that various factors may have contributed to this, including background processes, ongoing resource and memory cleanup, or the influence of other dependencies.

While it is challenging to unequivocally disprove the reported observation, it is evident that rclpy is generally not characterized by substantial slowness. The occasional variation in load times can often be attributed to the dynamic and complex nature of the system, with numerous elements influencing performance.

In conclusion, based on my testing, rclpy can be considered as performing reasonably well, and any observed fluctuations in load times are likely influenced by external factors rather than inherent issues with rclpy itself.

Recommend: Close the issue

Hello, thanks for the research you conducted. However, I strongly disagree with your conclusion. If what you are saying is right, then how is it possible, that the easiest and most simple example found on the doc - basic publisher and subscriber, if scaled to large numbers of refresh rate, does indeed perform EXTREMELY badly in python compared to C++?
The device running the test did not run anything else, the test conditions were completely the same and I was able to reproduce the same result over and over again.

[jhrncar]

i do not think it has to be exclusive, i am happy to review. but adding like rclpy is slower than rclcpp would generate other questions, what is the appropriate platform resource, how slower and so on? and that is gonna be dependent for the use case, i guess. besides, there are packages dependent on rclpy...

i am not sure where and how to describe this document for user. any recommendation or suggestions?

Hello, I am sorry I did not respond sooner, tbh I forgot. Yes, I think I am able to write down a proposal for the disclaimer that will suffice. However, of course there will be follow-up questions that will arise from this. I dont think it is a valid point to not act in any way because of this though. People need to be informed that to prevent endless frustration on this issue.

[ralbra]

So I had a look at the problem you had and did some test myself on an atom (Apollo Lake) quadcore box.

I also took the basic sub/pub tutorial for python and let it run with different publishing rates.(you linked the wrong tutorial)

With 2 and 10 Hz there was no noticeable load, with 100 Hz I was at 50 % load.
At 1000hz I was at nearly 100% load but the publish rate was not nearly at that level.

Then I deleted the log output of the publisher and voila the 1000hz ran at ~75% load with a little headroom.

That's when I thought to hell with it and deleted the timer and ran the timer_callback in an while loop.
This resulted in a publishing rate of roughly 10Khz which is actually quiet impressive for that little thing.
I suspect that the handover of the logger is the limiting factor here.

Bottom line is here, yeah python will be slower as C++ but if you avoid things like excessively writing logs and avoid constructs that are prone to eat cpu time like timers, sleeps, unnecessarily construct new objects etc., the impact shouldn't be that bad.

And yeah if you need the fastest and most efficient program don't use vanilla python.