rerun.io support: new log visualization tool

[adeebshihadeh]

https://www.rerun.io/ is a new robotics visualization project. It's beautiful, performant, and seems to have everything we need for debugging openpilot routes.

@savojovic got a proof of concept merged at tools/rerun. Try the demo out with: tools/rerun/run.py --demo.

PlotJuggler has been great, but it doesn't include the camera views, text logs, etc. Rerun should entirely replace PlotJuggler in our tools stack.

This is the thread for posting feedback, bug reports, and feature requests.

[polooner]

i can't get the demo to run, not sure where to go with these errors. any idea how to fix?

Getting route log paths
  0%|                                                                                                           | 0/13 [00:00<?, ?it/s][2024-05-20T15:33:11Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:11Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:11Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:11Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:12Z WARN  re_sdk_comms::buffered_client] Dropping messages because tcp client has timed out or quitting.
[2024-05-20T15:33:12Z WARN  re_sdk_comms::buffered_client] Dropping messages because tcp client has timed out or quitting.
[2024-05-20T15:33:12Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to connect to Rerun server at 127.0.0.1:9876: Connection refused (os error 111)
[2024-05-20T15:33:12Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:12Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:12Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to connect to Rerun server at 127.0.0.1:9876: Connection refused (os error 111)
[2024-05-20T15:33:12Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:12Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
[2024-05-20T15:33:12Z INFO  re_sdk_comms::server] New SDK client connected: 127.0.0.1:52584
Authorization required, but no authorization protocol specified
Error: winit EventLoopError: the requested operation is not supported by Winit -> the requested operation is not supported by Winit
[2024-05-20T15:33:12Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to connect to Rerun server at 127.0.0.1:9876: Connection refused (os error 111)
[2024-05-20T15:33:12Z WARN  re_sdk_comms::buffered_client] Dropping messages because tcp client has timed out or quitting.
[2024-05-20T15:33:12Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to send to Rerun server at 127.0.0.1:9876: Connection reset by peer (os error 104)
[2024-05-20T15:33:13Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to connect to Rerun server at 127.0.0.1:9876: Connection refused (os error 111)
[2024-05-20T15:33:13Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to connect to Rerun server at 127.0.0.1:9876: Connection refused (os error 111)
[2024-05-20T15:33:13Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to connect to Rerun server at 127.0.0.1:9876: Connection refused (os error 111)
[2024-05-20T15:33:13Z WARN  re_sdk_comms::buffered_client] Failed to send message after 3 attempts: Failed to connect to Rerun server at 127.0.0.1:9876: Connection refused (os error 111)
[2024-05-20T15:33:15Z WARN  re_sdk_comms::buffered_client] Dropping messages because tcp client has timed out.
[2024-05-20T15:33:15Z WARN  re_sdk_comms::buffered_client] Dropping messages because tcp client has timed out.
[2024-05-20T15:33:15Z WARN  re_sdk_comms::tcp_client] Tried to flush while TCP stream was still Pending. Data was possibly dropped.```

[BBBmau]

To help with improving rerun, I was able to reach out to the rerun team for feedback. From our code it didn't seem right that we were calling rr.init() on every single driving segment since it not only attempts to create the viewer multiple times but also attempts to allocate a new stream object and duplicate blueprint. This pretty much points to why we get a data leak.

and issue has been made here for it. rerun-io/rerun#6420

But for now this a good step forward towards refining rerun to just work.

[ntegan1]

Loading qcamera frames from comma api via their presentation timestamps converted to nanos

Untitled5.mp4

Untitled3.patch

[reddyn12]

Are you guys getting a memory leak? Mem keeps on creeping up, even at 10GB. Mem creeps up EVEN when I save the recording and load it in a new rerun instance. Also I remember @adeebshihadeh saying he gets 10 FPS. I'm getting 40 with rerun GUI app running through windows 'python3 -m rerun ws://localhost:9877'. The port is what is exposed when running the demo script. Also, the recording file size is 1.05GB, can anyone confirm?

[bongbui321]

tl;dr: we currently don't have batch logging for Scalar archetype and Scalar is intended to be used for a few hundreds to a thousand of log float values only, since it has all fancy features which requires a lot of additional memory usage then necessary

idea to explore (this is more of a hack in case we want to implement this fast) points3D supports batch logging. Log necessary data and use Transform of origin in rerun Space View to plot it in 2D

it seems that the Rerun team is addressing this issue and should be a quick fix from us once those are done (probably just change the logging of archetypes). These are the trackers for those issues:

rerun-io/rerun#5967
rerun-io/rerun#5904 (this is pretty much our issue rn)

From my testing, the thing that slows down logging and the bloated memory usage is from rr.log() of the rr.Scalar(). Upon further investigation, how we are using rr.Scalar() isn't really its intended usage (not for big number of time series data), and a lot of its available features are not used by us anyway. currently floating point batch logging is not supported in Rerun (something similar to rerun.points3D which could take in a matrix of 3D points which speeds up logging a ton.

[savojovic]

At the moment of writing code for POC, I couldn't find any major bottlenecks in code so I suspected that the problem with performance and memory could be an upstream problem. I guess this confirms it.

[adeebshihadeh]

Sounds like we should try points3D. The goal right now is just to be PlotJuggler-quality, then we can wait for rerun to add all these nice-to-haves on top.

[adeebshihadeh]

@bongbui321 if you're interested in the bounty, a solution like points3D would qualify as long as it's at least PlotJuggler quality

[bongbui321]

sure