Performance on different platforms

[petergerten]

@fnoop you mentioned you are targeting the system to work on a RPI. I agree it is a good idea due to cheap price, availability etc.

I am interested in what kind of platform people are currently using the system and what kind of performance you see. From reading though old issue reports I got:

• "We have only ~4 fps for 720p and ~8fps for 480p with track_targets."
• "On the Joule I had track_targets running about 380-390% on 4 cores"

I am curious: as I saw the 1920x1080 screenshot. Were you able to run 1080p at 30fps on a Joule?
For some reason I can't even get 1080p30 on my i7.

[fnoop]

Hah, no I wish! The 1080p screenshots were taken inside a VM by running a prerecorded video from the raspberry through vision_landing. vision_landing can take a video file as input rather than live camera and it will process it frame by frame, so it takes a while to process the video but you get a complete output. With live video, it just grabs the next frame available so will skip frames.

On the Joule, I was getting about 85fps @ 640x480, 30fps @ 720p (both close to the maximum of the cameras) and about 10-15fps @ 1080p, which is still pretty good.
On the raspberry last time I tried I was getting up to 20-23fps @ 640x480. The upcoming maverick 1.1.4 has a more optimised opencv for the Pi3 so hoping to get a bit more performance.
But it varies widely depending on the thermal load, other system processes, number of targets in view etc. And the system hasn't been optimised at all yet, in fact the codebase is in a bit of flux at the moment while I start that. I'm moving the primary queue listener from a fast main loop to an async callback (hopefully) and dropping the main loop to a sedentary 1hz, which will further improve performance particularly on slower platforms.

Further, the timesync and correlation changes to the main ardupilot firmware (waiting to be reviewed and merged) will allow it to cope with much bigger latencies and lower framerates, so hopefully slower platforms like the raspberry will be able to precision landing fine, just with lower performance. The lower the framerate and bigger the latency, the less able the platform is able to cope with fast/large movements and corrections.

[fnoop]

In addition, performance is more than just fps. It's much more important to have a good camera and a good connection. The realsense camera had a terrible rgb camera but excellent IR cameras which were used fo the stereo separation. But you could access them individually for vision and they were global shutter @ 90fps, and I got excellent results from them. The upcoming realsense D400 range has two initial models - the slightly more expensive D435 is the one to go for for applications like this as it has the global shutter cameras, as well as a wider FoV.
Working with USB3/CSI is a pain but worth it in the long run as you can get the raw video out at much (much) higher rates. Trying to work with compressed video over a usb2 link sucks, as does slow raw framerates over usb2 or the complexity and high latency of hdmi links.