Video streaming to onboard computer & to ground server

[hnqiu]

Hi @dji-dev,

We are planning to build applications using M350 RTK with onboard computer and I was reading this page https://developer.dji.com/doc/payload-sdk-tutorial/en/model-instruction/choose-develop-platform.html

According to the first table, some functionalities (e.g. flight control) are only available via the e port while some others (e.g. basic camera function, camera video stream) are available via the psdk port. So if I want all of these functionalities, how should I connect the onboard computer? Does the computer need two cables (e port development kit & psdk development kit) connected to the drone? 

Specifically, we would like to stream payload (H20T) videos to a ground server. How does the dataflow look like in this case? Is it like

• jetson sends 'stream video' request to drone via psdk kit
• then video stream goes from H20T -> (another) psdk port -> drone -> wifi -> ground server?

Is the video 1080p only or can be higher resolution?
Also, can we stream videos from multiple sources (e.g. H20T camera and FPV camera) at the same time?

Another question related to the onboard computer - my understanding is the computer can only get 1080p frames via the camera liveview interface. However, is there a way for the onboard computer to read higher resolution images from the SD card on H20T?

Thanks a lot

[uzgit]

@hnqiu are you saying you need to stream video to the controller after it has been processed by the Jetson? If so then I have done something similar and can give some pointers. If you just need to send the H20T video to a ground server then I think you can just pull it from the HDMI port on the controller when the H20T is selected, while pulling the same stream to the Jetson via the E-port.

[dji-dev]

Agent comment from Leon in Zendesk ticket #106946:

Dear hnqiu,

Greetings. Thank you for reaching out to DJI Innovations.

If you wish to utilize all the functions supported by PSDK, you will need the E-Port Development Kit and the SkyportV2 Adapter Ring Kit + Payload SDK 2.0 Development Kit to connect to the drone's E-Port and Payload Port respectively.

It is important to note that you still cannot run all functions simultaneously in the same PSDK program. For instance, if you intend to use a feature supported only by the PSDK port, even if you have connected the hardware to the Payload Port, you will not be able to use this feature on the PSDK running on the E-Port. You will need to run two PSDK programs on the onboard computer, each supporting independent functions, one corresponding to the E-Port and the other to the Payload Port.

Should you wish to transmit H20T data to a ground server, your requirement will necessitate secondary development based on the onboard computer. You can run PSDK on the E-Port to access the H20T video stream data through PSDK interfaces, then proceed with encoding and transmission. The general process is as follows: H20T -> Payload Port -> M350 -> Onboard Computer -> WiFi -> Ground Server.

We appreciate your support for DJI products! Wishing you every success!

Best Regards,

DJI Innovations SDK Technical Support

°°°

[hnqiu]

@hnqiu are you saying you need to stream video to the controller after it has been processed by the Jetson? If so then I have done something similar and can give some pointers. If you just need to send the H20T video to a ground server then I think you can just pull it from the HDMI port on the controller when the H20T is selected, while pulling the same stream to the Jetson via the E-port.

Thanks a lot @uzgit. Yeah that would be great if you could share some suggestions.
In terms of how we want to stream the video, both ways (ground server receiving videos processed by Jetson and/or from the controller) might be needed as we're still exploring the capability.
Ideally, we want the ground server to be able to process multiple sensor signals simultaneously (e.g. FPV cam + H20T CMOS cam + thermal cam), and we might want to obtain high resolution frames. Can these two requirements be achieved by pulling the data from the controller?
On the other hand, is there any limitation when we use Jetson to stream processed videos?

[hnqiu]

Thank you @dji-dev.

You will need to run two PSDK programs on the onboard computer, each supporting independent functions, one corresponding to the E-Port and the other to the Payload Port.

Should/Can we synchronise the two instances in this case?

---

You can run PSDK on the E-Port to access the H20T video stream data through PSDK interfaces, then proceed with encoding and transmission. The general process is as follows: H20T -> Payload Port -> M350 -> Onboard Computer -> WiFi -> Ground Server.

Is there any limitation when we use the onboard computer to stream processed videos? I understand the camera liveview functionality can only get 1080p even when H20T is capable of recording higher res videos - is this correct?
In addition, can we use the onboard computer to broadcast multiple sensor signals simultaneously (e.g. FPV cam + H20T CMOS cam + thermal cam)? - I believe this is how the onboard computer streams videos. Can we call this multiple times with different camera sources?

[uzgit]

@hnqiu so I have done something very similar to what Leon (@dji-dev) has suggested. Very nice of him to lay it out like that - could be good to provide a tutorial about these things on the PSDK site because it can be very confusing to develop this sort of use case. I will lay out my observations here so they can be open for others to use, or for DJI to tell everyone where I went wrong so others can have an easier time developing.

I am using a Matrice 350 with Raspberry Pi 5 top-mounted for easy access to both the E-port and PSDK port.

There's a couple of points that I would watch out for:

• Each application will require setting up the high bandwidth communication channel so it can transfer video. Startup order of the applications matters (at least in my case). I have to start the PSDK port application before starting the E-port application, otherwise the PSDK port application will break the bulk connection of the E-port application. I have used the USB bulk method on the E-port and the Ethernet method on the PSDK port, since it seems like the PSDK expansion board only provides an Ethernet interface. With the E-port expansion board you can use either Ethernet-over-USB (easier) or USB bulk (less well-documented).
• I would recommend doing most of your processing on the E-port application, and using the PSDK port application just to stream the video. Can transmit the video from the E-port application to the PSDK port application using a shared pipe after compression and actually implement the streaming using this approach here: https://sdk-forum.dji.net/hc/zh-cn/community/posts/7924808225561-Is-there-any-way-to-directly-stream-real-time-video-to-remote-controller-on-PSDK although it doesn't go into all the necessary details, particularly about initialization of the Camera EMU service. I have implemented my application to just stream the desktop of my Raspberry Pi to the controller, but you can use it for sanity checks if you want since you will have to implement similar functionality: https://github.com/uzgit/Payload-SDK/blob/master/src/rpi_desktop_streamer/main.cpp
• My E-port application can only receive Liveview from the H20T at 1440p, not 1080p, even though it is documented at 1080p on the PSDK site: https://developer.dji.com/doc/payload-sdk-tutorial/en/function-set/advanced-function/liveview.html. This actually makes it difficult in my use case because the Raspberry Pi's hardware decoder only goes to 1080p, so I have to decode the video stream in software. It's easier on something like the P1 which actually does stream in 1080p. Be aware also that all of the streams from the H20T will be received by your companion board in the same resolution (wide, zoom, and thermal) regardless of their native resolutions. This does make it simpler when switching between streams.
• As @dji-dev said, you will need a SkyPort V2 connector. Be careful with this because it has its own firmware and was released before the Matrice 350 with firmware that is incompatible with the Matrice 350. If you receive one with an early firmware version, you must use a Matrice 300 to update the firmware, otherwise you will brick it (as I did). You can find more info and specifics here: https://sdk-forum.dji.net/hc/en-us/community/posts/29885161236761-How-to-stream-video-to-controller
• I don't think you can pull multiple streams from the H20T simultaneously, but I'm not sure about this.
• As far as I know, the SkyPort V2 connector (same connector as on top of the H20T) has to plug into a gimbal port, and the gimbal port is what plugs into the PSDK port. This is what I did - maybe @dji-dev can confirm or deny. I got an upward gimbal mount for this purpose, and disassembled it to remove the electronics, so it would be less bulky.

My end product looks like this (E-port plug on the starboard side, PSDK port plug on the port side):

Hope this helps and maybe puts out some information for others who want to do this sort of thing. Some of this stuff was really a pain to figure out.

[hnqiu]

Thanks a lot @uzgit! These are so informative and yeah it'd have taken a lot to figure out. As you suggested, we're thinking to use the psdk port application to stream videos only (and potentially do more processing on the ground server). It's good to know this is achievable while the e-port application can also do something else.
Your product looks awesome 😄

Just wanted to confirm if my interpretation is correct cc: @dji-dev

Should you wish to transmit H20T data to a ground server, your requirement will necessitate secondary development based on the onboard computer. You can run PSDK on the E-Port to access the H20T video stream data through PSDK interfaces, then proceed with encoding and transmission. The general process is as follows: H20T -> Payload Port -> M350 -> Onboard Computer -> WiFi -> Ground Server.

The work @uzgit shared here is based on two applications, each connecting to e-port and psdk port respectively.
I wonder - if I "run PSDK on the E-Port to access the H20T video stream data through PSDK interfaces, then proceed with encoding and transmission. The general process is as follows: H20T -> Payload Port -> M350 -> Onboard Computer -> WiFi -> Ground Server." Can I just run the e-port application only (i.e. the psdk port application won't be necessary)?

[dji-dev]

Agent comment from Leon in Zendesk ticket #106946:

Dear hnqiu,

Greetings. We appreciate your contact with DJI - Da-Jiang Innovations.

Thank you for your patience. If you are running the PSDK program on the E-Port, the correct process is as follows: H20T -> Payload Port -> M350 -> Vehicle-mounted computer on the E-Port (running PSDK) -> WiFi -> Ground server.

You only need to run the PSDK program on the E-Port, there is no need to run it on the Payload Port.

Thank you for your support of DJI products! We wish you all the best!

Best Regards,

DJI - Da-Jiang Innovations SDK Technical Support

°°°

[hnqiu]

Awesome. Thanks both!

[dji-dev]

Agent comment from Leon in Zendesk ticket #106946:

Dear hnqiu,

Greetings! We appreciate your contact with DJI - Da-Jiang Innovations.

You're welcome. If you have no further inquiries, please feel free to close this ticket. Should you encounter any SDK-related issues in the future, please don't hesitate to consult us again. Thank you once more for reaching out to us.

Thank you for your support of DJI products! Wishing you all the best!

Best Regards,

DJI - Da-Jiang Innovations SDK Technical Support

°°°

[hnqiu]

Thank you @dji-dev. Closing this issue now.