PSDK 3.6 - 3.8 DjiLiveview_StartH264Stream Function causing M300 to be unstable in air.

[DominikWawak]

Hello,

This issue happens when using the DjiLiveview_StartH264Stream function on the latest psdk v3.8 but also from the versions 3.6 and 3.7.
The behavior observed is that when wanting to switch the camera (H20) lens by calling the function. We see the velocity commands experience small spikes that reflect in the drone becoming unstable for a split second performing random motion.
This behavior did not happen in PSDK v3.5 and we were able to call the same function to stop and/or start the streaming in order to change lenses from wide to zoom.

Essentially the velocity of the drone experiences a sudden change and I think it could be because when calling this function we dont receive some data coming from the psdk at the desired frequency.

Thank you.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Sorry, maybe due to a problem with the translation software, we may not be able to fully understand what you mean. Could you please shoot a simple video for us to help explain? Thanks~

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support

°°°

[DominikWawak]

Hi,

The behavior that I experience is calling the DjiLiveview_StartH264Stream function to turn on/off the stream causes the DJI M300 to be unstable for a second while flying. Only in psdk versions 3.6 - 3.8.

Thank you.

[DominikWawak]

This is also observed in the M30T, changing to zoom or wide lens, starting or stoping the streaming makes the drone tilt for a second.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Theoretically, switching the video source will not affect the body posture. Can you confirm that this phenomenon occurs every time it is called? Does this problem occur in real flight or in the simulator? If it also occurs in the simulator state, we can try to reproduce your problem.
If you only appear in actual flight scenarios, please help us collect your drone logs and tell us the time when the problem occurred so that we can quickly locate the logs. Thank you again~

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support

°°°

[DominikWawak]

This issue occurs in simulation and in real flights. The set up we have is a controller that uses the DJI controller and I also use the PSDK streaming. When I call the streaming function regardless if it is start true or false I see the drone pitch change mostly in the x axis. I have some images of the data being sent and received on the serial ports.

Port 8 is the advanced sensing and port 3 is the UART. The baud rate was increased to max.
This is the behavior with PSDK with streaming on and calling the streaming function:

When the Streaming is set to start_stop false and only the lens is changed there is no impact on the advanced sensing

Running with PSDK 3.5 Streaming and changing lens the drone does not do this tilting motion.

Thank you.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you for the information and image data you provided. We understand what you mean and plan to conduct a local simulator-based test to see whether the video stream has an impact on the drone's flight attitude.
I also want to confirm with you that you currently only have one drone, right? Have you tried testing on multiple drones? What is the firmware version number of your drone?

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support

°°°

[DominikWawak]

We are using the M300 and the M30, in both we see this issue, The M300 was flying with firmware version 0.7 and 0.9 and the M30T is running with version 9.01.

Thank you.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you for your additional information. We will arrange a retest locally. After the test is completed, we will synchronize the results to you. Please wait patiently. We will contact you again soon.

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support

°°°

[DominikWawak]

I was able to reproduce the error in the samples by editing the Flight Controller Sample - Take off position ctrl landing. In the test_flight_control.c file I edit this function mainly to trigger the start of the stream. It is visible in the simulator. If you need more information let me know,
Thank you.

bool
DjiTest_FlightControlMoveByPositionOffset(const T_DjiTestFlightControlVector3f offsetDesired, float yawDesiredInDeg,
                                          float posThresholdInM, float yawThresholdInDeg)
{
    int timeoutInMilSec = 20000;
    int controlFreqInHz = 50;  // Hz
    int cycleTimeInMs = 1000 / controlFreqInHz;
    int outOfControlBoundsTimeLimit = 10 * cycleTimeInMs;    // 10 cycles
    int withinControlBoundsTimeReqmt = 100 * cycleTimeInMs;  // 100 cycles
    int elapsedTimeInMs = 0;
    int withinBoundsCounter = 0;
    int outOfBounds = 0;
    int brakeCounter = 0;
    int speedFactor = 2;

    T_DjiReturnCode returnCode_live;

    static E_DjiMountPosition s_mountPosition = DJI_MOUNT_POSITION_PAYLOAD_PORT_NO1;

    //! get origin position and relative height(from home point)of aircraft.
    T_DjiFcSubscriptionPositionFused originGPSPosition = DjiTest_FlightControlGetValueOfPositionFused();
    dji_f32_t originHeightBaseHomePoint = DjiTest_FlightControlGetValueOfRelativeHeight();
    if (originHeightBaseHomePoint == -1) {
        USER_LOG_ERROR("Relative height is invalid!");
        return false;
    }

    T_DjiFlightControllerJoystickMode joystickMode = {
        DJI_FLIGHT_CONTROLLER_HORIZONTAL_POSITION_CONTROL_MODE,
        DJI_FLIGHT_CONTROLLER_VERTICAL_POSITION_CONTROL_MODE,
        DJI_FLIGHT_CONTROLLER_YAW_ANGLE_CONTROL_MODE,
        DJI_FLIGHT_CONTROLLER_HORIZONTAL_GROUND_COORDINATE,
        DJI_FLIGHT_CONTROLLER_STABLE_CONTROL_MODE_ENABLE,
    };
    DjiFlightController_SetJoystickMode(joystickMode);



    while (elapsedTimeInMs < timeoutInMilSec) {
        T_DjiFcSubscriptionPositionFused currentGPSPosition = DjiTest_FlightControlGetValueOfPositionFused();
        T_DjiFcSubscriptionQuaternion currentQuaternion = DjiTest_FlightControlGetValueOfQuaternion();
        dji_f32_t currentHeight = DjiTest_FlightControlGetValueOfRelativeHeight();
        if (originHeightBaseHomePoint == -1) {
            USER_LOG_ERROR("Relative height is invalid!");
            return false;
        }

        float yawInRad = DjiTest_FlightControlQuaternionToEulerAngle(currentQuaternion).z;
        //! get the vector between aircraft and origin point.

        T_DjiTestFlightControlVector3f localOffset = DjiTest_FlightControlLocalOffsetFromGpsAndFusedHeightOffset(
            currentGPSPosition,
            originGPSPosition,
            currentHeight,
            originHeightBaseHomePoint);
        //! get the vector between aircraft and target point.
        T_DjiTestFlightControlVector3f offsetRemaining = DjiTest_FlightControlVector3FSub(offsetDesired, localOffset);

        T_DjiTestFlightControlVector3f positionCommand = offsetRemaining;
        DjiTest_FlightControlHorizCommandLimit(speedFactor, &positionCommand.x, &positionCommand.y);

        T_DjiFlightControllerJoystickCommand joystickCommand = {positionCommand.x, positionCommand.y,
                                                                offsetDesired.z + originHeightBaseHomePoint,
                                                                yawDesiredInDeg};
        DjiFlightController_ExecuteJoystickAction(joystickCommand);

      
        if (elapsedTimeInMs%2000 == 0) {
            returnCode_live = DjiLiveview_StartH264Stream((E_DjiLiveViewCameraPosition) s_mountPosition, 0, DjiTest_PayloadCameraStreamCallback);
        }
        
        if (DjiTest_FlightControlVectorNorm(offsetRemaining) < posThresholdInM &&
            fabs(yawInRad / s_degToRad - yawDesiredInDeg) < yawThresholdInDeg) {
            //! 1. We are within bounds; start incrementing our in-bound counter
            withinBoundsCounter += cycleTimeInMs;
        } else {
            if (withinBoundsCounter != 0) {
                //! 2. Start incrementing an out-of-bounds counter
                outOfBounds += cycleTimeInMs;
            }
        }
        //! 3. Reset withinBoundsCounter if necessary
        if (outOfBounds > outOfControlBoundsTimeLimit) {
            withinBoundsCounter = 0;
            outOfBounds = 0;
        }
        //! 4. If within bounds, set flag and break
        if (withinBoundsCounter >= withinControlBoundsTimeReqmt) {
            break;
        }
        s_osalHandler->TaskSleepMs(cycleTimeInMs);
        elapsedTimeInMs += cycleTimeInMs;
    }

    while (brakeCounter < withinControlBoundsTimeReqmt) {
        s_osalHandler->TaskSleepMs(cycleTimeInMs);
        brakeCounter += cycleTimeInMs;
    }

    if (elapsedTimeInMs >= timeoutInMilSec) {
        USER_LOG_ERROR("Task timeout!");
        return false;
    }

    return true;
}

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you for the content you provided. In this case, we can directly replace the test and theoretically be able to reproduce the phenomenon you described, right?

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support

°°°

[DominikWawak]

Hi,

Yes thats right, the only thing you might be missing is the callback at the end.

static void DjiTest_PayloadCameraStreamCallback(E_DjiLiveViewCameraPosition position, const uint8_t *buf,
                                                uint32_t bufLen)
{
    FILE *fp = NULL;
    size_t size;

    fp = fopen(s_payloadCameraStreamFilePath, "ab+");
    if (fp == NULL) {
        printf("fopen failed!\n");
        return;
    }

    size = fwrite(buf, 1, bufLen, fp);
    if (size != bufLen) {
        fclose(fp);
        return;
    }

    fflush(fp);
    fclose(fp);
}

Thank you.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you again for your addition. We tried to embed your code into our sample program for testing. We noticed that when using the simulator, the drone flew with slight and frequent jitters. We would also like to ask you to help us confirm it. Are the phenomena consistent?

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support
20240315160540.webm

°°°

[DominikWawak]

Hi,

Yes, that is just simulate what we do when we fly the drone we would during flight call this function to change the camera lens and it would cause the drone to jitter. It happens both in simulator and in real flights every time.

Thank you,
Dominik.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you very much for your confirmation. Regarding this issue, we will first collect the logs and send them to the R&D team for analysis and confirmation, and at the same time help you confirm whether there is any way to prevent jitter.

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support

°°°

[DominikWawak]

Hi DJI Team,

Have you any updates on this issue?

Thank you.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you for your patience. We see that this issue is still under analysis and no preliminary conclusion has been given yet. We will help you continue to follow up on this issue and urge the relevant teams to provide a conclusion as soon as possible.

Thank you for your support of DJI products.

Best Regards,
DJI Dajiang innovation SDK technical support

°°°

[DominikWawak]

Hi DJI Team,

Just checking in if you might have some updates or possible fixes on how to stop the jitter.
Thank you.

Dominik.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings. We appreciate your contact with DJI - Da-Jiang Innovations.

We apologize for the delay. Our research and development team is still analyzing this issue, and we are awaiting any updates on the progress of this matter. We will urge the R&D team to provide a conclusion as soon as possible. Thank you once again for your patience.

Thank you for your support of DJI products! Wishing you all the best!

Best Regards,
DJI - Da-Jiang Innovations SDK Technical Support

°°°

[gsuess]

@dji-dev were you able to make any progress on this?

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings. We appreciate your contact with DJI - Da-Jiang Innovations.

Thank you for your patience. Your issue has been resolved. We have identified that the abnormal shaking is caused by a Yaw command sent by the PSDK, especially during the aircraft's sideways flight process. The PSDK rapidly sends a Yaw rotation command at the beginning, causing the aircraft's attitude to twist. We have confirmed this by printing the Yaw angle, which we found to be rapidly changing. You can check the settings related to this in your code.

Thank you for your support of DJI products! We wish you all the best.

Best Regards,

DJI - Da-Jiang Innovations SDK Technical Support
image.png

°°°

[DominikWawak]

Hi @dji-dev

Thank you for the response. When you say the issue is resolved will you be releasing a new version of the PSDK with the fix soon or do you maybe have some change that we can do on our side to resolve it.

Thank you,
Dominik.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings. Thank you for reaching out to DJI - Da-Jiang Innovations.

We apologize for any confusion in my previous communication. We would like to clarify that the issue at hand is not related to the PSDK code but rather to the input of flight commands. Could you please verify the input of flight commands or add print statements to confirm the changes in Yaw?

We appreciate your support for DJI products! Wishing you all the best!

Best Regards,

DJI - Da-Jiang Innovations SDK Technical Support

°°°

[gsuess]

@dji-dev just out of curiosity...

I understand how a configuration issue and flight-commands can affect the profile of the flight, but what I don't understand is why changing the camera lens mode is affecting flight at all, or how that is not an issue with the PSDK. Surely changing the camera lens should not affect the flight profile?

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings! Thank you for reaching out to DJI Innovations.

Switching the camera will not affect the flight attitude. Our research and development team has identified an abnormal YAW control command issued under the PSDK, which caused the aircraft to exhibit abnormal shaking. This shaking is unrelated to the video streaming function.

We appreciate your support for DJI products! Wishing you all the best!

Best Regards,

DJI Innovations SDK Technical Support

°°°

[biancabnd]

Hello @dji-dev,

Thanks for all the support and feedback on this issue.
From what you are saying, the issue is with the yaw control command that is being sent. To try to avoid that we did 2 tests:

1. Running the same sample code with the yaw control command being +30 deg and then -30 deg.
2. Running the same sample code with the yaw control command being always 0 deg. Constant command

We recorded the output yaw, roll, and pitch angles as well as the timestamp at which we trigger the camera stream to start (as one would do when needed to change the camera lens). Below, you can see the output we have obtained:

1st Test: with constant yaw control command +30deg, followed by constant yaw command -30deg.
Going to the first waypoint without triggering the camera stream, afterwards constantly call start main camera streaming

2nd Test: constant yaw control command 0 deg
Going to the first waypoint without triggering the camera stream, afterwards constantly call start main camera streaming

As you can see, the sample code runs normal, but once we start the camera streaming the drone has an abnormal shaking. There is a direct correlation with the action of starting the camera streaming and the drone's attitude being disrupted. (Disregard the attitude change from the beginning of the log because that is due to the rapid takeoff of the drone)
As was mentioned by @DominikWawak, this was not happening with the Payload-SDK v3.5.

Can you please try to reproduce this behavior and give more information on what could be the root cause? Have you tried to constantly start and stop the streaming of the main camera, while also controlling the drone? I think you would observer the same behavior.

Thank you in advance.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings. We appreciate your contact with DJI Innovations.

Thank you for your patience. As the country is currently observing a national holiday, our engineers are on leave. Your feedback has been noted, and the engineers are expected to respond to you by May 6th. We apologize for any inconvenience this may have caused.

Thank you for your support of DJI products! Wishing you all the best!

Best Regards,

DJI Innovations SDK Technical Support

°°°

[gsuess]

@dji-dev were you able to reproduce the problem now? If not, it would be good to see how you run a scenario where starting a camera stream does not affect the flight profile of the drone so that we can pinpoint the differences in your approach in comparison to our approach.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings. Thank you for reaching out to DJI - Da-Jiang Innovations.

We apologize for the delay and appreciate the additional information you provided. We believe it is necessary to conduct another round of testing based on PSDK 3.5 and PSDK 3.8.1. During the testing, we will keep the YAW flight control command unchanged and attempt to repeatedly call the video stream to see if we can replicate the issue you encountered. Due to the high volume of inquiries recently, we will strive to complete the retesting within this week. Once the retesting is completed, we will promptly synchronize the results with you. Thank you once again for your cooperation.

We appreciate your support for DJI products! Wishing you all the best.

Best Regards,

DJI - Da-Jiang Innovations SDK Technical Support

°°°

[gsuess]

@dji-dev do you have any new results on your tests?

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings,

Thank you for reaching out to DJI Innovations.

We appreciate your patience. In response to your inquiry, we have conducted further testing on the issue. We have implemented a multi-threaded approach to execute the task of retrieving video streams. You may replace the files in the attachment and utilize the C++ sample code. Please select the option "Flight controller sample - take off position ctrl landing" to run the code.

We have tested this on the M30/T M300+H20T models and did not encounter any shaking issues. Kindly review the differences between your setup and ours.

Thank you for your support of DJI products! Wishing you continued success!

Best Regards,

DJI Innovations SDK Technical Support
image.png
test_flight_control.c

°°°

[biancabnd]

Hello,
Thank you for the information. However, I cannot retrieve the files in the attachment. I have a message saying: Unauthorized.You are not authorized to see the file. Make sure that you are [logged in here]

I do have an account to DJI Support. Can you please indicate me how I can retrieve these attachments?

Thank you.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102271:

Dear DominikWawak,

Greetings,

Thank you for reaching out to DJI - Da-Jiang Innovations.

We appreciate your patience. The requested information has been uploaded to a cloud storage service, and you can access it from the following link:
https://pan-sec.djicorp.com/s/op4PPLZGz9ofPab

Thank you for your support of DJI products! We wish you all the best!

Best Regards,

DJI - Da-Jiang Innovations SDK Technical Support

°°°