Extended dshot telemetry data parsing and presentation #625

damosvil · 2023-02-01T21:46:02Z

This pr adds EDT parsing and presentation functionality to blackbox explorer.
EDT data is packed in debug fields in the following way:

bit 15 - alert event
bit 14 - warning event
bit 13 - error event
bits 11-8 - Max stress level during the whole flight
bits 7-0 - ESC data depending on the chosen debug mode

This pr is related to:
bird-sanctuary/bluejay#64
betaflight/betaflight#12170
betaflight/betaflight-configurator#3262

ESC temperature log files for testing:
BTFL_BLACKBOX_LOG_Meteor65_20230201_183226.zip
BTFL_BLACKBOX_LOG_Meteor65_20230202_183845.zip

ESC demag metric files for testing:
BTFL_BLACKBOX_LOG_Meteor65_20230203_224822.zip

ESC rpm log files for testing:
BTFL_BLACKBOX_LOG_Meteor65_20230203_232019.zip

damosvil · 2023-02-02T22:09:52Z

Replaced tabs by spaces in modified files

McGiverGim · 2023-02-03T10:33:57Z

@damosvil I don't understand too mucho about ESCs but an important question to get to the best solution:

This info about demag, desync, stall, etc. events. How important is? Is habitual to have them and not notice it or if it happens is sign that something is broken/misconfigured and needs to be fixed?

I say that because package it only in debug fields, will "hide" it from the major part of users, except if they configure this debug fields in the firmware and look at it expressly in the blackbox.

If this events are habitual, at least at some extent -> then use debug fields is ok to me.
If this events are not, and must be noticed by the user -> then I think is better to send them as events and not as debug fields, in this way they will be noticed inmediatelly in the blackbox (but we need to revisit the events fields because they are limited to 32 only I think remember). The debug fields may remain, they can give more information if a user needs it, but must be not the principal way of notification.

Regards!

McGiverGim · 2023-02-03T10:55:14Z

@damosvil looking deeply at the code, it seems you are using the bookmarks feature to mark them? Maybe you can publish some screen capture about how they look?

damosvil · 2023-02-03T23:23:00Z

Below I describe behavior of some of the events:

Demag events are pretty usual as they happen when motors change speed abruptly.
Desync events are triggered when many consecutive demag events happen (in Bluejay/Blheli_S):
-- 160 consecutive demag events for Low Demag Compensation trigger a Desync event
-- 130 consecutive demag events for High Demag Compensation trigger a Desync event
-- 255 consecutive demag events for Off Demag Compensation trigger a Desync event
Stall events happen when motor stalls
Demag Metric data is used to adjust commutation timing. It ranges from 120 to 255. When it is low it means that the ESC is capable of doing good commutation, the higher the worse commutation.
Max Demag Metric is a value between 0 and 15 that indicates how the esc carries on with the motor during flight. It is the result of the formula: MDM = (DM - 120) / 9. During the test flights in a brand new Meteor65 it had a value of 4. Normal values should be below 10, otherwise the ESC may have problems performing a good commutation.

Debugging EDT you will always have the following status data:

Demag event
Desync event
Stall event
Max Demag Metric

You will have an additional value depending on the debug mode used:

DSHOT_STATUS_N_TEMPERATURE: Status data and temperature
DSHOT_STATUS_N_VOLTAGE: Status data and temperature
DSHOT_STATUS_N_CURRENT: Status data and current
DSHOT_STATUS_N_DEBUG1: Status data and debug1 value
DSHOT_STATUS_N_DEBUG2: Status data and debug2 value
DSHOT_STATUS_N_DEMAG_METRIC: Status data and demag metric
DSHOT_STATUS_N_ERPM_FRACTION_18: Status data and RPM but with less resolution than when debugging DSHOT_RPM_TELEMETRY

This is the first time all this data is logged in Betaflight blackbox, so any idea regarding on how to display it is more than welcome.

Capture of temperature:
To the left you can read values. X is demag event [0,1], D is desync event [0,1], S is stall event [0,1], MAX_DEMAG is demag metric max [0-15], and then the chosen magnitude, in this case temperature.

Capture of demag metric on a brand new Meteor 65:

Capture of rpm:

damosvil · 2023-02-04T09:23:49Z

At this time I choose to draw the EDT selected magnitude in the graph, but I am thinking on printing a mark in the temperature graph for desync events and a blue vertical line from top to botom for stall events. Demag events are pretty usual so I would let them in the info to the right, but I would not print them. Max demag metric is interesting to be checked at the end of the flight, so I would not print this number in the graph, it can be simply cheched at the end of the flight. Other option is to print a mark when it is bigger than 9.
Please, let me know how you see this.

damosvil · 2023-02-04T09:51:57Z

In general all this data is interesting to technical users that want to know how the esc/motors are going during flight.
The most interesting to be tracked during the flight is max demag metric. It can provide information about the esc/motor health during the whole flight.
As I commented above event and demag data is new at this point, so we may create an initial version with some features, and later when the user base grows we may add some new features based on user experience and needs.

McGiverGim · 2023-02-07T07:25:13Z

Ok, understood. So the major part of values from the ESC are debug values and must be shown when the users logs it. As you say this PR is ok for that.

But it seems there are some information that can be important to show (maybe desync/stall events? can you confirm?) It seems to me, by your description, that they must not happen and if it does, is a good idea to inform the user. Am I right?

If yes, I think they must be shown as events in the timeline. Is similar to what we do with the flight modes, that we draw a "flag/mark" in the timeline:
https://github.com/betaflight/betaflight/blob/facf44b4066d726fafcc969a8fcb9afbeb3009ff/src/main/blackbox/blackbox.c#L354
but this can be left to a future PR. Adding the debug data as first step is ok to me.

McGiverGim · 2023-02-07T07:42:52Z

Looking at the code, the lapTimer must be taken as a good example too. But as I said, can be left for a future PR.

haslinghuis · 2023-02-12T05:19:38Z

@damosvil please remove dependency changes as they are not required for your (excellent) work.

damosvil · 2023-02-12T06:50:58Z

@haslinghuis I removed the changes over package.json and yarn.lock. Please, let me know if I have to change anything else.

js/flightlog_fields_presenter.js

McGiverGim · 2023-02-15T17:47:04Z

js/flightlog.js

+    /*
+     * Propagates array data fiels over empty fields
+     * Debug fields can be arrays with data, so it fills data gaps when no arrays are stored
+     */
+    function propagateArrayFields(chunks)
+    {


I'm not too sure about this. I think in the rest of the code we simply store the latest value, and add them when we process other kind of data, to add the values while processing. Maybe is clear with your method, with a little more process time (it needs to look at all the data chunk).

Can you look at the slow frames and gps frames chunks to see if they can be "filled" with this and remove the old implementation? Taking times of this method will be great to see if we loose too much time. If it takes not too much time, I prefer your implementation.

When there are errors in the blackbox (frames lost), this method will fill the data? I think remember that not, that we don't have data in the chunk when the frames are lost.

Can you look at the slow frames and gps frames chunks to see if they can be "filled" with this and remove the old implementation? Taking times of this method will be great to see if we loose too much time. If it takes not too much time, I prefer your implementation.

I have been looking for the current implementation. Is it the complete frame function set?

When there are errors in the blackbox (frames lost), this method will fill the data? I think remember that not, that we don't have data in the chunk when the frames are lost.

Still looking into this

Has this been resolved?

@McGiverGim are you okay with this - for a follow up PR?

js/flightlog_parser.js

js/flightlog.js

js/flightlog_fielddefs.js

js/flightlog_fields_presenter.js

haslinghuis · 2023-11-26T13:54:43Z

js/flightlog_parser.js

@@ -1108,8 +1141,8 @@ var FlightLogParser = function(logData) {
            predictor = frameDef.predictor,
            encoding = frameDef.encoding,
            values = new Array(8),
-            i, j, groupCount;
-
+            i, j, groupCount, updateCurrent;


Try not adding new vars - please declare / initialize at block scope.

The function where this i, j variables are defined use them in loops and after the loops, so in this case I recommend keeping them. They are defined that way because of the logic in this function.

haslinghuis · 2023-12-03T12:41:32Z

FYI have added relevant debug modes and fixed the build process.

js/graph_config.js

Requesting new review

Found a workaround to plot esc signal 4 Added basic ESC plot functionality updated project files Implemented array fiels propagation over empty fields Fixed bad FLIGHT_LOG_FIELD_ENCODING_PACK_1F_1F_1F_1G_4U_8U decoding Removed tabs on changed files Fixed some more review findings Updated DSHOT_STATUS_N_ERPM_FRACTION_18 graph ranges Reverted changes in package.json and yarn.lock Improved wording and graph ranges Updated concatenations to template literals If a field contain array values always use the last field for graph Added DSHOT motor events to graph Fixed review findings Added demag events to graph Changed speciffic Bluejay wording by general esc one Fixed sonarcloud false positive Updated debug modes to match the ones in bf master Added debug mode to match edt_events Betaflight branch Removed duplicated BARO graph config Fixed review findings Using let instead of var Refactored if to meet sonarcloud advise

sonarcloud · 2023-12-13T14:54:08Z

Quality Gate failed

Failed conditions

20.4% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

github-actions · 2023-12-13T15:03:16Z

Do you want to test this code? Here you have an automated build:
Betaflight-Blackbox-Explorer-Linux
Betaflight-Blackbox-Explorer-macOS
Betaflight-Blackbox-Explorer-Windows
WARNING: It may be unstable and result in corrupted configurations or data loss. Use only for testing!

damosvil marked this pull request as draft February 1, 2023 21:46

damosvil force-pushed the edt_events branch from 746473f to c823cc0 Compare February 2, 2023 18:18

This comment was marked as outdated.

Sign in to view

This comment has been minimized.

Sign in to view

damosvil force-pushed the edt_events branch from c823cc0 to 3172578 Compare February 2, 2023 22:07

This comment has been minimized.

Sign in to view

damosvil force-pushed the edt_events branch from 3172578 to fbcd458 Compare February 3, 2023 23:41

This comment has been minimized.

Sign in to view

damosvil force-pushed the edt_events branch from fbcd458 to 9f38efc Compare February 12, 2023 06:45

This comment has been minimized.

Sign in to view

damosvil mentioned this pull request Feb 13, 2023

Implemented pwm frequency configurable by eeprom setting bird-sanctuary/bluejay#65

Closed

McGiverGim previously requested changes Feb 15, 2023

View reviewed changes

damosvil mentioned this pull request Feb 16, 2023

Dynamic pwm bird-sanctuary/bluejay#67

Closed

damosvil force-pushed the edt_events branch 2 times, most recently from 1b23437 to e219afd Compare February 18, 2023 07:47

This comment has been minimized.

Sign in to view

damosvil force-pushed the edt_events branch from e219afd to 89c7d4c Compare February 18, 2023 21:26

This comment has been minimized.

Sign in to view

damosvil force-pushed the edt_events branch from 89c7d4c to e35090b Compare February 18, 2023 22:45

haslinghuis reviewed Feb 19, 2023

View reviewed changes

js/flightlog_parser.js Outdated Show resolved Hide resolved

damosvil requested a review from haslinghuis November 26, 2023 10:50

haslinghuis reviewed Nov 26, 2023

View reviewed changes

js/flightlog.js Outdated Show resolved Hide resolved

haslinghuis reviewed Nov 26, 2023

View reviewed changes

js/flightlog_fielddefs.js Outdated Show resolved Hide resolved

haslinghuis reviewed Nov 26, 2023

View reviewed changes

js/flightlog_fields_presenter.js Outdated Show resolved Hide resolved

haslinghuis reviewed Nov 26, 2023

View reviewed changes

js/flightlog_fields_presenter.js Outdated Show resolved Hide resolved

haslinghuis reviewed Nov 26, 2023

View reviewed changes

js/flightlog_fields_presenter.js Outdated Show resolved Hide resolved

haslinghuis reviewed Nov 26, 2023

View reviewed changes

haslinghuis added the rebase required label Dec 4, 2023

damosvil force-pushed the edt_events branch from 185f76d to ad3c11e Compare December 4, 2023 21:18

haslinghuis reviewed Dec 4, 2023

View reviewed changes

js/graph_config.js Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

damosvil force-pushed the edt_events branch from ad3c11e to c31e0d9 Compare December 4, 2023 21:27

haslinghuis removed the rebase required label Dec 4, 2023

This comment has been minimized.

Sign in to view

damosvil force-pushed the edt_events branch 2 times, most recently from 49b771d to ea0c470 Compare December 5, 2023 07:12

This comment has been minimized.

Sign in to view

haslinghuis requested review from McGiverGim and removed request for McGiverGim December 5, 2023 18:39

haslinghuis modified the milestones: 3.7.0, 3.8.0 Dec 6, 2023

damosvil force-pushed the edt_events branch from ea0c470 to 2d50013 Compare December 13, 2023 14:53

McGiverGim approved these changes Dec 13, 2023

View reviewed changes

haslinghuis added the rebase required label Mar 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extended dshot telemetry data parsing and presentation #625

Extended dshot telemetry data parsing and presentation #625

damosvil commented Feb 1, 2023 •

edited

This comment was marked as outdated.

This comment has been minimized.

damosvil commented Feb 2, 2023

This comment has been minimized.

McGiverGim commented Feb 3, 2023

McGiverGim commented Feb 3, 2023

damosvil commented Feb 3, 2023 •

edited

This comment has been minimized.

damosvil commented Feb 4, 2023 •

edited

damosvil commented Feb 4, 2023 •

edited

McGiverGim commented Feb 7, 2023

McGiverGim commented Feb 7, 2023

haslinghuis commented Feb 12, 2023

damosvil commented Feb 12, 2023

This comment has been minimized.

McGiverGim Feb 15, 2023

damosvil Feb 19, 2023 •

edited

haslinghuis Dec 5, 2023

damosvil Dec 5, 2023

haslinghuis Dec 5, 2023

This comment has been minimized.

This comment has been minimized.

haslinghuis Nov 26, 2023

damosvil Dec 5, 2023

haslinghuis commented Dec 3, 2023

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sonarcloud bot commented Dec 13, 2023

github-actions bot commented Dec 13, 2023

Extended dshot telemetry data parsing and presentation #625

Are you sure you want to change the base?

Extended dshot telemetry data parsing and presentation #625

Conversation

damosvil commented Feb 1, 2023 • edited

This comment was marked as outdated.

This comment has been minimized.

damosvil commented Feb 2, 2023

This comment has been minimized.

McGiverGim commented Feb 3, 2023

McGiverGim commented Feb 3, 2023

damosvil commented Feb 3, 2023 • edited

This comment has been minimized.

damosvil commented Feb 4, 2023 • edited

damosvil commented Feb 4, 2023 • edited

McGiverGim commented Feb 7, 2023

McGiverGim commented Feb 7, 2023

haslinghuis commented Feb 12, 2023

damosvil commented Feb 12, 2023

This comment has been minimized.

McGiverGim Feb 15, 2023

Choose a reason for hiding this comment

damosvil Feb 19, 2023 • edited

Choose a reason for hiding this comment

haslinghuis Dec 5, 2023

Choose a reason for hiding this comment

damosvil Dec 5, 2023

Choose a reason for hiding this comment

haslinghuis Dec 5, 2023

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

haslinghuis Nov 26, 2023

Choose a reason for hiding this comment

damosvil Dec 5, 2023

Choose a reason for hiding this comment

haslinghuis commented Dec 3, 2023

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sonarcloud bot commented Dec 13, 2023

Quality Gate failed

github-actions bot commented Dec 13, 2023

damosvil commented Feb 1, 2023 •

edited

damosvil commented Feb 3, 2023 •

edited

damosvil commented Feb 4, 2023 •

edited

damosvil commented Feb 4, 2023 •

edited

damosvil Feb 19, 2023 •

edited