SatNogs upload w/CRC does NOT appear on Grafana Dashboard

[K4KDR]

More of an FYI than an 'issue' or problem...

While decoding the AX100 downlink from GreenCube, I was of course using the 'Telemetry Submit' block to upload to SatNogs. My call sign appeared in the list of received uploads but did NOT appear on the Grafana Dashboard of contributed data.

Reviewing the dump of downlink packets from SatNogs, it was clear that some of the payloads included the trailing 4-byte CRC while most did not. Clearly my contributions via GNU Radio were the minority that were uploading decodes WITH the 4-byte CRC.

The issue was resolved by incorporating a 'PDU Head/Tail' block between the 'GOMspace AX100 Deframer' and the 'Telemetry Submit' block with the 'Mode' set to Head- (note the trailing '-') and the 'Number' field set to '4'. The resulting decodes were absent the 4-byte CRC and my decodes immediately appeared on the Grafana Dashboard for GreenCube.

Here are the 'before' and 'after' decodes for illustration:

Best regards,

-Scott, K4KDR

[N6RFM]

Thanks Scott for figuring this out! What's really interesting is that up until a few days ago, our GreenCube spots (which had the checksum) were listed on the Satnogs Grafana dashboard. Perhaps a recent Satnogs software update related to the dashboard changed the behavior.

Perhaps an opportunity in a future update of gr-satellites for the Telemetry Submit block to have an option, with or without the CRC.

[daniestevez]

Hi, thanks for reporting this.

The problem is mainly that the frame formats that should go into SatNOGS DB have never been standardized to ensure compatibility between different applications (SatNOGS Observations, gr-satellites, UZ7HO, DK3WN, etc.). I opened a thread about this in the Librespace forums years ago, but it never had much traction and not enough actions were taken. I understand that this is a lot of work, but there seems to be few support from the community to try to improve things.

In this case, for the AX100 modem gr-satellites outputs the full payload, which typically includes the CSP header and may or may not include a CRC-32C at the end. gr-satellites doesn't look at the payload or modifies it, except to print parsed telemetry or the parsed CSP header.

Historically (before v3.0.0), the flowgraphs in gr-satellites would check the CRC-32C (which is an optional part of CSP) if there was any. This was problematic, because:

1. There is a bit in the CSP header that indicates if CRC-32C is used, but some satellites didn't use this bit correctly.
2. Different satellites used different endianness for the CRC-32C.
3. Some satellites chose to include the CSP header in the CRC calculation and others chose not to.

This meant that the appropriate parameters to handle the CRC needed to be set by hand for each satellite. This required lots of book-keeping and caused problems if the settings changed in the middle of a mission. I don't recall if the CRC-32C was dropped after checking it.

This didn't provide many benefits, because when there is Reed-Solomon (which is often the case with the AX100), then if the Reed-Solomon decoder is successful, it is quite likely that the frame is correct (in deep space applications, CRCs are often omitted when Reed-Solomon is used, and used only with Turbo coding and other codings). For this reason, I decided not to handle the ckeck of CRC-32C's for CSP frames moving forward to v3.0.0. The block to check the CRC is still around and can be used if desired.

[daniestevez]

The issue was resolved by incorporating a 'PDU Head/Tail' block between the 'GOMspace AX100 Deframer' and the 'Telemetry Submit' block with the 'Mode' set to Head- (note the trailing '-') and the 'Number' field set to '4'.

I think that this is a very good solution to fix the problem on an individual basis by using a custom flowgraph.

Thinking about how to fix this for everyone, I notice that there is currently no way in SatYAML to indicate that the 4 last bytes of frames should be removed. Additionally, Bob mentioned that this used to work, and that maybe SatNOGS has changed recently. I wonder why we have to keep tracking what SatNOGS keeps changing for no apparent reason and without much discussion or heads up from their side.

Perhaps an opportunity in a future update of gr-satellites for the Telemetry Submit block to have an option, with or without the CRC.

Not sure if this is a good idea, since only some satellites have a CRC of 4 bytes at the end of the frames. Others have a CRC of a different size (2 bytes), and others have no CRC at all. And for others, it might depend on the type of the frame. Not sure what this particular option would do in all these other cases.

The solution of using PDU/Head Tail that Scott mentioned is much more flexible, and is probably a better way of stripping the last (or first) N bytes of a frame before telemetry submission if we want to do that for some reason.

[daniestevez]

So, as said, not sure what is the best action for gr-satellites to take here.

Perhaps a good idea is to reach out to SatNOGS folks to understand exactly what has changed recently and why (if anything at all has changed; this might be caused by a change in the satellite or something else).

[janvgils]

This was shared via the satnogs community portal

So apart from looking for some standardization, a question that has been brought forward multiple times, we can also be confronted with a possible change in the data send by the satellite.

[K4KDR]

Since this (a number of trailing bytes to remove) can vary from satellite to satellite - plus there could be other parameters that are specific to only one satellite, is it possible for GNU Radio "variables" to be set from entries in the SatYML file? If so, then flowgraphs could be standardized and values read from the SatYML file to customize the flowgraph on an as-needed basis.