Submit decoded frames to SatNOGS observation

[daniestevez]

This idea originated here. gr-satellites is sometimes used as a post-observation script in SatNOGS. It would be good if it can send the demodulated frames to SatNOGS so that they appear in the "data" tab of the corresponding observation.

[kerel-fs]

🪲 Missing frame reception timestamp for recordings

When uploading frames to satnogs-network they must be submitted with the timestamp of reception as part of the filename. On a normal station this timestamp is created from the system clock in the gr-satnogs.frame_file_sink#L71-80.

Thus we need some method to reconstruct this timestamp within gr-satellites for demodulated frames.

💡 Possible solution

gr-satellites already supports the generation of reception timestamps when using the KISS output, but those timestamps are only correct when it is run in real-time during an observation, as they are similarly created from the system clock in pdu_to_kiss.py#L40-L56.

For replayed recordings adding a --start-time TIMESTAMP parameter to the CLI which specifies the timestamp of the start of the recording would in principle allow gr-satellites to reconstruct the required timestamps. It would require us to get the elapsed time since the start of the recording in pdu_to_kiss.create_timestamp.

@daniestevez What do you think about this solution & do you think it is possible?

[daniestevez]

Thanks for your interest in this @kerel-fs!

First of, if gr-satellites is run in real-time, this is a not an issue, as you said. Currently I'm thinking that the best way to run gr-satellites for integration with SatNOGS is in real-time, alongside gr-satnogs, rather than making it run afterwards on the IQ file generated by gr-satnogs.

However, it would be nice to have some functionality to allow generating timestamps of recordings correctly. There is in fact an issue #92 open about this, but I haven't had time to work in it so far.

Your suggestion is in fact how the older versions v1 and v2 of gr-satellites worked. The disadvantage is that this approach needs to play back the recording at 1x speed (with a throttle block). This is not very convenient. No one really wants to wait 10 minutes for a full pass recording to process in order to have accurate timestamps. gr-satellites is often capable of processing the full file in a matter of seconds. (As background information, in v1 and v2 it wasn't possible to process recordings as fast as allowed by the CPU. The flowgraphs only supported input by streaming in samples with UDP, so samples were lost if you streamed too fast, and streaming the recording at 1x speed was the only "supported" way to work. Therefore, it was reasonable to implement the timestamp handling as you suggested).

As things work now, we read recordings as fast as the CPU allows (no throttle block at all in the flowgraph), so I think that timestamp handling should be implemented correctly by counting samples rather than using the UNIX clock. This is possible, but not so easy because GNU Radio doesn't have any notion of time or counting samples.

My idea about how to do this is roughly:

• At the input of the flowgraph have a blocks that knows the recording sample rate and inserts tags periodically (say every 1 second). This block is bootstrapped by indicating the recording start timestamp as a parameter, so that it generates the correct timestamp tags.
• Handle tag propagation correctly throughout the stream processing chain, specially where we decimate.
• At some point we detect syncwords and tags to extract PDUs with the packets. The blocks that do this should add tags with the timestamp corresponding to the packet so that these tags end up as metadata in the PDU. The naïve way to do this is just to insert the last timestamp tag that was seen by the block (so we can err by at most one second). The fancy way is to know the symbol rate and count samples after the last seen timestamp tag, so that we can accurately determine the timestamp corresponding to the beginning of the packet. In any case, we end up with a timestamp metadata in the PDUs.
• PDU metadata needs to be propagated correctly throughout all the chain.
• When PDUs arrive to the blocks that write KISS output or to send telemetry to SatNOGS DB, if there is timestamp metadata (i.e., if we're running from a recording and the user specified the starting timestamp), then we use that value, if we're not running from a recording, we the UNIX clock, else (if we're running from a recording but the user didn't give the timestamp information), we don't add a timestamp to the KISS output and we don't send the telemetry frame to SatNOGS DB.

So all this is doable but takes quite some time and I haven't set my head to doing it, so I guess it won't be done soon. While this approach makes total sense to me in applications much wider than gr-satellites, I haven't seen this done elsewhere. For bonus points this should be compatible with timestamp tags generated by UHD and similar radios. For example, if you have a meta-file including timestamps, then the timestamps should come from the file metadata rather than counting samples.

In any case, I think that >90% of the use cases will be real-time processing. For the remaining cases, we could do as in gr-satellites v2 or even the user could do some hacky setup by using libfaketime or similar (I have never used this; it just came out of a Google search) and arranging for the recording to be played back in real-time (for instance by streaming it in real-time by UDP).

So for the matter of the discussion in this thread, let's assume that gr-satellites is already producing good timestamps because it's running in real-time or the user has set-up something using libfaketime. Discussion about how is best to generate timestamps from a recording fits better in #92.

TLDR: This only affects recording processing, and to allow processing the recordings faster than 1x this should be done by counting samples rather than using the UNIX clock/elapsed time. However, this takes significant effort and will not be done soon. Still, the real-time case is probably the most popular.

[kerel-fs]

Counting samples is exactly what I envisioned when I wrote "get the elapsed time" (ignoring for the fact that the recording could be missing samples). Thanks for the detailed explanation on how this could be properly implemented! Tbh I didn't expect counting samples to be so complicated (my gnuradio experience is limited).

In any case, I think that >90% of the use cases will be real-time processing

Unfortunately I'll not be able to run gr-satellites in real-time (I'm not the station operator) so my use case here (and also the one described by @cpbridges elsewhere) is exactly one of the harder ones as it seems. :)

[daniestevez]

Thanks for the clarification. I understood "get the elapsed time" as calling clock_gettime() or similar (probably because I was biased, as this was what I did on gr-satellites v2).

The difficulty of counting samples in GNU Radio comes mainly from having to deal with sample rate changes (especially variable rate, such as you would have for clock recovery), and because at some point you got to PDUs, so there aren't samples any more.

As a workaround, would playing the recording at 1x speed be acceptable in your use-case? I'm thinking that it's easy to add a --throttle argument to gr-satellites to force playback at 1x and use --start-time TIMESTAMP by calling clock_gettime(). This may be a good compromise solution until the correct sample-counting approach is done, and having it like this will be easier to use than with libfaketime.

[kerel-fs]

As a workaround, would playing the recording at 1x speed be acceptable in your use-case?

Yes, this would be fully acceptable for now!

Running throttled is not a problem for me when dealing with a small number of observations to process (e.g. when running in a post-observation script or manually re-decoding selected recordings with gr-satellites), and the resulting accuracy wouldn't be much different than what we can expect when the timestamp was created by gr-satnogs (using system clock as well).

[daniestevez]

Then I'll set to implement this. Probably during this afternoon.

[daniestevez]

This is done now in 122ccea. I have tested that it is working with the KISS file sink. I haven't tested it with the SatNOGS DB submit block, since I don't have a recording with an accurate timestamp at hand and I don't want to fill the DB with tests. It should work, since most of the code is common.

To use this, you need to include both the --throttle parameter to limit playback speed to 1x and the --start_time parameter to specify the start of the recording (the format is YYYY-MM-DD HH:MM:SS).

Please let me know if this works well for you, and in that case I'll add the description of this functionality to the documentation.

[kerel-fs]

Thanks for implementing this! I did some first test with observation 2901793 and the results are as expected (ie. inital offset due to setup time and potentially a small drift) and acceptable for the use case imho.

original timestamp	reconstructed ts	delta/s
2020-09-28 20:45:56	2020-09-28 20:45:52	-4
2020-09-28 20:46:13	2020-09-28 20:46:09	-4
2020-09-28 20:46:43	2020-09-28 20:46:39	-4
2020-09-28 20:47:00	2020-09-28 20:46:55	-5
2020-09-28 20:47:01	2020-09-28 20:46:55	-6
2020-09-28 20:47:02	2020-09-28 20:46:56	-6
2020-09-28 20:47:03	2020-09-28 20:46:57	-6
not decoded	2020-09-28 20:46:58
2020-09-28 20:47:15	2020-09-28 20:47:10	-5

edit: The code for this short analysis can be found here.

[daniestevez]

Thank you for the test. I'm wondering about the -4 s initial delta (this means that gr-satellites decoded the packet earlier than it should). Maybe the throttle block let some seconds of the wav file rush in at the beginning (it only regulates average rate). I also think an error of a few seconds is acceptable, any way.

Keep me updated about how well this is working (I think there were some people interested in hooking up gr-satellites in the post-observation script) and if there is something else I can do to help.

[DL4PD]

Just my 2 cents on testing: you can always use db-dev.satnogs.org for that. No worries about illegal frames or the amount of tests you're running.