Range offset correction

[pasha-ponomarenko]

In a number of radars an incorrect value of the distance (prm.frang) and time delay (prm.lagfr) to the first range gate has been recorded. This may happen for a number of reasons and leads to errors in geolocation and virtual altitude estimates. Moreover, the resulting misidentification of samples blanked during transmitter emission creates data gaps at fixed range gates. This issue has been identified and resolved as early as in 2011-2012, and an on-line tool for estimating the range offset has been added to VT webpage. However, no effective way of post factum range correction in the data has been implemented. I suppose, the most direct solution would be to introduce an extra parameter in the hardware files, e.g., rdiff, that will be used for automatic correction of the recorded values of frang and lagfr before coordinates or group range are calculated.

[ksterne]

I thought we had a solution of modifying the rawacf files to put the correct values of frang and lagfr in. Is this a brainstorming of how it could be done without modifying the rawacf file? If so, I'm not sure if it's necessary since we've modified files in the past and it didn't seem to be an issue. Maybe I'm not aware of issues that have appeared.

[ecbland]

I'm interested in following this discussion, as I believe LYR is affected by this issue too (I think the offset is 2 samples)

@pasha-ponomarenko

an on-line tool for estimating the range offset has been added to VT webpage.

Which tool are you referring to? Can you provide the link?

@ksterne

I thought we had a solution of modifying the rawacf files to put the correct values of frang and lagfr in.

Are you referring to the lagfr_fix routine in RST? This requires the user to know (a) whether or not a correction is required for a particular radar, and (b) the input values to perform the correction. This information is not readily available to users.

I think the onus should be on the PI to provide accurate frang/lagfr information so that users can calculate correct group ranges without any extra work. Whether this is through modification of the rawacf files or a new entry in the hardware file is up for discussion. Since the information currently distributed in the rawacf files for the affected radars is incorrect, it seems sensible to correct the rawacf files directly. However, this is probably an enormous task, and may create issues with reproducability.

[pasha-ponomarenko]

@ksterne, yes I want to have a brainstorming session on this issue. My point here is that why should the range offset fix be any different from that of the phase offset, especially for the historic data? The rewriting of lagfr (and, thus, breaking the unwritten convention that the raw data should not be modified) was applied more out of desperation because at that time (2012-2013) it was virtually impossible to modify RST without breaking it. However, now, thanks to Evan and everyone else, we are in a very different situation and can apply a more robust solution which would not affect the data reproducibuility, as @ecbland pointed out above.

Can you provide the link?

@ecbland, the bad lag finder is here
http://vt.superdarn.org/tiki-index.php?page=Badlag_Find
Indeed, it shows a two-gate offset (the first range gate is ~90 km farther then assumed):
(1539707757.pdf)

[ksterne]

I think that's a good question of how is the range offset different than the phase offset. I think this gets to a fundamental question: do we want to distribute data with errors?

My thinking or explanation with that question is that the range offset is different than the phase offset because the phase offset applies to the elevation angle that's calculated in fit-level processing. I will admit, the offset wouldn't be needed if the acfs (or some other parameter) were recorded in the correct order. However, looking at documentation for this repo, I don't see anything in the definition of the xcfd array that assumes the alignment of the data array. I also don't know if it's possible in the current radar operation softwares (ROS) to introduce this kind of shift in recording. Maybe in some of the newer SDR-based codes.

On the other hand, it seems clear that the lagfr and frang parameters note that's the value to the first range gate. So I'd think keeping the incorrect values for these two parameters would be distributing erroneous data unless the documentation was updated to note with the exception of a range offset issue.

I'm somewhat confused on the data reproducibility point here. Wouldn't introducing a range offset parameter make the data not as reproducible when going to fit-level data? I understand this on a sense that it modifies rawacf files, but the DDWG has been good on the distribution and updates of new files.

[pasha-ponomarenko]

My thinking or explanation with that question is that the range offset is different than the phase offset because the phase offset applies to the elevation angle that's calculated in fit-level processing.

The range/time to the first range gate is used in:
(a) calculating samples affected by Tx-overalp, and these samples are removed (or mistakenly not removed!) during the fitting, so the fit-level data quality is affected as well
(b) calcualting of geolocation used for gridding/mapping.

I will admit, the offset wouldn't be needed if the acfs (or some other parameter) were recorded in the correct order etc...

Now I am confused... Order? Alignment? Do you mean time or range? If sampling starts earlier or later than it should, then we do have an offset. For example, in our case it was introduced while calculating latency of the digital filters, which at that stage could only be estimated within one range gate accuracy.

I opened this issue mainly because I feel somewhat uneasy about modifying the primary data, especially few years after they've been recorded, without leaving a comprehensive record on why (or at least when!) it has happened and what was the offset.

The reproducibility issues I mentioned are related to comparing the fit and grid/map data that are obtained before and after the correction. If we have an rdiff record in the hdw file, then we would be able to trace the cause of the observed discrepancy easily.

[ecbland]

Just wanted to add that the Longyearbyen PI is supportive of adding a new field to the hardware file, rather than modifying the rawacf files.

[egthomas]

Following the DAWG meeting at last week's workshop, I've searched through the ACF fitting libraries and it appears neither the rxrise value in the radar parameter block or the recrise value in the hdw file is used anywhere. Those values are (currently) only used within the RST for geolocation purposes. Hence my confusion when the discussion turned to using the recrise value in the hdw file to account for an incorrect frang/lagfr value.

If I understand @pasha-ponomarenko's proposed solution correctly, wouldn't that introduce additional fitting errors for those (many) radars that already have a non-zero value of recrise in their hdw files but a correct frang/lagfr?

[sshepherd]

It is used in the function slant_range() in the file cnvtcoord.c return (lagfr-rxris+(range_gate-1)*smsep+range_edge)*0.15; So the idea is that rxris is adjusted so that lagfr-rxris gives the correct value. Since rxris is in the hdw.dat files it is a "cheat", but avoids reprocessing rawacf files. There are, of course, dangers of doing this, so it should be well-documented.

…

On Mon, Jun 10, 2019 at 12:04 AM Evan Thomas ***@***.***> wrote: Following the DAWG meeting at last week's workshop, I've searched through the ACF fitting libraries and it appears neither the rxrise value in the radar parameter block or the recrise value in the hdw file is used anywhere. Those values are (currently) only used within the RST for geolocation purposes. Hence my confusion when the discussion turned to using the recrise value in the hdw file to account for an incorrect frang/ lagfr value. If I understand @pasha-ponomarenko <https://github.com/pasha-ponomarenko>'s proposed solution correctly, wouldn't that introduce additional fitting errors for those (many) radars that already have a non-zero value of recrise in their hdw files but a correct frang/lagfr? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#191?email_source=notifications&email_token=AAWYGP2EGYFW3PMY2HG2XMDPZXHDJA5CNFSM4F3VEVZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXI32UQ#issuecomment-500284754>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWYGP5VKXSN3VT3GCMLVSDPZXHDJANCNFSM4F3VEVZQ> .

[pasha-ponomarenko]

I have just looked through the FITACF2 and 3 codes and relaised that instead of rxris a value of 100 us was hardcoded while determining samples affected by the RX pulses
(line 70 in badlags.c)

t2 = t1 + 3*ptr->txpl/2 + 100; /* adjust for rx-on delay */

and the same has been reproduced in FITACF3
(line 655 in preprocessing.c )

t2 = t1 + 3.0 * fit_prms->txpl/2 + 100;

This means that these two files have to be changed accordingly before we start using prameter 14 in *.hdw files (analog Rx rise ) for correcting the range offset.

[egthomas]

@sshepherd and @pasha-ponomarenko thanks for the responses, I understand now why it wasn't obvious that rxrise has an effect in the FITACF software if the "standard" value of 100 us was hardcoded rather than read from the parameter block or hardware file.

So I guess my point is this - within the last ~2 years, we've changed the receiver rise time values in the hdw files for many radars (CLY, INV, PGR, RKN, SAS, MCM, SPS) from 100 us to 0 us (not to mention the Syowa pair of radars which have always used 50 us in both their hardware and site files). If those changes were correct and a hardcoded value of 100 us was still being used within FITACF 2.5 / 3.0, shouldn't that mean errors have been introduced into the fitacf-level files for each of those radars?

[pasha-ponomarenko]

@egthomas, in FITACF this has only effect on determining samples that are contaminated through the TX pulse overlap. The basic value of smsep is 300 us, therefore, to have any detrimental effect on the fitted data quality the offest should be equal to or exceed 300 us. This means that changes of 50-100 us are generally OK for 45-km data, but one needs to be careful with 15-km resolution.

My point above is that in order to correct larger offsets (one range gate or more) we need first to replace the hardcoded 100-us values with rxris.

[mts299]

@pasha-ponomarenko hardware files are now under DSWG and if you want to add another field to any file you need to raise the issue to them: https://github.com/SuperDARN/dswg-published-docs/issues

I think we can close this now?

[ecbland]

I agree that the original issue raised here should be handled by the DSWG. But perhaps we could keep the issue open until someone (probably me) submits it as an issue to DSWG? Just to have a smooth handover :)

We should still address the hard-coded rxrise issue that @pasha-ponomarenko identified.

[mts299]

@ecbland did you want to make the DSWG issue still?

[ecbland]

@mts299 Yes but I haven't got to it yet.

Also, we have not yet fixed the hard-coded rxrise issue.

[mts299]

Okay, if someone can explain the hard-coded value to me and how to fix it and where it needs to be fixed I could probably do it.

[mts299]

@pasha-ponomarenko @ecbland is this something DAWG can do since DSWG is limbo?