noise.lag0 and noise.vel in fitacf-format files

[egthomas]

I have been trying to better educate myself on the different noise parameters in the rawacf and fitacf format files, and was wondering about the noise.lag0 and noise.vel fields in fitacf files. As far as I can tell, these fields were populated by older versions of FITACF via the noise_acf function, which was removed from FITACF following discussions at the 2006 SD workshop, eg:

# $Log: version.info,v $
# Revision 2.0  2007/02/02 21:42:06  code
# Changes agreed to at the 2006 SuperDARN meeting.

and

 $Log: do_fit.c,v $
 Revision 1.16  2007/02/02 21:38:42  code
  Removing everything belonging to NOISE_ACF calculation and extraction
 (lines 153-160 and 198-251 in Revision 1.15)

(and likely influenced by the work of Ponomarenko and Waters [2006])

The "modern" fitting algorithms in the RST (eg, fitacf2, fitacf3, lmfit2) all seem to set noise.lag0 and noise.vel to zero (the noise.sky value, however, is of course still used).

So, here is a philosophical question - we have added several new fields to fitacf, map, etc files over the years, but would we ever remove a field that is no longer relevant? Is there value in reading/storing the noise.lag0 and noise.vel values from fitacf-format files (or more likely fit-format files) processed with pre-2007 versions of the software?

Note there are several fields from the old parameter block which were dropped with the conversion from dat/fit to rawacf/fitacf (e.g., nbaud and the various usr_*-defined parameters).

[pasha-ponomarenko]

@egthomas , from years ago my recollection is that there was some serious technical problem with removing the existing fields, but I never knew the exact details. I think then we were talking about removing fields related to XFC power fitting.

[ecbland]

@pasha-ponomarenko I removed all the unused XCF fields for fitacf3 files in #472, so there shouldn't be any technical issue here. I'm all in favour of @egthomas's suggestion to remove the unused fields.

[pasha-ponomarenko]

Gee, I am getting senile! 🤔
Of course, please go ahead with removing these fields.

[egthomas]

On a somewhat related note, I just realized that grid (and map) files use the noise.search parameter to calculate the noise_mean and noise_sd values. Not all radars seem to record noise.search values, so maybe this should be changed to something else like sky noise?

[pasha-ponomarenko]

Yep, this is probably a better solution. I was always uneasy with using the search noise level in conjunction with FITACF data as its magnitude is established using a very different algorithm and much shorter integration time. I believe that originally there was some magic scaling coefficient applied to make the search noise compatible with that estimated from low-power ACFs.

[egthomas]

@pasha-ponomarenko thanks for the response - I've just opened a pull request to change the underlying library to use the sky noise value rather than the search noise.

(I haven't yet had the time to examine removing the noise.lag0 and noise.vel fields to see if that causes any issues elsewhere in the RST)

[pasha-ponomarenko]

@egthomas, do you know how the noise level is utilised in gridding/mapping routines? I can only see that it is copied into the respective file formats.

[egthomas]

@pasha-ponomarenko as far as I can tell, the noise values recorded in the grid files are only for informative purposes and not used in any subsequent processing algorithms (eg Map Potential, or even plotting).

[pasha-ponomarenko]

Thanks, @egthomas , this was my impression too: just to maintain consistency between the formats.