Stereo offset for legacy data

[egthomas]

Buried deep in fitacf.c of FITACF 2.5 there is the following comment:

    /* need to incorporate Sessai's code for setting the offset
       for legacy data here.
    */

which, as far as I can tell, has been present in this file / FITACF since at least 2006 (e.g., https://github.com/SuperDARN/rst-archive/blob/rst.2.05/code/src.lib/superdarn/fitacf.1.22/src/fitacf.c#L169-L171). For context, our software currently assumes the offset value for Stereo data in dat- and fit-format files is stored in the usr_resL1 field.

In a copy of Sessai's code, here was his explanation of this issue:

We should be very careful for the fact that
some RCPs for mono mode use usr_resL1 for some other purposes!!!
also note that there was an period when
the 'CHANNEL' parameter had not been introduced in Radops parms,
in the very early stereo era.
So in order to know if the data is stereo or mono correctly,
we should check all of ST_ID, YEAR, CHANNEL and usr_resL1!!!!!!!

I'll try reaching out to Sessai, but my impression is that this fix needs to be implemented at a more fundamental level (ie at the I/O stage within OldRawReadCurrent of the oldraw library and OldFitReadCurrent of the oldfit library) rather than within a later processing stage such as FITACF. And I suppose the native IDL dat / fit code may require modifications too...

[egthomas]

Following up on this, at least one control program (cpid = 4204, from 1998-2009) used the usr_resL1 field to store information other than a Stereo offset value. This will affect FITACF 2.5 such that the wrong Badlags function will be called, eg (https://github.com/SuperDARN/rst/blob/main/codebase/superdarn/src.lib/tk/fitacf.2.5/src/do_fit.c#L195-L196)

  if (iptr->prm.offset==0) FitACFBadlags(&iptr->prm,&badsmp);
  else FitACFBadlagsStereo(&iptr->prm,&badsmp);

although FITACF 3.0 appears to be unaffected since it also compares against the channel number, eg (https://github.com/SuperDARN/rst/blob/main/codebase/superdarn/src.lib/tk/fitacf_v3.0/src/preprocessing.c#L609)

  if((offset != 0) && ((channel == 1) || (channel == 2))) {

As an aside, LMFIT (v1) appears to handle this completely incorrectly (comparing against channel only), eg (https://github.com/SuperDARN/rst/blob/main/codebase/superdarn/src.lib/tk/lmfit.1.0/src/lmfit.c#L392-L397)

  /*calculate noise levels*/
  if(prm->cp != -3310 && prm->cp != 3310 && prm->cp != 503 && prm->cp != -503)
  {
    if(fblk->prm.channel==0) FitACFBadlags(&fblk->prm,&badsmp);
    else FitACFBadlagsStereo(&fblk->prm,&badsmp);
  }

(edit note that FITACFEX2 also uses this incorrect logic)

while LMFIT v2 does not appear to treat Stereo data differently at all (at least based on either the channel or offset fields), which doesn't seem quite right.

Getting back to the early Stereo mode data, it appears the channel field was not introduced to the dat and fit parameter block at HAN/PYK until 2002-04-24. So prior to that Stereo HAN and PYK files have a (correct) non-zero offset but no channel information. As far as Tim and I can tell, the potentially very early Stereo data where neither the offset or channel information are recorded were either never distributed or do not exist.

[egthomas]

Also, because STO dat files from Jul 2006 - Jan 2009 were converted to rawacf with dattorawacf and we no longer have the originals, STO rawacf files on the data servers with the 4204 CPID also have a non-zero offset, eg (20090101.0000.11.sto.rawacf):

scalars:
	char	"radar.revision.major" = 1
	char	"radar.revision.minor" = 66
	char	"origin.code" = 1
	string	"origin.time" = "Wed Feb 25 15:55:37 2009"
	string	"origin.command" = "/home/dproc/bin/dattorawacf /data1/tmp/tmp.T30861"
	short	"cp" = -4204
	short	"stid" = 8
	short	"time.yr" = 2009
	short	"time.mo" = 1
	short	"time.dy" = 1
	short	"time.hr" = 0
	short	"time.mt" = 0
	short	"time.sc" = 11
	int	"time.us" = 0
	short	"txpow" = 9000
	short	"nave" = 28
	short	"atten" = 0
	short	"lagfr" = 1200
	short	"smsep" = 200
	short	"ercod" = 0
	short	"stat.agc" = 31744
	short	"stat.lopwr" = -8252
	float	"noise.search" = 434.000000
	float	"noise.mean" = 2261.000000
	short	"channel" = 0
	short	"bmnum" = 15
	float	"bmazm" = -34.325001
	short	"scan" = 1
	short	"offset" = 1
	short	"rxrise" = 100
	short	"intt.sc" = 3
	int	"intt.us" = 0
	short	"txpl" = 200
	short	"mpinc" = 2400
	short	"mppul" = 7
	short	"mplgs" = 18
	short	"nrang" = 70
	short	"frang" = 180
	short	"rsep" = 30
	short	"xcf" = 0
	short	"tfreq" = 13220
	int	"mxpwr" = 4400000
	int	"lvmax" = 20000
	int	"rawacf.revision.major" = 6
	int	"rawacf.revision.minor" = 5
	string	"combf" = "$Id: fte_4204.c,v 1.9 2000/11/29 17:16:37 barnes Exp $"
	float	"thr" = 3.000000

So either these files need to be corrected, or the rawacf reading software will need to be modified too. Or maybe the logic to handle this should go inside all of the fitting routines too?