rewind(fp) has no effect when reading from stdin

[ecbland]

BUG

A few routines in RST use rewind(fp) to navigate back to the start of a file. This works well when reading from an actual file, but it does not always work when reading from stdin. I discovered this while fixing #424.

Priority

• Minor - typo, incorrect naming, fails in a specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation of data

RST version

Does the bug occur in an official RST release?

• Yes. RST version: v4.6 and several previous versions
• No. Branch containing the bug (e.g. develop): ______

Example of the bug

Notice that the first two records (06:01:00 and 06:01:03) are missing when reading from stdin

# -----------------------------------------------------------
# trim_fit reading from a fitacf file
# -----------------------------------------------------------
make_fit 20150305.0601.00.inv.rawacf > 20150305.0601.00.inv.fitacf
trim_fit -st 06:00 -et 07:00 20150305.0601.00.inv.fitacf > from_file.fitacf
dmapdump from_file.fitacf | grep -A 2 time.hr | head -12

	short	"time.hr" = 6
	short	"time.mt" = 1
	short	"time.sc" = 0
--
	short	"time.hr" = 6
	short	"time.mt" = 1
	short	"time.sc" = 3
--
	short	"time.hr" = 6
	short	"time.mt" = 1
	short	"time.sc" = 6
--


# -----------------------------------------------------------
# trim_fit reading from stdin
# -----------------------------------------------------------
make_fit 20150305.0601.00.inv.rawacf | trim_fit -st 06:00 -et 07:00 > from_stdin.fitacf
dmapdump from_stdin.fitacf | grep -A 2 time.hr | head -12

	short	"time.hr" = 6
	short	"time.mt" = 1
	short	"time.sc" = 6
--
	short	"time.hr" = 6
	short	"time.mt" = 1
	short	"time.sc" = 9
--
	short	"time.hr" = 6
	short	"time.mt" = 1
	short	"time.sc" = 12
--

The following routines can read from stdin and use rewind(fp):

• trim_fit (see Fix trim_fit to allow removal of first record in fit file #158)
• trim_snd
• fit_speck_removal ("fixed" with Remove option to read from standard input in fit_speck_removal #454 by removing option to read from stdin)

Note that the other trim_* routines are not affected -- they just skip over the start of the file so we get the same output reading from a file and from stdin.

---

time_plot has a similar bug, introduced in #235 (by me 🤣 )
It doesn't explicitly use rewind(fp) but it does essentially the same thing with FitSeek/OldFitSeek

# this produces a blank plot...
make_fit 20150305.0401.00.inv.rawacf | time_plot -x -a -b 5

#if we specify -ex or -et, the routine works correctly
make_fit 20150305.0401.00.inv.rawacf | time_plot -x -a -b 5 -ex 02:00
make_fit 20150305.0401.00.inv.rawacf | time_plot -x -a -b 5 -et 06:00

Potential Bug Location

rst/codebase/superdarn/src.bin/tk/tool/trim_fit.1.20/trim_fit.c

Lines 389 to 390 in 5b133f3

	/* rewind file pointer in case we want to trim first record */
	rewind(fp);

rst/codebase/superdarn/src.bin/tk/tool/trim_snd.1.0/trim_snd.c

Lines 195 to 196 in 5b133f3

	/* rewind file pointer in case we want to trim first record */
	rewind(fp);

rst/codebase/superdarn/src.bin/tk/tool/fit_speck_removal.1.0/fit_speck_removal.c

Lines 236 to 237 in 5b133f3

	//** Read the file again, apply the median filter, and write a new file
	rewind(fp); // rewind the file pointer

rst/codebase/superdarn/src.bin/tk/plot/time_plot.1.8/time_plot.c

Lines 1108 to 1136 in 5b133f3

	if (etime !=-1) {
	if (edate==-1) etime+=stime - ( (int) stime % (24*3600));
	else etime+=edate;
	} else if (extime !=0) etime=stime+extime;
	else {
	/* determine end time from the last record in the file
	if the total time range is less than 10min then set it to 10min.
	this has been implemented only for fit/fitacf format files */
	int status=0;
	if (fitflg && old) {
	while (status!=-1) status=OldFitRead(oldfitfp,prm,fit);
	etime=TimeYMDHMSToEpoch(prm->time.yr,prm->time.mo,prm->time.dy,
	prm->time.hr,prm->time.mt,
	prm->time.sc+prm->time.us/1.0e6);
	TimeEpochToYMDHMS(stime,&yr,&mo,&dy,&hr,&mt,&sc);
	status=OldFitSeek(oldfitfp,yr,mo,dy,hr,mt,0,NULL);
	if (status==-1) {
	fprintf(stderr,"Error determining start/end time of file. Please specify using -st and -et options.\n");
	exit(-1);
	}
	status=OldFitRead(oldfitfp,prm,fit);
	} else if (fitflg) {
	double atme;
	status=FitFseek(fitfp,prm->time.yr+1,prm->time.mo,prm->time.dy,
	prm->time.hr,prm->time.mt,0,&atme,inx);
	if (status!=-1) { /* status should be -1 if end of file was reached successfully */
	fprintf(stderr,"Error determining start/end time of file. Please specify using -st and -et options.\n");
	exit(-1);
	}

Potential Solution(s)

I don't know whether this problem can actually be solved. It seems to be a limitation of reading from stdin, but I'll defer to the experts 😄