CALWF3 contains a hard-coded value for the maximum number of rows read in CTE table

[mdlpstsci]

The CTE calibration reference file contains two binary FITS tables and two images with the maximum number of traps set to 999 in cte.h. The traps value being set at 999 corresponds to the number of rows in the first binary table, QPROF, of the CTE FITS file, as well as to the first dimension (NAXIS1) of the RPROF and CPROF images found in the third and fourth extensions of the CTE FITS file. The traps value in CALWF3 cte.hfile SHOULD be >= with the number of rows in the QPROF table, and the NAXIS1 dimension of the RPROF/CPROF images in order to ensure the code works properly.

The root of the problem was the instrument team created a new candidate CTE reference file with greater than 999 row, and CALWF3 generated poor output without any complaint. Here is the structure of the CTE reference file as it is currently constructed.

EXT# FITSNAME FILENAME EXTVE DIMENS BITPI OBJECT

0 zcv2057mi_cte zcv2057mi_cte.fits 16
1 BINTABLE QPROF 1 4Fx999R
2 BINTABLE SCLBYCOL 1 6Fx8412R
3 IMAGE RPROF 1 999x100 -32
4 IMAGE CPROF 1 999x100 -32

There is a check in CALWF3 which is an attempt to determine if an updated CTE reference file were modified to have more rows in the QPROF table, and the RPROF/CPROF image dimensions were increased. The only items which are counted as traps are when (qlev_q < 999999) where qlev_q is the second column in the table. This check is NOT USEFUL if the reference file were extended to have more than 999 rows AND the number of (qlev_q values < 999999) exceeds 999. The problem is that CALWF3 will only check up to 999 rows. If there are more rows AND they have real trap values (not the 999999 indicator), then CALWF3 is basically not getting the proper information which will cause unpredictable results. The new candidate QPROF table contains more than 999 rows with the (qlev_q value < 999999) which silently caused a problem in CALWF3.

[mdlpstsci]

In order for the team members to continue their analysis, two quick solutions were pursued. I wrote the specific data the team members were interested in examining out to a pre-existing temporary output file (rsz data written to the *rac_tmp.fits output file). This was done on a private branch with the code being provided to the team members with the clear indication this code will be deleted (version CALWF3 3.4.2 TestCTE). At the same time, Jay exploited the use of the SCLBYCOL table to provide scaling factors for the actual traps in the data which allowed for a smaller (< 999) CTE table. Once it was understood the hard-coded value for traps (and hence rows read from table) was the issue, team members were able to continue their analysis.

Having said all this, the maximum number of rows to be read should be data-driven, but the code will need to checked carefully to ensure the assumption the maximum number of traps is not built-into the CTE algorithm.