Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move pixel_replace step #8340

Open
stscijgbot-jp opened this issue Mar 11, 2024 · 5 comments · May be fixed by #8409
Open

Move pixel_replace step #8340

stscijgbot-jp opened this issue Mar 11, 2024 · 5 comments · May be fixed by #8409

Comments

@stscijgbot-jp
Copy link
Collaborator

Issue JP-3566 was created on JIRA by David Law:

At present, the pixel_replace step happens in spec2, after photometric calibration and prior to writing out the cal.fits files.  There are two downsides to this: First, any new outliers detected in spec3 won't be able to run through pixel replacement prior to resampling/cubebuild into the final data products.  Second, any users wishing to try running with/without pixel replacement need to run both the spec2 and spec3 pipeline.

We should therefore move the pixel_replace step to immediately before resample/cube_build, in both spec2 and spec3 pipelines.  This would happen after producing the cal files, so that the cal files for a given exposure do not contain any replaced values.  If turned on for spec2, the s2d/s3d files would have thus been pixel-replaced prior to resampling.  If turned on for spec3, the s2d/s3d files would likewise have been pixel-replaced prior to resampling.  Essentially, while pixel_replace is a discrete step it can be thought of as an option to resample/cubebuild, either replacing just prior to resampling or not depending how the param ref files are set up.

Thoughts?  Tyler Pauly Greg Sloan Melanie Clarke 

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Greg Sloan on JIRA:

I agree with the reasoning. We should move pixel_replace so that the cal.fits files are saved before pixels are replaced.

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Melanie Clarke on JIRA:

I agree as well - I think this is a good first step toward making missing pixel handling more robust.  This would save interpolation until the last minute possible with the current implementation, and make it much easier to generate products with and without pixel replacement for comparison.

For more future improvements, we might want to consider moving the interpolation even further down in the processing, during spectral extraction and/or resampling.  It might be possible to make a more robust spatial profile to fit, for example, if all input data is available to the algorithm, or if a PSF model is provided for optimal extraction.

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Jane Morrison on JIRA:

Howard Bushouse or Greg Sloan I am updating the regression tests. I updated test_miri_lrs_slit_spec2.py to do the pixel_replacement and not skip it - since that is how the default pipeline is run.
I want to make sure the s2d results are the expected results. When a pixel is a nan in the input to resample it seems that the resulting s2d pixel value is 0 . The CON plane has a value of 0 in this case. I wanted to make sure that to be consistent with other products these pixels should be a NAN. This would be a nice JP-ticket but I thought since I saw this I would ask.

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Greg Sloan on JIRA:

Seems like NaNs should stay NaNs. Zeroes can be interpreted as valid data. But I want to be clear on what is happening. Is pixel_replace writing "0", or is resample_spec doing that?

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Howard Bushouse on JIRA:

Jane Morrison Greg Sloan The issue of zero vs. NaN in resampled images is the subject of the long-standing ticket JP-3281, which was just discussed at the JP coordination meeting yesterday. They come out zero in resampled images because of the setting of the FILLVAL=INDEF parameter in resample. Once the default gets changed for that to NaN, then resampled pixels that have no contribution from inputs will be set to NaN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant