JP-2922: Use new source_id syntax for level-3 products #8442

hbushouse · 2024-04-26T14:10:43Z

This PR updates all the source_id handling machinery within associations, calwebb_spec2 steps, and the calwebb_spec3 pipeline to handle the new source_id syntax for level-3 source-based products, which includes an increase in the number of source_id digits from 5 to 9, and implements the use of "b" and "v" source prefixes for NIRSpec MOS data extracted from background and virtual slits, respectively.

Note that the exp_to_source routine has been updated to use "source_name" as the key for sorting MOS datamodels, because it's the only piece of meta data available that properly distinguishes between "source", "background", and "virutal" slits. source_id values can overlap with one another for MOS now (e.g. a "real" source slit can have the same source_id value as a background slit). Other multi-slit modes (NIRSpec FS and NIRCam/NIRISS WFSS) still use source_id for sorting in exp_to_source, because they don't have source_name populated (names are unknown).

Checklist for maintainers

added entry in CHANGES.rst within the relevant release section
updated or added relevant tests
updated relevant documentation
added relevant milestone
added relevant label(s)
ran regression tests, post a link to the Jenkins job below.
How to run regression tests on a PR
Make sure the JIRA ticket is resolved properly

…jp-2922

codecov · 2024-05-09T15:32:02Z

Codecov Report

Attention: Patch coverage is 39.58333% with 29 lines in your changes are missing coverage. Please review.

Project coverage is 57.92%. Comparing base (781e0e0) to head (06bb487).
Report is 4 commits behind head on master.

Files	Patch %	Lines
jwst/pipeline/calwebb_spec3.py	5.26%	18 Missing ⚠️
jwst/assign_wcs/nirspec.py	58.33%	10 Missing ⚠️
jwst/exp_to_source/exp_to_source.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8442      +/-   ##
==========================================
- Coverage   57.93%   57.92%   -0.02%     
==========================================
  Files         387      387              
  Lines       38839    38847       +8     
==========================================
- Hits        22502    22501       -1     
- Misses      16337    16346       +9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…jp-2922

hbushouse · 2024-05-17T19:46:36Z

Latest regression test run is here: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1453/

Unfortunately it includes a large number of unrelated failures, due to an update to a NIRCam photom ref file earlier today. Remaining related failures are due to expected differences in file names and source_id values in multislit files. Will make another run once the truth files have been updated for the NIRCam photom change, just to make it simpler to sort out.

jwst/assign_wcs/nirspec.py

hbushouse · 2024-05-17T19:53:39Z

jwst/assign_wcs/tests/test_nirspec.py

@@ -310,8 +310,8 @@ def test_msa_configuration_all_background():
    dither_position = 1
    slitlet_info = nirspec.get_open_msa_slits(msaconfl, msa_meta_id, dither_position,
                                              slit_y_range=[-.5, .5])
-    ref_slit = trmodels.Slit(57, 8281, 1, 251, 23, -2.15, 2.15, 4, 0, '1x1', 'background_57', 'bkg_57',
-                             0, 0.0, 0.0)
+    ref_slit = trmodels.Slit(57, 8281, 1, 251, 23, -2.15, 2.15, 4, 57, '1x1', 'background_57', 'bkg_57',


Background slits now get the slitlet_id value assigned to source_id, which in this test case is 57.

hbushouse · 2024-05-17T19:55:42Z

jwst/outlier_detection/outlier_detection_step.py

@@ -156,7 +156,6 @@ def process(self, input_data):
                        model.meta.cal_step.outlier_detection = "SKIPPED"
                else:
                    self.input_models.meta.cal_step.outlier_detection = "SKIPPED"
-                self.skip = True


Needed to remove this, because outlier_detection gets run multiple times, once for each set of source/slit data in the observation. If one set has the step abort, due to their only being 1 set of input data, we don't want the step to be skipped for all remaining sets of inputs.

I thought I saw that once, reducing MOS data, and never got around to tracking it down. Thanks for the fix!

melanieclarke

This all looks good to me. I spot checked some MOS data with virtual and background slits, and found header values as described for SRCNAME, SOURCEID, and SRCALIAS. Filenames from spec3 also looked right.

nden

The new source names for virtual and background slits do not contain the program number any more. Is this on purpose? Is it what users would expect?

nden · 2024-05-23T11:40:26Z

jwst/assign_wcs/nirspec.py

+                source_alias = "vrt_{}".format(abs(source_id))
+                stellarity = 0.0
+                source_ra = 0.0
+                source_dec = 0.0


This is a change in behavior. Previously source_x/ypos and source_ra/dec for virtual slits came from the msa file. I think it makes sense to get them from the msa file because the assumption is there are sources in those slits. For a program I checked those are set in the msa file and this code resets them to 0 which we probably don't want to do.
In terms of setting these attributes should virtual slits be treated like source slits?

Wow, you're right! I was (blindly) assuming that slits with a virtual source, identified by their negative source_id value in the MSA shutter table, would not have any corresponding info in the MSA source table, and hence was not even bothering to load it. But they do have entries in the source table. For example, here are the shutter table rows for virtual slit 87, which has source_id=-70. And there is a corresponding row in the source table for source_id=-70:

FITS_rec([(87, 88, 4, 345, 1, -70, 'N', 'OPEN', 0.5, 0.5, 1, 'Y'), (87, 88, 4, 345, 1, 0, 'Y', 'OPEN', nan, nan, 2, 'N'), (87, 88, 4, 345, 1, 0, 'Y', 'OPEN', nan, nan, 3, 'N')], (1345, -70, '1345_-70', '-09:-039:0.0000 +53:00:57.29', 215.1912160802715, 53.01591471620116, 'None', 0.0)

The x/ypos values are set to 0.5 in the shutter table, so that won't have any change relative to the defaults I had been assigning, but it would result in real values for all the other items, like source_name (1345_-70), alias, ra, dec, and stellarity.

Good catch!!

JP-2922: Use new source_id syntax for level-3 products

a6d737a

hbushouse added this to the Build 11.0 milestone Apr 26, 2024

hbushouse requested a review from a team as a code owner April 26, 2024 14:10

github-actions bot added assign_wcs associations pipeline NIRSPEC spec3 pipeline labels Apr 26, 2024

hbushouse marked this pull request as draft April 26, 2024 14:11

hbushouse added the NIRSPEC MSA label Apr 26, 2024

Merge branch 'master' of https://github.com/spacetelescope/jwst into …

b52a6d2

…jp-2922

More updates to force unique names for all slits

c997cb8

github-actions bot added the exp_to_source label May 9, 2024

stscijgbot-jp mentioned this pull request May 10, 2024

Implement long-term fix for NIRSpec MOS source ID handling #7287

Open

hbushouse added 2 commits May 14, 2024 08:04

Merge branch 'master' of https://github.com/spacetelescope/jwst into …

f3dcac1

…jp-2922

Final (I hope) tweaks to the new code

7f3b1ab

github-actions bot added the outlier_detection label May 14, 2024

hbushouse added 5 commits May 14, 2024 15:31

fix style check

58686b3

fix bug discovered during testing

18f8038

add change log entry

65e1448

Fix bugs discovered during local regression testing

688ccd1

fix failing unit test

ca2f008

github-actions bot added the testing label May 16, 2024

Updates to spec2 and spec3 regtests

7e3b4ff

github-actions bot added spec2 pipeline regression_testing labels May 16, 2024

Update docs and fix regtests

ad74675

github-actions bot added NIRISS documentation labels May 17, 2024

github-actions bot added the Wide Field Slitless Spec (WFSS) label May 17, 2024

Yet another bug fix for side effects in WFSS modes

87ec5de

hbushouse mentioned this pull request May 17, 2024

Fix bug marking background slits as sources in jwst.assign_wcs.nirspec.py #8494

Closed

8 tasks

rebase from master

06bb487

hbushouse marked this pull request as ready for review May 17, 2024 19:46

hbushouse requested review from nden and melanieclarke May 17, 2024 19:47

hbushouse commented May 17, 2024

View reviewed changes

jwst/assign_wcs/nirspec.py Show resolved Hide resolved

hbushouse commented May 17, 2024

View reviewed changes

melanieclarke approved these changes May 17, 2024

View reviewed changes

nden reviewed May 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JP-2922: Use new source_id syntax for level-3 products #8442

JP-2922: Use new source_id syntax for level-3 products #8442

hbushouse commented Apr 26, 2024 •

edited

codecov bot commented May 9, 2024 •

edited

hbushouse commented May 17, 2024

hbushouse May 17, 2024

hbushouse May 17, 2024

melanieclarke May 17, 2024

melanieclarke left a comment

nden left a comment

nden May 23, 2024 •

edited

hbushouse May 23, 2024

JP-2922: Use new source_id syntax for level-3 products #8442

Are you sure you want to change the base?

JP-2922: Use new source_id syntax for level-3 products #8442

Conversation

hbushouse commented Apr 26, 2024 • edited

codecov bot commented May 9, 2024 • edited

Codecov Report

hbushouse commented May 17, 2024

hbushouse May 17, 2024

Choose a reason for hiding this comment

hbushouse May 17, 2024

Choose a reason for hiding this comment

melanieclarke May 17, 2024

Choose a reason for hiding this comment

melanieclarke left a comment

Choose a reason for hiding this comment

nden left a comment

Choose a reason for hiding this comment

nden May 23, 2024 • edited

Choose a reason for hiding this comment

hbushouse May 23, 2024

Choose a reason for hiding this comment

hbushouse commented Apr 26, 2024 •

edited

codecov bot commented May 9, 2024 •

edited

nden May 23, 2024 •

edited