L1 datamodel to ramp conversion

[tddesjardins]

Currently the logic to convert from a L1-like datamodel (e.g., WfiScienceRaw) to the Ramp model is contained in the DQInitStep:

        if not isinstance(input_model, RampModel):
            # Create base ramp node with dummy values (for validation)
            input_ramp = maker_utils.mk_ramp(shape=input_model.shape)
            # check if the input model has a resultantdq from SDF
            if hasattr(input_model, "resultantdq"):
                input_ramp.groupdq = input_model.resultantdq.copy()

            # Copy input_model contents into RampModel
            for key in input_model.keys():
                # check for resultantdq if present copy this to the emp
                # it to the ramp model, we don't want to carry this around
                if key != "resultantdq":
                    if key not in input_ramp:
                        input_ramp[key] = input_model.__getattr__(key)
                    elif isinstance(input_ramp[key], dict):
                        # If a dictionary (like meta), overwrite entires (but keep
                        # required dummy entries that may not be in input_model)
                        input_ramp[key].update(input_model.__getattr__(key))
                    elif isinstance(input_ramp[key], np.ndarray):
                        # Cast input ndarray as RampModel dtype
                        input_ramp[key] = input_model.__getattr__(key).astype(
                            input_ramp[key].dtype
                        )
                    else:
                        input_ramp[key] = input_model.__getattr__(key)

            # Create model from node
            output_model = RampModel(input_ramp)
        else:
            output_model = input_model

This works for WfiScienceRaw, but not for other L1-like data such as TVAC or FPS datamodels. Either the code in DQInit needs to be updated to accept the other L1-like data (not including GW files), or a suggestion was to modify the datamodel to include a method that handles the transformation to the Ramp model.

[tddesjardins]

Here's an example TVAC file that does not work with DQInitStep and the exception it produces. The file is at /grp/roman/TEST_DATA/RITA_DEV/TVAC_SAMPLES_RDM/TVAC1_NOMOPS_SCIMON_20231015055603_WFI01_uncal.asdf.

2024-04-26 12:07:48,957 - CRDS - ERROR -  Error determining best reference for 'pars-dqinitstep'  =   Unknown reference type 'pars-dqinitstep'
2024-04-26 12:07:48,957 - stpipe.DQInitStep - INFO - DQInitStep instance created.
2024-04-26 12:07:48,992 - stpipe.DQInitStep - INFO - Step DQInitStep running with args (<roman_datamodels.datamodels._datamodels.TvacModel object at 0x133828890>,).
2024-04-26 12:07:48,993 - stpipe.DQInitStep - INFO - Step DQInitStep parameters are:
  pre_hooks: []
  post_hooks: []
  output_file: None
  output_dir: None
  output_ext: .asdf
  output_use_model: False
  output_use_index: True
  save_results: False
  skip: False
  suffix: None
  search_output_file: True
  input_dir: ''
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In[8], line 1
----> 1 dq.call(test)

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/stpipe/step.py:686, in Step.call(cls, *args, **kwargs)
    683 name = config.get("name", None)
    684 instance = cls.from_config_section(config, name=name, config_file=config_file)
--> 686 return instance.run(*args)

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/stpipe/step.py:509, in Step.run(self, *args)
    507     self.prefetch(*args)
    508 try:
--> 509     step_result = self.process(*args)
    510 except TypeError as e:
    511     if "process() takes exactly" in str(e):

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/romancal/dq_init/dq_init_step.py:72, in DQInitStep.process(self, input)
     69                 input_ramp[key] = input_model.__getattr__(key)
     71     # Create model from node
---> 72     output_model = RampModel(input_ramp)
     73 else:
     74     output_model = input_model

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/roman_datamodels/datamodels/_datamodels.py:43, in _RomanDataModel.__init__(self, init, **kwargs)
     42 def __init__(self, init=None, **kwargs):
---> 43     super().__init__(init, **kwargs)
     45     if init is not None:
     46         self.meta.model_type = self.__class__.__name__

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/roman_datamodels/datamodels/_core.py:134, in DataModel.__init__(self, init, **kwargs)
    132 af = asdf.AsdfFile()
    133 af["roman"] = self._instance
--> 134 af.validate()
    135 self._asdf = af
    136 return

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/_asdf.py:628, in AsdfFile.validate(self)
    624 def validate(self):
    625     """
    626     Validate the current state of the tree against the ASDF schema.
    627     """
--> 628     self._validate(self._tree)

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/_asdf.py:619, in AsdfFile._validate(self, tree, custom, reading)
    614 with self._blocks.options_context():
    615     # If we're validating on read then the tree
    616     # is already guaranteed to be in tagged form.
    617     tagged_tree = tree if reading else yamlutil.custom_tree_to_tagged_tree(tree, self)
--> 619     schema.validate(tagged_tree, self, reading=reading)
    620     # Perform secondary validation pass if requested
    621     if custom and self._custom_schema:

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/schema.py:649, in validate(instance, ctx, schema, validators, reading, *args, **kwargs)
    646     ctx = AsdfFile()
    648 validator = get_validator({} if schema is None else schema, ctx, validators, None, *args, **kwargs)
--> 649 validator.validate(instance)
    651 additional_validators = [_validate_large_literals]
    652 if ctx.version >= versioning.RESTRICTED_KEYS_MIN_VERSION:

File ~/mambaforge/envs/rcaltest/lib/python3.11/site-packages/asdf/_jsonschema/validators.py:312, in create.<locals>.Validator.validate(self, *args, **kwargs)
    310 def validate(self, *args, **kwargs):
    311     for error in self.iter_errors(*args, **kwargs):
--> 312         raise error

ValidationError: mismatched tags, wanted 'asdf://stsci.edu/datamodels/roman/tags/exposure-1.0.0', got 'asdf://stsci.edu/datamodels/roman/tags/base_exposure-1.0.0'

Failed validating 'tag' in schema['properties']['meta']['allOf'][1]['properties']['exposure']:
    {'tag': 'asdf://stsci.edu/datamodels/roman/tags/exposure-1.0.0',
     'title': 'Exposure Information'}

On instance['meta']['exposure']:
    {'data_problem': False,
     'end_time': {'base_format': 'datetime',
                  'value': '2023-10-15T05:58:50.219'},
     'exposure_time': 167.21875,
     'frame_divisor': -99,
     'frame_time': 3.0403409090909093,
     'group_time': 3.0403409090909093,
     'groupgap': 0,
     'ma_table_name': '71_FULLMA510ALL',
     'ma_table_number': 510,
     'mid_time': {'base_format': 'datetime',
                  'value': '2023-10-15T05:57:26.610'},
     'nframes': 1,
     'ngroups': 56,
     'read_pattern': [[1],
                      [2],
                      [3],
                      [4],
                      [5],
                      [6],
                      [7],
                      [8],
                      [9],
                      [10],
                      [11],
                      [12],
                      [13],
                      [14],
                      [15],
                      [16],
                      [17],
                      [18],
                      [19],
                      [20],
                      [21],
                      [22],
                      [23],
                      [24],
                      [25],
                      [26],
                      [27],
                      [28],
                      [29],
                      [30],
                      [31],
                      [32],
                      [33],
                      [34],
                      [35],
                      [36],
                      [37],
                      [38],
                      [39],
                      [40],
                      [41],
                      [42],
                      [43],
                      [44],
                      [45],
                      [46],
                      [47],
                      [48],
                      [49],
                      [50],
                      [51],
                      [52],
                      [53],
                      [54],
                      [55]],
     'sca_number': 1,
     'start_time': {'base_format': 'datetime',
                    'value': '2023-10-15T05:56:03.000'},
     'type': 'WFI_IMAGE'}

[stscijgbot-rstdms]

This issue is tracked on JIRA as RCAL-833.

[stscijgbot-rstdms]

Comment by Jonathan Eisenhamer on JIRA:

Tyler Desjardins/Javier Sanchez The given example data appear to need to be updated to rad/rdm version 0.20. The current data give the following when attempting to read:

m = rdm.open('TVAC1_NOMOPS_WFISCIENCEMON_20231015055603_WFI01_uncal.asdf')
...blah...
ValidationError: mismatched tags, wanted 'asdf://stsci.edu/datamodels/roman/tags/tvac/calibration_software_version-1.0.0', got 'asdf://stsci.edu/datamodels/roman/tags/calibration_software_version-1.0.0'

Failed validating 'tag' in schema['properties']['meta']['allOf'][0]['allOf'][0]['properties']['calibration_software_version']:
{'description': 'The version number of the calibration software used '
'in processing this\n'
'file.\n',
'tag': 'asdf://stsci.edu/datamodels/roman/tags/tvac/calibration_software_version-1.0.0',
'title': 'Calibration Software Version Number'}

On instance['meta']['calibration_software_version']:
'0.0.0'

[stscijgbot-rstdms]

Comment by Javier Sanchez on JIRA:

Jonathan Eisenhamer , I just updated the file `███████████████████████████████████████████████████████████████████████████████████████████████████ to 0.20.0, and it validates correctly. Please let me know if it works for you.

[stscijgbot-rstdms]

Comment by Jonathan Eisenhamer on JIRA:

Much better. Thank you!

[stscijgbot-rstdms]

Comment by Jonathan Eisenhamer on JIRA:

Issue resolved by RCAL PR 1258 and RDM PR 352