Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datamodel.open does not open an associaiton file #1116

Open
stscijgbot-rstdms opened this issue Feb 27, 2024 · 9 comments
Open

Datamodel.open does not open an associaiton file #1116

stscijgbot-rstdms opened this issue Feb 27, 2024 · 9 comments

Comments

@stscijgbot-rstdms
Copy link
Collaborator

Issue RCAL-777 was created on JIRA by Nadia Dencheva:

The following command generates an error

% strun roman_hlp dither_asn.json --verbose


TypeError: Open requires a filepath, file-like object, or Roman datamodel ```
"strun" expects romancal.stpipe._datamodels_open to be able to open also association files. "_datamodels_open" is an abstract method in stpipe.Step which needs to be implemented in both romancal and jwst in order for both pipeline to work with stpipe. This method also needs to accept a keyword "asn_n_members" in order to allow opening a specified number of files in a model container, instead of opening all of them. This is important for e.g. getting the CRDS parameters. Currently the method in roman_datamodels accepts this keyword but deletes it. This ticket is for
 * implement rdm.open(ModelContainer)
 * implement asn_n_members in rdm.open when the input is a ModelContainer
 * ModelContainer should accept asn_n_members as well
@braingram
Copy link
Collaborator

This also interferes with pars- file fetching xref: #1111

@stscijgbot-rstdms
Copy link
Collaborator Author

Comment by David Davis on JIRA:

Can someone put the dither_asn.json and the input files somewhere they can be grabbed.

I tried,

 

strun roman_hlp L3_hlp_asn.json --verbose
2024-02-27 12:54:58,968 - stpipe - WARNING - **WARNING**: LOCAL JWST PRD VERSION PRDOPSSOC-062 CANNOT BE CHECKED AGAINST ONLINE VERSION
2024-02-27 12:54:59,006 - stpipe - DEBUG - PARS-HIGHLEVELPIPELINE: CRDS parameter reference retrieval disabled.
2024-02-27 12:54:59,010 - stpipe.HighLevelPipeline - INFO - HighLevelPipeline instance created.
...
2024-02-27 12:58:31,777 - stpipe.HighLevelPipeline.resample - INFO - Update S_REGION to POLYGON ICRS  80.022983496 29.980260247 79.869082420 30.057254612 79.793849602 29.944231263 79.947634092 29.867324371
2024-02-27 12:58:31,991 - stpipe.HighLevelPipeline.resample - INFO - Step resample done

 

It does fail at tweakreg on an issue that is also being worked.

@stscijgbot-rstdms
Copy link
Collaborator Author

stscijgbot-rstdms commented Mar 4, 2024 •
edited

Comment by Mairan Teodoro on JIRA:

ModelContainer already accepts {}asn_n_members{}.

@braingram
Copy link
Collaborator

The error here can be seen using the mosaic_asn.json in artifactory:
https://bytesalad.stsci.edu/ui/repos/tree/General/roman-pipeline/dev/WFI/image/mosaic_asn.json
Running the following with romancal main:

> strun roman_hlp mosaic_asn.json --verbose

Produces the error:

2024-03-04 16:28:01,331 - stpipe - WARNING - **WARNING**: LOCAL JWST PRD VERSION PRDOPSSOC-062 CANNOT BE CHECKED AGAINST ONLINE VERSION
2024-03-04 16:28:01,388 - stpipe - DEBUG - Retrieving all substep parameters from CRDS
Traceback (most recent call last):
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/roman_datamodels/datamodels/_utils.py", line 53, in _open_path_like
    asdf_file = asdf.open(init, **kwargs)
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/asdf/_asdf.py", line 1644, in open_asdf
    return AsdfFile._open_impl(
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/asdf/_asdf.py", line 916, in _open_impl
    return cls._open_asdf(
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/asdf/_asdf.py", line 825, in _open_asdf
    self._file_format_version = cls._parse_header_line(header_line)
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/asdf/_asdf.py", line 743, in _parse_header_line
    raise ValueError(msg)
ValueError: Does not appear to be a ASDF file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/bgraham/.pyenv/versions/romancal/bin/strun", line 26, in <module>
    step = Step.from_cmdline(sys.argv[1:])
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/stpipe/step.py", line 186, in from_cmdline
    return cmdline.step_from_cmdline(args)
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/stpipe/cmdline.py", line 375, in step_from_cmdline
    step, step_class, positional, debug_on_exception = just_the_step_from_cmdline(
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/stpipe/cmdline.py", line 308, in just_the_step_from_cmdline
    parameter_cfg = step_class.get_config_from_reference(
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/stpipe/pipeline.py", line 182, in get_config_from_reference
    with cls._datamodels_open(dataset, asn_n_members=1) as model:
  File "/Users/bgraham/projects/src/romancal/romancal/stpipe/core.py", line 37, in _datamodels_open
    return rdm.open(init, **kwargs)
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/roman_datamodels/datamodels/_utils.py", line 94, in rdm_open
    asdf_file = init if isinstance(init, asdf.AsdfFile) else _open_path_like(init, memmap=memmap, **kwargs)
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal/lib/python3.10/site-packages/roman_datamodels/datamodels/_utils.py", line 55, in _open_path_like
    raise TypeError("Open requires a filepath, file-like object, or Roman datamodel") from err
TypeError: Open requires a filepath, file-like object, or Roman datamodel

@ddavis-stsci would you try the above file and command to see if it produces an error for you?

@ddavis-stsci
Copy link
Collaborator

ddavis-stsci commented Mar 5, 2024 via email

@braingram
Copy link
Collaborator

Thanks for checking.

I just tried again with romancal main and I get the same issue. Setting a breakpoint before the error I can confirm that the failure is during an attempt to open the association file (not one of the files in the association). I can also confirm that all files in the association are in the same directory and readable.

If I run the command you suggested (thanks!) strun --disable-crds-steppar roman_hlp mosaic_asn.json --verbose the pipeline starts and I do not see the above error (it fails at a later step but the association is loaded without issue).

I'm using the latest main for:

pip freeze alabaster==0.7.13 appnope==0.1.3 asdf==3.1.0 asdf-astropy==0.5.0 asdf-coordinates-schemas==0.2.0 asdf-standard==1.0.3 asdf-transform-schemas==0.3.0 asdf-unit-schemas==0.1.0 asdf-wcs-schemas==0.3.0 astropy==5.3 astropy-sphinx-theme==1.1 astroquery==0.4.6 asttokens==2.2.1 attrs==23.1.0 Babel==2.12.1 backcall==0.2.0 beautifulsoup4==4.12.3 bytecode==0.15.1 cachetools==5.3.1 cattrs==23.2.3 certifi==2023.5.7 cfgv==3.4.0 chardet==5.1.0 charset-normalizer==3.1.0 ci-watson==0.6.1 colorama==0.4.6 contourpy==1.1.0 coverage==7.2.7 crds==11.17.0 cycler==0.11.0 ddsketch==2.0.4 decorator==5.1.1 deepdiff==6.5.0 Deprecated==1.2.14 distlib==0.3.6 docutils==0.18.1 drizzle==1.14.3 envier==0.5.1 et-xmlfile==1.1.0 exceptiongroup==1.1.1 executing==1.2.0 filelock==3.12.2 fonttools==4.40.0 gwcs==0.20.0 html5lib==1.1 hypothesis==6.79.1 identify==2.5.29 idna==3.4 imagesize==1.4.1 importlib-metadata==6.7.0 iniconfig==2.0.0 ipython==8.14.0 jaraco.classes==3.3.1 jedi==0.19.0 Jinja2==3.1.2 jmespath==1.0.1 jsonschema==4.17.3 keyring==24.3.0 kiwisolver==1.4.4 lxml==4.9.3 MarkupSafe==2.1.3 matplotlib==3.7.1 matplotlib-inline==0.1.6 metrics-logger==0.1.0 mistune==3.0.1 more-itertools==10.2.0 nodeenv==1.8.0 numpy==1.25.0 numpydoc==1.5.0 opencv-python-headless==4.7.0.72 openpyxl==3.1.2 opentelemetry-api==1.23.0 ordered-set==4.1.0 packaging==23.1 Parsley==1.3 parso==0.8.3 pexpect==4.8.0 photutils==1.11.0 pickleshare==0.7.5 Pillow==10.0.0 platformdirs==3.8.0 pluggy==1.0.0 poppy==1.1.1 pre-commit==3.4.0 prompt-toolkit==3.0.39 protobuf==4.25.3 psutil==5.9.5 ptyprocess==0.7.0 pure-eval==0.2.2 pyerfa==2.0.0.3 Pygments==2.15.1 pyparsing==3.1.0 pyproject-api==1.5.2 pyrsistent==0.19.3 pysiaf==0.20.0 pytest==7.3.2 pytest-arraydiff==0.5.0 pytest-astropy==0.10.0 pytest-astropy-header==0.2.2 pytest-cov==4.1.0 pytest-doctestplus==0.13.0 pytest-filter-subpackage==0.1.2 pytest-mock==3.11.1 pytest-openfiles==0.5.0 pytest-remotedata==0.4.0 python-dateutil==2.8.2 pyvo==1.5 PyYAML==6.0 rad @ git+https://github.com/spacetelescope/rad.git@ae17bd05aab7f592cb015d364e0a10306c7c6835 requests==2.31.0 roman_datamodels @ git+https://github.com/spacetelescope/roman_datamodels.git@ef56c9a3742e2619098f5ea6197de8260a925166 -e git+ssh://git@github.com/braingram/romancal@829233d2535c976104f469bd58ee96c899876a6e#egg=romancal scipy==1.11.2 semantic-version==2.10.0 six==1.16.0 snowballstemmer==2.2.0 soc-roman-tools==0.1.0 sortedcontainers==2.4.0 soupsieve==2.5 spherical-geometry==1.2.23 Sphinx==6.2.1 sphinx-asdf==0.2.0 sphinx-astropy==1.9.1 sphinx-automodapi==0.16.0 sphinx-bootstrap-theme==0.8.1 sphinx-gallery==0.13.0 sphinx-rtd-theme==1.2.2 sphinxcontrib-applehelp==1.0.4 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.1 sphinxcontrib-jquery==4.1 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlparse==0.4.4 stack-data==0.6.2 stcal @ git+https://github.com/spacetelescope/stcal.git@4ddddf8ed0e035b1524c4255abea286971c730eb stpipe==0.5.0 stsci-rtd-theme==1.0.0 stsci.imagestats==1.6.3 stsci.stimage==0.2.6 synphot==1.2.1 toml==0.10.2 tomli==2.0.1 tox==4.6.3 traitlets==5.9.0 tweakwcs==0.8.6 typing_extensions==4.10.0 urllib3==2.0.3 virtualenv==20.23.1 wcwidth==0.2.6 webbpsf==1.2.1 webencodings==0.5.1 wrapt==1.16.0 xmltodict==0.13.0 zipp==3.15.0

Would you check if you have the STPIPE_DISABLE_CRDS_STEPPARS environment variable set? The log message you shared contains:

2024-03-05 10:28:22,984 - stpipe - DEBUG - PARS-HIGHLEVELPIPELINE: CRDS
parameter reference retrieval disabled.

which could indicate that this environment variable is set. If it's set, the code that triggers the error will be skipped which would explain why you're not seeing the same error.

@ddavis-stsci
Copy link
Collaborator

ddavis-stsci commented Mar 6, 2024 via email

@braingram
Copy link
Collaborator

Thanks for the link.

The docs mention does not yet use CRDS parameters. This suggests that they will be used and presumably will be fetched by stpipe. The current implementation of Datamodel.open does not allow this for association files which this issue describes.

@ddavis-stsci
Copy link
Collaborator

ddavis-stsci commented Mar 6, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@braingram @ddavis-stsci @stscijgbot-rstdms