Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update NWM Client New interfaces and bring up to feature parity with old client #202

Closed
wants to merge 56 commits into from

Conversation

jarq6c
Copy link
Collaborator

@jarq6c jarq6c commented Aug 8, 2022

This turned out to be a bit bigger than I had intended, but it is the result of months of experimentation as well as feedback from frequent users of National Water Model output. I'll summarize changes for each component.

FileDownloader.py

Added an overwrite boolean parameter that will warn and skip similarly named files.

NWMFileProcessor.py

Updated get_dataset to accept the xarray.open_mfdataset paths argument. Essential coordinates (reference_time, time, and feature_id) are always returned.

NWMClient.py

Updated abstract interface to accept an nwm_feature_ids parameter to facilitate retrieval of smaller amounts of data. Unfortunately, retrieving data is still relatively slow, but this new interface accommodates the most common use-case (retrieval of time series data from a few sites).

NWMFileClient.py

Added unit_system parameter to match older nwm_client unit handling. Replaced get_dataset method with get_files that just downloads files and returns their local paths. get method works with several configurations and reference_times by default. Implements nwm_feature_ids parameter.

ParquetStore.py

Formerly ParquetCache.py, I decided to rename this module and reimplement as a MutableMapping similar to pandas.HDFStore. It can still be used as a context manager, but now it also includes key-access and inherits useful methods like get and pop from MutableMapping. As a consequence, the old get method was removed. This could live somewhere else, eventually.

NWMClientDefaults.py

Added defaults for handling units, variables, and the new ParquetStore.

UnitHandler.py

Taken verbatim from the older nwm_client. This could potentially live somewhere else (eventually).

_version.py

bumped to 7.0.0 due to breaking interface changes

Checklist

  • PR has an informative and human-readable title
  • PR is well outlined and documented. See #12 for an example
  • [?] Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (see CONTRIBUTING.md)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
  • Placeholder code is flagged / future todos are captured in comments
  • Reviewers requested with the Reviewers tool ➡️

@jarq6c jarq6c requested a review from hellkite500 August 8, 2022 19:41
@jarq6c jarq6c self-assigned this Aug 8, 2022
@jarq6c
Copy link
Collaborator Author

jarq6c commented Aug 8, 2022

Ping @aaraney

@aaraney
Copy link
Member

aaraney commented Aug 8, 2022

@jarq6c, thanks for pinging me. Im excited to start reviewing this tomorrow!

Copy link
Member

@aaraney aaraney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing this, @jarq6c! I got about halfway through the review and I need to step away to do some other tasks. Ill be back to finish this up either this weekend or sometime next week. Have a nice weekend!

@jarq6c jarq6c removed the request for review from hellkite500 May 19, 2023 16:15
@aaraney
Copy link
Member

aaraney commented Jun 9, 2023

@jarq6c, it looks like newer commits got rebased on top of this PR yesterday. I think you can do an interactive rebase (git rebase -i HEAD~4) on your nwm-new-units branch, drop the three new commits, and then force push to get us back to where we were before. Then, I think you need to rebase your nwn-new-units branch on top of main, resolve any conflicts, and force push again. The first force push is optional, but I think its worth it just to make sure the history lines up on GitHub.

@jarq6c
Copy link
Collaborator Author

jarq6c commented Jun 13, 2023

@jarq6c, it looks like newer commits got rebased on top of this PR yesterday. I think you can do an interactive rebase (git rebase -i HEAD~4) on your nwm-new-units branch, drop the three new commits, and then force push to get us back to where we were before. Then, I think you need to rebase your nwn-new-units branch on top of main, resolve any conflicts, and force push again. The first force push is optional, but I think its worth it just to make sure the history lines up on GitHub.

Started a new PR from a new branch over on #227

@jarq6c jarq6c closed this Jun 13, 2023
@jarq6c jarq6c deleted the nwm-new-units branch June 14, 2023 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants