Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve OGE validation compared to eGRID #282

Draft
wants to merge 5 commits into
base: development
Choose a base branch
from

Conversation

grgmiller
Copy link
Collaborator

@grgmiller grgmiller commented Feb 15, 2023

This PR improves and automates the validation of OGE output data compared to eGRID data (fixes CAR-1893, CAR-1894, CAR-1892, CAR-1887).

Updates:

  • update download_data.download_egrid_file() to only download the single eGRID file for the year for which the pipeline is being run. Previously, we had downloaded all eGRID files 2018-2021. Note: until we start expanding historical coverage, I am not planning to add urls for eGRID versions prior to 2018.
  • Moves all of the egrid validation functions from src.validation.py to a new file src.validate_egrid.py - we might want to name this something different?
  • Creates a new constants.py file to hold constants referenced across the repo (prevents circular imports)
  • Started work on a new automated logic to identify why discrepancies exist between eGRID data and OGE data

To do:

  • check that all eGRID column mapping / loading is correct
  • Calculate and/or load _for_electricity and _for_electricity_adjusted values
  • Make dataframe filters more legible when filtering plant data errors
  • When there is a mismatched set of plants, examine how much of difference results from mismatch, then if there is remaining difference for matched set of plants
  • When flagging issues, tag how many plants have that issue so that we make sure we are capturing all issues and there are no problem plants that are uncategorized.
  • Consider changing plant comparison status to a numeric percent instead of text category (or have both).

Where to pick up work next time (3/11/23): started transferring code to a fresh notebook to put it all in order. Need to re-build the plant comparison df

@grgmiller grgmiller changed the title Improve OGE validation against eGRID Improve OGE validation compared to eGRID Mar 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant