Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix incomplete timeseries warnings #316

Open
grgmiller opened this issue Nov 29, 2023 · 0 comments
Open

Fix incomplete timeseries warnings #316

grgmiller opened this issue Nov 29, 2023 · 0 comments
Labels
bug Something isn't working data cleaning Cleaning and standardizing data emissions Accuracy/completeness of emission mass data

Comments

@grgmiller
Copy link
Collaborator

When running the data pipeline for 2021, there are several warnings appearing for various data outputs regarding incomplete timeseries data:

Data quality metric export:

2023-06-09 20:20:56,968 [INFO] oge.output_data:158 Exporting hourly_profile_method to data/results/2021/data_quality_metrics
2023-06-09 20:20:58,949 [WARNING] oge.validation:792 There are incomplete timeseries for the following ['plant_id_eia'] groups in shaped_eia_data
2023-06-09 20:20:58,949 [WARNING] oge.validation:795 
                          count                       min                       max  expected_num_hours
plant_id_eia report_date                                                                               
903212       2021-11-01     719 2021-11-01 08:00:00+00:00 2021-12-01 08:00:00+00:00                 721
909312       2021-11-01     719 2021-11-01 05:00:00+00:00 2021-12-01 05:00:00+00:00                 721
913612       2021-11-01     719 2021-11-01 03:00:00+00:00 2021-12-01 03:00:00+00:00                 721
[...]

Upon combining hourly plant data:

2023-06-09 20:21:59,875 [INFO] oge.data_pipeline:542 16. Combining plant-level hourly data
2023-06-09 20:24:04,908 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['plant_id_eia'] groups in combined_plant_data
2023-06-09 20:24:04,970 [WARNING] oge.validation:776 
              datetime_utc  expected_num_hours
plant_id_eia                                  
9                     5136                8760
47                    6553                8760
51                    6529                8760
59                    7296                8760
124                   5808                8760
170                   5064                8760
201                   8017                8760
203                   6552                8760
[...]

For certain BA-level results:

2023-06-09 20:24:22,416 [INFO] oge.output_data:131 Exporting AECI to data/results/2021/power_sector_data/annual/
2023-06-09 20:24:22,448 [INFO] oge.validation:149 Checking that fuel and emissions values are positive...  
2023-06-09 20:24:22,463 [INFO] oge.validation:175 OK
2023-06-09 20:24:22,463 [INFO] oge.validation:181 Checking that no values are missing...  
2023-06-09 20:24:22,463 [INFO] oge.validation:198 OK
2023-06-09 20:24:22,683 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['fuel_category'] groups in power sector hourly ba table
2023-06-09 20:24:22,683 [WARNING] oge.validation:776 
               datetime_utc  expected_num_hours
fuel_category                                  
wind                   8758                8760
2023-06-09 20:26:00,361 [INFO] oge.output_data:131 Exporting LDWP to data/results/2021/power_sector_data/annual/
2023-06-09 20:26:00,392 [INFO] oge.validation:149 Checking that fuel and emissions values are positive...  
2023-06-09 20:26:00,408 [INFO] oge.validation:175 OK
2023-06-09 20:26:00,408 [INFO] oge.validation:181 Checking that no values are missing...  
2023-06-09 20:26:00,408 [INFO] oge.validation:198 OK
2023-06-09 20:26:00,612 [WARNING] oge.validation:773 There are incomplete timeseries for the following ['fuel_category'] groups in power sector hourly ba table
2023-06-09 20:26:00,612 [WARNING] oge.validation:776 
               datetime_utc  expected_num_hours
fuel_category                                  
natural_gas            8761                8760
total                  8761                8760
@grgmiller grgmiller added bug Something isn't working data cleaning Cleaning and standardizing data emissions Accuracy/completeness of emission mass data labels Nov 29, 2023
@grgmiller grgmiller added this to the 2022 Data Release milestone Nov 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working data cleaning Cleaning and standardizing data emissions Accuracy/completeness of emission mass data
Projects
None yet
Development

No branches or pull requests

1 participant