Some fields become null in later report entries in the pudl sqlite db

[stephendeyoung]

I'm working with a nightly build of the pudl sqlite db and I've noticed in the generators tables some fields have a non-null value but then in a later report it becomes null. In the example below you can see that minimum_load_mw has a value for 2021 and 2022 but becomes null in later years.

Is this expected behaviour?

Here's the SQL query so you can copy/paste if easier:

SELECT report_date , minimum_load_mw, plant_name_eia, plant_id_eia, generator_id  
FROM out_eia__yearly_generators oeyg 
WHERE plant_id_eia = '141' AND generator_id = 'AFSB1'
ORDER BY report_date DESC

[zaneselvans]

The null values in 2023 and 2024 aren't surprising, because the full, annual EIA-860 for 2023 hasn't been released yet -- there's the EIA-860M through the first months of 2024, but it only includes a small subset of the data that's available in the full EIA-860. An early release version of the full EIA-860 for 2023 should come out in June I think, with the data typically being finalized in like October.

The full year of monthly EIA-923 data for 2023 also isn't available yet (it only goes through November right now) so in this annually aggregated table, you won't find any values that are dependent on the EIA-923 data either.

Not sure what's up with the null value in 2020 though. Is that the first year this plant shows up in? If so it might be in there with a non-operational status.

[stephendeyoung]

Thank you. Can I assume that if an older row had a value for a column it will be the same for the newer row?

[zaneselvans]

For any of the columns that are ultimately coming from the "slowly changing dimension" (SCD) tables, this is likely a good assumption, and minimum_load_mw appears to be one of them:

https://data.catalyst.coop/pudl/core_eia860__scd_generators