Pandas reporter final_df_output event resolution

[victorgarcia98]

Add the field output_event_resolution field to PandasReporterConfigSchema used to set the event_resolution of the final_df_output. This is useful for cases in which we want to modify the event_resolution using methods that are not part of timely-beliefs (e.g: resample).

In addition, this PR solves the issue of having NaN DataSources in the returned timely-beliefs.

[Flix6x]

Why can't we get the resolution of the final output from self.sensor.event_resolution instead?

[victorgarcia98]

Why can't we get the resolution of the final output from self.sensor.event_resolution instead?

The idea is, by default, check that the output event_resolution == self.sensor.event_resolution, happening in the Reporter class. If we set the output event_resolution to self.sensor.event_resolution, we always bypass this check.

[Flix6x]

So now you can pass the output resolution explicitly, but it must match the output sensor resolution. ...

Can we think of some alternatives? Is this resolution only passed to make that check succeed? If so, maybe we want that check only on proper BeliefsDataFrames, and want to have a different check on regular DataFrames (maybe checking the index frequency instead)?

[Flix6x]

bdf.lineage.sources might also work.

https://github.com/SeitaBV/timely-beliefs/blob/main/timely_beliefs/beliefs/__init__.py#L109

I wouldn't have expected any NaN sources, actually. How come there are any?

[victorgarcia98]

Using bdf.lineage.sources also yields a NaN source.

The following code snippet should be useful to reproduce this behavior using the data from 2023.

from datetime import datetime, timedelta
import pytz

from flexmeasures.data.models.time_series import Sensor

sensor = Sensor.query.get(6)

timezone = pytz.timezone("Europe/Amsterdam")

event_starts_after = tz.localize(datetime(2023, 1, 2))
event_ends_before = tz.localize(datetime(2023, 1, 23))
resolution = timedelta(seconds=3600)

bdf = sensor.search_beliefs(
    event_starts_after=event_starts_after,
    event_ends_before=event_ends_before,
    resolution=resolution,
)

>> bdf.sources.isna().sum()
37

[victorgarcia98]

Can we think of some alternatives? Is this resolution only passed to make that check succeed? If so, maybe we want that check only on proper BeliefsDataFrames, and want to have a different check on regular DataFrames (maybe checking the index frequency instead)?

Good point! I've updated the code to update the event_resolution of the output BeliefDataFrame to the event_start frequency in case this doesn't match the output sensor's.

[Flix6x]

I think this could be problematic. The suggested refactoring of the reporter class may shine some light, but there are some other things we also need to consider:

• The distinction between frequency and resolution, at least in the TimelyBeliefsReporter.
• FlexMeasures stores resolutions in the database as intervals, which allows distinguishing between a calendar day (interval '1 day') and a 24-hour day (interval '24 hours'), but uses mostly Python datetime.timedelta objects in the code, which doesn't allow expressing a calendar day (timedelta(days=1) is defined to be exactly timedelta(hours=24)). So Pandas offsets won't always end up correctly in our database. And even within Pandas, some assumptions regarding these conversions can be challenged. For example, in the case of "24H" = timedelta(hours=24).

We could use some test cases to see what works and what doesn't. And maybe we should prioritize the refactoring.