Prefer monotonic clock for timestamp in Sample and all other FooMetrics

[christianparpart]

What's needed?

Hi,

while migrating to BatteryPool again, I noticed that data types like Sample, CapacityMetrics, PowerMetrics, SoCMetrics all feature a timestamp member of type datetime. I think it should not be of type datetime (e.g. be initialized to the result of the call datetime.now() because that is subject to jump arbitrarily forth and back at the will of the system administrator (e.g. sudo date -s ..., which potentially could break assumptions by a callee that is getting new metrics/samples being passed with the time jumping out of nowhere.

Proposed solution

Prefer a strong data type to enforce monotonic time clock use

Use cases

Improved resillience against external clock changes (sysop setting time, or computer going to sleep for various reasons)

Alternatives and workarounds

No response

Additional context

to guard against time jumps, it's usually typical to always favor monotonic clocks. I am not sure how far Frequenz-SDK users actually rely on forward-ticking clocks or if these timestamp values matter at all (for resampling I think they might, and there we should actually prefer monotonic clock, IMHO)

[llucax]

Converted to a discussion because this is not a trivial change, and would require a very careful consideration. Once we agree on a path, we can create an issue (or more) with the concrete actions we need to take.

[llucax]

I don't think using a monotonic clock is a viable option here, because this timestamp will be used for data science all over, and I guess they need concrete dates. Not sure if there is anything we can do for users that need a strong protection against time jumps.

We had a discussion in the past about this, but I can't find where. Maybe it was in the internal chat.

FYI @frequenz-floss/python-sdk-team @thomas-nicolai-frequenz @frequenz-floss/datasci-team

[llucax]

I will copy here some old internal discussion we had about this. We didn't bring up the specific issue about a controller operator willingly messing up with the controller's clock, we only talked about daylight saving time for local time and leap seconds for UTC.

The consensus was to ignore time jumps as, at least for data science, it shouldn't have an impact.

---

@llucax
Hello everyone. We've been discussing about how to handle timers in the SDK and to make a good decision it would be useful to know how time data will be used. The problem is there are different ways to think about time:

• Monotonic (the times just increases monotonically and we only care about time differences, how much time it passed)
• UTC (it has the notion of absolute time, or date, like now is 2023-03-29 11:02:00 UTC). The problem with this is there is something called "leap second". TL;DR: the earth doesn't rotate uniformly around the sun so from time to time one extra second needs to be added, so UTC can be "jumpy", either there is a second 60 at some point, or a second could be skipped or be "produced" twice)
• Local time (same as UTC, but we also have time jumps when switching to/from DST (Daylight Savings Time)).

I guess to make predictions you need local time, at least to predict human behavior (maybe for solar panels it could be UTC). What do you normally do with those time jumps?

https://www.timeanddate.com/time/leapseconds.html

@thomas-nicolai-frequenz
IGNORE FOR NOW! There has been no issue with this so far. If it comes up we'll address it.
The data pipeline should always stay UTC, whatever happens on top shouldn't be a worry of the SDK imho

@cwasicki
I agree to use UTC. For most of our projects we work on a much less granular time scale (e.g. 15 min) and leap seconds would be a minor issue compared to other uncertainties we have. UTC is convenient to work with and we can shift it to local time as required. @david-natingga-frequenz do you see any problem with that for your work?

@david-natingga-frequenz
I agree with using UTC and not care about leap seconds (e.g. skip it). As Christoph pointed out, most of our use cases do not require so detailed data. If one second is skipped (or included in a mean) once in a while, that is ok.

[Marenz]

I am not sure I get the problem case to be honest. Are you saying a once created datetime could change its value if the system time changes?

If so, that would be a problem, though I doubt that is the case..

As for sudden jumps when using datetime.now(ts=..), we hardly create datetimes in our code as they all come from nitrogen.
A quick grep through our SDK reveals maybe two classes that use now().

It maybe worth creating a scenario where this happens to see possible problems, but otherwise, I am not yet completely understanding the problem :)

[llucax]

I guess the problem is d1 = datetime.now(utc); d2 = datetime.now(utc); assert d2 > d1 might assert if somebody updated the clock.

[Marenz]

If it's that specific case, again, we hardly create our own timestamps in code, we get it all from the network..

[llucax]

The network is still not something magical, it is a server that it is very likely living in the same controller, but it is a good point that if changed, it needs to be changed in the server, as we can't really turn a timestamp into a monotonic clock value.