What happens if a participant's system clock is wrong?

[peterbourgon]

It seems like timestamps, and particularly timeout timestamps, are derived from time.Now(), but that's not reliable — what would happen if a node's system clock is far in the future, or far in the past?

[ebuchman]

Hey Peter great to see you here asking questions about time :D

If the state machine is using time.Now() anywhere, it's a bug. Do you have an instance of this?

The canonical time for a block is provided in a BFT way by the validator set. It's included in the block header. Currently its the median timestamp of all the votes (see spec) but we are moving to a "proposer based timestamp model" (see spec) which provides some stronger guarantees but also introduces a new clock synchrony assumption into the consensus.

For a recent bug involving time.Now() in the state machine, see here

[peterbourgon]

Thanks for the link to the spec! That's clarifying, and the median approach is clever.

I think I'm less concerned about Byzantine actors, and more concerned about inadvertent clock skew among otherwise Good™ participants. Correct me if I'm wrong, but in both the existing and proposer-based timestamp model, there seems to be the assumption that (non-Byzantine) validators will all report the same system time to within some precision. Or, as you express in the system model of the new spec,

There is a reference Newtonian real-time t (UTC).

Every correct validator V maintains a synchronized clock C_V that ensures:

[PBTS-CLOCK-PRECISION.0]

There exists a system parameter PRECISION such that for any two correct validators V and W, and at any real-time t,
|C_V(t) - C_W(t)| < PRECISION

I'm not an expert in this domain, but I did stay at a Holiday Inn Express last night, and I'm pretty sure that you can't actually assert any constraints on PRECISION, because you can't make any reliable claims about the relationship between any C_V and t. Said another way, t is actually unknowable! A system knows it's own clock, but that clock can very easily be very wrong.

As defined in the protocol, timestamps ratchet upwards, so that's cool and means very bad precisions will only break things in one direction. But what happens if that happens? If timestamps aren't used for correctness or other important purposes, and/or if far-future timestamps don't break anything, then maybe nothing! No problem! But I wasn't able to figure out the implications after a bit of digging. So maybe another way of phrasing my question would be: what consumes these timestamps, and what would happen to those things if the timestamps were wrong?

[peterbourgon]

If the state machine is using time.Now() anywhere, it's a bug. Do you have an instance of this?

I think I just saw this bit, not sure if it's meaningful.

[ebuchman]

So with the median timestamps, theres no clock sync assumption, the clocks can be where ever. But it does mean +1/3 can move the clock whereever they want by colluding, and this might actually be a problem since its not accountable (we can't prove they were misbehaving, they could claim their clocks were borked). Hence the move to the proposer-based system, which requires +2/3 to collude, but it does introduce a clock sync assumption. If the clock sync assumption fails, we just won't be able to make blocks and will need some manual intervention.

So maybe another way of phrasing my question would be: what consumes these timestamps, and what would happen to those things if the timestamps were wrong?

Right, one of the main things that consumes this is the staking logic for unbonding periods. So being able to fast forward the time basically means validators could make evidence of misbehaviour no longer valid. This is a major reason why we wanted to move away from the median system where +1/3 could pull off this attack towards the proposer based system which requires +2/3 to attack.

More generally time can be used by any of the application logic, so it needs to be careful to reason for itself about what happens if things go wrong. Though by moving to proposer timestamps, we can provide much stronger guarantees on the time's correctness (ie. otherwise +2/3 are colluding, which means we have bigger problems anyways), of course at the expense of a new kind of liveness fault (if clocks go out of sync and we can't make a block because we can't agree on time). But that's consistent with general approach to prefer safety over liveness.

I think I just saw this bit, not sure if it's meaningful.

That's local client code so its not part of consensus.

[peterbourgon]

the proposer-based system . . . requires +2/3 to collude, but it does introduce a clock sync assumption. If the clock sync assumption fails, we just won't be able to make blocks and will need some manual intervention.

Nodes don't need to be acting maliciously to yield incorrect timestamps — it's a common pathology of distributed systems that rely on system clocks. Leap years, leap seconds, faulty NTP servers, dead batteries on rack servers, you name it. And it's entirely possible that all the validators agree on an incorrect, far-future timestamp. So they commit blocks and all that with monotonically increasing, but also increasingly wrong, timestamps. Does that break anything?

—

clock sync assumption

Hmm, does that mean that there's an expected minimum PRECISION (or ACCURACY) defined by the protocol (or in the code) which triggers errors if not satisfied?

[ebuchman]

it's a common pathology of distributed systems that rely on system clocks.

Of course. But most of the analysis is based on worst case malicious behaviour. In the median timestamp model, the clock can drift arbitrarily based on independent drift in validator clocks. In the proposer based model, the validator clocks need to stay within some accuracy of each other, so if they just randomly drift the chain will likely halt until validators fix it.

There's nothing to guarantee that whatever time the validators agree on really is some universally true clock time, and in principle it can break things for applications, depending on the application. Lots of applications are designed to use block height rather than timestamp to avoid any issues, but this comes with other problems.