[Q] Is it safe to whisper-fill.py while carbon is active and the source file contains datapoints which are preceding those in destination?

[interfan7]

We're running a Python carbon for years.
Recently we've added go-carbon server.
Both are ingesting same metrics.

Ultimately we want to shut-down the old Python carbon.
Thus we need to migrate history from the old Whisper to new:

whisper-fill.py old_whisper new_whisper

While go-carbon is still running (we can disable the Python carbon if that's beneficial).

Is it safe? I'm afraid maybe not because they have overlapping time periods of ingestion.

If not safe, can we use advisory locking? whisper-fill.py seems to support it, and go-carbon provides the following in it's configuration:

# use flock on every file call (ensures consistency if there are concurrent read/writes to the same file)
flock = true

[deniszh]

IIRC, whisper-fill will copy only absent data, so, it will replace only None's, and not data. And yes, if both go-carbon and whisper-fill use locks it should be honored. I.e. it should be safe to use whisper-fill with go-carbon, and it should not cause data corruption.

[interfan7]

@deniszh

Do you think it even necessary to use the flock as a safeguard in this case?
I wonder whether somehow there could be created race conditions.

On paper, absent data can only be history which exists in the older file and absent in the new.
And the go-carbon write only new data.
Can I conclude then that the 2 writes occur on different locations of the file and thus cannot collide?

[deniszh]

Do you think it even necessary to use the flock as a safeguard in this case?

Yes, it is.

I wonder whether somehow there could be created race conditions.

No race conditions, it can just corrupt data if both processes will write it. Whisper is not thread safe.

On paper, absent data can only be history which exists in the older file and absent in the new.
And the go-carbon write only new data.
Can I conclude then that the 2 writes occur on different locations of the file and thus cannot collide?

That part I completely didn't understand, sorry. Could you please elaborate what you mean?

[deniszh]

Ah, I got what you mean. Well, I don't know that part of kernel very well, but usually everyone recommends protecting write to the same file from two processes by lock. If you're feeling adventurous and doesn't really care about your data, you can try to not use lock.

[interfan7]

Thanks @deniszh !!