During eviction, set is_replaying and raise special exception

[cretz]

What was changed

• Set workflow.unsafe.is_replaying() to True during eviction
• Raise a special BaseException-based exception during eviction instead of RuntimeError which has an Exception base

Checklist

1. Closes [Bug] Make sure is_replaying is True upon eviction, and only throw certain exception types out on eviction #523

[Sushisource]

Something I don't quite understand about the statement in the issue:

anyone in a finally or except Exception can catch exceptions occurring during eviction and do side-effecting things like logging

Why would the user code be woken up at all during eviction? We shouldn't, ideally, even run the tasks containing the user code at all. Then we wouldn't need to do anything special here.

[cretz]

Why would the user code be woken up at all during eviction? We shouldn't, ideally, even run the tasks containing the user code at all. Then we wouldn't need to do anything special here.

Because that's the only way you can collect paused coroutines in many languages (including Java, Go, and Python). The whole issue with #499 is that we were letting Python GC wake these things up in whatever thread. Now we wake them up in our controlled environment. This is the same as Java where you could technically catch a io.temporal.internal.sync.DestroyWorkflowThreadError.

[Sushisource]

Ah, right, now I remember that. I was curious about if Python really must raise that, and it seems like it maaaaybe doesn't, it happens as a consequence of close() getting called on generators: https://docs.python.org/3/reference/expressions.html#generator.close

I wonder if we could override close for the generators which constitute user's workflow code and tasks started therein to simply... not do that? Maybe something worth considering, but I can imagine it might have a bunch of unexpected consequences too.

Anyway, this works for me but it might be worth checking out

[cretz]

I was curious about if Python really must raise that, and it seems like it maaaaybe doesn't, it happens as a consequence of close() getting called on generators: https://docs.python.org/3/reference/expressions.html#generator.close

There is no way to prevent it. We've tried everything. The proper way, like Go and Java, is to force coroutines to complete themselves and ignore side effects.

I wonder if we could override close for the generators which constitute user's workflow code and tasks started therein to simply... not do that? Maybe something worth considering, but I can imagine it might have a bunch of unexpected consequences too.

https://github.com/python/cpython/blob/v3.12.3/Lib/_collections_abc.py#L176 occurs and there's nothing we can find to prevent this. And yes, I think they require it to ensure garbage collection of the coroutine so unsure what consequences it would have if we even could prevent it.