Let ERT be able to stop experiment when all realizations are pending

[berland]

This allows the base_run_model to also being able to check regularly if there are other tasks at hand, like terminating the experiment.

Issue
Resolves #7871

Approach
Add heartbeat to monitor.track()

Add a test to certify that monitor.track() is able to emit None events, aka "heartbeats". A better test would be to test that BaseRunmodel.run_monitor() would be able to exit at any time, but that looks like a big task. If run_monitor() is changed not to request hearbeats, the original bug would reappear but this test will not catch it.

• PR title captures the intent of the changes, and is fitting for release notes.
• Added appropriate release note label
• Commit history is consistent and clean, in line with the contribution guidelines.
• Make sure tests pass locally (after every commit!)

When applicable

• When there are user facing changes: Updated documentation
• New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
• Large PR: Prepare changes in small commits for more convenient review
• Bug fix: Add regression test for the bug
• Bug fix: Create Backport PR to latest release

[berland]

todo: write test triggering the original bug. kind-of-done

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.86%. Comparing base (b63ebb1) to head (7d46284).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7924      +/-   ##
==========================================
+ Coverage   85.82%   85.86%   +0.04%     
==========================================
  Files         378      378              
  Lines       23069    23074       +5     
  Branches      636      625      -11     
==========================================
+ Hits        19798    19813      +15     
+ Misses       3198     3180      -18     
- Partials       73       81       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[xjules]

Just wondering if we need the receiver_timeout at all when having the heartbeat in the first place.

[berland]

the heartbeat_interval is None by default, so there is not really a heartbeat in place unless you ask for it.

The receiver timeout is 60 second, while a heartbeat is very short. I thought that this could risk logging the error "Evalutor did not send the TERMINATED event" too often and when it should not.

[berland]

There is only one consumer of the track() function except for tests, so the heartbeat could be there by default. It would require modifications in ca 15 tests though to ignore the None event, but that is doable.

[xjules]

I see. Then it makes sense to keep it - maybe. Nevertheless the None is there only for the sake of tests. It very much depends how trivial this modification of tests would be.

[xjules]

Let's keep it for now. I'll create an issue if it is doable to remove it.

[xjules]

Nice job @berland ! 🚀

[xjules]

Relates to #7993