Deregister Runner Application when Spot Interruption signal is received

[dimisjim]

to prevent issues like this: #84

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.

[dimisjim]

Any news regarding this?

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.

[dimisjim]

bump

[ScottGuymer]

Hi

Do you have any thoughts on how you might see this working?

[dimisjim]

Hi @ScottGuymer ,

yeah, one idea would be to install a cron inside the runners that checks regularly for spot interruption notices (see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html), then it would deregister itself from the available pool of runners, or via a lambda that would be delegated to perform this action.

Ideally it would wait until a job is finished, if it currently runs one. Of course it would be difficult to predict when a certain job is expected to finish, so it would be enough to wait for a reasonable amount of time (which would be before the end of the termination timestamp that was given by the notice) so that a job has more chances on finishing.

These would hopefully minimize the runners being shut down while running a job.
What do you think?

[lmeynberg]

When I understand this correctly part of this module could be a connection to the SNS topic and listen for these events. And then shutdown the runner using the normal shutdown runner.

[tedparagon]

We went months with only getting a handful of evictions. So it was definitely worth rerunning a couple jobs to get that huge spot instance savings. However over the last month or so the evictions have increased significantly to the point where we are considering other options. I think this feature would help a lot.

These are just some of our evictions in the last 30 minutes. They are becoming more the norm than exception.

[npalm]

In PR #3789 we adding a first step to handle spot termination events.