Skip to content
This repository has been archived by the owner on Oct 3, 2023. It is now read-only.

Soft delete server on downscale #154

Open
aniketmaurya opened this issue Oct 7, 2022 · 0 comments
Open

Soft delete server on downscale #154

aniketmaurya opened this issue Oct 7, 2022 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@aniketmaurya
Copy link
Contributor

aniketmaurya commented Oct 7, 2022

In the case of downscaling the servers, we don't give any grace time to process the backlogs it might have.
This results in the loss of some client requests.

A better way would be to soft delete the server first, wait for a minute, and finally, stop the LightningWork.

Soft delete: just remove the server from the LoadBalancer.servers list but don't stop the ModelServing work.

Here is how Kubernetes do it.

IMO, the best way to do this would be overriding the on_exit method.

@aniketmaurya aniketmaurya added the enhancement New feature or request label Oct 7, 2022
@aniketmaurya aniketmaurya self-assigned this Oct 13, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant