Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent concurrent restart #173

Open
nvx opened this issue May 8, 2018 · 0 comments
Open

Prevent concurrent restart #173

nvx opened this issue May 8, 2018 · 0 comments

Comments

@nvx
Copy link

nvx commented May 8, 2018

Currently there is no way to ensure only N of M servers running the same process restart after a config change concurrently.

There is a splay option to add a random delay which can partially address this issue, but randomness is a part of it and it is ill suited to processes that are slow to start.

As mentioned in #53 using a Consul lock could be used to implement this reliably with the lock needing to be acquired before the process is restarted (the lock would not be used during startup, only during a restart/reload after config change).

The lock should be combined with a fixed timeout (at a minimum as a failover, but also in case there isn't any good way to signal that the service is "ready") combined optionally with other checks such as that the process has started listening on a certain port, or looking for a certain stdout/stderr line. Perhaps Consul's own service health checking could be utilised in some fashion to signal this. This would be a once off check to release the lock (unless the timeout happens first).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants