Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

need to handle ci leaking cluster somehow, as well as parallel jobs #8

Open
rhs opened this issue Aug 20, 2017 · 2 comments
Open

need to handle ci leaking cluster somehow, as well as parallel jobs #8

rhs opened this issue Aug 20, 2017 · 2 comments

Comments

@rhs
Copy link
Contributor

rhs commented Aug 20, 2017

kubernaut claim
You attempted to kubernaut claim a cluster but you are already at your maximum claim limit (1). Please release your existing claim kubernaut discard or wait until the existing claim expires.

I ran into the above error message when setting up a travis job. The job barfed before I had a chance to discard my cluster. This seems like a thing that would commonly happen during initial setup.

I think documenting this somewhere, and how to properly release resources in a travis job would be useful. (There is an after_script that I think always runs which can do cleanup even if the job fails.)

Parallel jobs would be an interesting problem as well, given this limit.

@rhs
Copy link
Contributor Author

rhs commented Aug 20, 2017

What might work here is some kind of option that would tell claim to wait a configurable time for the cluster to be released and then forcibly steal the claim if it is not available by then. That way multiple concurrent jobs would wait for each other to finish, but you wouldn't have to worry about leaks.

@plombardi89
Copy link
Contributor

I like the idea of having an example .travis.yml file kicking around in perhaps an examples/travis directory in this repository. We could also consider a Travis integration that allows us to automatically kill Kubernaut instances when a Travis job finishes via some kind of Travis API polling mechanism for projects.

While that approach would work for multiple concurrent jobs it means your CI process for unrelated projects is blocked by one of the other projects which holds the claim. That is probably undesirable.

Another option is increasing the claim # for a user or some combination of both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants