-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Process webhook refresh in background to not block the request (#14269) #18173
base: master
Are you sure you want to change the base?
Conversation
0db36bd
to
81d564b
Compare
…rgoproj#14269) Signed-off-by: dhruvang1 <dhruvang1@users.noreply.github.com>
81d564b
to
409a7c2
Compare
h.Add(1) | ||
go func() { | ||
defer h.Done() | ||
h.HandleEvent(payload) | ||
}() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a chance that this can be spammed, we should keep the no. of possible routines to a bounded no. configurable by the user. We should follow a worker pool model here instead of spawning 1:1 go routines for every request.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a problem with existing implementation as well. This changes simply terminates the go routine handling the request and creates a new one, instead of continuing the request go routine. IMO, this change is no worse at handling the spam than existing implementation (since go can create thousands of go routines without breaking a sweat). We are more likely to get OOM in case of spam, since each request independently pulls all Apps/AppSet in memory than noticing any problems with go routines.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's good point though the no. of simultaneous connections that can possibly be created by the webhook server is restricted by the no. of available sockets/fds for the container, which is bounded and as far as I know can be configured if required. With this change the active connection doesn't live for too long even though a background job has been spun up which is eating the resources behind the scene so we should have the option to limit the no. of active routines. The limit can be hundreds of thousands but that's dependent on the usage of the actual user and the resources they are willing to provide to the controller.
To better handle spam we can probably dedup requests that come in at a quick succession.
This PR moves the webhook processing for both Application and ApplicationSet to a goroutine. This would allow the server to quickly send HTTP 200 to the webhook request adhering to the quick response guidelines set by GitHub, GitLab, etc.
Checklist:
Fixes #14269