-
-
Notifications
You must be signed in to change notification settings - Fork 753
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement graceful shutdown on Mac #534
Comments
I am fairly new to the language with 3 months experience and I diagnosed that the context.WithTimeout is causing it. I tried it with context.TODO() and it works fine now |
Had similar issue, just use root = "."
testdata_dir = "testdata"
tmp_dir = "tmp"
[build]
bin = "./tmp/my-api-go"
cmd = "pkill my-api-go; go build -o ./tmp/my-api-go ."
delay = 1000
exclude_dir = ["assets", "tmp", "vendor", "testdata", "node_modules"]
exclude_file = []
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
full_bin = ""
include_dir = []
include_ext = ["go", "tpl", "tmpl", "html"]
kill_delay = "0s"
log = "build-errors.log"
send_interrupt = false
stop_on_error = true
[color]
app = ""
build = "yellow"
main = "magenta"
runner = "green"
watcher = "cyan"
[log]
time = false
[misc]
clean_on_exit = false
[screen]
clear_on_rebuild = false |
@hamza72x I have similar settings but I'm seeing different behavior (the ^C is being ignored on MacOS). When I run my server locally I see it exit from an interrupt as expected: I0304 12:51:10.617018 47315 server.go:142] updating pod info
I0304 12:51:10.617172 47315 server.go:108] starting server at port 7442
^C
I0304 12:51:14.062000 47315 server.go:67] quitting server
E0304 12:51:14.062116 47315 server.go:59] http: Server closed Doing the same thing with air i see:
but when I check lsof i still see the process running and bound to a port (and it still logs to my local terminal):
here is my config
|
The key here is to set |
I'm able to reproduce this error even when When air is running the bin and a ctrl+c comes in cleanup() is executed. At this line in cleanup we call to have the bin stopped. When this call happens we aren't waiting for it to finish. https://github.com/cosmtrek/air/blob/master/runner/engine.go#L582-L586 The air program typically exits before it has the chance to stop the bin and get to https://github.com/cosmtrek/air/blob/master/runner/engine.go#L521-L522 |
I can confirm that the server is not killed even when |
This is also happening to me using ZSH. I have As a bit of a work around until this is resolved, I have set the following config to automatically kill the process using a specific port: post_cmd = ["lsof -i tcp:8080 | awk 'NR==2{print $2}' | xargs kill"] |
I have a 5 second delay for connections to close on graceful shutdown. If you're listening for an interrupt then be sure to set a kill_delay as well otherwise air will close faster than your cleanup process.
|
Be sure to capture syscall.SIGTERM also. systemd/docker uses that signal for stop and restart. Save yourself some headaches on deployment.
|
@taylow Improved on your solution a little bit. I ensure I reap the process with a given listening port before every reload and when you quit post_cmd = [
"lsof -i :8080 -sTCP:LISTEN | sed 1d | awk '{print $2}' | xargs kill -9"
]
pre_cmd = [
"lsof -i :8080 -sTCP:LISTEN | sed 1d | awk '{print $2}' | xargs kill -9"
] This part of the lsof -sTCP:LISTEN Hope it helps and I hope this will be fixed soon so we don't have to use a workaround. ❤️ |
I was having the same issue as you. I have a docker container running a go service where I listen for SIGINT and SIGTERM after starting the server so I can gracefully shutdown. I was able to resolve with the following:
If you don't set something for kill delay it seems like Air eats the signal. |
Just writing in to say that even with |
Referring to the example toml https://github.com/cosmtrek/air/blob/master/air_example.toml At least set your delay to 100 milliseconds | kill_delay = 100000000 Just timing my DB and app close takes nearly 500 microseconds so 100 milliseconds is pretty safe for Testing/Development. EDIT: Here is my gracefull shutdown portion for reference.
Output: |
@figuerom16 I've tried delay values of |
@peterldowns Are you on the latest air also? My air -v is v1.51.0, built with Go go1.22.2. |
@figuerom16 I'll set up a minimal reproduction case and share it here within the next day or so. Sorry for not doing so earlier, I realize it's hard to debug without an easy way to reproduce the problem. I'm using air installed via nix, but it's 1.51.0 built with go1.22.1. I'm on macOS Monterey version 12.5.1 on a M1 pro:
|
Could folks having this problem give this a try? |
@sethbrasile I have the same problem and #575 solved the issue for me 👍 |
This repo is a reproducible example, if anyone needs: https://github.com/sa-/goth-counter |
@sethbrasile I can confirm that this fixed the issue for me as well. I tested by cloning your fork, building the One suggestion — can you make it so that if the backend server has terminated,
|
Also, thank you for the fix! |
I had some issues with this as well. I tried your branch @sethbrasile, it helps but doesn't fix it entirely for me, there were times I had to manually kill the process. This code here could be more resilient: https://github.com/cosmtrek/air/blob/master/runner/util_linux.go#L12. It's ignoring some errors, and in my opinion the delay between SIGINT and SIGKILL should be the default, now the user has to know about the |
I had the same issue and was able to permanently fix it by adding:
|
|
Wild guess, but it seems that upgrading MacOS from 14.4.1 to 14.5 seems to have resolved this for me. Can anyone else confirm? Previously I reproduced this problem of failing to kill process with SIGINT/SIGTERM with both air and modd, so I don't think air is to blame specifically. |
I temporarily workaround this by using an older version of air (v1.46.0) and the graceful shutdown on mac works for me |
I usually unconsciously do Ctrl-C and I found this what I consider a bug or maybe it could just because of my code. I'm not really sure. So when the server is up and running and everything goes perfect right? and when I exit with Ctrl-C it does not finish the exiting process causing port already in use on next start.
My code on graceful shutdown
The text was updated successfully, but these errors were encountered: