Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cron jobs are not signaled SIGTERM on terminated and jobs are still started while waiting for termination #226

Open
Tofandel opened this issue Jul 4, 2023 · 0 comments

Comments

@Tofandel
Copy link

Tofandel commented Jul 4, 2023

When stopping an ofelia container, the running jobs do not receive a SIGTERM signal which would tell them the container is about to stop, instead it just waits for them to finish normally (which can take an undefined amount of time) then 10 seconds later docker SIGKILLs the ofelia container which prevents them from shutting down correctly and doing the cleanup that they should be doing, this is a problem for very long jobs

A SIGTERM would normally tell those jobs that they are about to stop and that they should save their current state, they otherwise loose the progress they made

Also while waiting for jobs to stop, new jobs are being picked up, which would also get killed without due process

2023-07-05 01:58:08 2023-07-04T23:58:08.141Z  daemon.go:69 ▶ WARNING Signal received: terminated, shutting down the process
2023-07-05 01:58:08 2023-07-04T23:58:08.141Z  daemon.go:83 ▶ WARNING Waiting running jobs.
2023-07-05 01:58:16 2023-07-04T23:58:16.004Z  common.go:125 ▶ NOTICE [Job "jobqueue" (3982f5ca83d2)] Started - /var/www/html/bin/console jms-job-queue:run -vv -r 300
2023-07-05 01:58:16 2023-07-04T23:58:16.016Z  common.go:125 ▶ NOTICE [Job "mailqueue" (c7855fbff257)] Started - /var/www/html/bin/console swiftmailer:spool:send --mailer=queue --message-limit=30
2023-07-05 01:58:16 2023-07-04T23:58:16.441Z  common.go:125 ▶ NOTICE [Job "mailqueue" (c7855fbff257)] StdOut: 
2023-07-05 01:58:16  [2023-07-04 23:58:16] Processing queue mailer spool... 
2023-07-05 01:58:16  0 emails sent
2023-07-05 01:58:16 2023-07-04T23:58:16.441Z  common.go:125 ▶ NOTICE [Job "mailqueue" (c7855fbff257)] Finished in "425.510373ms", failed: false, skipped: false, error: none

PS: While the docker API currently does not support killing an exec, we can still create a new exec with the command kill -n 9 $execId, I would attempt a PR, but I have never touched go before

@Tofandel Tofandel changed the title Cron jobs are killed with SIGKILL instead of SIGTERM Cron jobs are not signaled SIGTERM on terminated and jobs are still started while waiting for termination Jul 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant