Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[5.x] Ensure graceful termination of workers marked for termination #1433

Draft
wants to merge 1 commit into
base: 5.x
Choose a base branch
from

Conversation

tarexme
Copy link

@tarexme tarexme commented Apr 27, 2024

Fix #1432

Description

There appears to be an issue where workers marked for termination while processing jobs do not terminate gracefully when horizon:terminate is subsequently invoked. These workers, while still actively running, are overlooked during the supervisor's termination process. As a result, instead of terminating gracefully, they are killed upon the supervisor's exit.

Steps To Reproduce

  1. Launch the master supervisor with the fast_termination option set to false using horizon command.
  2. Send a long-running job to the queue. Ensure that this job is being processed by a worker.
  3. Wait for scaleDown() method to be triggered on ProcessPool, ensuring that the process handling the long-running job is marked for termination. For consistent test results, use the code snippet below to simulate a supervisor restart during which all worker processes are marked for termination by scaling process pools down to 0.
  4. Terminate horizon using horizon:terminate command.
// Dispatch a long-running job that sleeps for 60 seconds
SleepJob::dispatch(60);
sleep(10); // Make sure that job is picked up

// Trigger a restart on all supervisors, marking workers for termination
foreach (app(SupervisorRepository::class)->names() as $name) {
    app(HorizonCommandQueue::class)->push(
        $name, Restart::class
    );
}

sleep(10); // Make sure that all supervisors have restarted

// Call the terminate command
Artisan::call(TerminateCommand::class);
class SleepJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct(protected readonly int $sleepDuration)
    {
        // Nothing
    }

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        for ($i = 0; $i < $this->sleepDuration; $i++) {
            sleep(1);
        }

        Log::debug('Job finished.');
    }
}

@driesvints
Copy link
Member

Please add a thorough description to your PR and not just link to an issue. This will help the people reviewing your PR.

@tarexme tarexme changed the title [5.x] Fix #1432 [5.x] Ensure graceful termination of workers marked for termination Apr 29, 2024
@taylorotwell taylorotwell marked this pull request as draft April 29, 2024 14:48
@art-vanesyan
Copy link

are there any updates? the same situation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Graceful termination fails for workers marked for termination during job processing
3 participants