You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There appears to be an issue where workers marked for termination while processing jobs do not terminate gracefully when horizon:terminate is subsequently invoked. These workers, while still actively running, are overlooked during the supervisor's termination process. As a result, instead of terminating gracefully, they are killed upon the supervisor's exit.
Steps To Reproduce
Launch the master supervisor with the fast_termination option set to false using horizon command.
Send a long-running job to the queue. Ensure that this job is being processed by a worker.
Wait for scaleDown() method to be triggered on ProcessPool, ensuring that the process handling the long-running job is marked for termination. For consistent test results, use the code snippet below to simulate a supervisor restart during which all worker processes are marked for termination by scaling process pools down to 0.
Terminate horizon using horizon:terminate command.
// Dispatch a long-running job that sleeps for 60 seconds
SleepJob::dispatch(60);
sleep(10); // Make sure that job is picked up
// Trigger a restart on all supervisors, marking workers for termination
foreach (app(SupervisorRepository::class)->names() as $name) {
app(HorizonCommandQueue::class)->push(
$name, Restart::class
);
}
sleep(10); // Make sure that all supervisors have restarted
// Call the terminate command
Artisan::call(TerminateCommand::class);
class SleepJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public function __construct(protected readonly int $sleepDuration)
{
// Nothing
}
/**
* Execute the job.
*/
public function handle(): void
{
for ($i = 0; $i < $this->sleepDuration; $i++) {
sleep(1);
}
Log::debug('Job finished.');
}
}
The text was updated successfully, but these errors were encountered:
Horizon Version
5.24.3
Laravel Version
11.5.0
PHP Version
8.3.6
Redis Driver
PhpRedis
Redis Version
7.2.4
Database Driver & Version
No response
Description
There appears to be an issue where workers marked for termination while processing jobs do not terminate gracefully when
horizon:terminate
is subsequently invoked. These workers, while still actively running, are overlooked during the supervisor's termination process. As a result, instead of terminating gracefully, they are killed upon the supervisor's exit.Steps To Reproduce
fast_termination
option set tofalse
usinghorizon
command.scaleDown()
method to be triggered onProcessPool
, ensuring that the process handling the long-running job is marked for termination. For consistent test results, use the code snippet below to simulate a supervisor restart during which all worker processes are marked for termination by scaling process pools down to 0.horizon:terminate
command.The text was updated successfully, but these errors were encountered: