-
Notifications
You must be signed in to change notification settings - Fork 629
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
laravel horizon jobs stuck in pending #1280
Comments
Hey there, Can you first please try one of the support channels below? If you can actually identify this as a bug, feel free to open up a new issue with a link to the original one and we'll gladly help you out. Thanks! |
ok thanks |
Got the same thing happening, latest Laravel & Horizon versions. Jobs even though processed correctly keep accumulating on the Pending list in the Dashboard, although the queues are empty. I could not find more accurate reports or solutions specifically to this situation. @khalidgxg did you learn what was causing it for you? And @driesvints I assume if one can showcase a repo with Laravel+Horizon that could consistenly show/reproduce the issue happening as reported, it would then qualify for you guys to pursue it as a bug, right? I might try to put something together. |
Hey all. This should have been fixed already in 4.x, does that not work for you? laravel/telescope#1349 |
Telescope that is. |
@driesvints #1185 |
Thank you for reporting this issue! As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub. If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team. Thank you! |
Thank you. We'd appreciate any help through a PR for this. |
@driesvints is there a way to release these reserved jobs to they can be processed again? |
I don't know sorry. |
@themsaid do you maybe know if it's possible to release these jobs that get stuck as |
I'd like to add here as well that we're experiencing the same issue with jobs being stuck in 'pending' even if completed_at is present in horizon dashboard, and the jobs actually completed successfully. We have not been able to determine the root cause. We dispatch millions of jobs a month, and it affects only approximately 20 per day. Still, it's rather unsettling and we'd love to find a solution. |
@pnlinh you are WAY out of date with laravel, horizon & php. Please update to most recent versions and test again. There is no sense in asking for help with such outdated versions. |
Thanks for your suggestion but my project cannot upgrade now. I added delay value to jobs, it seems it works. |
@pnlinh please try to focus the discussion on supported Laravel/Horizon versions, thanks. |
Even with updated versions I still have this issue. |
i set 'TELESCOPE_JOB_WATCHER' to false in config, and they all came flooding back into completed. "Watchers\JobWatcher::class => env('TELESCOPE_JOB_WATCHER', false)" mentioned laravel/telescope#1349 (comment) |
same issue.. |
Hey all. Extra messages that you're experiencing this issue aren't really helpful. Instead, please try posting extra findings around the issue or help out with a PR, thanks. |
I've been facing this problem for almost 3 years, where the internal solution provided is sleep(3) inside all jobs. What does the answer above make sense, since with sleep(3) the events have time to orchestrate themselves normally. Would this be a possible point of investigation? |
This worked for me for a couple of months, but since the application scaled up and we had more workload this became a huge headhache. What I'm doing right now it's migrating everything for serverless services, jobs, and isolated applications with C#. The only way to stop this behavior is to stop using Horizon for huge workloads. |
It can be temporarily avoided in this way. |
This is a good solution tho. But have you tested with a huge workload? My workload's very big and when I started adding 2000 sleep everything started to overlap. I didn't try with 5k, might be it's a good workaround but anyway, it's not a good solution. |
I this is within 1 second to detect whether it is a pending state, every 5ms cycle detection, if it is a pending state means that the event has been executed, you can carry out the next operation, I also do the anti-dumbness, if 1 second after the event has not been executed, the task will not care about it, so that he stays in the pending list, but this extreme case is almost zero! |
Horizon Version
5.15
Laravel Version
10.09.0
PHP Version
8.1
Redis Driver
Predis
Redis Version
2.1.2
Database Driver & Version
No response
Description
Hi all,
I am writing to report a bug related to the job processing in Laravel Horizon. Specifically, I have encountered a situation where a job appears to be stuck in the "pending" status despite having a "completed_at" timestamp, and another job is stuck in the "reserved" status without progressing further.
Here are the details of the problematic jobs:
Job with "pending" status but "completed_at" :
"status": "pending",
"completed_at": "1685113859.5327",
"reserved_at": "1685113859.5284",
Job with "reserved" status :
"status": "reserved",
"completed_at": null,
"reserved_at": "1685110558.3953",
In the first case, the job has a "completed_at" timestamp indicating its successful completion, but it remains in the "pending" status. On the other hand, the second job is stuck in the "reserved" status without progressing further.
Could you please investigate this issue and provide guidance on how to resolve it? It seems that the job statuses are not being updated correctly, causing confusion in monitoring and processing.
Thank you for your attention to this matter. I look forward to your response and assistance in resolving this bug.
Best regards,
khalid
Steps To Reproduce
The text was updated successfully, but these errors were encountered: