-
-
Notifications
You must be signed in to change notification settings - Fork 36
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workflow backoff #178
Comments
@rommeA I did not roll back that feature. The problem is with the database migration that creates the jobs table.
It seems that this table is using a signed smallinit which limits the max value to 32767 and therefore 32768 is too big. First I would ask that you check if 32768 attempts seems like too many. If you think that an activity/workflow should retry more than 32767 times then you can increase the size of that column. I hope this has answered your question. If not please feel free to ask again. |
@rmcdaniel yeah, I understand that the problem is in the database. But 32768 attempts really is too many for a workflow (not an activity). I really cannot find the reason why only one specific workflow retries so many times and I struggle to reproduce this bug (if it really is a bug). When this "stuck" job occurres in jobs table, the workflow is waiting for acceptance (line 86):
To fix a stuck job I manually change max_retries value for the job in the database, then the job fails, I run
and the job completes successfully. I'll give my feedback If I manage to reproduce it locally. P.S.: maybe it has smth to do with the cache. My project implements custom replication - I replicate workflows from the other instance of my app - models are created silently. I have many other workflows in my project and this problem with too many attempts occures only for replicated (exchanged) workflow class. |
Yes there are events https://laravel-workflow.com/docs/features/events that get created. |
@rommeA After doing some investigating it seems Workflows don't have a backoff and never did, it is only activities that do, The documentation is wrong. It seems that in some cases a workflow can spinwait when there are idle workers and this can trigger the maximum count but this is intentional. Perhaps we should suggest that the migrations should be modified. But we should confirm the spinwait issue is what's happening first. |
I would also point out that if an activity fails to save its state when returning, it keeps trying despite of maximum tries, basically looping the workflow forever, without apparent failures. This issue could be the root cause of workflows trying too many times for no apparent reason. Hope this helps! |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Thank you for this package, I've been using it for a while now.
In one of my Workflows I constantly get this error in jobs (don't know why yet):
I've seen you've implemented backoff for Workflow, but did you decide to rollback this feature? Thanks!
The text was updated successfully, but these errors were encountered: