Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add queuing_timeout and execution_timeout option for job #457

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

bluzky
Copy link

@bluzky bluzky commented Oct 7, 2021

This PR implement 2 timeout options for exq.

  • queuing_timeout (in second): TTL for job in queue, timed out job will be
    discard using a middle ware
  • execution_timeout (in second): if execution time is longer than
    execution_timeout the worker is killed and not retry any more.

This could be used as an example for those who want to implement timeout.
This PR does not included test and execution

- `queuing_timeout` (in second): TTL for job in queue, timed out job will be
discard using a middle ware
- `execution_timeout` (in second): if execution time is longer than
`execution_timeout` the worker is killed and not retry any more.
@ananthakumaran
Copy link
Collaborator

Both of these can be implemented easily with existing APIs/middleware provided by Exq. I am not convinced about the benefit of adding this in Exq itself.

expiration

def perform() do
  job = Exq.worker_job()
  if !expired?(job.enqueued_at) do
    do_work()
  end
end

timeout

def perform() do
  {:ok, tref} = :timer.kill_after(:timer.seconds(timeout))

  try do
    do_work()
  after
    :timer.cancel(tref)
  end
end

Disable retry

for certain errors like the kill above

defmodule RetryMiddleware do
  def before_work(pipeline), do: pipeline
  def after_processed_work(pipeline), do: pipeline

  def after_failed_work(pipeline) do
    if non_retryable?(pipeline.assigns.error) do
      put_in(pipeline.assigns.job.retry, pipeline.assigns.job.retry_count || 0)
    else
      pipeline
    end
  end
end

@bluzky
Copy link
Author

bluzky commented Oct 8, 2021

@ananthakumaran wow thank you for your simple solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants