Batch retry inside of super job

[j-mcnally]

If an item in a batch fails and is retried, the batch finalizes and next task is run before the batch completes.

Superworker.create(:ImageImportSuperWorker, :staging_image_ids, :valid_only, :started_at, :staging_image_group_id) do
  batch staging_image_ids: :staging_image_id do
    ImageImportWorker :staging_image_id, :valid_only
  end
  FinalizeImageImportWorker :staging_image_group_id, :valid_only, :started_at
end

In the example FinalizeImageImportWorker can be called before all the batch ImageImportWorkers are actually done, if an ImageImportWorker is retried.

[j-mcnally]

The work around to this is to disable the child worker retries completely and keep retrying on the same worker/thread.

The following module which can be included in sidekiq workers does the trick

module Sidekiq
  module RetryManually
    def with_retry(count=10, &block)
      retries = 0
      begin
        yield
      rescue Exception => e
        if (retries <= count)
          retries += 1
          sleep (retries * 1)
          retry
        else
          puts "Sidekiq Failed after #{count} retries::: #{e.inspect}"
          raise e
        end
      end
    end
  end
end

and can be called around the perform code

  def perform(staging_image_id, valid_only, time=0, staging_group=0)
    with_retry 5 do
      staging_image_group = nil
      StagingImage.find(staging_image_id).import!
    end
  end

[brodock]

does it still apply?