sidekiq-merger

Merge sidekiq jobs occurring before the execution times. Inspired by sidekiq-grouping.

Demo

Use Case

Cancel Task

Bulk Notification

Installation

Add this line to your application's Gemfile:

gem 'sidekiq-merger'

And then execute:

$ bundle

Or install it yourself as:

$ gem install sidekiq-merger

Usage

Add merger option into your workers.

class SomeWorker
  include Sidekiq::Worker

  sidekiq_options merger: { key: -> (args) { args[0] } }

  def perform(*merged_args)
    merged_args.each do |args|
      # Do something
    end
  end
end

Then, enqueue jobs by perform_in or perform_at.

SomeWorker.perform_in 100, 4
SomeWorker.perform_in 100, 3
SomeWorker.perform_in 100, 5
# Passed 100 seconds from the first enqueue.
SomeWorker.perform_in 100, 6
SomeWorker.perform_in 100, 1

SomeWorker will be executed in 100 seconds with args of [4], [3], [5], then with args of [6], [1].

perform_async works without merging args.

SomeWorker.perform_async 4
SomeWorker.perform_async 3
SomeWorker.perform_async 5

In this case, SomeWorker will be executed 3 times with args of [4], [3] and [5].

Quick Check

Run docker containers to check the behavior of this gem.

$ docker-compose up

Then, open http://localhost:3000/. You can push jobs from the UI and see what happens in the sidekiq console.

Options

key (optional, default: nil)

Defines merge key so different arguments can be merged.

Format: String or Proc

e.g. sidekiq_options merger: { key: -> (args) { args[0..1] } }

unique (optional, default: false)

Prevents enqueue of jobs with identical arguments.

Format: Boolean

e.g. true

batch_size (optional, default: nil)

Allow to specify how many jobs max to provide as arguments per aggregation

Format: Int

e.g. 50

Web UI

Add this line to your config/routes.rb to activate web UI:

require "sidekiq/merger/web"

Test

$ bundle exec appraisal rspec

The test coverage is available at ./coverage/index.html.

Lint

$ bundle exec appraisal rubocop

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Copyright

Copyright (c) 2017 dtaniwaki. See LICENSE for details.