Pace provides a high-throughput way to process Resque jobs inside an EventMachine reactor.
When combined with EM::HttpRequest you can send thousands of requests per second to a remote web service.
Tested under:
- REE 1.8.7
- MRI 1.9.2 (best memory performance)
To have fun with the examples, fire one up and then start enqueuing Resque jobs:
$ rake examples:http
$ irb
> require "rubygems"
> require "resque"
> class MyJob; def self.queue; "normal"; end; end
> Resque.enqueue(MyJob)
> 10.times { |n| Resque.enqueue(MyJob, :n => n) }
In a separate process, start up a worker:
require 'pace'
worker = Pace::Worker.new(:queue => "normal")
worker.start do |job|
klass = job["class"]
options = job["args"].first
# do work with options
end
Pace connects to Redis with a URI that's looked up in the following order:
- Pace.redis_url attr_accessor
- REDIS_URL environment variable
- Defaults to 127.0.0.1:6379/0
It's very easy to overwhelm a remote service with pace. You can specify the maximum number of jobs to consume per second.
Pace::Worker.new(:queue => "normal", :jobs_per_second => 100)
If you need to pause a worker (for example, during remote service failure):
worker.pause
And when ready:
worker.resume
You can also pause for a set period of time. The worker will resume automatically.
worker.pause(0.5) # 500ms
Pace attempts to keep the reactor going at all costs with explicit rescues and EM's catch-all error handler. A hook is provided for errors so that action can be taken:
worker.add_hook(:error) do |json, error|
message = error.message
# The job JSON can be nil if the error is raised in a callback.
message << json if json
Pace.logger.warn(message)
end
Hooks can also be attached at the class-level, which affects all workers.
Pace::Worker.add_hook(:error, handler)
Finally, an Airbrake hook is provided that will notify Airbrake on all exceptions:
require "pace/airbrake"
Pace::Worker.add_hook(:error, Pace::Airbrake.hook)