Message deduplication

[hmarr]

When we're using mirrored queues, there is a chance of receiving a message multiple times. Prevent this by storing message ids / hashes in Redis or Zookeeper, and checking if messages have already been processed before handing them off to a consumer.

See https://github.com/xing/beetle/blob/master/lib/beetle/deduplication_store.rb

[ms-ati]

+1