Prevent double-acking or acking on wrong channel in rabbitmq

[jeanetteclark]

After deploying on production a new rabbitmq issue popped up. I'm seeing some error messages that look like

 RabbitMQ connection error: channel is already closed due to channel error; protocol method:
 #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED -
 unknown delivery tag 1, class-id=60, method-id=80) [edu.ucsb.nceas.mdqengine.Worker:450]

SO says this usually happens because of double ack-ing, ack-ing on wrong channels or ack-ing messages that should not be ack-ed. I think that this is possibly double acking, but it could also be acking on the wrong channel, especially now that I have the system fixed so that rabbitmq can open connections/channels better if they get closed. Possibly what is happening is that a channel times out, so the connection gets closed, then a new channel/connection is opened and when trying to ack the message we hit the exception because it is the wrong channel.