An example of a RabbitMQ instance running on Docker with custom configuration. This can be loaded with Docker or Docker Compose.
This is designed as a tutorial and template that you can customise and extend for the implementation you desire.
This Readme will cover:
- Useful docker commands
- RabbitMQ key concepts
- How the configuration files in the
rabbitmq
folder configure the RabbitMQ instance - Using docker and docker-compose to run and remove the RabbitMQ images and containers
- How to use the RabbitMQ UI to view, create and delete exchanges, queues, bindings
- How to configure work, dead and retry exchanges and queues
Useful commands:
docker ps
view running containersdocker ps -a
view all containers, including stopped onesdocker rm [CONTAINER]
Removes a containerdocker images
lists imagesdocker image rm [IMAGE]
There are scripts in the bin
folder to build, run, stop and remove containers and images. See how to use them below.
- www.rabbitmq.com AMQP Concepts
- www.rabbitmq.com AMQP Quick Reference Guide
- www.rabbitmq.com Get Started
- www.rabbitmq.com Documentation
- www.cloudamqp.com blog RabbitMQ for Beginners: What is RabbitMQ?
This contains the definitions.json
and rabbitmq.conf
files that are loaded into the RabbitMQ instance. The configuration be overwritten directly in the files. Alternatively log into RabbitMQ UI at localhost:15672, amend then Export definitions
with
Filename for download: definitions.json
from the Overview
tab and save them into this folder. The current configuration creates:
- Admin user with username
me
and passwordme
(use this to log into the RabbitMQ UI at localhost:15672) - Exchanges:
work
(with routing key 'work'),retry
anddead
- Queues:
work
,retry
anddead
- Bindings for queues to exchanges
See below for more detail on the configuration of the bindings and arguments for the exchanges and queues.
There are bash scripts in the bin
folder to:
run.sh
- This will stop and remove existingrabbitmq-docker
images and container, then build a new image and run the containerrabbitmq-docker
build.sh
- builds the imagerabbitmq-docker
using the Dockerfilestop.sh
- stops the containerrabbitmq-docker
. It takes an optional--silent
flag if you don't want it to log output, egbin/stop --silent
.remove.sh
- stops and removes therabbitmq-docker
containers and images. It also has an optional--silent
flag if you don't it them to log output, egbin/remove --silent
.
The Dockerfile
will create an image from rabbitmq:3.7.3-management
and loads the configuration defined in the rabbitmq
folder.
COPY ./rabbitmq/rabbitmq.conf ./etc/rabbitmq/rabbitmq.conf
COPY ./rabbitmq/definitions.json ./etc/rabbitmq/definitions.json
This uses the docker-compose.yml
, which also uses the image rabbitmq:3.7.3-management
and loads the definitions.json
and rabbitmq.conf
files into the container.
docker-compose up
Start the RabbitMQ instance with docker-composedocker-compose up -d
Start RabbitMQ on docker to run without logging to the current terminal sessiondocker-compose down -v
To shut down the RabbitMQ instance and remove the volumes (thedefinitions.json
andrabbitmq.conf
configuration files)docker-compose logs rabbitmq-docker
View the logsdocker-compose logs --follow rabbitmq-docker
To view logs and follow output
You can also use the bin
scripts to stop and remove the containers and images here too.
bin/remove.sh
- Stops and removes the rabbitmq-docker container and imagebin/stop.sh
- will just stop the rabbitmq-docker container
Open the definitions.json
file and replace the names of user, vhost, exchanges, queues and bindings with the ones you need.
It can be easier to use the RabbitMQ UI to configure these
- Log into RabbitMQ UI at
localhost:15672
using the loginme
for username and password (the default isguest
if you are not using thedefinitions.json
file in therabbitmq
folder) - amend the configuration
- On the
Overview
tab, click onExport definitions
and setFilename for download: definitions.json
, and thenDowload broker definitions
and save them into therabbitmq
folder
- Create and run the RabbitMQ instance using Docker or Docker Compose
- login at
localhost:15672
- username and password are set in the
definitions.json
, currently "me" and "me"
This configuration shows you how to use exchanges and queues with three main purposes:
work
the main exchange and queue that an application would consume messages from.retry
publish a message here for it to wait for the time to live before routing the message to thework
exchange. This is useful if a message throws an error or cannot be consumed at the time the message is fetched from thework
queue.dead
a place to store messages that could not be delivered or have expired their time to live on thework
queue.
- Click on the
Exchanges
tab to view exchanges. The ones defined inrabbitmq/definitions.json
should be listed - Publish a message on
my-exchange.work
exchange with routing key: "work" - Click on the queues tab.
my-queue.work
should show a spike of activity and the message count should increase by 1 for the configured time to live value (currently set at 5 seconds).- After the time to live has expired, the message will be considered dead
- It will be routed to the dead letter exchange:
my-exchange.dead
- This places the message in the queue
my-queue.dead
my_queue.dead
should show an additional message. View it by clicking on the queue, thenGet messages
->Get messages
- Click on the
Exchanges
tab, then click on themy-exchange.retry
exchange - Publish a message on
my-exchange.retry
exchange with routing key: "work" - Click on the queues tab. The
my-queue.retry
message count should increase by 1 for the configured time to live value (currently set at 5 seconds). - After the time to live has expired, the message will be routed to the
my-exchange.work
- This places the message in the queue
my-queue.work
for the time to live, before routing to themy-exchange.dead
, which sends it tomy-queue.dead
- Publish a message on
my-exchange.work
without a routing key or one not defined in the configuration. - This should be routed to
my-queue.dead
Below explains how to set up the exchanges, queues and bindings.
Create an exchange "my-exchange.work". One of the simplest ways of routing is using a direct type exchange, where you define the "routing_key" in the bindings when linking the exchange to a queue. So for this example, specify the type as "direct".
Define a queue, such as "my-queue.work", then the bindings between them with the source being the exchange and the destination the queue. The destination_type is queue. Then in the arguments state the routing key, for example "routing_key":"work"
(the routing key can be anything you want).
- On your main queue, eg
my_queue.work
, add arguments
- "x-dead-letter-exchange": "my-exchange.dead"
- "x-message-ttl": 5000
x-message-ttl
is the variable that stores the amount of time a message should be kept on a queue before being considered dead and routed to the dead-letter-exchange
. The 5000 here will keep it here for 5 seconds, which is useful for testing, but should be longer when used in production.
- Define your dead letter exchange, eg
my-exchange.dead
, and set it as typefanout
(fanout exchanges will send all messages onto queues without needing a routing key) - Define your dead letter queue, eg
my-queue.dead
- Bind the dead letter exchange to the dead letter queue
- Create
my-queue.retry
queue bound tomy-exchange.retry
- Create exchange
my-exchange.retry
with arguments:- Set
x-dead-letter-exchange
tomy-exchange.work
- Set
x-message-ttl
to a time you want it to wait on the retry queue before being sent back to thework
exchange eg 5000 (5 seconds)
- Set
To create an exchange and queue for messages that cannot be routed due to not having a defined routing key or headers:
- Add as an argument to your
work
exchange:"alternate-exchange": "my-exchange.dead"
- https://medium.com/@thomasdecaux/deploy-rabbitmq-with-docker-static-configuration-23ad39cdbf39
- https://stackoverflow.com/questions/40280293/defining-a-queue-with-a-config-file-in-rabbitmq
- https://www.rabbitmq.com/configure.html
- https://devops.datenkollektiv.de/creating-a-custom-rabbitmq-container-with-preconfigured-queues.html
- https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html
- https://medium.com/@kiennguyen88/rabbitmq-delay-retry-schedule-with-dead-letter-exchange-31fb25a440fc