Skip to content

Micro-services as modules with Docker and pyzmq (ZeroMQ) basic examples

License

Notifications You must be signed in to change notification settings

NumesSanguis/pyzmq-docker

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Docker & ZeroMQ

Update 2019-08-17

Simplified sample code:

  • Removed *-no-compose folders; Python scripts, docker-compose and Docker run now use same code base.
  • IP changed from * to 0.0.0.0. Prevent errors on some OSes.
  • pub/sub main.py now works with named arguments.
  • Updated this README.md to be more extensive and clear.

Overview

Example repository to demonstrate how you can turn Python scripts into micro-services in Docker containers, which can communicate over ZeroMQ. The examples here can be run as just Python-Python, Docker-Docker (docker-compose) or Docker-Python (docker run).

Examples are using a Publisher-Subscriber pattern to communicate. This means that the publisher micro-service just send messages out to a port, without knowing who is listening and a subscriber micro-service receiving data, without knowing where the data comes from.

With ZeroMQ, only 1 micro-service can socket.bind(url) to 1 address. However, you can have unlimited micro-services socket.connect(url) to an address. This means that you can either have many-pub to 1-sub (examples in this Git repo) or 1-pub to many-sub on 1 ip:port combination.

Install Docker

1. Python-Python

  1. Open a terminal and navigate to folder pyzmq-docker/sub
  2. Execute python main.py
  3. Open a terminal and navigate to folder pyzmq-docker/pub
  4. Execute python main.py
  5. See subscriber receiving messages from the publisher!

Notes:

  • Steps 1-2, can be reversed with steps 3-4.
  • Make sure you've installed PyZMQ in your Python installation (conda install pyzmq or pip install pyzmq)

2. Docker-Docker with docker-compose

  1. Open a terminal and navigate to folder pyzmq-docker
  2. Executedocker-compose up --build
  3. See a Dockerized subscriber receiving messages from a Dockerized publisher! (That's really everything? 0.o)

Notes:

  • If you didn't make any changes to your Docker container, you can Execute docker-compose up without --build to skip the build process.
  • Advantages of docker-compose:
    • You need only 1 docker-compose.yml to start multiple Docker micro-services
    • It connects the pub micro-service to the sub micro-service with tcp://sub:5550. Docker automatically turns sub into the IP of the subscriber micro-service.

3. Docker-Python with docker run

Notes:

  • Make sure you've installed PyZMQ in your Python installation (conda install pyzmq or pip install pyzmq)

3a. pub-Docker, sub-Python

  1. Open a terminal and navigate to folder pyzmq-docker/sub
  2. Execute python main.py
  3. Open file pub/Dockerfile and change "yo.ur.i.p" to your machine IP (something similar to: "192.168.99.1")
  4. Open a terminal and navigate to folder pyzmq-docker/pub
  5. Execute docker build . -t foo/pub
  6. Execute docker run -it foo/pub
  7. See that your subscriber receives messages from your Dockerized publisher.

Notes:

  • Step 5 can be skipped after the first time if no changes were made to the Docker/Python files.
  • Steps 1-2 can be reversed with steps 3-6.

3b. pub-Python, sub-Docker

  1. Open a terminal and navigate to folder pyzmq-docker/sub
  2. Execute docker build . -t foo/sub
  3. Execute docker run -p 5551:5551 -it foo/sub (maps port of Docker container to localhost)
  4. Open a terminal and navigate to folder pyzmq-docker/pub
  5. Execute python main.py
  6. See that your Dockerized subscriber receives messages from your publisher.

Notes:

  • Steps 1-3 can be reversed with steps 4-5.
  • Add a name to a container by adding --name foo-sub to docker run
  • In case of container name already in use, remove that container with: docker rm foo-sub

Other

Inspiration

Stackoverflow question: https://stackoverflow.com/questions/53802691/pyzmq-dockerized-pub-sub-sub-wont-receive-messages

Useful Docker commands

sudo usermod -a -G docker $USER  # add current user to group docker on Linux systems (Ubuntu)

docker build . -t foo/sub  # build docker image
docker run -it foo/sub  # run build docker image and enter interactive mode
docker run -p 5551:5551 -it foo/sub  # same as above with mapping Docker port to host
docker run -p 5551:5551 --name foo-sub -it foo/sub  # same as above with naming container
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' pyzmq-docker_sub_1  # get ip of container
docker rm foo-sub  # remove container by name

docker-compose up  # run docker-compose.yml
docker-compose build / docker-compose up --build  # rebuild images in docker-compose.yml

docker image ls  # show docker images
docker container ls  # show docker containers
docker exec -it pyzmq-docker_pub_1 bash  # enter bash in container
docker attach pyzmq-docker_sub_1  # get

To detach the tty without exiting the shell, use the escape sequence Ctrl+p + Ctrl+q

docker rm $(docker ps -a -q)  # Delete all containers
docker rmi $(docker images -q)  # Delete all images

Debug docker-machine IP not found (probably not necessary)

Docker machine working check:

  • Open a terminal and Execute command: docker-machine ip
    • Should return a Docker machine IP (likely 192.168.99.100)
    • If not, see section "Debug" (e.g. Error: No machine name(s) specified and no "default" machine exists)

Debug attempts:

About

Micro-services as modules with Docker and pyzmq (ZeroMQ) basic examples

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published