Pika performace

[Gsantomaggio]

I am opening this issue after a conversation with @lukebakken
here the thread

RabbitMQ test with different libraries

I made a few tests with different client libraries.

Environment:

• Laptop 12 cores, 32 gb ram
• Two rabbitmq nodes in cluster ( same machine)
• RabbitMQ version 3.7.13 - Erlang 21.2.4
• 64 async threads for each server
• Ubuntu 18.04
• Two Python processes running

(.venv3) ➜  py-amqp git:(master) ✗ pip freeze
amqp==2.4.2
atomicwrites==1.3.0
attrs==19.1.0
librabbitmq==2.0.0
more-itertools==6.0.0
pika==0.13.1
pluggy==0.9.0
py==1.8.0
PyAMQP==0.0.7.1
pytest==4.3.1
six==1.12.0
vine==1.3.0
(.venv3) ➜  py-amqp git:(master) ✗

Here is the code:

import time
import uuid
import sys
class PyAmqpTest:

    def publish(self, rm):
        c = amqp.Connection(host=rm)
        channel = c.channel()
        qname = str(uuid.uuid4())
        message = amqp.Message(
            channel=channel,
            body='the quick brown fox jumps over the lazy dog',
            properties=dict(content_type='application/json',
                            content_encoding='utf-8'))

        channel.queue_declare(queue=qname, auto_delete=False)
        print("start: %s" % (time.ctime(time.time())))
        for i in range(1, 900000):
            channel.basic_publish(message, routing_key=qname)
        print("end: %s" % (time.ctime(time.time())))

    def thread_publish(self, rm):
        for i in range(1, 15):
            _thread.start_new_thread(self.publish, (rm,))


print('starting .. %s' % sys.argv[1])
x = PyAmqpTest()
x.thread_publish(sys.argv[1])

input("Press Enter to continue...")

py-amqp

import amqp as amqp

and then

python3 py_amqp_publish.py localhost:5674 
python3 py_amqp_publish.py localhost:5672

around 18.000 messages per second

librabbitmq

import librabbitmq as amqp

and then

python3 py_amqp_publish.py localhost:5674 
python3 py_amqp_publish.py localhost:5672

around 102.000 messages per second

Pika

import _thread

import pika
import time
import uuid
import sys


class PyPikaTest:

    def publish(self, rm):
        c = pika.BlockingConnection(pika.ConnectionParameters(port=rm))

        channel = c.channel()
        qname = str(uuid.uuid4())
        channel.queue_declare(queue=qname, auto_delete=False)
        _properties = pika.BasicProperties(
            content_type='application/json',
            content_encoding='utf-8'
        )
        print("start: %s" % (time.ctime(time.time())))
        for i in range(1, 900000):
            channel.basic_publish(
                exchange='',
                routing_key=qname,
                properties=_properties,
                body='the quick brown fox jumps over the lazy dog'
            )
        print("end: %s" % (time.ctime(time.time())))

    def thread_publish(self, rm):
        for i in range(1, 15):
            _thread.start_new_thread(self.publish, (rm,))


print('starting .. %s' % sys.argv[1])
x = PyPikaTest()
x.thread_publish(sys.argv[1])

input("Press Enter to continue...")

So:

python3 py_pika_publish.py 5672
python3 py_pika_publish.py 5674

about 11.000 messages per second

---

Thank you

[ponach]

Just to shed more light into this yet to be explained the huge difference in throughput, I found that using librabbitmq, there's unexplainable message loss meaning not all the messages are received by the messaging queue ( I experimented with 1000000, at the end of the experiment the messaging queue only got 999,820 )

[Gsantomaggio]

@ponach Thank you for the feedback.

I don't use librabbitmq in production, but during my tests, I never lost messages.
Here an example:

A common error during the tests, is to close the client/connections after the last publish.
The publish is asycronous so you should wait a bit before close it.

Btw I really suggest to open an issue with your problem to the librabbitmq repository, and please attach also code you are using for your tests.

[auvipy]

librabbitmq is not recommended for production. we have a plan for improving py-amqp further as it is more feature-rich and less buggy.

[Gsantomaggio]

librabbitmq is not recommended for production.

Yes but I am more interested to understand the performance different

[michaelklishin]

The only fair comparison here is with py-amqp. A couple of profiler sessions will yield relevant data. I don't know what other recommendation there can be.

[Gsantomaggio]

We can close the issue if you want @michaelklishin @lukebakken
It is not important anymore ( at least for me :) )