Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speed up check_rabbitmq_objects: reduce amount of metadata requested #98

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

kjetilho
Copy link

@kjetilho kjetilho commented Sep 2, 2019

Parsing big JSON documents can take a long time, and we only want
the number of objects. "name" is a valid attribute for all
endpoints except bindings, so we ask for "vhost" instead for that case.

On our OpenStack using RabbitMQ, CPU time for this check was reduced
from 52 seconds to 2 seconds.

Parsing big JSON documents can take a long time, and we only want
the number of objects.  "name" is a valid attribute for all
endpoints except bindings, so we ask for "vhost" instead for that case.

On our OpenStack using RabbitMQ, CPU time for this check was reduced
from 52 seconds to 2 seconds.
@kjetilho kjetilho changed the title check_rabbitmq_objects: reduce amount of metadata requested speed up check_rabbitmq_objects: reduce amount of metadata requested Sep 2, 2019
@kjetilho
Copy link
Author

kjetilho commented Sep 2, 2019

Here are two runs with added instrumentation - the first is the original code. Notice how most time is spent after body was fetched ("is done" is emitted at end of for-loop).

19:58:53  fetch https://rmq:15671/api/vhosts
19:58:53  body length: 971
19:58:53  vhost is done
19:58:53  fetch https://rmq:15671/api/exchanges
19:58:53  body length: 247777
19:58:54  exchange is done
19:58:54  fetch https://rmq:15671/api/bindings
19:58:54  body length: 1790778
19:58:56  binding is done
19:58:56  fetch https://rmq:15671/api/queues
19:58:58  body length: 4778028
19:59:34  queue is done
19:59:34  fetch https://rmq:15671/api/channels
19:59:35  body length: 2931379
19:59:48  channel is done
RABBITMQ_OBJECTS OK - Gathered Object Counts | vhost=0;; exchange=530;; binding=6176;; queue=3074;; channel=2732;;

And with restricted columns:

20:00:03  fetch https://rmq:15671/api/vhosts?columns=name
20:00:03  body length: 14
20:00:03  vhost is done
20:00:03  fetch https://rmq:15671/api/exchanges?columns=name
20:00:03  body length: 25733
20:00:03  exchange is done
20:00:03  fetch https://rmq:15671/api/bindings?columns=vhost
20:00:03  body length: 86479
20:00:04  binding is done
20:00:04  fetch https://rmq:15671/api/queues?columns=name
20:00:04  body length: 215866
20:00:05  queue is done
20:00:05  fetch https://rmq:15671/api/channels?columns=name
20:00:05  body length: 162218
20:00:05  channel is done
RABBITMQ_OBJECTS OK - Gathered Object Counts | vhost=0;; exchange=530;; binding=6176;; queue=3074;; channel=2731;;

@kjetilho
Copy link
Author

kjetilho commented Dec 4, 2019

I wonder if the Travis CI has been fixed and I could retry the job?

I see from the logs that the previous successful test was done in 2017 on a worker running Ubuntu Trusty with RabbitMQ Version 3.6.14

The run above was done on an Ubuntu Xenial worker without RabbitMQ installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant