Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question regarding anomalous RabbitMQ data #67

Closed
febrezo opened this issue Oct 10, 2022 · 0 comments
Closed

Question regarding anomalous RabbitMQ data #67

febrezo opened this issue Oct 10, 2022 · 0 comments
Labels
inactivity question Further information is requested

Comments

@febrezo
Copy link

febrezo commented Oct 10, 2022

I'm experiencing an issue in our OpenCTI deployment in which the data volume of the RabbitMQ service is enormous:

109G	./data/amqpdata/mnesia/rabbit@rabbit_node_1/msg_stores/vhosts/<LONG_HEX_NUMBER>/queues/<OTHER_LONG_HEX_NUMBER>
109G	./data/amqpdata/mnesia/rabbit@rabbit_node_1/msg_stores/vhosts/<LONG_HEX_NUMBER>/queues
17G	./data/amqpdata/mnesia/rabbit@rabbit_node_1/msg_stores/vhosts/<LONG_HEX_NUMBER>/msg_store_persistent
125G	./data/amqpdata/mnesia/rabbit@rabbit_node_1/msg_stores/vhosts/<LONG_HEX_NUMBER>

The configuration of the docker-compose.yml says the following for the service:

  rabbitmq:
    image: rabbitmq:3.10-management
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
    volumes:
      - amqpdata:/var/lib/rabbitmq
    restart: always
    hostname: rabbit_node_1 # Line added to avoid having different identifiers for the node that are kept and no longer used.

The addition of the hostname line did not solve the problem since data in that volume keeps increasing under the data folder for data node.

I guess that my issue is probably not related with OpenCTI but with tasks that are being queued and kept pending (or something similar) but at the same time I'm not experimenting losses of data since things seem (I highlight, seem) to being ingested normally and enrichments are performed properly using VT, AbuseIPDB and others.

Do you have any idea of how to deal with this space issues? Do you know if there is something that I can add to limit the space and discard, for example, old tasks or something? Since I'm not an expert on RabbitMQ and my background with managing these services has been limited to the official docs and generic documents, my only temporal workaround to keep the internal service alive (don't laugh) has been to periodically freeing the data manually to let the platform be alive but, as you can imagine, I'm not comfortable at all since I'm sure that by doing so I'm forcing the deletion of queued tasks and, probably, streaming issues (which, as I'm not using them, I'm not experimenting).

@SamuelHassine SamuelHassine added the question Further information is requested label Aug 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
inactivity question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants