Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to GZIP messages on queue #559

Open
khepin opened this issue Jul 17, 2023 · 1 comment
Open

Option to GZIP messages on queue #559

khepin opened this issue Jul 17, 2023 · 1 comment

Comments

@khepin
Copy link
Collaborator

khepin commented Jul 17, 2023

@adm-bome @M-Porter @mattgrande

Would like your opinion on adding this as an option.

When doing more intensive usage of RabbitMQ, memory management on the broker can become an issue. At work, we've had good success extending the classes in this library to gzip the message bodies on the queues. This can in some cases have a very dramatic and beneficial effect on the cluster.

I think more apps could benefit from this, and this library could (should?) offer an easy option to turn this on.

A previous PR I had made for this (#447) enabled it as a connection level setting.

However at work, we've done this on a per-job basis instead.

That PR also only allowed a single compression algorithm (gzip) whereas if I made this to merge in this lib now, I would prefer we use a configurable compression algorithm.

Before committing more work to this though, I'd like to gather your thoughts on whether yes or no this should be part of this library.

@adm-bome
Copy link
Collaborator

adm-bome commented Jul 18, 2023

Sounds fine by me. I would like to see some proposal in a branch.

As I understand this now:
Zip the payload of a AMQPMessage, when creating and Unzip when retrieved, right?

Some shout-outs

'connections' => [
    // ...

    'rabbitmq' => [
        // ...

        'options' => [
            'compressor' => 'gzip'  // or a Compressor::class
        ],
    ],

    // ...    
],

When compressor is omitted, the default should be 'false' in QueueConfig and should be to not compress or decompress. (disabled by default)

When compressor is set anything other then false, for example to gzip, the correct compression should be applied.

Optional: (like to see this)
Maybe create a abstract CompressorClass with compress/decompress methods/interface, and a compressorFactory, so alternate compressors can be added by config.

Implement a:

  • NullCompressor::class, as a default in the QueueConfig (won't compress)
  • GzipCompressor:class, (chosen in the factory when 'gzip' is set into the connections::rabbitmq::options:compressor config.)
  • ...
  • ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants