Dynamic Producer/Consumer Creation

[praveengopalan]

Hi team,
We are handling different events in our system, and we creats separate stream to handle each event. Only at runtime we decides (based on some configuration) which producer can send the message. So currently we creating producer at the time of message publish and closing after that, is it a good approch. Is there any way we can reuse already created producer so that we can avoid create/close producer for each message.

[michaelklishin]

Modern messaging protocols delegate topology management to applications for exactly this reason. However, creating a new stream "per event" sounds excessive. But as long as the stream churn is low and there is a reasonable number of them (since each stream is a stateful entity consumes a certain amount of resources on every node it has a replica on), there's nothing wrong with that.

Creating a new publisher, a new connection, a new channel (in AMQP 0-9-1) per message is very much NOT how messaging or streaming protocols are meant to be used:

• Connections
• Channels

Opening a new connection is multiple round trips, opening a new channel is a single round trip and adding a stream publisher is a single round trip. Doing so for every message is guaranteed to have drastic negative effects on throughput and completely throw out the window all the benefits that messaging and streaming tools have to offer.