How to handle addition/deletion of streams for super streams?

[aeb-dev]

Is it possible to add or delete a stream from a super stream while the producer is running?

I made a simple test:
I have one producer that sends to super.stream.test super stream

super.stream.test has 3 partitions as s1, s2, s3.

The system is running and I want to add one more partition(s4) to the super.stream.test. How can I handle this?

[Gsantomaggio]

You can add or remove stream to the super stream using AMQP by adding the right bindings:

The client is not aware of this change at the moment the only way is to restart the client.
Super stream configuration is usually static.

We could introduce some refresh_topology on demand on time-based, but I am not totally sure about that

[aeb-dev]

I think in terms of flexibility/ergonomics having hard coded partitions does not help with scalability. If I can not scale then it means I am not using my full potential. Also not having such feature also creates other problems for maintenance, deployment.

Let me give an example then:

Assume that the business domain requires messages to be delivered to the services with low latency. Imagine that you have 3 services consuming from super streams. At one point in time, the number packages send to super stream increased and your services can not keep up(lagging behind the stream) which introduces latency. In that case you would like to scale the service but the situation does not let you.

[Gsantomaggio]

If I can not scale

Sorry @aeb-dev, but I didn't say that is not possible to scale the super-stream. I said that it is possible using AMQP bindings.

hard coded partition

We don't have hard-coded partitions, The client reads the partitions when it starts.

At the moment, if you add or remove partitions, you need to restart the client.

We will evaluate a way to handle this kind of situation.

[aeb-dev]

Sorry I could not explained myself clearly. What I meant by scale was scaling on the fly. What I meant by hard coded was requiring a restart. So we are on the same page.

If you need any further test or information do not hesitate to ask

[Gsantomaggio]

Thank you @aeb-dev

We will think about that.

[Gsantomaggio]

We can expose an API the user can call externally to check the topology changes and re-create the super stream consumers/producers

[aeb-dev]

How will the user understand when to call this API?

[Gsantomaggio]

No way at the moment. Maybe using a timer or based on some event in your system

[aeb-dev]

As a user, can I have a separate amqp connection to listen changes on exchange? Does amqp send such events and can I listen to those events with amqp client?

[aeb-dev]

Can this be used for listening on changes on stream and call new API accordingly?