Do we want this behavior? I'm thinking that if you setup a consumer that says x-stream-offset=10 and it start consumes to offset=20 if I restart that consumer then I would expect it to continue from 21, not 10, even though the x-stream-offset says 10.
Otherwise I have to specify x-stream-offset first time and then somehow remove the argument if the consumer is restarted and should continue. WDYT?

It's a bit tricky IMO. But in general I think that if you specify x-stream-offset when consuming, that should take precedent over other things.

Let's say you get messages 10-20, and then decide you want to get them again for whatever reason (or jump to another part of the stream), there would be no way for you to do that if we use our tracking.

An alternative could be to use an extra argument (x-stream-auto-offset or something), where we think of x-stream-offset as a starting point, and then use our internal tracking for any subsequent requests.

How this works in Kafka is that offsets are stored based on consumer group name, so if you want to start over using a new offset you need to rename the group. Changing offset when using a group that has already a stored offset on the broker has no effect. They do have an API call into kafka to reset a consumer group offset so you can tell a running consumer to start reading from somewhere else, but that is just a side note.

For rabbitmq Streams it looks like the offset it stored based on consumer name (same feature where you can group consumers on) so if order to "reset" the offset and use another value you need to rename the consumer. Do we have a name of something for a consumer that we can use in this case?

For rabbitmq Streams it looks like the offset it stored based on consumer name (same feature where you can group consumers on) so if order to "reset" the offset and use another value you need to rename the consumer. Do we have a name of something for a consumer that we can use in this case?

We use the consumer tag, so you can always "start over" by just setting a new tag.

Okay, then if we can reset by changing consumer tag I think the broker should use stored offsets if there are any and if not use the one given by the consumer using the argument x-stream-offset

So use stored offset if exists for consumer tags, otherwise use the offset given by consumer

Hmm, I'm not sure I agree - I think that if you provide x-stream-offset you explicitly say what you want. That would also change existing behavior where you always get what you request when providing an x-stream-offset. But let's discuss more during tomorrows meeting!

@snichme added x-stream-use-automatic-offset so we can have it both ways :)

yay, nice!

RabbitMQ says that if x-stream-offset isn't specified it uses "next" as detault:

If it is unspecified the consumer will start reading from the next offset written to the log after the consumer starts

Oh, yeah that's right. I've noticed that as well but forgot to do anything about it 🙈

I guess it's been like that since we released streams though - should we change our behavior to match, or should we note that as something that we do differently?

Well, that quote is actually from the stream client, as rabbit doesn't support this for the amqp clients.
So based on that we can do what we want. So if we have one way it works now, leave it at that for now might be good.

yay, nice!