Support history

[tomchristie]

Something that'd be really valuable would be to allow subscriptions to start up with some initial history.

For example:

• Subscribe to "logs", with the last 1000 records, so that a front-end can display a log console, that's populated with the latest set of logs.
• Subscribe to "http-stats" with the last 24 hours of records, so that a front-end can display a realtime status monitoring graph, that's populated with a window of the latest history.

Not all backends will support this, but Redis Streams, and Apache Kafka do. I think the implementation may be a bit fiddly to get right, but before we even get there let's discuss the API.

One option here might be...

async with broadcaster.subscribe("logs", history=1000):
    ...

async with broadcaster.subscribe("http-stats", history=timedelta(hours=24)):
    ...

• Are there any API alternatives we ought to be considering?
• Would Kafka and Redis Streams properly support both the int and the timedelta styles, and what commands would we need to be mapping them onto?

[woile]

kafka has offsets so it should be possible, also for time it has offsets_for_times which returns the offset inside that timestamp. It's a bit tricky because of the kafka partitions, but I'll investigate further.

And redis stream have xrange but not sure if they support datetime.

Another thing to think about is how to retrieve everything. Let's say you have a topic where the content expires daily "notifications", so you want to read it all, how would it look like?