Start reading logs from current time

[amrmotan]

I have an issue where the plugin starts pulling logs from the start time of creating the log sink/topic, and it takes hours and sometimes days before reaching the logs of the current time, which makes it not reliable. The current time was like 11 am and the plugin started reading logs from 4pm two days ago.

To overcome that I had to create a new sink and start reading logs from it, but it is not reliable every time I want to read from an existing sink to start from the first day where the sink was create, also it is a bad idea to create a new sink everytime we want to configure a new pubsub channel to get real-time logs.

It would be awesome if we can have a parameter to tell pub/sub plugin to start reading logs starting from current time and ignore old logs that are already in the sink

[josephlewis42]

The best solution to this is probably to delete and re-create sinks as necessary if you don't want the durability Pub/Sub provides. Right now Pub/Sub can't flush or start at a specific point (see this StackOverflow post by one of the Pub/Sub developers) so the other option is for the Logstash plugin pull down and ignore all the results up to that point in time.

I'd definitely recommend checking out the Pub/Sub architecture doc if you're interested in the nitty-gritty details and why this is a hard problem; it's a fun read. :-)

[amrmotan]

Thanks @josephlewis42 for the reply. I will check the document.