To describe the issue a bit differently, this output plugin is using its own buffering mechanism that may delay (a lot) sending some log records, waiting for BatchSize size to be reached.

At the moment, this plugin will send a SQS message batch when, and only when, there are BatchSize records to batch together. Which means if fluentbit flushes its memory buffer to send the corresponding in-flight records and if the records count isn't an exact multiple of BatchSize, there will be pending records in the plugin memory. When fluentbit will flush new records, those pending records will be part of the first batch sent (if any).

There are several concerns here :

• if fluentbit stops or crashes while there are pending records in the plugin memory, they'll be lost
• if the source is emitting logs very slowly, some logs may be sent very late; plugin could even wait indefinitely before sending them if BatchSize is never reached
• fluentbit has many parameters to tweak buffering and backpressure, and the plugin logic should follow the same principles without introducing new ones (this is obviously largely opinionated 😄)

Doing some experiments around this issue, I came up with some code updates. PR is coming.