Provide a way to implement conditional routing of events to specific outputs

[ppf2]

Would be nice for beats to provide a feature to allow conditional routing of events to outputs, eg. type A log events to be routed to Logstash A instance, type B log events to be routed to Logstash B instance. Perhaps we can extend the generic filtering capability to provide this type of conditional routing. Currently, to implement this, the user will have to configure multiple beats instances, or send the events to a Logstash instance and perform the routing from there.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mbarretta]

Still a common ask, reopening for PM visibility

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[jlind23]

ping @nimarezainia is this something you have in your radar?

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[ruflin]

I think with the v2 architecture of Elastic Agent his will become possible. Each input can select its output. I would not directly call it conditional routing but if you have 2 logfile inputs, each could use its own output.

[nimarezainia]

@ruflin @cmacknz,
Running thistrain of thought past you: on the standalone agent for an input one could configure "use_output" to satisfy this requirement. of course we currently support Elasticsearch and Logstash only. But the user has the option of defining multiple Logstash instance (as an example) and then reference these in the input section of the configuration file.

To address our Beats users who need this feature _ I would say migration to standalone is a viable option.

However we do want to provide this functionality for the fleet managed agents also, where the output is configured at the datastream level (discussion point? i believe it can't be at the integration level. Users would need more granularity logs vs metrics as an example). Perhaps short term, the first step here would be to apply the config in an advanced yaml box with guidence on how it should be done.

Proper solution imo is to have a UI configurable under each datastream to choose the output for that datastream. I'd be curious to hear if for this we need the V2 shipper to complete?

[ruflin]

Proper solution imo is to have a UI configurable under each datastream to choose the output for that datastream. I'd be curious to hear if for this we need the V2 shipper to complete?

I personally think this is overkill. I would rather set it on the level of the integration. If a user wants to ship logs and metrics to two different outputs, the integration has to be configured twice, once with logs and once with metrics enabled for example.

[fholzer]

of course we currently support Elasticsearch and Logstash only.
Not sure i follow.

I have a use case where we have different types logs and modules on a single server. Ideally I'd like to configure the module and logfile A to be shipped to elasticsearch, with the module using the resp. ingest pipeline. While having log file B shipped to Kafka, where a logstash cluster read from for further processing and enrichment of the data.

While it's absolutely doable to setup 2 or even 3 filebeat instances, from a maintenance overhead POV it just seems like overkill. The configuration and automation effort is just really high. The filebeat RPM comes with all the right directories, config files, and the systemd unit. To setup an additional beat, we'd have to change all of this substantially. Having dedicates state directory (/var/lib), config directory (/etc/filebeat - which, on every upgrade we'd probably have to manually check and compare files to the previous version), the systemd unit file we'd also need to duplicate or change the existing one to handle instance (%i).

Generally speaking, it's just really not convenient to run multiple instances of filebeat. (Which, to be fair, isn't a problem specific to filebeat, but a general "problem" for software that comes pre-packaged for running a single instance of that software, when you want/need to run multiple instances.)

Maybe an alternative to supporting multiple outputs in filebeat could be to support the use case where you have one or more regular inputs shipped to a non-elasticsearch output, while also having module enabled which ship to elasticsearch. While technically this would be a 2-output setup, it's actually only the modules which use the ES output.

Or, simply making it easier to run a multi-instance setup with the pre-built RPM provided by elastic. (Having an instance based systemd unit file, maybe also scripts that prepare necessary directories and default config files on first run, etc.)

Curious to hear you thoughts on this.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[zez3]

This is still relevant.
My use case is a bit different described here: https://alexmarquardt.com/2021/03/15/driving-filebeat-data-into-separate-indices-uses-legacy-index-templates/
But I get some permission issues when I try to route to a different datastream

[zez3]

My use case would be foe reroute processor in filebeat
https://www.elastic.co/guide/en/elasticsearch/reference/master/reroute-processor.html

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[zez3]

Still relevant