Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow control the order of Output Plugin #1115

Open
sugaf1204 opened this issue Apr 8, 2024 · 2 comments
Open

Allow control the order of Output Plugin #1115

sugaf1204 opened this issue Apr 8, 2024 · 2 comments
Labels
help wanted Extra attention is needed

Comments

@sugaf1204
Copy link

sugaf1204 commented Apr 8, 2024

Is your feature request related to a problem? Please describe.

In the Fluentd, I can't completely control the order of output plugin.

The order of plugins cannot be controlled between them that have different values for the tag field of output plugin.
For example, when using rewrite_tag_filter, the problem arises because we cannot control the order of customPlugin that matches the tag before rewritten and s3 plugin that matches it after rewritten.

apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: example
spec: 
  outputs: 
  - tag: "k8s.log.*"
     customPlugin:
       config: |
         <match k8s.log.*>
           @type rewrite_tag_filter
           <rule>
             key $.type
             pattern ^(.+)$
             tag $1.log
           </rule>
         </match>
  - tag: "A.log"
     s3:
     ......
  - tag: "B.log" 
     forward:
     ......
  - tag: "*.log" 
     elasticsearch:
     ......
  - tag: "k8s.**" # catch all
     stdout:
     ......

The order in which these are written out to config is undifined.

Describe the solution you'd like

In the current implementation, within a group of plugins with the same tag field value, the order of them can probably be controlled by metadata.name of CR.
(grouping by tag field value: https://github.com/fluent/fluent-operator/blob/v2.7.0/apis/fluentd/v1alpha1/helper.go#L296)
(sorted by metadata.name: https://github.com/fluent/fluent-operator/blob/v2.7.0/apis/fluentd/v1alpha1/helper.go#L124)

I can solve this problem by writing everything in single customPlugin, but it is not good because it raises the possibility of managing a very huge single config in the future.
I want to control the order of all plugins according to the order of the outputs array defined in one CR, not just the metadata.name.

Additional context

No response

@benjaminhuo
Copy link
Member

@sugaf1204 #1106 pls take a look if this PR from @antrema fixed the problem you mentioned

@benjaminhuo benjaminhuo added the help wanted Extra attention is needed label Apr 9, 2024
@sugaf1204
Copy link
Author

sugaf1204 commented Apr 9, 2024

Sorry, I just misunderstood the behavior of rewrite_tag_filter.

I don't know how to determine the order of the rewrite_tag_filter plugin and the plugins that catch the tags that did not match any of the others.

Is there way to resolve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants