Unknown output plugin 'elasticsearch'

[vasanthaneppali]

i am trying to copy a volume and not able to make the container up

/usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/config/types.rb:96: warning: regular expression has ']' without escape
2022-05-07 16:38:16 +0530 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins"

command
docker run -itd --user root --restart always -v fluentd-volume:/fluentd/etc --name fluentd -p "24225:24225" fluent/fluentd:v1.14-debian-1

[rkorzeniewski]

I think a default fluentd image does not include elasticsearch plugin and you have to add it to a custom image.

[junejie]

@vasanthaneppali
sample here

FROM fluent/fluentd:v1.6-debian-1
USER root

RUN apt update -y
RUN apt install -y ubuntu-dev-tools

RUN gem install elasticsearch -v 7.17.1 --no-document
RUN gem install fluent-plugin-elasticsearch -v 5.0.3 --no-document
RUN gem install fluent-plugin-prometheus -v 2.0.1 --no-document

USER fluent

[artheus]

@junejie This actually will not work properly, as the plugin gems will not be readable/executable by the fluent user in the correct way. I just tried this exact way, and this resulted in errors like this:

<internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- /usr/local/bundle/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions (LoadError)
        from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:103:in `block in search'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:100:in `each'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:100:in `search'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:44:in `lookup'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/plugin.rb:169:in `new_impl'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/plugin.rb:114:in `new_output'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:130:in `add_match'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:74:in `block in configure'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:64:in `each'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:64:in `configure'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/root_agent.rb:149:in `configure'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/engine.rb:105:in `configure'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/engine.rb:80:in `run_configure'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/supervisor.rb:708:in `run_supervisor'
        from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/command/fluentd.rb:366:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/local/bundle/gems/fluentd-1.15.1/bin/fluentd:15:in `<top (required)>'
        from /usr/local/bundle/bin/fluentd:25:in `load'
        from /usr/local/bundle/bin/fluentd:25:in `<main>'

EDIT: Reason being the read permissions of the gem files:

fluent@07e77483ed31:/usr/local/bundle/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin$ ls -lia
total 32
3849008 drwxr-xr-x 2 root root  4096 Aug 24 15:33 .
3849007 drwxr-xr-x 3 root root  4096 Aug 24 15:33 ..
3849010 -rw-r----- 1 root root 12916 Aug 24 15:33 exception_detector.rb
3849011 -rw-r----- 1 root root  4614 Aug 24 15:33 out_detect_exceptions.rb

EDIT 2: My suggestion is to either add documentation that the files mode/ownership also needs to be updated. Or that --user-install should be supported (if it is not already)

[kenhys]

It may depend on your case, daemonset image may be useful for you. (they provides image for elasticsearch 7)
https://github.com/fluent/fluentd-kubernetes-daemonset