out_file:Inconsistent path settings and symlink_path settings.

[Shingo-Nakayama]

Describe the bug

If tag is set in the chunk key, We will get an error unless we specify the tag in the path setting of the out_file plugin.
However, there is no such check in the symlink_path setting. (i.e., no error occurs even if no tag is specified.)
Also, this check is a setting to make the pass unique, but if we use a part of tag such as tag[0] instead of tag, no error will occur.

Is this an intended spec? Or is there a check omitted?

To Reproduce

For example, use the following input.

<source>
  @type sample
  sample {"message":"hoge"}
  tag test.hoge
</source>

<source>
  @type sample
  sample {"message":"fuga"}
  tag test.fuga
</source>

At this time, setting the out_file plugin as follows will result in an error.

<match test.**>
  @type file
  path /tmp/test
 symlink_path /tmp/test/link/${tag}
  append true

  <buffer time,tag>
    @type file
    path /tmp/test/buffer
    timekey 60 
    timekey_wait 3
  </buffer>
</match>

 $ bundle exec fluentd -c /Users/fluentd/configuration.txt
2024-04-26 10:20:51 +0900 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-04-26 10:20:51 +0900 [info]: parsing config file is succeeded path="/Users/fluentd/configuration.txt"
2024-04-26 10:20:51 +0900 [info]: gem 'fluentd' version '1.16.5'
2024-04-26 10:20:51 +0900 [error]: config error file="/Users/fluentd/configuration.txt" error_class=Fluent::ConfigError error="Parameter 'path: /tmp/test/test.%Y%m%d%H%M.log' doesn't have tag placeholder"

However, if the out_file plugin is set up as follows, there is no error and two outputs compete for a single path.

<match test.**>
  @type file
  path /tmp/test/${tag[0]}
 symlink_path /tmp/test/link/${tag}
  append true

  <buffer time,tag>
    @type file
    path /tmp/test/buffer/tag
    timekey 60 
    timekey_wait 3
  </buffer>
</match>

/tmp/test $ ls -R
buffer			test.202405091047.log	test.202405091049.log
link			test.202405091048.log

./buffer:
tag

./buffer/tag:
buffer.b617fba30cfcdcb64c554db644d9ef7c5.log
buffer.b617fba30cfcdcb64c554db644d9ef7c5.log.meta
buffer.b617fba30dd0f05a6f9a793e93f657172.log
buffer.b617fba30dd0f05a6f9a793e93f657172.log.meta

./link:
test		test.fuga	test.hoge
/tmp/test $ tail test.202405091048.log
2024-05-09T10:48:00+09:00	test.fuga	{"message":"fuga"}
2024-05-09T10:48:01+09:00	test.fuga	{"message":"fuga"}
2024-05-09T10:48:02+09:00	test.fuga	{"message":"fuga"}
...
2024-05-09T10:48:59+09:00	test.fuga	{"message":"fuga"}
2024-05-09T10:48:00+09:00	test.hoge	{"message":"hoge"}
2024-05-09T10:48:01+09:00	test.hoge	{"message":"hoge"}
...
2024-05-09T10:48:59+09:00	test.hoge	{"message":"hoge"}

Furthermore, if the out_file plugin is set up as follows, there will be no error and the two will compete for a single symlink_path.

<match test.**>
  @type file
  path /tmp/test/${tag[0]}
 symlink_path /tmp/test/link
  append true

  <buffer time,tag>
    @type file
    path /tmp/test/buffer
    timekey 60
    timekey_wait 3
  </buffer>
</match>

$ ls -R
buffer  link

./buffer:
buffer.b616f84e16b5e9575a8a61ca0cbf34536.log        buffer.b616f84e16b968e52f8d0c2bc37bdb9d9.log
buffer.b616f84e16b5e9575a8a61ca0cbf34536.log.meta   buffer.b616f84e16b968e52f8d0c2bc37bdb9d9.log.meta

./link:
test
/tmp/test $ tail link/test
2024-04-26T13:26:39+09:00   test.fuga   {"message":"fuga"}
2024-04-26T13:26:40+09:00   test.fuga   {"message":"fuga"}
2024-04-26T13:26:41+09:00   test.fuga   {"message":"fuga"}
...
/tmp/test $ tail link/test
2024-04-26T13:27:00+09:00   test.hoge   {"message":"hoge"}
2024-04-26T13:27:01+09:00   test.hoge   {"message":"hoge"}
2024-04-26T13:27:02+09:00   test.hoge   {"message":"hoge"}
...

Expected behavior

In either case, an error should occur. (Or this is the intended specification.)

Your Environment

- Fluentd version:1.17.0
- Operating system:
ProductName:		macOS
ProductVersion:		13.0
BuildVersion:		22A380
- Kernel version:22.1.0

Your Configuration

See To Reproduce section.

Your Error Log

See To Reproduce section.

Additional context

No response

[daipom]

Thanks! There are 2 issues:

1. path setting validation issue.
   • It seems that validation for tag placeholder is not enough.
2. symlink_path setting validation issue.
   • We can completely ignore the chunk keys and set the static path for symlink.
   • It causes multiple chunks to compete for the same symlink, and some logs can't be taken from it.

symlink_path seems to be intentionally unvalidated.

• Add in current logfile symlink #83
• out_file: Support placeholders in symlink_path parameter #2254

However, some logs can't be taken from it if the placeholders are not enough.
We should do something about it.

[Shingo-Nakayama]

I put out a PR about the symlink_path issue.
#4502