Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document havester_limit for Filestream input and fix typo #39244

Merged
merged 1 commit into from Apr 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
24 changes: 24 additions & 0 deletions filebeat/docs/inputs/input-filestream-file-options.asciidoc
Expand Up @@ -517,6 +517,30 @@ less than or equal to `prospector.scanner.check_interval`
If `backoff.max` needs to be higher, it is recommended to close the file handler
instead and let {beatname_uc} pick up the file again.

[float]
[id="{beatname_lc}-input-{type}-harvester-limit"]
===== `harvester_limit`

The `harvester_limit` option limits the number of harvesters that are started in
parallel for one input. This directly relates to the maximum number of file
handlers that are opened. The default for `harvester_limit` is 0, which means
there is no limit. This configuration is useful if the number of files to be
harvested exceeds the open file handler limit of the operating system.

Setting a limit on the number of harvesters means that potentially not all files
are opened in parallel. Therefore we recommended that you use this option in
combination with the `close.on_state_change.*` options to make sure
harvesters are stopped more often so that new files can be picked up.

Currently if a new harvester can be started again, the harvester is picked
randomly. This means it's possible that the harvester for a file that was just
closed and then updated again might be started instead of the harvester for a
file that hasn't been harvested for a longer period of time.

This configuration option applies per input. You can use this option to
indirectly set higher priorities on certain inputs by assigning a higher
limit of harvesters.

[float]
===== `file_identity`

Expand Down
5 changes: 3 additions & 2 deletions filebeat/docs/inputs/input-filestream.asciidoc
Expand Up @@ -11,8 +11,9 @@ Use the `filestream` input to read lines from active log files. It is the
new, improved alternative to the `log` input. It comes with various improvements
to the existing input:

1. Checking of `close_*` options happens out of band. Thus, if an output is blocked,
{beatname_uc} can close the reader and avoid keeping too many files open.
1. Checking of `close.on_state_change.*` options happens out of
band. Thus, if an output is blocked, {beatname_uc} can close the
reader and avoid keeping too many files open.

2. Detailed metrics are available for all files that match the `paths` configuration
regardless of the `harvester_limit`. This way, you can keep track of all files,
Expand Down