Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve reloader loop pre-startup fluentd #276

Open
Cryptophobia opened this issue Oct 21, 2021 · 1 comment
Open

Improve reloader loop pre-startup fluentd #276

Cryptophobia opened this issue Oct 21, 2021 · 1 comment
Assignees

Comments

@Cryptophobia
Copy link
Contributor

Cryptophobia commented Oct 21, 2021

Reported by @alex-vmw:

There needs to be some different logic for first start of the reloader versus a standard control loop once it is already running. Logically, you can probably set some firstStart=true variable and then set it to false when first startup loop finished.

a) time="2021-10-14T05:42:05Z" level=info msg="Sleeping for 30 seconds in order for fluentd to be ready." - Reloader should NOT wait for 30 seconds (it shouldn't wait at all) for fluentd to come up. During first boot, fluentd can NOT come up without reloader providing it the fluentd.conf file, so reloader should immediatelly get to work and NOT wait.

b) time="2021-10-14T05:46:41Z" level=error msg="cannot notify fluentd: Post \"http://127.0.0.1:24444/api/config.gracefulReload\": dial tcp 127.0.0.1:24444: connect: connection refused" - Reloader control loop should NOT try to issue a fluentd reload during first startup because it is NOT required and will always fail.

Update on 1b:
I was replacing the reloader image just now and was thinking. I think if we remove the http://127.0.0.1:24444/api/config.gracefulReload call during the first boot there will be an issue if the reloader was restarted due to some issue. If reloader was restarted due to some issue (OOM, etc.) it would in fact need to reload fluentd in case any configs have changed during the time reloader was down.

@Cryptophobia Cryptophobia self-assigned this Oct 21, 2021
@alex-vmw
Copy link

b) time="2021-10-14T05:46:41Z" level=error msg="cannot notify fluentd: Post \"http://127.0.0.1:24444/api/config.gracefulReload\": dial tcp 127.0.0.1:24444: connect: connection refused" - Reloader control loop should NOT try to issue a fluentd reload during first startup because it is NOT required and will always fail.

I now think that if we remove the http://127.0.0.1:24444/api/config.gracefulReload call during the first boot there will be an issue if the reloader was restarted due to some issue. If reloader was restarted due to some issue (OOM, etc.) it would in fact need to reload fluentd in case any configs have changed during the time reloader was down. It appears that we better keep that call even if it will fail 99% of the time. The other 1% of the time it would actually avoid an issue with updating the config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants