Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shutting down tempo always takes at least 30 seconds #2353

Closed
bouk opened this issue Apr 20, 2023 · 5 comments · Fixed by #3687
Closed

Shutting down tempo always takes at least 30 seconds #2353

bouk opened this issue Apr 20, 2023 · 5 comments · Fixed by #3687
Labels
good first issue Good for newcomers keepalive Label to exempt Issues / PRs from stale workflow operations

Comments

@bouk
Copy link

bouk commented Apr 20, 2023

Describe the bug

Shutting down the tempo service takes 30 seconds. This is annoying when rebooting because it holds up the reboot. I assume it's because of this code in the distributor that just sleeps for 30 seconds

To Reproduce
Steps to reproduce the behavior:

  1. Run tempo
  2. SIGTERM tempo
  3. It takes 30 seconds

Expected behavior

It should shut down when all requests have been properly processed

Environment:

I'm running tempo in a systemd service on NixOS, the configuration for it can be found here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/tracing/tempo/default.nix

tempo, version 2.0.0 (branch: <release>, revision: 2.0.0)
  build user:
  build date:
  go version:       go1.19.5
  platform:         linux/amd64
@joe-elliott
Copy link
Member

joe-elliott commented Apr 24, 2023 •
edited

Oof, that's no good. Ideally we'd review the comment and determine if that sleep is even necessary anymore. For an easier win, I'm fine with making that configurable. 30s is a LONG time to wait for requests to finish processing.

Good find.

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

@github-actions github-actions bot added the stale Used for stale issues / PRs label Jun 24, 2023
@joe-elliott joe-elliott added keepalive Label to exempt Issues / PRs from stale workflow good first issue Good for newcomers and removed stale Used for stale issues / PRs labels Jun 26, 2023
@Joffref Joffref mentioned this issue Jul 2, 2023
Closed
3 tasks
@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

@github-actions github-actions bot added the stale Used for stale issues / PRs label Oct 12, 2023
@bouk
Copy link
Author

bouk commented Oct 12, 2023

Still a problem I believe

@github-actions github-actions bot removed the stale Used for stale issues / PRs label Oct 13, 2023
@litetex
Copy link

litetex commented May 16, 2024 •
edited

Just stumble upon this and yes this problem is still present in the latest version

The default timeout also conflicts with docker compose (if you deploy it that way) default stop_grace_period of 10s which always causes a SIGKILL to be performed...

Would be great if someone could give #2604 another go, so we can at least manage the hardcoded value

@chodges15 chodges15 mentioned this issue May 18, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers keepalive Label to exempt Issues / PRs from stale workflow operations
Projects
None yet
Milestone
No milestone
3 participants
@bouk @joe-elliott @litetex