Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix span-metrics' subprocessors bug #3612

Merged
merged 7 commits into from May 6, 2024
Merged

Conversation

mapno
Copy link
Member

@mapno mapno commented Apr 26, 2024

What this PR does:
Fixes a bug in which a shared map in the metrics-generator override config was not properly copied to avoid modifying the config of other tenants.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@mapno mapno changed the title Fix span-metrics' subprocessors bug that applied wrong configs when r… Fix span-metrics' subprocessors Apr 26, 2024
@mapno mapno changed the title Fix span-metrics' subprocessors Fix span-metrics' subprocessors bug Apr 26, 2024
@joe-elliott
Copy link
Member

How difficult is it to construct a test that reproes and then confirm this fixes it?

@mapno
Copy link
Member Author

mapno commented Apr 26, 2024

How difficult is it to construct a test that reproes and then confirm this fixes it?

I think it's feasible. I'll build one 👍

Copy link
Member

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it make more sense to break the shared memory here:

https://github.com/grafana/tempo/blob/main/modules/generator/config.go#L66

where we copy the config?

CHANGELOG.md Outdated Show resolved Hide resolved
Co-authored-by: Joe Elliott <joe.elliott@grafana.com>
@mapno
Copy link
Member Author

mapno commented Apr 30, 2024

does it make more sense to break the shared memory here:

https://github.com/grafana/tempo/blob/main/modules/generator/config.go#L66

where we copy the config?

Yes, but we only need the copy if there are changes to the subprocessors. What you suggests is more consistent, but also creates more allocations 🤷
I'm ok with doing either. If we changed it I don't think the number of allocations would be a major problem.

@joe-elliott
Copy link
Member

Yes, but we only need the copy if there are changes to the subprocessors. What you suggests is more consistent, but also creates more allocations 🤷

This is just a few more allocations every couple minutes when it refreshes user overrides? That should be fine. I'd definitely lean towards code clarity over performance in this case.

@mapno
Copy link
Member Author

mapno commented Apr 30, 2024

Changed 👍

Copy link
Member

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a simple test failure, but i'm good on this approach

@mapno mapno enabled auto-merge (squash) May 6, 2024 08:04
@mapno mapno disabled auto-merge May 6, 2024 08:12
@mapno mapno merged commit 37f3308 into grafana:main May 6, 2024
14 checks passed
@mapno mapno deleted the subprocessors-map-bug branch May 6, 2024 08:39
joe-elliott added a commit to joe-elliott/tempo that referenced this pull request May 7, 2024
* Fix span-metrics' subprocessors bug that applied wrong configs when running multiple tenants

* Add test

* Fix race condition

* Update CHANGELOG.md

Co-authored-by: Joe Elliott <joe.elliott@grafana.com>

* Copy when copying the config

* Fix bug

---------

Co-authored-by: Joe Elliott <joe.elliott@grafana.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

2 participants