feat(query): add TenantIngestionMetering query throttle #1310
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
alextheimer
force-pushed
the
card-throttle
branch
2 times, most recently
from
aea4659
to
ca17e67
Compare
alextheimer
changed the title
alextheimer
force-pushed
the
card-throttle
branch
from
ca17e67
to
f76b59c
Compare
alextheimer
changed the title
alextheimer
force-pushed
the
card-throttle
branch
from
f76b59c
to
4dd70ae
Compare
alextheimer
commented
Dec 15, 2021
coordinator/src/main/scala/filodb.coordinator/TenantIngestionMetering.scala
Outdated
Show resolved
Hide resolved
alextheimer
changed the title
alextheimer
force-pushed
the
card-throttle
branch
from
49abfae
to
07cd6cd
Compare
alextheimer
force-pushed
the
card-throttle
branch
from
07cd6cd
to
072a254
Compare
alextheimer
force-pushed
the
card-throttle
branch
from
072a254
to
6ad22b4
Compare
alextheimer
commented
Dec 25, 2021
| settings, | ||
| () => { _datasetInfo.map{ case (dsRef, _) => dsRef}.toIterator }, | ||
| () => { _coordinators.head._2 }) | ||
| inst.schedulePeriodicPublishJob() |
There was a problem hiding this comment.
No real reason to require this method to be called after initialization.
alextheimer
commented
Dec 25, 2021
coordinator/src/main/scala/filodb.coordinator/TenantIngestionMetering.scala
Outdated
Show resolved
Hide resolved
alextheimer
commented
Dec 25, 2021
Comment on lines
37
to
43
| private val SCHED_INIT_DELAY = ASK_TIMEOUT // time until first job is scheduled | ||
| private val SCHED_DELAY = ASK_TIMEOUT // time between all jobs after the first | ||
|
|
||
| private val CLUSTER_TYPE = settings.config.getString("cluster-type") | ||
|
|
||
| private val METRIC_ACTIVE = "active_timeseries_by_tenant" | ||
| private val METRIC_TOTAL = "total_timeseries_by_tenant" |
There was a problem hiding this comment.
Moved to companion object.
alextheimer
changed the title
alextheimer
force-pushed
the
card-throttle
branch
from
f1bfaed
to
6cb0c28
Compare
alextheimer
changed the title
alextheimer
commented
Jan 4, 2022
| object TenantIngestionMetering { | ||
| protected val METRIC_ACTIVE = "active_timeseries_by_tenant" | ||
| protected val METRIC_TOTAL = "total_timeseries_by_tenant" | ||
| protected val PARALLELISM = 8 // number of datasets queried in parallel |
There was a problem hiding this comment.
Unsure what this should be-- should I make it configurable? Or get the number of cores from the JRE?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request checklist
Current behavior :
TenantIngestionMeteringqueries might consistently take longer than the static timeout duration. No timeseries would be published.New behavior :
Adds a throttling subclass to monitor query timeouts. This class adjusts the query frequency and timeout duration according to the count of timeouts in a lookback window.