getCompactionTaskCapacity is not worker category aware. #15847
Labels
Comments
|
My overlord dynamic config looks like this, overlord dynamic config{
"type": "equalDistributionWithCategorySpec",
"workerCategorySpec": {
"categoryMap": {
"kill": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"compact": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"single_phase_sub_task": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_dimension_cardinality": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_index_generate": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_index_generic_merge": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_range_index_generate": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_dimension_distribution": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"index_kafka": {
"defaultCategory": "xxx_category",
"categoryAffinity": {
"datasource_2": "yyy_category",
"datasource_3": "zzz_category"
}
}
},
"strong": true
}
} |
|
We worked around this by explicitly setting the See Update capacity for compaction tasks Example, |
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
The getCompactionTaskCapacity function which is used to ascertain that the Druid cluster has enough task slots before the coordinator schedules additional compaction tasks doesn't take into consideration overlord dynamic config. The overlord dynamic config can prevent compaction tasks from running on specific categories of workers. This way, the compaction task capacity is incorrectly overestimated.
For example, I have two worker categories,
compaction-categorywith a total of600task slots and anotheringestion-categorywith2000slots(high number because of multiple ingestion task replicas).Using the overlord dynamic config,
compaction-categoryis configured to run the following task types,ingestion-categoryis configured to run,Now, getCompactionTaskCapacity would return
2600as the total capacity, which is inaccurate since only600slots are actually available for compaction tasks. While this might not pose a problem in a healthy cluster, it becomes critical during compaction task failures. The oversight leads to the coordinator creating excessive compaction tasks, resulting in contention on compaction slots and slowing down all compaction tasks. This creates a feedback loop where the increasing number of compaction tasks exacerbates contention, ultimately overwhelming the overlord with too many tasks to handleAffected Version
Saw this on Druid 25. Is also present in master.
The text was updated successfully, but these errors were encountered: