-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
To add random exchange in NLJ and SPF when right child is DAS #1484
Open
zhangyuqin1998
wants to merge
7
commits into
oceanbase:master
Choose a base branch
from
zhangyuqin1998:random_shuffle
base: master
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
48b8129
add random exchange in NLJ and SPF when right child is DAS
91f69a9
fix alloc_by_reference_child_distribution
8b47c33
add comment
24c7d5a
add test case
73efbce
add test case
27cbef0
add test case
ecd7cfe
fix
File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4504,6 +4504,8 @@ int ObLogPlan::compute_join_exchange_info(JoinPath &join_path, | |
} else { | ||
left_exch_info.dist_method_ = ObPQDistributeMethod::PARTITION_HASH; | ||
right_exch_info.dist_method_ = ObPQDistributeMethod::HASH; | ||
// The hash join dfo should rely on the child of the local shuffle side. | ||
right_exch_info.is_related_child_ = true; | ||
left_exch_info.slave_mapping_type_ = sm_type; | ||
right_exch_info.slave_mapping_type_ = sm_type; | ||
} | ||
|
@@ -4535,6 +4537,8 @@ int ObLogPlan::compute_join_exchange_info(JoinPath &join_path, | |
} else { | ||
left_exch_info.dist_method_ = ObPQDistributeMethod::HASH; | ||
right_exch_info.dist_method_ = ObPQDistributeMethod::PARTITION_HASH; | ||
// The hash join dfo should rely on the child of the local shuffle side. | ||
left_exch_info.is_related_child_ = true; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. slave mapping 中也可以用 is_related_child_ 这个标记,表示父 dfo 依赖子 dfo 进行构造 |
||
left_exch_info.slave_mapping_type_ = sm_type; | ||
right_exch_info.slave_mapping_type_ = sm_type; | ||
} | ||
|
@@ -4626,6 +4630,17 @@ int ObLogPlan::compute_join_exchange_info(JoinPath &join_path, | |
if (join_path.right_path_->is_sharding() && !join_path.right_path_->contain_fake_cte()) { | ||
right_exch_info.dist_method_ = ObPQDistributeMethod::LOCAL; | ||
} | ||
} else if (DistAlgo::DIST_NONE_ALL == join_path.join_dist_algo_ | ||
&& JoinAlgo::NESTED_LOOP_JOIN == join_path.join_algo_ | ||
&& join_path.right_path_->is_access_path() | ||
&& get_optimizer_context().get_parallel() > 1) { | ||
/* | ||
When the data volume is small, the GI operator can only divide a small number of granules, resulting in | ||
task skew. We insert a random shuffle operator on GI to shuffle the data and achieve load balancing. We | ||
construct the nlj dfo relied on the left child, to minimize network transfer caused by random shuffling. | ||
*/ | ||
left_exch_info.dist_method_ = ObPQDistributeMethod::RANDOM; | ||
left_exch_info.is_related_child_ = true; | ||
} else if (DistAlgo::DIST_NONE_ALL == join_path.join_dist_algo_ || | ||
DistAlgo::DIST_ALL_NONE == join_path.join_dist_algo_) { | ||
// do nothing | ||
|
@@ -9116,7 +9131,29 @@ int ObLogPlan::create_subplan_filter_plan(ObLogicalOperator *&top, | |
} | ||
if (OB_SUCC(ret)) { | ||
ObExchangeInfo exch_info; | ||
if (DistAlgo::DIST_BASIC_METHOD == dist_algo || | ||
if (DistAlgo::DIST_NONE_ALL == dist_algo && get_optimizer_context().get_parallel() > 1) { | ||
/* | ||
When the data volume is small, the GI operator can only divide a small number of granules, resulting in | ||
task skew. We insert a random shuffle operator on GI to shuffle the data and achieve load balancing. We | ||
construct the spf dfo relied on the left child, to minimize network transfer caused by random shuffling. | ||
*/ | ||
exch_info.dist_method_ = ObPQDistributeMethod::RANDOM; | ||
exch_info.is_related_child_ = true; | ||
if (OB_FAIL(allocate_exchange_as_top(top, exch_info))) { | ||
LOG_WARN("failed to allocate exchange as top"); | ||
} else if (OB_FAIL(allocate_subplan_filter_as_top(top, | ||
subquery_ops, | ||
query_ref_exprs, | ||
params, | ||
onetime_exprs, | ||
initplan_idxs, | ||
onetime_idxs, | ||
filters, | ||
dist_algo, | ||
is_update_set))) { | ||
LOG_WARN("failed to allocate subplan filter as top", K(ret)); | ||
} | ||
} else if (DistAlgo::DIST_BASIC_METHOD == dist_algo || | ||
DistAlgo::DIST_PARTITION_WISE == dist_algo || | ||
DistAlgo::DIST_NONE_ALL == dist_algo) { | ||
// is basic or is_partition_wise | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里之前是个bug。父dfo依赖子dfo进行构造时,没有生成p2p id。这会导致slave mapping和join filter一起使用时报4016。在这里生成p2p id之后可以解决这个bug