Skip to content

Commit

Permalink
[SPARK-48027][SQL][FOLLOWUP] Add comments for the other code branch
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
This PR propose to add comments for the other code branch.

### Why are the changes needed?
#46263 missing the comments for the other code branch.

### Does this PR introduce _any_ user-facing change?
'No'.

### How was this patch tested?
N/A

### Was this patch authored or co-authored using generative AI tooling?
'No'.

Closes #46536 from beliefer/SPARK-48027_followup.

Authored-by: beliefer <beliefer@163.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
beliefer authored and cloud-fan committed May 14, 2024
1 parent e7a1efd commit 0ea8088
Showing 1 changed file with 12 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -123,21 +123,20 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with PredicateHelper with J
case ExtractEquiJoinKeys(joinType, lkeys, rkeys, _, _, left, right, _) =>
// Runtime filters use one side of the [[Join]] to build a set of join key values and prune
// the other side of the [[Join]]. It's also OK to use a superset of the join key values
// (ignore null values) to do the pruning.
// (ignore null values) to do the pruning. We can also extract from the other side if the
// join keys are transitive, and the other side always produces a superset output of join
// key values. Any join side always produce a superset output of its corresponding
// join keys, but for transitive join keys we need to check the join type.
// We assume other rules have already pushed predicates through join if possible.
// So the predicate references won't pass on anymore.
if (left.output.exists(_.semanticEquals(targetKey))) {
extract(left, AttributeSet.empty, hasHitFilter = false, hasHitSelectiveFilter = false,
currentPlan = left, targetKey = targetKey).orElse {
// We can also extract from the right side if the join keys are transitive, and
// the right side always produces a superset output of join left keys.
// Let's look at an example
// An example that extract from the right side if the join keys are transitive.
// left table: 1, 2, 3
// right table, 3, 4
// left outer join output: (1, null), (2, null), (3, 3)
// left key output: 1, 2, 3
// Any join side always produce a superset output of its corresponding
// join keys, but for transitive join keys we need to check the join type.
// right outer join output: (3, 3), (null, 4)
// right key output: 3, 4
if (canPruneLeft(joinType)) {
lkeys.zip(rkeys).find(_._1.semanticEquals(targetKey)).map(_._2)
.flatMap { newTargetKey =>
Expand All @@ -152,7 +151,11 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with PredicateHelper with J
} else if (right.output.exists(_.semanticEquals(targetKey))) {
extract(right, AttributeSet.empty, hasHitFilter = false, hasHitSelectiveFilter = false,
currentPlan = right, targetKey = targetKey).orElse {
// We can also extract from the left side if the join keys are transitive.
// An example that extract from the left side if the join keys are transitive.
// left table: 1, 2, 3
// right table, 3, 4
// left outer join output: (1, null), (2, null), (3, 3)
// left key output: 1, 2, 3
if (canPruneRight(joinType)) {
rkeys.zip(lkeys).find(_._1.semanticEquals(targetKey)).map(_._2)
.flatMap { newTargetKey =>
Expand Down

0 comments on commit 0ea8088

Please sign in to comment.