Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] [SPARK-46743][SQL] Count bug after constant folding #10762

Open
NVnavkumar opened this issue May 2, 2024 · 1 comment
Open

[FEA] [SPARK-46743][SQL] Count bug after constant folding #10762

NVnavkumar opened this issue May 2, 2024 · 1 comment
Labels
audit_4.0.0 Audit related tasks for 4.0.0 feature request New feature or request Spark 4.0+ Spark 4.0+ issues

Comments

@NVnavkumar
Copy link
Collaborator

apache/spark@b2f6474848f

Spark introduced a fix for Count bug handling in subqueries. Spark will prevent constant folding in subqueries that contain a COUNT-bug succeptible aggregate. There is a new configuration flag spark.sql.optimizer.decorrelateSubqueryPreventConstantHoldingForCountBug.enabled that is defaulted to true. We should ensure that our behavior is consistent with the fix that is now the default behavior.

We could fallback to GPU when the flag is set to false, but that logic is complicated since it's only COUNT in subqueries.

@NVnavkumar NVnavkumar added feature request New feature or request ? - Needs Triage Need team to review and classify audit_4.0.0 Audit related tasks for 4.0.0 Spark 4.0+ Spark 4.0+ issues labels May 2, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label May 7, 2024
@mattahrens
Copy link
Collaborator

Priority is to investigate that we are not falling back with this new default state (true).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit_4.0.0 Audit related tasks for 4.0.0 feature request New feature or request Spark 4.0+ Spark 4.0+ issues
Projects
None yet
Development

No branches or pull requests

2 participants