[FEA] [SPARK-46743][SQL] Count bug after constant folding #10762
Labels
audit_4.0.0
Audit related tasks for 4.0.0
feature request
New feature or request
Spark 4.0+
Spark 4.0+ issues
apache/spark@b2f6474848f
Spark introduced a fix for Count bug handling in subqueries. Spark will prevent constant folding in subqueries that contain a COUNT-bug succeptible aggregate. There is a new configuration flag
spark.sql.optimizer.decorrelateSubqueryPreventConstantHoldingForCountBug.enabled
that is defaulted totrue
. We should ensure that our behavior is consistent with the fix that is now the default behavior.We could fallback to GPU when the flag is set to false, but that logic is complicated since it's only COUNT in subqueries.
The text was updated successfully, but these errors were encountered: