[SPARK-36438][PYTHON] Support list-like Python objects for Series comparison #34114

itholic · 2021-09-27T10:39:35Z

What changes were proposed in this pull request?

This PR proposes to implement Series comparison with list-like Python objects.

Currently Series doesn't support the comparison to list-like Python objects such as list, tuple, dict, set.

Before

>>> psser
0    1
1    2
2    3
dtype: int64

>>> psser == [3, 2, 1]
Traceback (most recent call last):
...
TypeError: The operation can not be applied to list.
...

After

>>> psser
0    1
1    2
2    3
dtype: int64

>>> psser == [3, 2, 1]
0    False
1     True
2    False
dtype: bool

This was originally proposed in databricks/koalas#2022, and all reviews in origin PR has been resolved.

Why are the changes needed?

To follow pandas' behavior.

Does this PR introduce any user-facing change?

Yes, the Series comparison with list-like Python objects now possible.

How was this patch tested?

Unittests

itholic · 2021-09-27T10:47:02Z

cc @ueshin @HyukjinKwon @xinrong-databricks

python/pyspark/pandas/base.py

SparkQA · 2021-09-27T11:10:53Z

Test build #143644 has finished for PR 34114 at commit d930f89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-27T11:38:04Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48157/

SparkQA · 2021-09-27T12:42:57Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48157/

SparkQA · 2021-09-29T04:55:36Z

Test build #143700 has finished for PR 34114 at commit cbe62da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

python/pyspark/pandas/base.py

SparkQA · 2021-09-29T05:10:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48215/

python/pyspark/pandas/tests/test_ops_on_diff_frames.py

SparkQA · 2021-09-29T05:56:18Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48215/

SparkQA · 2021-09-30T07:54:27Z

Test build #143760 has finished for PR 34114 at commit fb555b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-30T08:36:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48271/

SparkQA · 2021-09-30T09:25:01Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48271/

SparkQA · 2021-10-01T04:51:43Z

Test build #143780 has finished for PR 34114 at commit 5199684.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

python/pyspark/pandas/data_type_ops/base.py

SparkQA · 2021-10-01T05:08:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48292/

python/pyspark/pandas/data_type_ops/base.py

SparkQA · 2021-10-01T06:08:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48292/

SparkQA · 2021-10-01T06:55:37Z

Test build #143784 has finished for PR 34114 at commit 5c9b168.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-01T07:07:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48296/

SparkQA · 2021-10-01T07:53:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48296/

SparkQA · 2021-10-05T06:17:53Z

Test build #143836 has finished for PR 34114 at commit 57b3c73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-05T06:35:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48349/

SparkQA · 2021-10-05T07:37:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48349/

SparkQA · 2021-10-05T12:14:14Z

Test build #143851 has finished for PR 34114 at commit 15a6c20.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-05T12:32:28Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48364/

SparkQA · 2021-10-05T13:30:08Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48364/

HyukjinKwon

The change looks fine. cc @ueshin

SparkQA · 2021-10-06T03:21:18Z

Test build #143867 has finished for PR 34114 at commit 92753a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-06T03:34:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48380/

SparkQA · 2021-10-06T04:32:30Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48380/

HyukjinKwon · 2021-10-13T06:25:29Z

Merged to master.

…6438

SparkQA · 2021-10-13T07:03:30Z

Test build #144189 has finished for PR 34114 at commit cf9cc33.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class NettyLogger
class IndexNameTypeHolder(object):
new_class = type(NameTypeHolder.short_name, (NameTypeHolder,),
new_class = param.type if isinstance(param, np.dtype) else param
class Database(NamedTuple):
class Table(NamedTuple):
class Column(NamedTuple):
class Function(NamedTuple):
protected class YarnSchedulerEndpoint(override val rpcEnv: RpcEnv)
public final class TableIndex
public final class AlwaysFalse extends Filter
public final class AlwaysTrue extends Filter
public final class And extends BinaryFilter
abstract class BinaryComparison extends Filter
abstract class BinaryFilter extends Filter
public final class EqualNullSafe extends BinaryComparison
public final class EqualTo extends BinaryComparison
public abstract class Filter implements Expression, Serializable
public final class GreaterThan extends BinaryComparison
public final class GreaterThanOrEqual extends BinaryComparison
public final class In extends Filter
public final class IsNotNull extends Filter
public final class IsNull extends Filter
public final class LessThan extends BinaryComparison
public final class LessThanOrEqual extends BinaryComparison
public final class Not extends Filter
public final class Or extends BinaryFilter
public final class StringContains extends StringPredicate
public final class StringEndsWith extends StringPredicate
abstract class StringPredicate extends Filter
public final class StringStartsWith extends StringPredicate
public class ColumnarBatch implements AutoCloseable
public final class ColumnarBatchRow extends InternalRow
class IndexAlreadyExistsException(message: String, cause: Option[Throwable] = None)
class NoSuchIndexException(message: String, cause: Option[Throwable] = None)
case class Sec(child: Expression)
case class Csc(child: Expression)
trait OperationHelper extends AliasHelper with PredicateHelper
case class AsOfJoin(
case class SetCatalogAndNamespace(child: LogicalPlan) extends UnaryCommand
case class CreateTempFunction(
case class CreateFunction(
class SQLOpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](
case class OptimizeSkewedJoin(
case class SkewJoinChildWrapper(plan: SparkPlan) extends LeafExecNode
case class SimpleCostEvaluator(forceOptimizeSkewedJoin: Boolean) extends CostEvaluator
case class SetCatalogCommand(catalogName: String) extends LeafRunnableCommand
case class SetNamespaceCommand(namespace: Seq[String]) extends LeafRunnableCommand
case class ShowCurrentNamespaceCommand() extends LeafRunnableCommand
case class WriterBucketSpec(
case class EnsureRequirements(

SparkQA · 2021-10-13T07:34:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48668/

SparkQA · 2021-10-13T08:24:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48668/

HyukjinKwon · 2021-10-13T09:31:22Z

Merged to master.

[SPARK-36438] Support list-like Python objects for Series comparison

d930f89

github-actions bot added CORE PYTHON labels Sep 27, 2021

HyukjinKwon reviewed Sep 27, 2021

View reviewed changes

python/pyspark/pandas/base.py Outdated Show resolved Hide resolved

Resolved comments & test failure

cbe62da

HyukjinKwon reviewed Sep 29, 2021

View reviewed changes

python/pyspark/pandas/base.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Sep 29, 2021

View reviewed changes

python/pyspark/pandas/tests/test_ops_on_diff_frames.py Outdated Show resolved Hide resolved

Use collect_list and zip_with, explode

fb555b4

itholic added 2 commits October 1, 2021 13:12

move codes to proper place

e890e81

rollback meaningless clange

5199684

HyukjinKwon reviewed Oct 1, 2021

View reviewed changes

python/pyspark/pandas/data_type_ops/base.py Outdated Show resolved Hide resolved

Handling None

5c9b168

HyukjinKwon reviewed Oct 1, 2021

View reviewed changes

python/pyspark/pandas/data_type_ops/base.py Outdated Show resolved Hide resolved

Comapre the length

57b3c73

fix nullability

15a6c20

fix test failure

92753a2

HyukjinKwon approved these changes Oct 6, 2021

View reviewed changes

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

cf9cc33

…6438

HyukjinKwon closed this in 46bcef7 Oct 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-36438][PYTHON] Support list-like Python objects for Series comparison #34114

[SPARK-36438][PYTHON] Support list-like Python objects for Series comparison #34114

itholic commented Sep 27, 2021 •

edited

itholic commented Sep 27, 2021

SparkQA commented Sep 27, 2021

SparkQA commented Sep 27, 2021

SparkQA commented Sep 27, 2021

SparkQA commented Sep 29, 2021

SparkQA commented Sep 29, 2021

SparkQA commented Sep 29, 2021

SparkQA commented Sep 30, 2021

SparkQA commented Sep 30, 2021

SparkQA commented Sep 30, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

HyukjinKwon left a comment

SparkQA commented Oct 6, 2021

SparkQA commented Oct 6, 2021

SparkQA commented Oct 6, 2021

HyukjinKwon commented Oct 13, 2021

SparkQA commented Oct 13, 2021

SparkQA commented Oct 13, 2021

SparkQA commented Oct 13, 2021

HyukjinKwon commented Oct 13, 2021

[SPARK-36438][PYTHON] Support list-like Python objects for Series comparison #34114

[SPARK-36438][PYTHON] Support list-like Python objects for Series comparison #34114

Conversation

itholic commented Sep 27, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

itholic commented Sep 27, 2021

SparkQA commented Sep 27, 2021

SparkQA commented Sep 27, 2021

SparkQA commented Sep 27, 2021

SparkQA commented Sep 29, 2021

SparkQA commented Sep 29, 2021

SparkQA commented Sep 29, 2021

SparkQA commented Sep 30, 2021

SparkQA commented Sep 30, 2021

SparkQA commented Sep 30, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 1, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

SparkQA commented Oct 5, 2021

HyukjinKwon left a comment

Choose a reason for hiding this comment

SparkQA commented Oct 6, 2021

SparkQA commented Oct 6, 2021

SparkQA commented Oct 6, 2021

HyukjinKwon commented Oct 13, 2021

SparkQA commented Oct 13, 2021

SparkQA commented Oct 13, 2021

SparkQA commented Oct 13, 2021

HyukjinKwon commented Oct 13, 2021

itholic commented Sep 27, 2021 •

edited