Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-36435][PYTHON] Implement MultIndex.equal_levels #34113

Closed
wants to merge 7 commits into from
Closed

[SPARK-36435][PYTHON] Implement MultIndex.equal_levels #34113

wants to merge 7 commits into from

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Sep 27, 2021

What changes were proposed in this pull request?

This PR proposes implementing MultiIndex.equal_levels.

>>> psmidx1 = ps.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])
>>> psmidx2 = ps.MultiIndex.from_tuples([("b", "y"), ("a", "x"), ("c", "z")])
>>> psmidx1.equal_levels(psmidx2)
True

>>> psmidx1 = ps.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z"), ("a", "y")])
>>> psmidx2 = ps.MultiIndex.from_tuples([("a", "y"), ("b", "x"), ("c", "z"), ("c", "x")])
>>> psmidx1.equal_levels(psmidx2)
True

This was originally proposed in databricks/koalas#1789, and all reviews in origin PR has been resolved.

Why are the changes needed?

We should support the pandas API as much as possible for pandas-on-Spark module.

Does this PR introduce any user-facing change?

Yes, the MultiIndex.equal_levels API is available.

How was this patch tested?

Unittests

@SparkQA
Copy link

SparkQA commented Sep 27, 2021

Test build #143642 has finished for PR 34113 at commit 0b79e4a.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • public class NettyLogger
  • class IndexNameTypeHolder(object):
  • new_class = type(\"NameType\", (NameTypeHolder,),
  • new_class = param.type if isinstance(param, np.dtype) else param
  • public final class AlwaysFalse extends Filter
  • public final class AlwaysTrue extends Filter
  • public final class And extends BinaryFilter
  • abstract class BinaryComparison extends Filter
  • abstract class BinaryFilter extends Filter
  • public final class EqualNullSafe extends BinaryComparison
  • public final class EqualTo extends BinaryComparison
  • public abstract class Filter implements Expression, Serializable
  • public final class GreaterThan extends BinaryComparison
  • public final class GreaterThanOrEqual extends BinaryComparison
  • public final class In extends Filter
  • public final class IsNotNull extends Filter
  • public final class IsNull extends Filter
  • public final class LessThan extends BinaryComparison
  • public final class LessThanOrEqual extends BinaryComparison
  • public final class Not extends Filter
  • public final class Or extends BinaryFilter
  • public final class StringContains extends StringPredicate
  • public final class StringEndsWith extends StringPredicate
  • abstract class StringPredicate extends Filter
  • public final class StringStartsWith extends StringPredicate
  • public class ColumnarBatch implements AutoCloseable
  • case class Sec(child: Expression)
  • case class Csc(child: Expression)
  • trait OperationHelper extends AliasHelper with PredicateHelper
  • class SQLOpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](
  • case class OptimizeSkewedJoin(
  • case class SkewJoinChildWrapper(plan: SparkPlan) extends LeafExecNode
  • case class SimpleCostEvaluator(forceOptimizeSkewedJoin: Boolean) extends CostEvaluator
  • case class WriterBucketSpec(
  • case class EnsureRequirements(

@SparkQA
Copy link

SparkQA commented Sep 27, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48154/

@SparkQA
Copy link

SparkQA commented Sep 27, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48154/

@itholic
Copy link
Contributor Author

itholic commented Sep 27, 2021

cc @ueshin @HyukjinKwon @xinrong-databricks

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Test build #143702 has finished for PR 34113 at commit 6306fb1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Test build #143704 has finished for PR 34113 at commit 1f51541.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48218/

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Test build #143706 has finished for PR 34113 at commit f19e365.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48220/

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48218/

@SparkQA
Copy link

SparkQA commented Sep 29, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48220/

@SparkQA
Copy link

SparkQA commented Oct 1, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48290/

@SparkQA
Copy link

SparkQA commented Oct 1, 2021

Test build #143778 has finished for PR 34113 at commit 188f9e7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 1, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48290/

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants