Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][DO NOT MERGE] Draft implementation of the new PySpark API for support of both Spark Classic and Spark Connect #814

Draft
wants to merge 16 commits into
base: sparkConnect
Choose a base branch
from

Commits on Dec 5, 2023

  1. Update codeql.yml

    sonalgoyal committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    a83a2c3 View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2024

  1. Merge branch '0.4.0'

    sonalgoyal committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    ad9446c View commit details
    Browse the repository at this point in the history
  2. Merge branch '0.4.0'

    sonalgoyal committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    318a1cd View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2024

  1. Make sphinx works

    A temporary solution, based on the following:
    
    - introducing a new env variable ZINGG_DRY_RUN
    - if the variable is set:
      + mimic globally used JVM-stuff
      + otherwise do nothing
    
    ++ slightly update ignore and docs/Makefile
    ++ apply formatting to client.py
    
     On branch 762-fix_sphinx_build
     Changes to be committed:
    	modified:   .gitignore
    	modified:   python/docs/Makefile
    	new file:   python/pyproject.toml
    	modified:   python/zingg/client.py
    SemyonSinchenko committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    7df3540 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Configuration menu
    Copy the full SHA
    e02e25c View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. Initial work on the spark-conect compatible flow

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   python/requirements.txt
    	new file:   python/zingg_v2/__init__.py
    	new file:   python/zingg_v2/client.py
    	new file:   python/zingg_v2/errors.py
    	new file:   python/zingg_v2/structs.py
    SemyonSinchenko committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    b8041c9 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2024

  1. Update

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   python/zingg_v2/structs.py
    SemyonSinchenko committed Apr 6, 2024
    Configuration menu
    Copy the full SHA
    7ab47be View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. Update the new implementation

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	deleted:    python/zingg_v2/client.py
    	new file:   python/zingg_v2/pipes.py
    	modified:   python/zingg_v2/structs.py
    SemyonSinchenko committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    25f6ccd View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. Update the implementation DRAFT

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   python/pyproject.toml
    	new file:   python/zingg_v2/client.py
    	modified:   python/zingg_v2/errors.py
    	new file:   python/zingg_v2/models.py
    	modified:   python/zingg_v2/pipes.py
    	deleted:    python/zingg_v2/structs.py
    SemyonSinchenko committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    d0c0aed View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. Update from comments in PR

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   python/zingg_v2/models.py
    	modified:   python/zingg_v2/pipes.py
    SemyonSinchenko committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    23df1ee View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Batch of changes

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	new file:   buf.gen.yaml
    	new file:   buf.work.yaml
    	new file:   protobuf/connect_plugins.proto
    	modified:   python/zingg_v2/client.py
    	new file:   python/zingg_v2/connect.py
    	modified:   python/zingg_v2/models.py
    	new file:   python/zingg_v2/proto/connect_plugins_pb2.py
    	new file:   python/zingg_v2/proto/connect_plugins_pb2.pyi
    	new file:   python/zingg_v2/proto/connect_plugins_pb2_grpc.py
    SemyonSinchenko committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    71bf65f View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Batch of changes

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   python/zingg_v2/client.py
    SemyonSinchenko committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    ecb06b9 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2024

  1. Batch of changes

     On branch main
     Your branch is ahead of 'origin/main' by 1 commit.
       (use "git push" to publish your local commits)
    
     Changes to be committed:
    	modified:   .gitignore
    	modified:   buf.gen.yaml
    	modified:   common/core/src/main/java/zingg/common/core/executor/LabelUpdater.java
    	modified:   protobuf/connect_plugins.proto
    	modified:   python/pyproject.toml
    	modified:   python/zingg_v2/client.py
    	modified:   python/zingg_v2/models.py
    	modified:   python/zingg_v2/proto/connect_plugins_pb2.py
    	new file:   scripts/get-spark-connect-local.sh
    	new file:   scripts/run-spark-connect-local.sh
    	new file:   spark/client/src/main/java/zingg/spark/connect/ZinggConnectPlugin.java
    	new file:   spark/client/src/main/java/zingg/spark/connect/proto/ConnectPlugins.java
    	new file:   spark/client/src/main/java/zingg/spark/connect/proto/SubmitZinggJob.java
    	new file:   spark/client/src/main/java/zingg/spark/connect/proto/SubmitZinggJobOrBuilder.java
    	modified:   spark/pom.xml
    SemyonSinchenko committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    a035c91 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Semi-working version of

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   buf.gen.yaml
    	new file:   python/test_spark_connect.py
    	modified:   python/zingg_v2/client.py
    	modified:   python/zingg_v2/models.py
    	modified:   scripts/run-spark-connect-local.sh
    	deleted:    spark/client/src/main/java/zingg/spark/connect/ZinggConnectPlugin.java
    	new file:   spark/client/src/main/scala/zingg/spark/connect/ZinggConnectPlugin.scala
    	modified:   spark/pom.xml
    
     Untracked files:
    	spark-3.5.1-bin-hadoop3.tgz
    	spark-3.5.1-bin-hadoop3/
    SemyonSinchenko committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    9b8ec9f View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. Semi-working version of Connect

     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   .gitignore
    	modified:   python/requirements.txt
    	modified:   python/test_spark_connect.py
    	modified:   python/zingg_v2/client.py
    	modified:   python/zingg_v2/errors.py
    	modified:   scripts/run-spark-connect-local.sh
    	modified:   spark/client/pom.xml
    	modified:   spark/client/src/main/scala/zingg/spark/connect/ZinggConnectPlugin.scala
    	modified:   spark/pom.xml
    SemyonSinchenko committed May 1, 2024
    Configuration menu
    Copy the full SHA
    7b4e781 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2024

  1. Switch to 4.0

    + drop scala from spark-client
    + rewrite plugin in java
    + update to scala 2.13 and corresponding fixes
    + small changes
    
     On branch main
     Your branch is up to date with 'origin/main'.
    
     Changes to be committed:
    	modified:   pom.xml
    	modified:   spark/client/pom.xml
    	new file:   spark/client/src/main/java/zingg/spark/connect/ZinggConnectPlugin.java
    	deleted:    spark/client/src/main/scala/zingg/spark/connect/ZinggConnectPlugin.scala
    	modified:   spark/core/src/main/java/zingg/spark/core/block/SparkBlockFunction.java
    	modified:   spark/core/src/test/java/zingg/TestUDFDoubleWrappedArr.java
    SemyonSinchenko committed May 5, 2024
    Configuration menu
    Copy the full SHA
    0ebdc84 View commit details
    Browse the repository at this point in the history