Add Spark 3.5.0 support #436
Conversation
Codecov Report
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. @@ Coverage Diff @@
## master #436 +/- ##
=======================================
Coverage 91.30% 91.30%
=======================================
Files 18 18
Lines 828 828
Branches 55 55
=======================================
Hits 756 756
Misses 72 72 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
@WeichenXu123 PySpark 3.5.0 has been released, graphframes is ready to move to Spark 3.5.0. |
|
@EnricoMi Thanks! I just released graphframe 0.8.3 for spark 3.4 yesterday. Would you help adding CI for spark 3.5 ? then I will release a new version for spark 3.5 |
Sure, what else is needed other than this PR? |
Nothing else is needed. CI is for confirming graphframe is compatible with spark 3.5. |
| .setIntermediateStorageLevel(storageLevel) | ||
| .run() | ||
| assertComponents(components, expected) | ||
| // disabling adaptive query execution helps assertComponents |
There was a problem hiding this comment.
The need for this change here looks like a correctness bug in Spark 3.5.0. I created a PR with a attempted fix here: apache/spark#43188
Waiting for Spark 3.5.0 release...