Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Add pyspark null functions #11940

Merged
merged 2 commits into from May 15, 2024

Conversation

mariotaddeucci
Copy link
Contributor

@mariotaddeucci mariotaddeucci commented May 5, 2024

Adding some functions to work with nulls.

I tried to add the coalesce function directly and use it on nvl and ifnull but I got the error saying the coalesce function is not a scalar function.

This PR could be converted to a draft and then adding a issue and just merge when the coalesce be added to python relational api.

@Tishj
Copy link
Contributor

Tishj commented May 5, 2024

Thanks!
Feel free to make a PR extending our native python expression API as well when necessary.
Taking #11941 as an example shows how easy these additions can be :)

@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 7, 2024 20:41
@Tishj
Copy link
Contributor

Tishj commented May 8, 2024

Please merge with main and address the colesce todo, the linked PR has been merged

Thanks for your efforts to extend the support of the spark API!

@hannes hannes requested a review from Tishj May 8, 2024 07:15
@mariotaddeucci mariotaddeucci force-pushed the feature-pyspark-functions-nulls branch from 9c35594 to 8998e1d Compare May 8, 2024 12:13
@mariotaddeucci mariotaddeucci marked this pull request as draft May 8, 2024 12:14
@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 8, 2024 12:14
@mariotaddeucci mariotaddeucci marked this pull request as draft May 8, 2024 12:18
@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 8, 2024 12:19
@mariotaddeucci mariotaddeucci marked this pull request as draft May 8, 2024 19:18
@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 8, 2024 19:19
@mariotaddeucci
Copy link
Contributor Author

Hey @Tishj , I synced the branch but still getting the same error, now I don’t know what to do.

@Tishj
Copy link
Contributor

Tishj commented May 12, 2024

Please look at the PR, it's not a function expression, it has its own method

@duckdb-draftbot duckdb-draftbot marked this pull request as draft May 14, 2024 11:41
@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 14, 2024 11:41
@duckdb-draftbot duckdb-draftbot marked this pull request as draft May 14, 2024 12:33
@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 14, 2024 12:49
@mariotaddeucci mariotaddeucci force-pushed the feature-pyspark-functions-nulls branch from ef505bf to 2dd7a18 Compare May 14, 2024 13:19
@duckdb-draftbot duckdb-draftbot marked this pull request as draft May 14, 2024 13:19
@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 14, 2024 13:20
@mariotaddeucci
Copy link
Contributor Author

Got it, now it worked.
@Tishj thanks for all the support o/

Copy link
Contributor

@Tishj Tishj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, in the future please try to not make unnecessary changes, like the formatting ;)

@Mytherin
Copy link
Collaborator

Could you just have a look at the merge conflicts?

@duckdb-draftbot duckdb-draftbot marked this pull request as draft May 15, 2024 15:12
@mariotaddeucci mariotaddeucci marked this pull request as ready for review May 15, 2024 15:12
@mariotaddeucci
Copy link
Contributor Author

@Mytherin done o/

@Mytherin Mytherin merged commit 5b0058f into duckdb:main May 15, 2024
17 checks passed
@Mytherin
Copy link
Collaborator

Thanks!

@mariotaddeucci mariotaddeucci deleted the feature-pyspark-functions-nulls branch May 15, 2024 20:42
github-actions bot pushed a commit to duckdb/duckdb-r that referenced this pull request May 16, 2024
Merge pull request duckdb/duckdb#12083 from carlopi/fix_checkpoint
Merge pull request duckdb/duckdb#11940 from mariotaddeucci/feature-pyspark-functions-nulls
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants