Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

partial list of columns in spark_read_x produces unexpected behaviors #3421

Open
ahcyip opened this issue Feb 8, 2024 · 0 comments
Open

Comments

@ahcyip
Copy link

ahcyip commented Feb 8, 2024

A partial named columns list in spark_read_x seems to rewrite the names of the first x columns up to the length of the partial list and thereby likely refer to the wrong columns. So it seems like a partial columns list would not be useful and could be misleading / produce unexpected results for most users.

Suggestion: sparklyr should disable this partial list option (error out) or warn that not all columns are included or the columns you thought you were selecting and specifying type for are not the actual ones you are reading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants