Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure that repack collections only return tuple if necessary #11004

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

fjetter
Copy link
Member

@fjetter fjetter commented Mar 14, 2024

I hit this more often than I'd care to admit when trying to write the dask-expr scheduler integration. This integration inevitably has to touch the code that is dealing with collections parsing and this is part of it.

What's most concerning is that this unpack/repack code is not used consequently such that the unpack/repack in distributed, for instance, is behaving differently than the dask/dask equivalents.

Most endusers will notice this change iff they are using base functions optimize, persist, compute, etc. Most users will not interact with this but instead with the methods on the collections.

# before will always return a tuple no matter what
(d_opt,) = dask.optimize(d)

# With this PR
d_opt = dask.optimize(d)

This will help me streamline implementation a little and I'm sure that new users will appreciate this. However, I'm very certain this will cause mild breakage somewhere.

Copy link
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

     15 files  ±0       15 suites  ±0   3h 19m 15s ⏱️ + 1m 17s
 13 105 tests ±0   12 082 ✅  -  77     930 💤 ± 0     93 ❌ +   77 
162 239 runs  ±0  141 046 ✅  - 941  20 144 💤  - 92  1 049 ❌ +1 033 

For more details on these failures, see this check.

Results for commit 43fc1d7. ± Comparison against base commit 5949e54.

@mrocklin
Copy link
Member

FWIW this seems off to me. I've been out of it for long enough though that I don't feel qualified to push back strongly.

@fjetter fjetter marked this pull request as draft March 14, 2024 18:11
@fjetter
Copy link
Member Author

fjetter commented Mar 14, 2024

FWIW this seems off to me. I've been out of it for long enough though that I don't feel qualified to push back strongly.

I'm not pushing on this, yet. The current behavior is confusing to me but I'm not sure yet if this change improves anything

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants