Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sparse masking throws error #10980

Open
ilan-gold opened this issue Mar 6, 2024 · 1 comment
Open

Sparse masking throws error #10980

ilan-gold opened this issue Mar 6, 2024 · 1 comment
Labels
needs triage Needs a response from a contributor

Comments

@ilan-gold
Copy link

ilan-gold commented Mar 6, 2024

Describe the issue:
Computing a masked version of a sparse array throws an error on compute. I am aware than sparse support may not be official but figured I'd report anyway.

Minimal Complete Verifiable Example:

from scipy import sparse
import dask.array as da

X = sparse.random(10_000, 1_000, format="csr", density=0.1)
X_dask = da.from_array(X)
X_dask[X_dask > 0].compute()
X_dask[X_dask > 0] = 1000
X_dask.compute()

Anything else we need to know?:

I am happy to help with this (and other things we are loosely aware of surrounding sparse arrays) if you would accept a PR or investigation at least.

Environment:

  • Dask version: 2024.2.1
  • Python version: 3.9.18
  • Operating System: Mac
  • Install method (conda, pip, source): pip
@github-actions github-actions bot added the needs triage Needs a response from a contributor label Mar 6, 2024
@ilan-gold
Copy link
Author

ilan-gold commented Mar 8, 2024

So the first issue seems to have a host of issues (X_dask[X_dask > 0].compute()) including lack of 1D support in scipy.sparse and also a general incompatibility with dask and np/sp.matrix but the second issue has a more direct cause because there is no np.where equivalent in scipy.sparse. This can be easily reproduced with

np.where(X>0, 10, X)

But this sort of setting does work in general

mask = X>0
X[X>0] = 10
X[mask] # all 10's

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs triage Needs a response from a contributor
Projects
None yet
Development

No branches or pull requests

1 participant