sparse + dask array not supported

[shoyer]

cc @hameerabbasi

dask+sparse works, but not sparse+dask.

I think dask should probably implement this case, via it's __array_ufunc__ method. Here's the traceback:

In [1]: import sparse

In [2]: import dask.array as da

In [3]: import numpy as np

In [4]: x = np.arange(3)

In [5]: y = sparse.COO(x)

In [6]: z = da.from_array(y, chunks=3)

In [7]: z + y
Out[7]: dask.array<add, shape=(3,), dtype=int64, chunksize=(3,)>

In [8]: y + z
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-ff96b8185f8d> in <module>
----> 1 y + z

~/miniconda3/envs/xarray-py37-dev/lib/python3.7/site-packages/numpy/lib/mixins.py in func(self, other)
     23         if _disables_array_ufunc(other):
     24             return NotImplemented
---> 25         return ufunc(self, other)
     26     func.__name__ = '__{}__'.format(name)
     27     return func

TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'add'>, '__call__', <COO: shape=(3,), dtype=int64, nnz=2, fill_value=0>, dask.array<array, shape=(3,), dtype=int64, chunksize=(3,)>): 'COO', 'Array'

[mrocklin]

What do you think would be the right approach here? Call da.from_array on all non-dask arrays?

Currently we have this code, which seems problematic:

dask/dask/array/core.py

Lines 1106 to 1110 in 6f38240

	def __array_ufunc__(self, numpy_ufunc, method, *inputs, **kwargs):
	out = kwargs.get("out", ())
	for x in inputs + out:
	if not isinstance(x, (np.ndarray, Number, Array)):
	return NotImplemented

I wonder if just removing that check would make things work

[hameerabbasi]

I think probably adding a hasattr(..., '__array_ufunc__') would help here.

[shoyer]

Xarary defines __array_ufunc__, so that wouldn't be enough. Dask needs to defer to xarray, but handle sparse arrays itself. I don't know if there's a good way to do that without an explicit list of valid or invalid "array chunk types".

…

On Mon, Aug 12, 2019 at 6:48 AM Hameer Abbasi ***@***.***> wrote: I think probably adding a `hasattr(..., '*array_ufunc*') would help here. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5259?email_source=notifications&email_token=AAJJFVVTDWXY2MOJU2XHYITQEFSZTA5CNFSM4IK5QBXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CSSZA#issuecomment-520431972>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJJFVWRCWNCCUNN5YV7VHLQEFSZTANCNFSM4IK5QBXA> .

[hameerabbasi]

Maybe there needs to be a way to declare something as “ndarray-compatible”, this has come up before.

[shoyer]

Maybe there needs to be a way to declare something as “ndarray-compatible”, this has come up before.

Right, but remember that other "ndarray-compatible" arrays might need to wrap dask arrays. For example, if someone wrote a duck type compatible version of autograd.

[jakirkham]

What do you think would be the right approach here? Call da.from_array on all non-dask arrays?

We call da.asarray in other places to handle non-Dask Arrays. Seems reasonable to do it here too.

[shoyer]

We call da.asarray in other places to handle non-Dask Arrays. Seems reasonable to do it here too.

yes, though with overriding numpy ufuncs we still have to be careful to explicitly skip types like xarray.DataArray

[hameerabbasi]

Right, but remember that other "ndarray-compatible" arrays might need to wrap dask arrays. For example, if someone wrote a duck type compatible version of autograd.

Wouldn’t autograd be wrapped by Dask Array instead?

[jakirkham]

We call da.asarray in other places to handle non-Dask Arrays. Seems reasonable to do it here too.

yes, though with overriding numpy ufuncs we still have to be careful to explicitly skip types like xarray.DataArray

This is what I'm hoping the asduckarray function will do for us in libraries like Dask. 😉

xref: numpy/numpy#13831 (for others interested in learning more)