Making numba an optional dependency?

[arogozhnikov]

Hi, I've been looking for an alternative to scipy.sparse with more consistent interface.
Surprised I didn't find this earlier, but anyway.

There is one thing I'm quite worried before making sparse a dependency, i.e. numba.
In the past couple of years it was constantly lagging behind numpy, at least a couple of releases.
It looks that numba project had lost some energy and struggles with updating (that's a challenging task they try to attack, so no surprise here). I'd like to not lock myself in a position where I can't remove it from the list of dependencies.

So, wondering if I'm not alone, and maybe there are ways to make numba optional and e.g. have pure Cython-based or numpy-based implementations (with reasonable reduction in speed).

Thank you!

[hameerabbasi]

Hello, and thank you for the comment.

We are working on a pure C++-based implementation that may or may not use cppyy as a dependency. The prototype is very WIP, and there is no ETA as of now. You can find it here to help contribute: https://github.com/hameerabbasi/xsparse

Why cppyy and not, e.g. Cython, pybind11 or the like? Because we need runtime template instantiation in C++. This can also be achieved with a custom version of cling, for instance.

[wlav]

To your last point and FWIW (I understand that this is the opposite from what OP is asking, but it's all loosely coupled, no requirements): I've started a Numba extension for cppyy. For now, all it can handle is free functions (incl. run-time instantiations of templated ones), but eventual goal is to inline (templated) C++ into Numba python at the IR level. My targets are Eigen and Nvidia's CUTLASS, but could try xsparse as well if there's interest in such.

A few basic examples are in the test suite:
https://github.com/wlav/cppyy/blob/master/test/test_numba.py

[hameerabbasi]

Many thanks for the reply, it certainly looks interesting. Definitely, it can help with an incremental switch, if, for example, NumPy arrays can be mapped to pointers or similar.

However, I don't see XSparse at that stage yet. What would help us more is conda-forge packages for Windows at this point. I know there was an effort to create those but they don't seem to be supported yet. I also think it might more generally help with adoption.

[wlav]

Yes, the conda-forge folks want to use Clang/LLVM from upstream, but they only support 32b Windows, whereas conda-forge supports only 64b Windows. In any case, upstream does not have all necessary Windows patches in Clang, so only LLVM builds could be reused. Upstream is closing in on LLVM13 and has another GSoC on PCMs on Windows, which may improve matters.

Still, I'd figure using conda to install the compiler (installing MSVC is the hardest problem on Windows) and using cppyy from PyPI should work (that one has both 32b and 64b wheels for the backend).