Draft: Follow-up numpy 2.0 (DLLAPI)

[duburcqa]

No description provided.

[duburcqa]

I think I will try to use numpy/_core/include/numpy/npy_2_compat.h instead of dealing with backward compatibility manually. It sounds more reasonable.

[duburcqa]

I changed my mind. Since npy_2_compat.h must be vendored to get backward compatibility. The current MR with the previous one is basically the same without dedicated header.

[duburcqa]

For now, something is broken, and it is not clear to me why. I will ask on numpy repo:

 ImportError: /home/runner/.local/lib/python3.10/site-packages/jiminy_py/core/core.cpython-310-x86_64-linux-gnu.so: undefined symbol: EIGENPY_ARRAY_APIPyArray_RUNTIME_VERSION

here is the corresponding issue on Numpy.

[duburcqa]

After a long discussion with numpy team, it turns out that numpy API should not be exposed the way it is to avoid compatibility issues. Actually every import of numpy headers should be moved to the source file. It is the only way to ensure that EigenPy public API does not leak numpy internal symbols.

[seberg]

Maybe moving the discussion here (also). I am still happy to mitigate the linking issue in NumPy if you/eigenpy devs think it would be a serious help. However, while NumPy may not make things easy, I don't think the setup where eigenpy imports NumPy for a downstream library is great.

In a sense, the obvious solution I (and others) on NumPy see right now is to ask users to do a normal NumPy include with the typical NumPy pattern:

#define Py_ARRAY_UNIQUE_SYMBOL MyModule
#define NPY_NO_IMPORT
#include "numpy/ndarrayobject.h"

in most files, and:

#define Py_ARRAY_UNIQUE_SYMBOL MyModule
#include "numpy/ndarrayobject.h"


module_init_function() {
    import_array()
}

in the main module file before any import eigenpy import. Which should "detach" this transitive use of the NumPy API.

The question is whether that seems easy enough for the eigenpy use-case to avoid the issues (if you compile and run eigenpy together with the downstream library there are no issues).
(I would also be interested to discuss other ways we can make this nicer by reorganizing the NumPy headers. That might not help us right now but in the future.)