Inconsistent numbering of numpy requirement

[ultra-andy]

There appear to be inconsistencies in the numpy version dependencies in the cmsisdsp wheel files.

The cmsisdsp source code on GitHub specifies numpy 1.22.x in pyproject.toml:

requires = [
    ...,
    "numpy>=1.22, < 1.23"
]

...and also in setup.py:

    install_requires=['numpy>=1.22, < 1.23 ',
    ...
    ],

When installing cmsisdsp in Python 3.10 on MacOS using a pyproject.toml file which specifies the following:

dependencies = [
	"numpy == 1.22",
	"cmsisdsp == 1.9.7",
]

I get the following error message:

ERROR: Cannot install ace-implementation and ace-implementation==0.0.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    ace-implementation 0.0.1 depends on numpy==1.22
    cmsisdsp 1.9.7 depends on numpy<1.24 and >=1.23

Investigating the source wheel file on PyPi (cmsisdsp-1.9.7-cp310-cp310-macosx_10_9_universal2.whl), by converting it to a zip and unpacking, the METADATA file contains the following line:

Requires-Dist: numpy (<1.24,>=1.23)

This is inconsistent with the source code above.

Looking at the METADATA file in other wheel files:

cmsisdsp-1.9.7-cp310-cp310-win_amd64.whl => Requires-Dist: numpy (<1.23,>=1.22)
cmsisdsp-1.9.7-cp311-cp311-win_amd64.whl => Requires-Dist: numpy (<1.24,>=1.23)
cmsisdsp-1.9.7-cp311-cp311-macosx_10_9_universal2.whl => Requires-Dist: numpy (<1.24,>=1.23)

So it looks like the METADATA files are inconsistent with each other & with the cmsisdsp source code for:

MacOS Python 3.10, 3.11
Windows Python 3.11

Would someone please investigate and work out which is right and make everything consistent.

[christophe0606]

@ultra-andy I have not been able to find a version of NumPy that can be used for all computer + Python configurations and TensorFlow / PyTorch for those configurations.

So indeed, I have had to change the NumPy version when preparing the wheel before uploading so that cmsisdsp can be used with TensorFlow / PyTorch. So, the numpy versions of the wheel are not always consistent with the version in the github repo.

And what it means is that there will be configuration of packages where you'll hit an incompatibility and depending on the computer and python version it will be different.

I don't think it is possible to solve the problem since you often find packages that require different NumPy versions.

(But I am far from being an expert in Python package generation).

[ultra-andy]

@christophe0606 - thanks for the quick reply, so it's an issue with clashing Numpy requirements across external packages.

It can't be the first time this sort of thing has happened, so I'll have a look around the web & investigate - will post here if I learn anything useful!

[ultra-andy]

@christophe0606, I've had a look at PyTorch and TensorFlow NumPy requirements. The latest version of PyTorch does not appear to require NumPy, and TensorFlow uses NumPy >= 1.23.5 across all platforms (MacOS x86, MacOS ARM, Win AMD64, Linux aarch64 & Linux x86) and Python versions (3.9, 3.10 & 3.11) (at least for the latest release).

So, I think there's a missing bit of information, or I'm misunderstanding something.

Would you please describe an example of a problem that you run into when you pin NumPy to >= 1.23.5 for cmsisdsp for all architectures and Python versions?

[christophe0606]

@ultra-andy I don't remember the details. I think one problem (but not the only one) is that for some configurations there were no NumPy binaries available and rebuilding from source failed.
But there were other problems. Next time I prepare some new release, I'll log the issues that I had.

You could rebuild the package yourself from source if it can unlock you.

Not ideal but not that hard:
First build the CMSIS-DSP binary by going to any of the PythonWrapper/build folder.
Customize the create.bat (or create.sh) with the right paths.
Then type make once the cmake command from the script has succeeded.

Once you have built the CMSIS-DSP binary for Python, you can go to the root and build the wheel with :

pip wheel . --no-deps -w dist

and then install, it with pip install dist/name_of_the_wheel

[ultra-andy]

@christophe0606, Thanks for all that. The issue we have is that we are a team using this library in a repo. We have at least two Python versions so far (3.10 and 3.11), and two architectures (MacOS ARM and Intel x86). We can standardise on a Python version to knock out the first issue, but the mutually breaking split in the Numpy requirement across ARM and x86 is harder to resolve. Your approach means doing a build for MacOS ARM and also for Intel x86, and storing those builds somewhere for use of our repo. Very clunky.

Removing the numpy version pinning altogether will likely fix the immediate issue, since the Python version resolvers should be able to identify the compatible NumPy versions on each architecture, but this means different installs on different architectures are by definition using different Numpy library versions - not ideal ground to be on for a shared repo.

If, the next time you do a build, you could write a new ticket and link this one, or just extend this on, then we'll have more specific information that we can use to come up with a fix to the underlying problem.

Thanks much!

[christophe0606]

@ultra-andy I'll link this ticket when I make a new release.
Sorry for the inconvenience.