Specify pybind11 as a build-time dependency

[agriyakhetarpal]

Description

This PR adds pybind11 to the list of build-time dependencies in pyproject.toml (#3301); which means it is picked up, downloaded, and installed by pip automatically and therefore made available to users. CMake can now find the package instead of using the git-cloned directory. Contributors and developers will no longer have to git-clone the pybind11 repository from GitHub. All instances of pybind11 have been moved to a unified location for other workflows to use.

Closes #3480

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist:

• No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
• All tests pass: $ python run-tests.py --all (or $ nox -s tests)
• The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).

Further checks:

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[agriyakhetarpal]

I will run the wheel-building workflow on my fork just to make sure these changes are okay.

[agriyakhetarpal]

It passes locally for me, strangely! Edit: okay, now passing for Linux and macOS, I just need to fix the Windows tests. We don't compile the IDAKLU solver in PR tests for Windows but we do it when building wheels, so we can check for the CIBUILDWHEEL environment variable in this case. Similarly, the Read the Docs configuration does not compile the IDAKLU solver either, so I can use the READTHEDOCS environment variable for this. The goal is to ensure that the compilation of the IDAKLU solver remains optional and does not break anything if installing from a source distribution.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e2d8792) 99.59% compared to head (e80aaef) 99.59%.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3560   +/-   ##
========================================
  Coverage    99.59%   99.59%           
========================================
  Files          258      258           
  Lines        20755    20755           
========================================
  Hits         20670    20670           
  Misses          85       85           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[agriyakhetarpal]

So I am having a bit of trouble making the compilation of the IDAKLU solver optional on macOS and Linux platforms, especially because it breaks the installation on source distributions (CMake is installed by default since it is present in the build-system dependencies in pyproject.toml). Doing so on Windows is easy since we mention that the IDAKLU solver is not available on Windows (users can follow along with our complicated development procedure for the Windows wheels with MSVC and so on if they want to, though)

• One option I am considering is looking for the relevant SUNDIALS and SuiteSparse .dylib or .so files in the $HOME/.local/lib folder or a user-specified SUNDIALS_INST environment variable on macOS and Linux. The issue with this is that it can cause issues with the pybamm_install_odes entry point that is reserved for user installations via PyPI – SUNDIALS installed using that entry point does not come with KLU, which would further break someone's IDAKLU installation if they try to install from source sometime later (SUNDIALS + KLU are not compatible with scikits.odes because of double vs extended precision constraints). We could document this in our instructions but it doesn't feel like a permanent solution and it feels like we would be then going the other way since the goal is to simplify the installation.
• Another method is to use some build option that gets passed to setup.py—pip has a --global-option for this purpose as far as I am aware—which turns on the IDAKLU solver compilation procedure, but it is still difficult to ensure that the compilation remains an opt-in feature rather than an opt-out one (see also: https://discuss.python.org/t/optional-c-extension-handling/1595/7)
• We can scan for most environment variables during setup.py (it is a Python script after all and is meant for Python trickery like this) and turn compilation on or off based on that, I did that for Read the Docs but the doctests are broken currently and we will soon run out of options like those
• Another way: ensure that we can support more platforms (Build wheels for more platforms (Linux aarch64 wheels, MUSL wheels, etc.) #3462) either via cross-compilation or with the use of native external CI runners (Cirrus CI has had arm64 runners for both Linux and macOS since some while now) until GitHub Actions gets them – this way we will have wheels for every major platform and architecture and we will not need to bother with setup.py (wheels don't contain a setup.py).
• The best option I can think of at this moment is to remove CMake from the build-time dependencies and ask users to install it on their own in the docs when doing the source installation (it is pretty easy to do so via pip or brew so it doesn't defeat the purpose of making pybind11 a build-time dependency). We check for it in the compile_KLU() function and return False if it is not found – the CI can be configured to fix that pretty easily.

I think I will go with the last option for now unless there is something I have been missing or anything has been overlooked (cc: @kratman @Saransh-cpp).

Edit: never mind, removing CMake from the build-time dependencies makes it unavailable for installations locally and fails tests in CI due to build isolation (see e80aaef). This is turning out to be harder than I thought