MWE of skbuild issues

[Erotemic]

I'm not sure exactly what to call this. The motivation is that I'm having a very difficult time getting scikit-build to work robustly for a package that has a few different things going on. The best way that I think I can explain it is to show a minimal working example of a somewhat complex scikit-build project setup, and then enumerate the issues I'm having with it. For each issue, I'm either doing something wrong, or there is a bug in scikit-build. I think there might be a mix of both cases.

My idea is that after fixing these issues in this MWE / scikit-build itself, this semi-complex, but also minimal example will be a good sample project for the test suite.

The layout of the project is

• A src/cxx folder with code for a standalone C++ library. My thought is that this library should demo how to link to external dependencies like zlib or pthread (std::thread nowadays?).
• A src/python folder with a few things going on...
• A cython module that wraps the cxx library
• A submodule that contains a resource file that importlib should be able to help us access
• A root setup.py and CMakeLists file that should allow scikit-build to "just build" the project.

I cannot figure out how to get the setup.py to work correctly both when building wheels python setup.py bdist_wheel and installing in development mode pip install -e .. Each mode of using this has its own issues. I have a root-level demo_issue.sh script that provides commands to demonstrate these.

A list of the issues I'm having are:

• I can't figure out how to get the external c library to build as a shared library, and have scikit-build correctly include it in the wheel (or in the development-mode editable install). Currently it is doing static-linking. I would like to have a CMakeLists condition that demos how to work with a repo structure like this when doing static or dynamic linking.
• Running pip install -e . installs a my_skb_mod.egg-link that points to the wrong location when, the packages argument to setup in setup.py contains more than one package. I think this is due to a bug in the scikit-build code in skbuild/command/egg_info. I can hack this by including a [''], package, but skbuild also warns me not to do that.
• Only using one arg to packages=['my_skb_mod'] works with editable installs, but when I build a wheel, it does not include the submodules.
• Package data does not seem to be registered or included correctly.
• The Cython docstrings are getting dropped. I have no idea why this is. IIUC Cython defaults to keeping them. I don't see any logic that adds the --no-docstring flag.

Sorry this is a bit big / not submitted as an issue first, but the actual issues I'm having seem to be fairly interconnected, so I think this is as small as I can make this project. Also, I think this will be a valuable sample for the tutorial if we can get it working. I've found that the existing samples don't really show you how to work with scikit-build in more complex real-world cases. So I'm hopping this can fill that gap.

In summary I want, a working MWE that demonstrates how to build a scikit-build project with:

• A standalone C++ module that links to some external dependency.
• A cython module in a non-root main python module (i.e. src/python/<modname>) that links to the standalone C++ module either statically or dynamically, depending on CMake flags.
• Submodules as part of the main python module.
• Resources files that are registered with importlib.
• Cython functions should have docstrings

If there is any easy way to change this without modifying skbuild and get all of the features I want, please let me know. Also, because I do want this to be a useful scikit-build sample, I'm looking for input for:

• Name of this sample package test? Right now its issue-xxx-multi-package-with-package-dir, but maybe hello-complex-01 is better?
• Name of the "demo" main python module (currently is my_skb_mod)
• Name of the standalone C++ lib (currently mecl - for my external c library).
• Any other cases I should be considering that would make this more comprehensive, while still being reasonably minimal.

[henryiii]

python setup.py bdist_wheel is very much deprecated, and I'd not worry too much about getting it to work. As long as setuptools is imported first, I think it should probably work, but I'd not worry about it too much.

I think the missing doctoring problem is due to a fix that's not in a released version yet.

Can you compare with https://github.com/scikit-build/scikit-build-sample-projects ? Specifically, one of the new examples at https://github.com/scikit-build/scikit-build-sample-projects/pulls might be close? That's also where this should go, probably.

If we can work on fixing one thing at a time, that would be ideal. I'm hoping (#601) to have a lot more time next year to work on an overhaul, but improving the current situation is great in the mean time!

[henryiii]

#582 might be released - if you could figure out how to make #585 fail, that would make it much easier to test and verify if that fix works.

[Erotemic]

Using scikit-build-sample-projects makes sense. I can port this PR over there. I did take a look at the other examples, and there isn't anything that handles the case of a pure C++ library and a Cython wrapper. Looking at the PRs, there is a similar one (from me, a few months ago that I forgot about!). That one is specifically looking at the shared library issue, whereas this one is more comprehensive. I think both have merit. Obviously simple samples are good, but it's also valuable to have a more integration-test style of sample.

I'm only using bdist_wheel as an example to build a wheel. Using pip wheel . has the same effect. The wheel is not build correctly. It is missing the package data.

I'll take a look at #582 and see if I can make #585 fail.

[henryiii]

I'd use

Suggested change

	pip wheel .
	pipx run build --wheel