I'd be happy to have them covered somehow, but IMO we need to be careful not to significantly increase the runtime of the test suite any further - our tests are already sufficiently slow that it's a problem.

Happy to help out with this in whatever way possible, just let me know.

Instead of doing this as a part of the default CI, maybe we can run this periodically instead (on a less frequent schedule) and create an issue if it fails?

Let's see how long it will take first, if it's less than a minute it seems like it would be worth catching on each PR.

The CI tooling already missed that removing setup.py broke many tests due to not running.

Okay, I gave this a try and I have this experimentally working with pip.

Using a very naive check I have 26 failing tests and 66 passing tests, but I have not done any debugging yet to see if I'm doing something wrong, so I may have a draft PR ready to go this weekend.

The total time on my PC takes 67 seconds to run and requires Python 3.12+. I am considering making it it's own seperate runner, as I don't think the tests need to run against every platform?

Okay, I've created a draft PR: #12580

There's still work to do on some scenarios/tests, but I think the general approach is there.

I would appreciate feedback from both pip and packse maintainers.