TST: tox option to exclude scipy

[pllim]

Description

This pull request is to add tox option to exclude scipy so we can test devdeps without scipy-dev. I opted to run this combo in weekly cron. The setup and cadence are all negotiable.

Also cc @neutrinoceros

Fixes #15701

• By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

[github-actions]

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• Is a milestone set? Milestone must be set but we cannot check for it on Actions; do not let the green checkmark fool you.
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

[github-actions]

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

[pllim]

I have no idea why flynt is failing here. Looks unrelated. @nstarman or @WilliamJamieson , what is going on with flynt here?

EDIT: OK, nvm. It passed again after I downgrade it back to Python 3.10. Didn't know it is sensitive to Python version. 🤷

[pllim]

OK, looks like I got it: https://github.com/astropy/astropy/actions/runs/7217133913/job/19664457683?pr=15742

Package versions: 
Numpy: 2.0.0.dev0+git20231212.9726755
Scipy: not available
Matplotlib: not available
h5py: not available
Pandas: not available
PyERFA: 2.0.1.2.dev9+g0d5ddca
Cython: not available
Scikit-image: not available
pyarrow: not available
asdf-astropy: not available

p.s. Link check failure is unrelated. You can ignore it.

[neutrinoceros]

It's a great idea, thank you @pllim
I note that I was expecting this combo to run alongside the "all dev devs" one, not to replace it, but maybe there's a cost to having both that I'm not seeing ?

[pllim]

run alongside the "all dev devs" one

I guess we can. I just thought we already have enough jobs as it is, and we have so much traffic that the devdeps in CI for PR and push basically runs constantly, so no point also calling it again in cron job. But perhaps having both in cron is safer in case this repo gets a slow period.

What do others think?

[pllim]

@dhomeier , do you have any opinions here?

[dhomeier]

I guess we can. I just thought we already have enough jobs as it is, and we have so much traffic that the devdeps in CI for PR and push basically runs constantly, so no point also calling it again in cron job. But perhaps having both in cron is safer in case this repo gets a slow period.

I don't have a strong opinion, but also think this can be read both ways – we typically have a dozen or more full runs of ci_workflows every day, so any standard job on top of that in cron_weekly or actually even cron_daily has fairly negligible impact. There is the same job just with py312 indeed (and even with remote data), but I'd tend to test those for at least two Python versions (though actually the combination of newest devdeps with oldest supported Python may be more interesting).
But I also don't know why e.g. mpldev warrants its own extra job in workflows...

Bottom line is probably I'd see cron_weekly as primarily serving the really long and expensive jobs like the exotic archs, while this job could well go into cron_daily – along with the old py311-test-devdeps or perhaps in fact a py310-test-devdeps.

[pllim]

I thought about it but I only want to deal with devdeps failures once a week, so I opted to keep it in weekly but make it a new job with older Python 3.10. Testing cutting age stuff with oldest supported Python seems like we are just asking for trouble.

[pllim]

@neutrinoceros and @dhomeier , what do you think now?

https://github.com/astropy/astropy/actions/runs/7465080616/job/20313469760?pr=15742

[neutrinoceros]

pluggy._manager.PluginValidationError: Plugin 'pytest_mpl' for hook 'pytest_report_header'
hookimpl definition: pytest_report_header(config, startdir)
Argument(s) {'startdir'} are declared in the hookimpl but can not be found in the hookspec

If I'm reading this right, this means that pytest_mpl isn't compatible with pytest-dev at the moment. At a first glance it doesn't seem like it was reported upstream yet.

[pllim]

You can ignore devinfra failures. I have a patch already at matplotlib/pytest-mpl#219 that is being tested at #15820

[pllim]

linkcheck and predeps failures are also unrelated. What is important here is that the new job is successful and did not pull in scipy.

[neutrinoceros]

Ah sorry I completely missed the point !
Can we add a step to invoke pip freeze right before tests so it's easier to track down in logs ?

[pllim]

There is a pip freeze right before the test starts, see https://github.com/astropy/astropy/actions/runs/7465080616/job/20313469760?pr=15742

[neutrinoceros]

Please ignore everything I said (apparently) 😅

[dhomeier]

Testing cutting age stuff with oldest supported Python seems like we are just asking for trouble.

That was the point of testing it. ;-)
I was rather thinking of py311-test-devdeps-noscipy + py310-test-devdeps, but should not matter too much.

And I take your point that it would be enough to deal with this once a week, had only thought about CI resources.

[pllim]

@dhomeier , OK I flipped the Python versions as you requested. If it looks good now, can you please approve? Thanks!

[dhomeier]

Thanks, remaining weekly jobs should be no issue!

[dhomeier]

How do we enable auto-merge here?

[pllim]

Auto merge won't make sense since the new job isn't required. Thanks for the approval. I will keep an eye on it and merge when the optional new job passes.

[dhomeier]

Has already, just waiting for the slower archs, so I thought about shortening that...

[pllim]

I cancelled the really long jobs and merged.

Hmm... I guess backport isn't critical since auto-backport failed. I'll update the milestone.