Revisit test matrix: remove python 3.7

[wtgee]

          You should probably drop Python 3.7. It is picking up a very old `astropy` (4.3.1).

Originally posted by @pllim in #554 (comment)

EOL for Python 3.7 is 2023-06-27 (schedule) so we should consider removing support from our test mtarix.

The tox.ini file also specifies a number of versions of astropy that are quite old and doesn't specifically support the 5.x branch, so we should clean that up as well.

[wtgee]

I'll try to sit down this weekend and work on some of this and get a PR submitted.

[pllim]

FWIW, astropy 5.3 also dropped Python 3.8 to follow numpy, see astropy/astropy#14286 .