TST: rewrite doctest in a floating-point-comparison friendly way

[neutrinoceros]

Description

Fixes #16319, similar to #16329, but with stricter margins than the defaults.

This test worked fine on every archs for at least 5 years, so maybe it's hinting at a real bug that should be inspected more closely, but I'm really not sure it's worth it.

• By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

[github-actions]

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

[github-actions]

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

[pllim]

But would scipy/scipy#20569 also fixes this problem without us having to do anything here?

[neutrinoceros]

It wouldn't. I tested it locally and that's the reason why #16319 and #16320 are separate issues.

[eerovaher]

This is user-facing documentation, so having to import numpy and specify rtol and atol is distracting. It would be much better to display what d3d is when it is computed, then display matches.separation_3d(c) and let the reader compare the results themselves. Then we could also let the +FLOAT_CMP directive take care of any floating point rounding differences in a way that is invisible to the readers.

Ideally the entire section would be rewritten in a similar manner, but it might be best to make a minimal change here just to solve the immediate problems in CI.

[neutrinoceros]

Sounds reasonable. I agree that invoking np.allclose is distracting. I just pushed your suggestion !

[eerovaher]

I still think the entire section should be reworked, but fixing CI problems is urgent and doing a minimal change can be good enough for that purpose. That being said, @mhvk is very responsive so we can afford to wait for his review. Having more than one review is especially important for user-facing documentation because readability for non-experts cannot be judged with automated tests.

[mhvk]

Let's go with this for now! Thanks, @neutrinoceros!

[pllim]

Thanks, all!