Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: bump tolerance to address local test_axis_nan_policy failures #20748

Merged
merged 2 commits into from
May 19, 2024

Conversation

rgommers
Copy link
Member

@rgommers rgommers commented May 19, 2024

The normaltest tests need 4e-15 to pass in my x86-64 linux conda-forge environment; the kurtosistest tests need 6e-15. These tolerances don't need to be this tight and my setup is unlikely to be worst-case, so add a bit of margin on top.

Failures:

E   AssertionError:
E   Not equal to tolerance rtol=1e-15, atol=0
E
E   Mismatched elements: 3 / 6 (50%)
E   Max absolute difference: 7.99360578e-15
E   Max relative difference: 3.95721074e-15
E    x: array([[5.141364, 0.673337],
E          [4.039859, 2.172997],
E          [4.832018, 5.96047 ]])
E    y: array([[5.141364, 0.673337],
E          [4.039859, 2.172997],
E          [4.832018, 5.96047 ]])
        args       = (<function assert_allclose.<locals>.compare at 0x7ef45c1d6e80>, array([[5.14136359, 0.67333671],
       [4.03985923, 2...18, 5.96046984]]), array([[5.14136359, 0.67333671],
       [4.03985923, 2.17299682],
       [4.83201818, 5.96046984]]))
        func       = <function assert_array_compare at 0x7ef488122f20>
        kwds       = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-15, atol=0', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7ef48811bc50>
========================================================= short test summary info ==========================================================
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
========================================== 36 failed, 3096 passed, 54 skipped in 66.33s (0:01:06) ==========================================

Also skip these tests unless SCIPY_XSLOW is set. This one parametrized test takes about a minute to run.

The `normaltest` tests need 4e-15 to pass in my x86-64 linux conda-forge
environment; the `kurtosistest` tests need 6e-15. These tolerances don't
need to be this tight and my setup is unlikely to be worst-case, so
add a bit of margin on top.

Failures:
```
E   AssertionError:
E   Not equal to tolerance rtol=1e-15, atol=0
E
E   Mismatched elements: 3 / 6 (50%)
E   Max absolute difference: 7.99360578e-15
E   Max relative difference: 3.95721074e-15
E    x: array([[5.141364, 0.673337],
E          [4.039859, 2.172997],
E          [4.832018, 5.96047 ]])
E    y: array([[5.141364, 0.673337],
E          [4.039859, 2.172997],
E          [4.832018, 5.96047 ]])
        args       = (<function assert_allclose.<locals>.compare at 0x7ef45c1d6e80>, array([[5.14136359, 0.67333671],
       [4.03985923, 2...18, 5.96046984]]), array([[5.14136359, 0.67333671],
       [4.03985923, 2.17299682],
       [4.83201818, 5.96046984]]))
        func       = <function assert_array_compare at 0x7ef488122f20>
        kwds       = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-15, atol=0', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7ef48811bc50>
========================================================= short test summary info ==========================================================
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--3-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--2-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite--1-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-0-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-1-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-propagate-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-propagate-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-omit-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-omit-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-raise-kurtosistest-args41-kwds41-1-2-False-None] - AssertionError:
FAILED scipy/stats/tests/test_axis_nan_policy.py::test_axis_nan_policy_full[all_finite-2-raise-normaltest-args42-kwds42-1-2-False-None] - AssertionError:
========================================== 36 failed, 3096 passed, 54 skipped in 66.33s (0:01:06) ==========================================
```

[skip ci]
@rgommers rgommers added scipy.stats maintenance Items related to regular maintenance tasks labels May 19, 2024
@rgommers rgommers added this to the 1.14.0 milestone May 19, 2024
@rgommers
Copy link
Member Author

rgommers commented May 19, 2024

@mdhaber these tests are also super slow, test_axis_nan_policy.py takes about 10% of the total runtime of the full SciPy test suite. That doesn't seem reasonable - testing this highly parametrized is best done during development only. How about marking this test as xslow instead of slow?

@rgommers rgommers mentioned this pull request May 19, 2024
3 tasks
@mdhaber
Copy link
Contributor

mdhaber commented May 19, 2024

K

@mdhaber mdhaber changed the title TST: bump tolerance to address test_axis_nan_policy failures TST: bump tolerance to address local test_axis_nan_policy failures May 19, 2024
>3,000 tests is not only slow to skip (see code comment) but results in
3000 `S` characters of test log output that is much nicer to avoid.

[skip cirrus] [skip circle]
@mdhaber mdhaber merged commit 94895f2 into scipy:main May 19, 2024
30 checks passed
@rgommers rgommers deleted the fix-axis-nanpolicy-failures branch May 19, 2024 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Items related to regular maintenance tasks scipy.stats
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants