[JOSS] tests fail locally

[sappelhoff]

• xref: [REVIEW]: mTRFpy: A Python package for temporal response function analysis openjournals/joss-reviews#5657

Running the testsuite in a fresh environment locally, I get failures (linux, python 3.11).

I see that Python 3.11 is not yet covered in your test suit, I'll add it in #9

(mtrf) stefanappelhoff@arc-lin-004309:~/Desktop/mTRFpy$ pytest --verbose
======================================================================================= test session starts ========================================================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /home/stefanappelhoff/miniconda3/envs/mtrf/bin/python3.11
cachedir: .pytest_cache
rootdir: /home/stefanappelhoff/Desktop/mTRFpy
collected 12 items                                                                                                                                                                                 

tests/test_basics.py::test_check_data PASSED                                                                                                                                                 [  8%]
tests/test_basics.py::test_lag_matrix PASSED                                                                                                                                                 [ 16%]
tests/test_basics.py::test_arithmatic PASSED                                                                                                                                                 [ 25%]
tests/test_matlab_examples.py::test_encoding PASSED                                                                                                                                          [ 33%]
tests/test_matlab_examples.py::test_decoding PASSED                                                                                                                                          [ 41%]
tests/test_matlab_examples.py::test_transform FAILED                                                                                                                                         [ 50%]
tests/test_model.py::test_train PASSED                                                                                                                                                       [ 58%]
tests/test_model.py::test_predict PASSED                                                                                                                                                     [ 66%]
tests/test_model.py::test_test PASSED                                                                                                                                                        [ 75%]
tests/test_model.py::test_save_load PASSED                                                                                                                                                   [ 83%]
tests/test_stats.py::test_crossval PASSED                                                                                                                                                    [ 91%]
tests/test_stats.py::test_permutation PASSED                                                                                                                                                 [100%]

============================================================================================= FAILURES =============================================================================================
__________________________________________________________________________________________ test_transform __________________________________________________________________________________________

    def test_transform():
        transform_results = np.load(  # expected results
            root / "results" / "transform_results.npy", allow_pickle=True
        ).item()
    
        t = transform_results["t"]
        w = transform_results["w"]
        direction = transform_results["dir"][0, 0]
    
        trf_decoder = TRF(direction=-1)
        tmin, tmax = -0.1, 0.2
        trf_decoder.train(stimulus, response, fs, tmin, tmax, 100)
        trf_trans_enc = trf_decoder.to_forward(response)
    
        scale = 1e-5
>       np.testing.assert_almost_equal(trf_trans_enc.weights * scale, w * scale, decimal=11)

tests/test_matlab_examples.py:79: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../miniconda3/envs/mtrf/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
../../miniconda3/envs/mtrf/lib/python3.11/contextlib.py:81: in inner
    return func(*args, **kwds)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<function assert_array_almost_equal.<locals>.compare at 0x7f4aafab5da0>, array([[[-1.63891735e-01, -1.75839906e-01, -... 3.50612939e-03,  2.12364773e-02,  3.57734865e-02, ...,
         -1.11449796e-02, -2.29089935e-02, -2.88814115e-02]]]))
kwds = {'err_msg': '', 'header': 'Arrays are not almost equal to 11 decimals', 'precision': 11, 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Arrays are not almost equal to 11 decimals
E           
E           Mismatched elements: 1 / 81920 (0.00122%)
E           Max absolute difference: 1.52442529e-11
E           Max relative difference: 1.7259e-06
E            x: array([[[-1.63891734619e-01, -1.75839906370e-01, -1.81312251773e-01,
E                    ..., -6.04563894823e-02, -3.13816863207e-02,
E                     6.20892295951e-03],...
E            y: array([[[-1.63891734619e-01, -1.75839906370e-01, -1.81312251773e-01,
E                    ..., -6.04563894823e-02, -3.13816863207e-02,
E                     6.20892295950e-03],...

../../miniconda3/envs/mtrf/lib/python3.11/contextlib.py:81: AssertionError
========================================================================================= warnings summary =========================================================================================
tests/test_matlab_examples.py::test_encoding
tests/test_matlab_examples.py::test_decoding
tests/test_matlab_examples.py::test_transform
tests/test_model.py::test_predict
  /home/stefanappelhoff/Desktop/mTRFpy/mtrf/matrices.py:139: RuntimeWarning: invalid value encountered in matmul
    cov_xx[i_x] = x_lag.T @ x_lag

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== short test summary info ======================================================================================
FAILED tests/test_matlab_examples.py::test_transform - AssertionError: 
======================================================================= 1 failed, 11 passed, 4 warnings in 134.96s (0:02:14) =======================================================================

[OleBialas]

I can reproduce this with the newest Python and Numpy versions. It seems the error is just numerical inaccuracy. After restricting the equality test to the first 10 decimals, the error disappeared.

By the way, the matrix multiplication warning seems to be due to some bug in Numpy as well:
numpy/numpy#24067