Integration test currently broken

[KirstieJane]

We have a green tick on our CI but that's kind of a lie..... this line in the .travis.yml file is currently ignoring the integration test 😭

Everything runs but we get a different hash for the test_lm_against_fixture test: https://travis-ci.org/WhitakerLab/scona/jobs/529092803#L548 (lm stands for local measures). That Travis report is from the kw-test-tests branch which adds the regression test back in.

From my recollection this runs correctly on @Islast's local machine....but gets an error on Travis. So a first step is for someone to run these tests locally and see if they pass or fail on their own machine.

Checkout the development guide for a lot of instructions on how to run the code locally 👾

[sgibson91]

Hi @KirstieJane 👋 Integration test fails on my iMac as well 😞There's also a whole bunch of warnings from running the test suite too, some deprecation warnings and some RemovedInPytest4Warning. I'll create an issue with my output so we can see where the warnings are coming from and what might need updating. (Update: Issue #111 )

[wingedRuslan]

I run integration tests locally. But besides test_lm_against_fixture the following tests test_rich_against_fixture and test_gm_against_fixture failed as well.

[sgibson91]

So far, only test_lm_against_fixture is failing for me. Unless I set up a Python 3.6 environment, then 6 other tests fail. But none of those 6 are test_rich_against_fixture or test_gm_against_fixture.

[wingedRuslan]

Well, at least I figured out why 3 regression tests failing when I run them locally.
I used python3.5 to run regression tests, and I had 3 of them fail.
Switching to the python3.6 has helped to have only test_lm_against_fixture failing, the one described in the issue.

Failed tests with python3.5

python -m pytest tests/regression_test.py

================================== test session starts ===================================
platform linux -- Python 3.5.2, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
rootdir: /home/pilot/GSoC/scona
collected 4 items                                                                        

tests/regression_test.py .FFF                                                      [100%]

======================================== FAILURES ========================================
__________________________ FixturesTest.test_gm_against_fixture __________________________

self = <tests.regression_test.FixturesTest testMethod=test_gm_against_fixture>

    def test_gm_against_fixture(self):
        # test new global measures against fixture
        print('\ntesting new global measures against fixture')
        self.assertEqual(self.hash_dict_new[self.gm],
>                        self.hash_dict_original[self.gm])
E       AssertionError: 'd1f7c8c09b10695fa7de23227474f6a9290a75276eec595757e263219db56bad' != '1953e23e11a620c02ed4f8eb10aef2c566bd5479bd9528707ade96f1f5775439'
E       - d1f7c8c09b10695fa7de23227474f6a9290a75276eec595757e263219db56bad
E       + 1953e23e11a620c02ed4f8eb10aef2c566bd5479bd9528707ade96f1f5775439

tests/regression_test.py:43: AssertionError
---------------------------------- Captured stdout call ----------------------------------

testing new global measures against fixture
__________________________ FixturesTest.test_lm_against_fixture __________________________

self = <tests.regression_test.FixturesTest testMethod=test_lm_against_fixture>

    def test_lm_against_fixture(self):
        # test new local measures against fixture
        print('\ntesting new nodal measures against fixture')
        self.assertEqual(self.hash_dict_new[self.lm],
>                        self.hash_dict_original[self.lm])
E       AssertionError: '9327e95709a7526b6a9ef90657402ea36b1d9dae4aadd5ee4b4764c708be2c9c' != 'dc585cb573b87b1099c034f7c6991c410fca3f6878541e3caa2c7ac0816957dc'
E       - 9327e95709a7526b6a9ef90657402ea36b1d9dae4aadd5ee4b4764c708be2c9c
E       + dc585cb573b87b1099c034f7c6991c410fca3f6878541e3caa2c7ac0816957dc

tests/regression_test.py:37: AssertionError
---------------------------------- Captured stdout call ----------------------------------

testing new nodal measures against fixture
_________________________ FixturesTest.test_rich_against_fixture _________________________

self = <tests.regression_test.FixturesTest testMethod=test_rich_against_fixture>

    def test_rich_against_fixture(self):
        # test rich club against fixture
        print('\ntesting rich club against fixture')
        self.assertEqual(self.hash_dict_new[self.rich],
>                        self.hash_dict_original[self.rich])
E       AssertionError: '8665acdcd33037f088a5b50d281b487dc65e08eb905cf6a814034c2fdcb46ab7' != 'afa434e97c44762397972abd39a142060c77cb89b1b41503aa8b8a8fc03d9e8f'
E       - 8665acdcd33037f088a5b50d281b487dc65e08eb905cf6a814034c2fdcb46ab7
E       + afa434e97c44762397972abd39a142060c77cb89b1b41503aa8b8a8fc03d9e8f

tests/regression_test.py:49: AssertionError
==================== 3 failed, 1 passed, 2 warnings in 121.77 seconds ====================

1 Failed test with python3.6

python3.6 -m pytest tests/regression_test.py
================================== test session starts ===================================
platform linux -- Python 3.6.8, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /home/pilot/GSoC/scona, inifile:
collected 4 items                                                                         

tests/regression_test.py ..F.

======================================== FAILURES ========================================
__________________________ FixturesTest.test_lm_against_fixture __________________________

self = <tests.regression_test.FixturesTest testMethod=test_lm_against_fixture>

    def test_lm_against_fixture(self):
        # test new local measures against fixture
        print('\ntesting new nodal measures against fixture')
        self.assertEqual(self.hash_dict_new[self.lm],
>                        self.hash_dict_original[self.lm])
E       AssertionError: '3ce452749b52e3a60b7779ffe3c43e1de8219cb42715aeabd28b94363b59e9bb' != 'dc585cb573b87b1099c034f7c6991c410fca3f6878541e3caa2c7ac0816957dc'
E       - 3ce452749b52e3a60b7779ffe3c43e1de8219cb42715aeabd28b94363b59e9bb
E       + dc585cb573b87b1099c034f7c6991c410fca3f6878541e3caa2c7ac0816957dc

tests/regression_test.py:37: AssertionError
---------------------------------- Captured stdout call ----------------------------------

testing new nodal measures against fixture
========================== 1 failed, 3 passed in 102.61 seconds ==========================

[Islast]

Thanks @wingedRuslan 👍. The failing in 3.5 is an issue I understand. The procedure we're testing involves randomisation and due to differences in pythons 3.5 and 3.6 even when we set a seed we don't get exactly the same thing. We used to have test fixtures for each python version, but while troubleshooting the local measures problem I've been just testing python 3.6.
The test fixtures can be a bit of a pain to produce. Currently when I make them I create a minimal python environment in Anaconda, install only the dependencies of scona and then generate the fixtures. This can be a bit time consuming- does anyone have a smarter way of doing this?