Better document how to run the tests

[ExplodingCabbage]

When I run tox on my Ubuntu 14.04 machine, before giving up I get:

• a bunch of failures
• a bunch of LookupErrors
• loads of tracebacks with errors saying NameError: global name 'pyparsing' is not defined
• a bunch of other NameErrors
• tracebacks with errors saying OSError: Senna executable expected at /usr/share/senna-v3.0/senna-linux64 but not found
• multiple screens of nltk_data downloads that still haven't finished after many hours

I've consequently given up ensuring that the tests in my PR can run as part of the tox run, and will just try to run them individually. But shouldn't the process for running the tests be less painful than this? The CONTRIBUTING.md guidelines say

just execute tox

and that's what I'd expect to have to do; the reality is that the tests seem basically unrunnable. Do the core contributors have development setups that let them run the full test suite easily? If so, what're they doing differently to me? And what output should I expect to see when I run tox?

[iliakur]

I'm sad to say I have the same problem as you. Trying to actually make "just execute tox" run was really frustrating. @alvations at the time told me to only worry about the relevant tests for what I was working on. I guess that makes sense since we don't really have integration or E2E tests, and ideally unit and doctests should be testing isolated components. I say "ideally" because I personally have a couple of essentially integration tests inside my unit tests precisely due to lack of official places to put them :(

At some point when I have time I'd like to bring up the issue of our testing architecture in the mailing list.

[ExplodingCabbage]

It seems worth noting that there are really two actionables here, one quick and easy and one long and difficult:

• 1. amend the contributing guidelines to make clear that tox won't work on (most? all?) contributors' machines, and that they are only expected to run tests relevant to the modules they are modifying - the rest can run in CI
• 2. make it easy for new contributors to run the full test locally

As well as being easier, just doing 1 immediately provides about 90% of the value of 2. Not being able to run the full test suite on a dev machine really isn't a big deal; that in combination with guidelines indicating that I'm expected to run it is, and led to me wasting hours more time on a fool's errand than I would've done had the docs let me know that I didn't really have to run tox.

[iliakur]

@ExplodingCabbage do you have time to tackle the docs?

[ExplodingCabbage]

@copper-head time, yes, but I feel like it would be better for one of the core contributors who knows the current state of the tests to do it.

[stevenbird]

I'm sorry that I don't have any resources at my disposal in order to do this...

[fievelk]

@ExplodingCabbage We do not have test failures anymore. Do you think this issue can be marked as solved?

[ExplodingCabbage]

The tests ran for me this time around, so I guess this is resolved!