Add coverage checking to the tests.

[j-towns]

Firstly, you'll now need to have the coverage module installed in order to run the tests.

Running

nosetests tests

will now automatically generate a file .coverage in the root directory of the repo, containing coverage data.

To compile this file into user friendly html, run

coverage html

from the root directory.

I've also added a flag variable to autograd/util.py to enable more comprehensive coverage checking.

[j-towns]

I'm thinking about doing another 'spring cleaning' pr tidying up the tests a bit.

I think the contents of test_binary_ops.py, test_numpy.py and test_scalar_ops.py should probably all be in one file. Then there are quite a few tests that are effectively duplicated within those three files, and tests which are only testing scalar functionality (in test_scalar_ops) which should be using unary_ufunc_check or binary_ufunc_check.

[j-towns]

For example, arcsin, arccos, and arctan are currently only tested on scalars but should be tested as unary_ufuncs.

[coveralls]

Changes Unknown when pulling 8c3e4e5 on j-towns:coverage into ** on HIPS:master**.

[j-towns]

I've disabled those autogenerated coveralls comments.

[mattjj]

Wow, awesome work here. Improving the tests and estimating coverage would be really valuable. The tests have saved me from committing bad or incomplete changes on many occasions.

Is that estimate of the total test coverage accurate? I thought we had tests for pretty much every vjp and the core functionality as well, but maybe this total is including other code.

[j-towns]

Yeah you're right, it's not accurate at all. It only reflects code which is run by the check_fun_and_grads function during testing. Many tests use check_grads or instead. I thought that adding coverage checking to check_grads by default would be a mistake because it would add dependency on coverage.py to Autograd itself (as opposed to just the tests - note the different locations of check_grads and check_fun_and_grads). Maybe this distinction isn't important and coverage should be on in check_grads by default. You're correct about coverage - very nearly all the vjps appear to be touched by the tests, the exceptions are things like conjugate, which is just an alias for conj. However as I pointed out above some of the ufuncs are being tested by (presumably old) scalar only tests.

…

On Wed, 10 May 2017 at 18:00, Matthew Johnson ***@***.***> wrote: Wow, awesome work here. Improving the tests and estimating coverage would be really valuable. The tests have saved me from committing bad or incomplete changes on many occasions. Is that estimate of the total test coverage accurate? I thought we had tests for pretty much every vjp and the core functionality as well, but maybe this total is including other code. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOjgu56Pb-TDH7uLMeIu07Kf3LmQ1hS-ks5r4e03gaJpZM4NWtCK> .

[j-towns]

Maybe one solution would be to move check_grads from autograd/util.py into tests/numpy_utils.py, and turn on coverage checking there by default. Then you could keep an alias to check_grads in util.py for backward compatibility, which, when called, would have coverage checking turned off by default. Perhaps numpy_utils.py could then be renamed to testing_utils.py as well? On Wed, 10 May 2017 at 19:32, Jamie Townsend <jamiehntownsend@gmail.com> wrote:

…

Yeah you're right, it's not accurate at all. It only reflects code which is run by the check_fun_and_grads function during testing. Many tests use check_grads or instead. I thought that adding coverage checking to check_grads by default would be a mistake because it would add dependency on coverage.py to Autograd itself (as opposed to just the tests - note the different locations of check_grads and check_fun_and_grads). Maybe this distinction isn't important and coverage should be on in check_grads by default. You're correct about coverage - very nearly all the vjps appear to be touched by the tests, the exceptions are things like conjugate, which is just an alias for conj. However as I pointed out above some of the ufuncs are being tested by (presumably old) scalar only tests. On Wed, 10 May 2017 at 18:00, Matthew Johnson ***@***.***> wrote: > Wow, awesome work here. Improving the tests and estimating coverage would > be really valuable. The tests have saved me from committing bad or > incomplete changes on many occasions. > > Is that estimate of the total test coverage accurate? I thought we had > tests for pretty much every vjp and the core functionality as well, but > maybe this total is including other code. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#224 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AOjgu56Pb-TDH7uLMeIu07Kf3LmQ1hS-ks5r4e03gaJpZM4NWtCK> > . >

[mattjj]

I like the suggestions in your most recent comment (wow, has it already been 25 days?). I think it is important to have check_grads accessible from the autograd package, since it's such a handy function when writing custom VJPs or even when convincing yourself that autograd is doing the right thing on your code. However, I'm not sure how to make an alias to tests/numpy_utils.py:check_grads without also including the tests directory in the autograd package.

Maybe we can have an alias of check_grads in numpy_utils.py somehow, with the definition still in the autograd package?

[j-towns]

Yeah actually doing it that way around makes more sense given what you've pointed out about the packages.

I'll have a go at implementing that when I've got time (am pretty busy at the moment).