Testing model implementations

[pminervini]

Write unit tests inspired by "How to unit test machine learning code" [1]

[1] https://medium.com/@keeper6928/how-to-unit-test-machine-learning-code-57cf6fd81765

[pminervini]

@dirkweissenborn I would really like to sysematically test our models with the following two functions:

https://github.com/uclmr/jack/blob/master/jack/debug/base.py

One checks whether all layers are updated during training, and the other verifies that the loss is never zero.

How would you proceed for implementing the tests? (The first one seems especially useful)

[dirkweissenborn]

Sounds like a good idea! I don't really understand though what you mean by implementing the tests? Maybe we need a registry for models with a corresponding default dataset. For those we load a small part of the dataset (or the whole dataset and then just cut the list of QASettings), run the readers train function and compare parameters before and after. That requires you to call setup from data before the train method (otherwise you cannot get the initial parameters), but this is fine, since a flag in train checks whether the reader is already setup.

[pminervini]

@dirkweissenborn no I mean - I just need the training operator (train_op) and a feed_dict: where could I get those?

Or maybe I can just train in the normal way, and check before and after whether all layers got updated - what do you think ?

[dirkweissenborn]

it is easier if you just use the train method of the reader. You can also look at the train method to figure out how to train a TF reader, i.e. create feed deict etc. Pasquale Minervini <notifications@github.com> schrieb am Mo., 30. Okt. 2017 um 20:24 Uhr:

…

@dirkweissenborn <https://github.com/dirkweissenborn> no I mean - I just need the training operator (train_op) and a feed_dict: where could I get those? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#318 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABU6G3aBkmNYyrp7aU4TDOmiK3RNjKDhks5sxiJlgaJpZM4QKYr3> .

[riedelcastro]

Maybe worthwhile meeting for a chat between you two to clarify some of these questions? Github issues aren't the most effective way to communicate...