This repository contains resources for finding the best model with multiple model comparison. The algorithm is described in our paper,
Kernel Stein Tests for Multiple Model Comparison
Jen Ning Lim, Makoto Yamada, Bernhard Schölkopf, Wittawat Jitkrittum
NeurIPS 2019
Requires numpy, autograd, matplotlib and SciPy.
The package can be installed
with pip command.
pip install -e /path/to/the/folder/of/this/repo/after/clone
Or alternatively,
pip install git+https://github.com/jenninglim/model-comparison-test.git
Once installed, you should be able to do import reltest without any error.
See notebooks/demo_reltest.ipynb.
See reproduce-results.
The current implementation relies on an accurate approximation of the inverse CDF in the tail regions of the truncated normal (see 6). The implementation uses the inverse CDF to calculate the rejection threshold. If the test statistic is greater than the rejection threshold, we reject the null hypothesis.
The p-values produced from the modules may not be accurate. At the time of
writting this code there were several problems with
the scipy.stats.truncnorm module. See
1,
2,
3 and
4. This may have been fixed in
the pull request 5 in SciPy 1.4.0.