Skip to content

ngmarchant/oasis

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

46 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OASIS

image

image

image

OASIS is a tool for evaluating binary classifiers when ground truth class labels are not immediately available, but can be obtained at some cost (e.g. by asking humans). The tool takes an unlabelled test set as input and intelligently selects items to label so as to provide a precise estimate of the classifier's performance, whilst minimising the amount of labelling required. The underlying strategy for selecting the items to label is based on a technique called adaptive importance sampling, which is optimised for the classifier performance measure of interest. Currently, OASIS supports estimation of the weighted F-measure, which includes the F1-score, precision and recall.

Documentation: https://ngmarchant.github.io/oasis

Source: https://www.github.com/ngmarchant/oasis

Technical paper: https://arxiv.org/pdf/1703.00617.pdf

Example

See the Jupyter notebook under docs/tutorial/tutorial.ipynb:

>>> import oasis
>>> data = oasis.Data()
>>> data.read_h5('Amazon-GoogleProducts-test.h5')
>>> def oracle(idx):
>>>     return data.labels[idx]
>>> smplr = oasis.OASISSampler(alpha, data.preds, data.scores, oracle)
>>> smplr.sample_distinct(5000) #: query labels for 5000 distinct items
>>> print("Current estimate is {}.".format(smplr.estimate_[smplr.t_ - 1]))

License and disclaimer

The code is released under the MIT license. Please see the LICENSE file for details.

About

A Python package for efficient evaluation based on OASIS (Optimal Asymptotic Sequential Importance Sampling).

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages