tutorial_multiclass_hyp.ipynb

{
  "metadata": {
    "name": "Part X: Hypothesis testing"
  }, 
  "nbformat": 3, 
  "nbformat_minor": 0, 
  "worksheets": [
    {
      "cells": [
        {
          "cell_type": "heading", 
          "level": 1, 
          "metadata": {}, 
          "source": [
            "Part X: Hypothesis testing"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Let's revisit the classification problem from ", 
            "*the chapter on classifiers*."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "from mvpa2.tutorial_suite import *\n", 
            "ds = get_haxby2001_data_alternative(roi='vt', grp_avg=False)\n", 
            "print ds.sa['targets'].unique"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf = kNN(k=1, dfx=one_minus_correlation, voting='majority')\n", 
            "cv = CrossValidation(clf, NFoldPartitioner(), errorfx=mean_mismatch_error,\n                     enable_ca=['stats'])\n", 
            "cv_results = cv(ds)\n", 
            "print '%.2f' % np.mean(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "So here we have an 8-way classification problem, and during the cross-validation\nprocedure the chosen classifiers makes correct predictions for approximately\nhalf of the data points. The big question is now: ", 
            "**What does that tell us?**\n\n", 
            "There are many scenarios that could lead to this prediction performance. It\ncould be that the fitted classifier model is very good, but only capture the\ndata variance for half of the data categories/classes. It could also be that\nthe classifier model quality is relatively poor and makes an equal amount of\nerrors for all classes. In both cases the average accuracy will be around 50%,\nand most likely ", 
            "**highly significant**, given a chance performance of 1/8.  We\ncould now spend some time testing this significance with expensive permutation\ntests, or making assumptions on the underlying distribution. However, that\nwould only give us a number telling us that the average accuracy is really\ndifferent from chance, but it doesn't help with the problem that the accuracy\nreally doesn't tell us much about what we are interested in.\n\n", 
            "Interesting hypotheses in the context of this dataset could be whether the data\ncarry a signal that can be used to distinguish brain response patterns from\nanimate vs.  inanimate stimulus categories, or whether data from object-like\nstimuli are all alike and can only be distinguished from random noise, etc. One\ncan imagine to run such an analysis on data from different parts of the brain\nand the results change -- without necessarily having a big impact on the\noverall classification accuracy.\n\n", 
            "A lot more interesting information is available from the confusion matrix, a\ncontingency table showing prediction targets vs. actual predictions."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cv.ca.stats.matrix"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "We can see a strong diagonal, but also block-like structure, and have to\nrealize that simply staring at the matrix doesn't help us to easily assess the\nlikelihood of any of our hypothesis being true or false. It is trivial to do a\nChi-square test of the confusion table..."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print 'Chi^2: %.3f (p=%.3f)' % cv.ca.stats.stats[\"CHI^2\"]"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "... but, again, it doesn't tell us anything other than the classifier not just\ndoing random guesses. It would be much more useful, if we could estimate how\nlikely it is, given the observed confusion matrix, that the employed classifier\nis able to discriminate ", 
            "*all* stimulus classes from each other, and not just a\nsubset. Even more useful would be, if we could relate this probability to\nspecific alternative hypotheses, such as an animate/inanimate-only distinction.\n\n", 
            "*Olivetti et al. (2012)* have devised a method that allows for\ndoing exactly that. The confusion matrix is analyzed in a Bayesian framework\nregarding the statistical dependency of observed and predicted class labels.\nConfusions within a set of classes that cannot be discriminated should be\nindependently distributed, while there should be a statistical dependency of\nconfusion patterns within any set of classes that can all be discriminated from\neach other.\n\n", 
            "This algorithm is available in the\n", 
            "[None](http://pymvpa.org/generated/mvpa2.clfs.transerror.BayesConfusionHypothesis.html#mvpa2-clfs-transerror-bayesconfusionhypothesis) node."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "cv = CrossValidation(clf, NFoldPartitioner(),\n                     errorfx=None,\n                     postproc=ChainNode((Confusion(labels=ds.UT),\n                                         BayesConfusionHypothesis())))"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Most likely hypothesis to explain this confusion matrix\n\n", 
            "print cv_results.sa.hypothesis[np.argsort(cv_results.samples[:,1])[-1]]\n[['bottle'], ['cat'], ['chair'], ['face'], ['house'], ['scissors'], ['scrambledpix'], ['shoe']]"
          ]
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "References"
          ]
        }
      ], 
      "metadata": {}
    }
  ]
}