tutorial_classifiers.ipynb

{
  "metadata": {
    "name": "Classifiers -- All Alike, Yet Different"
  }, 
  "nbformat": 3, 
  "nbformat_minor": 0, 
  "worksheets": [
    {
      "cells": [
        {
          "cell_type": "heading", 
          "level": 1, 
          "metadata": {}, 
          "source": [
            "Classifiers -- All Alike, Yet Different"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "In this chapter we will continue our work from ", 
            "*chap_tutorial_mappers* in\norder to replicate the study of ", 
            "*Haxby et al. (2001)*. For this\ntutorial there is a little helper function to yield the dataset we generated\nmanually before:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "from mvpa2.tutorial_suite import *\n", 
            "ds = get_haxby2001_data()"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "The original study employed a so-called 1-nearest-neighbor classifier, using\ncorrelation as a distance measure. In PyMVPA this type of classifier is\nprovided by the ", 
            "[kNN](http://pymvpa.org/generated/mvpa2.clfs.knn.kNN.html#mvpa2-clfs-knn-knn) class, that makes it possible to specify\nthe desired parameters."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf = kNN(k=1, dfx=one_minus_correlation, voting='majority')"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "A k-Nearest-Neighbor classifier performs classification based on the similarity\nof a sample with respect to each sample in a ", 
            "[training dataset](http://pymvpa.org/glossary.html#term-training-dataset).  The\nvalue of `k` specifies the number of neighbors to derive a prediction,\n`dfx` sets the distance measure that determines the neighbors, and `voting`\nselects a strategy to choose a single label from the set of targets assigned to\nthese neighbors."
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n**Exercise**", 
            "\n\n", 
            "Access the built-in help to inspect the `kNN` class regarding additional\nconfiguration options."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "# you can use this cell for this exercise"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n", 
            "\n\n", 
            "Now that we have a classifier instance, it can be easily trained by passing the\ndataset to its `train()` method."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf.train(ds)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "A trained classifier can subsequently be used to perform classification of\nunlabeled samples. The classification performance can be assessed by comparing\nthese predictions to the target labels."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "predictions = clf.predict(ds.samples)\n", 
            "np.mean(predictions == ds.sa.targets)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "We see that the classifier performs remarkably well on our dataset -- it\ndoesn't make even a single prediction error. However, most of the time we would\nnot be particularly interested in the prediction accuracy of a classifier on the\nsame dataset that it got trained with."
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n**Exercise**", 
            "\n\n", 
            "Think about why this particular classifier will always perform error-free\nclassification of the training data -- regardless of the actual dataset\ncontent. If the reason is not immediately obvious, take a look at chapter\n13.3 in ", 
            "*The Elements of Statistical Learning*. Investigate how\nthe accuracy varies with different values of `k`. Why is that?"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "# you can use this cell for this exercise"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n", 
            "\n\n", 
            "Instead, we are interested in the generalizability of the classifier on new,\nunseen data. This would allow us, in principle, to use it to assign labels to\nunlabeled data. Because we only have a single dataset, it needs to be split\ninto (at least) two parts to achieve this. In the original study, Haxby and\ncolleagues split the dataset into patterns of activations from odd versus\neven-numbered runs. Our dataset has this information in the `runtype` sample\nattribute:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print ds.sa.runtype"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Using this attribute we can now easily split the dataset in half. PyMVPA\ndatasets can be sliced in similar ways as ", 
            "[NumPy](http://numpy.scipy.org)'s ", 
            "`ndarray`. The following\ncalls select the subset of samples (i.e. rows in the datasets) where the value\nof the `runtype` attribute is either the string 'even' or 'odd'."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "ds_split1 = ds[ds.sa.runtype == 'odd']\n", 
            "len(ds_split1)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "ds_split2 = ds[ds.sa.runtype == 'even']\n", 
            "len(ds_split2)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Now we could repeat the steps above: call `train()` with one dataset half\nand `predict()` with the other, and compute the prediction accuracy\nmanually.  However, a more convenient way is to let the classifier do this for\nus.  Many objects in PyMVPA support a post-processing step that we can use to\ncompute something from the actual results. The example below computes the\n", 
            "*mismatch error* between the classifier predictions and the ", 
            "*target* values\nstored in our dataset. To make this work, we do not call the classifier's\n`predict()` method anymore, but \"call\" the classifier directly with the test\ndataset. This is a very common usage pattern in PyMVPA that we shall see a lot\nover the course of this tutorial.  Again, please note that we compute an error\nnow, hence lower values represent more accurate classification."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf.set_postproc(BinaryFxNode(mean_mismatch_error, 'targets'))\n", 
            "clf.train(ds_split2)\n", 
            "err = clf(ds_split1)\n", 
            "print np.asscalar(err)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "In this case, our choice of which half of the dataset is used for training and\nwhich half for testing was completely arbitrary, hence we could also estimate\nthe transfer error after swapping the roles:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf.train(ds_split1)\n", 
            "err = clf(ds_split2)\n", 
            "print np.asscalar(err)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "We see that on average the classifier error is really low, and we achieve an\naccuracy level comparable to the results reported in the original study."
          ]
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "Cross-validation"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "What we have just done was to manually split the dataset into combinations of\ntraining and testing datasets, given a specific sample attribute -- in this\ncase whether a ", 
            "*pattern of activation* or ", 
            "[sample](http://pymvpa.org/glossary.html#term-sample) came from ", 
            "*even* or\n", 
            "*odd* runs.  We ran the classification analysis on each split to estimate the\nperformance of the classifier model. In general, this approach is called\n", 
            "[cross-validation](http://pymvpa.org/glossary.html#term-cross-validation), and involves splitting the dataset into multiple\npairs of subsets, choosing sample groups by some criterion, and estimating the\nclassifier performance by training it on the first dataset in a split and\ntesting against the second dataset from the same split.\n\n", 
            "PyMVPA provides a way to allow complete cross-validation procedures to run\nfully automatic, without the need for manual splitting of a dataset. Using the\n", 
            "[CrossValidation](http://pymvpa.org/generated/mvpa2.measures.base.CrossValidation.html#mvpa2-measures-base-crossvalidation) class, a cross-validation is set up by\nspecifying what measure should be computed on each dataset split and how\ndataset splits should be generated. The measure that is usually computed is the\ntransfer error that we already looked at in the previous section. The second\nelement, a ", 
            "[generator](http://pymvpa.org/glossary.html#term-generator) for datasets, is another very common tool in\nPyMVPA. The following example uses\n", 
            "[HalfPartitioner](http://pymvpa.org/generated/mvpa2.generators.partition.HalfPartitioner.html#mvpa2-generators-partition-halfpartitioner), a generator that, when called\nwith a dataset, marks all samples regarding their association with the first or\nsecond half of the dataset. This happens based on the values of a specified\nsample attribute -- in this case `runtype` -- much like the manual dataset\nsplitting that we have performed earlier.\n", 
            "[HalfPartitioner](http://pymvpa.org/generated/mvpa2.generators.partition.HalfPartitioner.html#mvpa2-generators-partition-halfpartitioner) will make sure to subsequently\nassign samples to both halves, i.e. samples from the first half in the first\ngenerated dataset will be in the second half of the second generated dataset.\nWith these two techniques we can replicate our manual cross-validation easily\n-- reusing our existing classifier, but without the custom post-processing\nstep."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf.set_postproc(None)\n", 
            "hpart = HalfPartitioner(attr='runtype')\n", 
            "cv = CrossValidation(clf, hpart)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n**Exercise**", 
            "\n\n", 
            "Try calling the `hpart` object with our dataset. What happens? Now try\npassing the dataset to its `generate()` method. What happens now?\nMake yourself familiar with the concept of a Python generator. Investigate\nwhat the code snippet `list(xrange(5))` does, and try to adapt it to the\n`HalfPartitioner`."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "# you can use this cell for this exercise"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "- - -\n", 
            "\n\n", 
            "Once the `cv` object is created, it can be called with a dataset, just like\nwe did with the classifier before. It will internally perform all the dataset\npartitioning, split each generated dataset into training and testing sets\n(based on the partitions), and train and test the classifier repeatedly.\nFinally, it will return the results of all cross-validation folds."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "cv_results = cv(ds)\n", 
            "np.mean(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Actually, the cross-validation results are returned as another dataset that has\none sample per fold and a single feature with the computed transfer-error per\nfold."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "len(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "cv_results.samples"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "Any classifier, really"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "A short summary of all code for the analysis we developed so far is this:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf = kNN(k=1, dfx=one_minus_correlation, voting='majority')\n", 
            "cvte = CrossValidation(clf, HalfPartitioner(attr='runtype'))\n", 
            "cv_results = cvte(ds)\n", 
            "np.mean(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Looking at this little code snippet we can nicely see the logical parts of\na cross-validated classification analysis.", 
            "\n\n1. ", 
            "Load the data", 
            "\n\n2. ", 
            "Choose a classifier", 
            "\n\n3. ", 
            "Set up an error function", 
            "\n\n4. ", 
            "Evaluate the error in a cross-validation procedure", 
            "\n\n5. ", 
            "Inspect results", 
            "\n\n", 
            "Our previous choice of the classifier was guided by the intention to replicate\n", 
            "*Haxby et al. (2001)*, but what if we want to try a different\nalgorithm? In this case another nice feature of PyMVPA comes into play. All\nclassifiers implement a common interface that makes them easily interchangeable\nwithout the need to adapt any other part of the analysis code.  If, for\nexample, we want to try the popular ", 
            "[svm](http://pymvpa.org/generated/mvpa2.clfs.svm.html#mvpa2-clfs-svm) (Support Vector\nMachines) on our example dataset it looks like this:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "clf = LinearCSVMC()\n", 
            "cvte = CrossValidation(clf, HalfPartitioner(attr='runtype'))\n", 
            "cv_results = cvte(ds)\n", 
            "np.mean(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Instead of k-nearest-neighbor, we create a linear SVM classifier,\ninternally using the popular LIBSVM library (note that PyMVPA provides\nadditional SVM implementations). The rest of the code remains identical.\nSVM with its default settings seems to perform slightly worse than the\nsimple kNN-classifier. We'll get back to the classifiers shortly. Let's\nfirst look at the remaining part of this analysis.\n\n", 
            "We already know that ", 
            "[CrossValidation](http://pymvpa.org/generated/mvpa2.measures.base.CrossValidation.html#mvpa2-measures-base-crossvalidation) can be used to\ncompute errors. So far we have only used the mean number of mismatches between\nactual targets and classifier predictions as the error function (which is the\ndefault).  However, PyMVPA offers a number of alternative functions in the\n", 
            "[errorfx](http://pymvpa.org/generated/mvpa2.misc.errorfx.html#mvpa2-misc-errorfx) module, but it is also trivial to specify custom\nones.  For example, if we do not want to have error reported, but instead\naccuracy, we can do that:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "cvte = CrossValidation(clf, HalfPartitioner(attr='runtype'),\n                       errorfx=lambda p, t: np.mean(p == t))\n", 
            "cv_results = cvte(ds)\n", 
            "np.mean(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "This example reuses the SVM classifier we have create before, and\nyields exactly what we expect from the previous result.\n\n", 
            "The details of the cross-validation procedure are also heavily\ncustomizable. We have seen that a ", 
            "[Partitioner](http://pymvpa.org/generated/mvpa2.generators.partition.Partitioner.html#mvpa2-generators-partition-partitioner) is\nused to generate training and testing dataset for each cross-validation\nfold. So far we have only used ", 
            "[HalfPartitioner](http://pymvpa.org/generated/mvpa2.generators.partition.HalfPartitioner.html#mvpa2-generators-partition-halfpartitioner) to\ndivide the dataset into odd and even runs (based on our custom sample\nattribute `runtype`). However, in general it is more common to perform so\ncalled leave-one-out cross-validation, where ", 
            "*one* independent part of a\ndataset is selected as testing dataset, while the other parts constitute the\ntraining dataset. This procedure is repeated till all parts have served as\nthe testing dataset once. In case of our dataset we could consider each of\nthe 12 runs as independent measurements (fMRI data doesn't allow us to\nconsider temporally adjacent data to be considered independent).\n\n", 
            "To run such an analysis, we first need to redo our dataset preprocessing,\nas, in the current one, we only have one sample per stimulus category for\nboth odd and even runs. To get a dataset with one sample per stimulus\ncategory for each run, we need to modify the averaging step. Using what we\nhave learned from the ", 
            "*last tutorial part* the\nfollowing code snippet should be plausible:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "datapath = os.path.join(tutorial_data_path, 'haxby2001')\n", 
            "ds = load_tutorial_data(roi='vt')\n", 
            "poly_detrend(ds, polyord=1, chunks_attr='chunks')\n", 
            "zscore(ds, param_est=('targets', ['rest']))\n", 
            "ds = ds[ds.sa.targets != 'rest']\n", 
            "run_averager = mean_group_sample(['targets', 'chunks'])\n", 
            "ds = ds.get_mapped(run_averager)\n", 
            "ds.shape"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Instead of two samples per category in the whole dataset, now we have one\nsample per category, per experiment run, hence 96 samples in the whole\ndataset. To set up a 12-fold leave-one-run-out cross-validation, we can\nmake use of ", 
            "[NFoldPartitioner](http://pymvpa.org/generated/mvpa2.generators.partition.NFoldPartitioner.html#mvpa2-generators-partition-nfoldpartitioner). By default it is\ngoing to select samples from one `chunk` at a time:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "cvte = CrossValidation(clf, NFoldPartitioner(),\n                       errorfx=lambda p, t: np.mean(p == t))\n", 
            "cv_results = cvte(ds)\n", 
            "np.mean(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "We get almost the same prediction accuracy (reusing the SVM classifier and\nour custom error function). Note that this time we performed the analysis on\na lot more samples that were each was computed from just a few fMRI volumes\n(about nine each).\n\n", 
            "So far we have just looked at the mean accuracy or error. Let's investigate\nthe results of the cross-validation analysis a bit further."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "type(cv_results)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cv_results.samples"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "The returned value is actually a ", 
            "[Dataset](http://pymvpa.org/generated/mvpa2.datasets.base.Dataset.html#mvpa2-datasets-base-dataset) with the\nresults for all cross-validation folds. Since our error function computes\nonly a single scalar value for each fold the dataset only contains a single\nfeature (in this case the accuracy), and a sample per each fold."
          ]
        }, 
        {
          "cell_type": "heading", 
          "level": 2, 
          "metadata": {}, 
          "source": [
            "We Need To Take A Closer Look"
          ]
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "By now we have already done a few cross-validation analyses using two\ndifferent classifiers and different pre-processing strategies. In all these\ncases we have just looked at the generalization performance or error.\nHowever, error rates hide a lot of interesting information that is very\nimportant for an interpretation of results. In our case we analyzed a\ndataset with eight different categories. An average misclassification rate\ndoesn't tell us much about the contribution of each category to the\nprediction error. It could be that ", 
            "*half of the samples of each category*\nget misclassified, but the same average error might be due to ", 
            "*all samples\nfrom half of the categories* being completely misclassified, while\nprediction accuracy for samples from the remaining categories is perfect.\nThese two results would have to be interpreted in totally different ways,\ndespite the same average error rate.\n\n", 
            "In psychological research this type of results is usually presented as a\n", 
            "[contingency table](http://en.wikipedia.org/wiki/Contingency_table) or ", 
            "[cross tabulation](http://en.wikipedia.org/wiki/Cross_tabulation) of expected vs. empirical\nresults. ", 
            "[Signal detection theory](http://en.wikipedia.org/wiki/Detection_theory) offers a whole range of techniques to\ncharacterize such results. From this angle a\nclassification analysis is hardly any different from a psychological\nexperiment where a human observer performs a detection task, hence the same\nanalysis procedures can be applied here as well.\n\n", 
            "PyMVPA provides convenient access to\n", 
            "[confusion matrices](http://pymvpa.org/glossary.html#term-confusion-matrix), i.e.  contingency tables of\ntargets vs. actual predictions.  However, to prevent wasting CPU-time and\nmemory they are not computed by default, but instead have to be enabled\nexplicitly. Optional analysis results like this are available in a dedicated\ncollection of ", 
            "[conditional attribute](http://pymvpa.org/glossary.html#term-conditional-attribute)s -- analogous to `sa` and\n`fa` in datasets, it is named `ca`. Let's see how it works:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "cvte = CrossValidation(clf, NFoldPartitioner(),\n                       errorfx=lambda p, t: np.mean(p == t),\n                       enable_ca=['stats'])\n", 
            "cv_results = cvte(ds)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "Via the `enable_ca` argument we triggered computing confusion tables for\nall cross-validation folds, but otherwise there is no change in the code.\nAfterwards the aggregated confusion for the whole cross-validation\nprocedure is available in the `ca` collection. Let's take a look (note\nthat in the printed manual the output is truncated due to page-width\nconstraints -- please refer to the HTML-based version full the full matrix)."
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cvte.ca.stats.as_string(description=True)"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "This output is a comprehensive summary of the performed analysis. We can\nsee that the confusion matrix has a strong diagonal, and confusion happens\nmostly among small objects. In addition to the plain contingency table\nthere are also a number of useful summary statistics readily available --\nincluding average accuracy.\n\n", 
            "Especially for multi-class datasets the matrix quickly becomes\nincomprehensible. For these cases the confusion matrix can also be plotted\nvia its ", 
            "[plot()](http://pymvpa.org/generated/mvpa2.clfs.transerror.ConfusionMatrix.plot.html#mvpa2-clfs-transerror-confusionmatrix-plot) method. If the\nconfusions shall be used as input for further processing they can also be\naccessed in pure matrix format:"
          ]
        }, 
        {
          "cell_type": "code", 
          "collapsed": false, 
          "input": [
            "print cvte.ca.stats.matrix"
          ], 
          "language": "python", 
          "metadata": {}, 
          "outputs": []
        }, 
        {
          "cell_type": "markdown", 
          "metadata": {}, 
          "source": [
            "\n\n", 
            "The classifier confusions are just an example of the general mechanism of\nconditional attribute that is supported by many objects in PyMVPA."
          ]
        }
      ], 
      "metadata": {}
    }
  ]
}