baseline_nb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Baseline Model\n",
    "## Game of Thrones Text Classification\n",
    "### T. P. Goter\n",
    "### W266 Final Project\n",
    "### Fall 2019\n",
    "\n",
    "This notebook is used to generate a Naive Bayes model for text classification using training data generated for the Game of Thrones novel. A simple GridSearch with cross validation is done for each model in order to empirically determine the best parameter of for smoothing (i.e., alpha). Both unigram and bigram models are considered. Additionally, consideration is given to a tfidf vectorication vice a simple word count vectorization. The results of these studies show that all four models perform about the same and have precision, recall and f1 scores near 0.70. Class 5 (i.e., Book 5) shows as the most easily predicted. This isn't really surprising given books four and five were focused on a subset of the characters. Thus, word counts of character names likely are better indicators for these books. With 70% accuracy as our baseline score on the development set, there is plenty of room for further improvement with BERT and the UDA techniques."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This tells matplotlib not to try opening a new window for each plot.\n",
    "%matplotlib inline\n",
    "\n",
    "# General libraries.\n",
    "import re\n",
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "# SK-learn libraries for learning.\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# SK-learn libraries for evaluation.\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# SK-learn libraries for feature extraction from text.\n",
    "from sklearn.feature_extraction.text import *\n",
    "\n",
    "# import the tokenization module\n",
    "from utils import tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "For consistency between our BERT, XLNet and baseline models, we want to use the same subsets of training examples. We do this by reading in the already pre-processed datasets that were generated for the BERT evaluations. As such these examples are already tokenized. For a Naive-Bayes classifier, this should not be an issue, but it is something to be aware of."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['train_20', 'train_200', 'train_2000', 'train_5000', 'train_12000', 'dev', 'test'])\n"
     ]
    }
   ],
   "source": [
    "# Read the data into dataframes stored in a dictionary\n",
    "dfs = {}\n",
    "for case in 'train dev test'.split():\n",
    "    if case == 'train':\n",
    "        for exs in '20 200 2000 5000 12000'.split():\n",
    "            case = 'train_{}'.format(exs)\n",
    "            dfs[case] = pd.read_pickle(os.path.join('Data/proc_data/GoT',case,case + '.pkl'))\n",
    "    else:\n",
    "        dfs[case] = pd.read_pickle(os.path.join('Data/proc_data/GoT',case,case + '.pkl'))\n",
    "\n",
    "print(dfs.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_basic_model(df_train, df_dev, bigram=False, tfidf=False, tokenize=False):\n",
    "    '''\n",
    "    Function to train and evalute a multinomial naive bayes model.\n",
    "    :param: bigram: Boolean, use unigram and bigrams\n",
    "    :param: tfidf: Boolean, use tfidf weighting during feature vectorization\n",
    "    :param: tokenize: Boolean, use WordPiece tokenization with BERT input vocabulary\n",
    "    '''\n",
    "    # Set up a range of alphas to test\n",
    "    alphas = {'alpha': [0.0001, 0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 10.0]}\n",
    "     \n",
    "    # Get data and labels from dataframe   \n",
    "    train_data = df_train.seq\n",
    "    train_y = df_train.label\n",
    "    dev_data = df_dev.seq\n",
    "    dev_y = df_dev.label\n",
    "    \n",
    "    # Instantiate the count vectorizer\n",
    "    if bigram:\n",
    "        if tfidf:\n",
    "            vectorizer = TfidfVectorizer(ngram_range=(1,2))\n",
    "        else:\n",
    "            vectorizer = CountVectorizer(ngram_range=(1,2))\n",
    "    else:\n",
    "        if tfidf:\n",
    "            vectorizer = TfidfVectorizer()\n",
    "        else:\n",
    "            vectorizer = CountVectorizer()\n",
    "    \n",
    "    # Generate the Feature Vectors\n",
    "    train_X = vectorizer.fit_transform(train_data)    \n",
    "    \n",
    "    # Generate the vocabulary for the dev data\n",
    "    dev_X = vectorizer.transform(dev_data)\n",
    "       \n",
    "    # Fit the model\n",
    "    clf = MultinomialNB()\n",
    "    \n",
    "    # Run a grid search over alpha (smoothing) values to determine best \n",
    "    gs_clf = GridSearchCV(clf, param_grid=alphas, cv=4, return_train_score=True)\n",
    "    gs_clf.fit(train_X, train_y)\n",
    "    \n",
    "    # Display the best parameter\n",
    "    print(50 * \"=\")\n",
    "    print(\"The best alpha value was determined to be {}\".format(gs_clf.best_params_['alpha']))\n",
    "    print(50 * \"=\")\n",
    "\n",
    "    # Let's make some predictions using the best classifier\n",
    "    y_pred = gs_clf.best_estimator_.predict(dev_X)\n",
    "    \n",
    "    # Calculate Accuracy and Error Rate\n",
    "    acc = sum(dev_y==y_pred)/len(dev_y)\n",
    "    err_rate = 1- acc\n",
    "       \n",
    "    print(\"Accuracy: {:0.3f}\".format(acc))\n",
    "    print(50 * \"=\")\n",
    "    print(classification_report(y_pred, dev_y))\n",
    "    print(confusion_matrix(dev_y, y_pred))\n",
    "    \n",
    "    return err_rate\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Unigram Model\n",
    "\n",
    "Metrics we will consider are precision, recall, and F1-score. Remember that:\n",
    "\n",
    "- Precision: Number of items labeled as class A that are actually class A. - How many false positives?\n",
    "- Recall: Number of items labeled as class A normalized to all things that are class A. - How many false negatives?\n",
    "- F1 Score: Harmonic mean of Precision and Recall\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 1.0\n",
      "==================================================\n",
      "Accuracy: 0.665\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.79      0.62      0.70       536\n",
      "           2       0.53      0.61      0.57       426\n",
      "           3       0.56      0.64      0.60       520\n",
      "           4       0.78      0.65      0.71       493\n",
      "           5       0.70      0.78      0.74       526\n",
      "\n",
      "   micro avg       0.66      0.66      0.66      2501\n",
      "   macro avg       0.67      0.66      0.66      2501\n",
      "weighted avg       0.68      0.66      0.67      2501\n",
      "\n",
      "[[335  36  29  13  11]\n",
      " [ 92 261  71  38  27]\n",
      " [ 70  84 333  55  50]\n",
      " [ 11  16  37 321  26]\n",
      " [ 28  29  50  66 412]]\n"
     ]
    }
   ],
   "source": [
    "uni_error = create_basic_model(dfs['train_12000'], dfs['dev'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bigram Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 0.5\n",
      "==================================================\n",
      "Accuracy: 0.705\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.83      0.65      0.72       542\n",
      "           2       0.59      0.66      0.63       439\n",
      "           3       0.61      0.72      0.66       500\n",
      "           4       0.84      0.67      0.75       514\n",
      "           5       0.72      0.83      0.77       506\n",
      "\n",
      "   micro avg       0.71      0.71      0.71      2501\n",
      "   macro avg       0.72      0.71      0.70      2501\n",
      "weighted avg       0.72      0.71      0.71      2501\n",
      "\n",
      "[[350  39  21   9   5]\n",
      " [ 95 290  54  32  18]\n",
      " [ 60  70 359  58  45]\n",
      " [ 13  17  16 346  19]\n",
      " [ 24  23  50  69 419]]\n"
     ]
    }
   ],
   "source": [
    "bi_error = create_basic_model(dfs['train_12000'], dfs['dev'], bigram=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    592\n",
       "5    585\n",
       "2    489\n",
       "1    424\n",
       "4    411\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfs['dev'].label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unigram Model with Tf-Idf Weighting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 0.5\n",
      "==================================================\n",
      "Accuracy: 0.656\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.80      0.61      0.69       559\n",
      "           2       0.52      0.62      0.57       410\n",
      "           3       0.58      0.63      0.60       545\n",
      "           4       0.76      0.64      0.70       489\n",
      "           5       0.67      0.79      0.73       498\n",
      "\n",
      "   micro avg       0.66      0.66      0.66      2501\n",
      "   macro avg       0.67      0.66      0.66      2501\n",
      "weighted avg       0.67      0.66      0.66      2501\n",
      "\n",
      "[[340  33  32  10   9]\n",
      " [100 254  74  37  24]\n",
      " [ 72  79 341  56  44]\n",
      " [ 17  17  36 313  28]\n",
      " [ 30  27  62  73 393]]\n"
     ]
    }
   ],
   "source": [
    "uni_tfidf_error = create_basic_model(dfs['train_12000'], dfs['dev'], tfidf=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bigram Model with Tf-Idf Weighting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 0.1\n",
      "==================================================\n",
      "Accuracy: 0.711\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.83      0.65      0.73       539\n",
      "           2       0.59      0.68      0.63       423\n",
      "           3       0.63      0.70      0.66       528\n",
      "           4       0.83      0.68      0.75       506\n",
      "           5       0.73      0.84      0.78       505\n",
      "\n",
      "   micro avg       0.71      0.71      0.71      2501\n",
      "   macro avg       0.72      0.71      0.71      2501\n",
      "weighted avg       0.73      0.71      0.71      2501\n",
      "\n",
      "[[351  34  26   8   5]\n",
      " [ 98 288  58  31  14]\n",
      " [ 58  67 371  55  41]\n",
      " [ 13  16  19 343  20]\n",
      " [ 19  18  54  69 425]]\n"
     ]
    }
   ],
   "source": [
    "bi_tfidf_error = create_basic_model(dfs['train_12000'], dfs['dev'], tfidf=True, bigram=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Evaluations\n",
    "Based on the simple model sensitivites performed above, we will use a bigram NaiveBayes model with a simple CountVectorizer as our baseline model. We will evaluate baseline performance on all subsets of training data and generate results based on evaluation on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 0.5\n",
      "==================================================\n",
      "Accuracy: 0.678\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.84      0.62      0.71       428\n",
      "           2       0.57      0.65      0.61       345\n",
      "           3       0.54      0.62      0.58       383\n",
      "           4       0.81      0.69      0.75       367\n",
      "           5       0.70      0.80      0.74       415\n",
      "\n",
      "   micro avg       0.68      0.68      0.68      1938\n",
      "   macro avg       0.69      0.68      0.68      1938\n",
      "weighted avg       0.70      0.68      0.68      1938\n",
      "\n",
      "[[265  25  19   3   2]\n",
      " [ 76 225  53  24  19]\n",
      " [ 41  63 239  46  50]\n",
      " [ 12  10  25 255  14]\n",
      " [ 34  22  47  39 330]]\n"
     ]
    }
   ],
   "source": [
    "bi_error_12000 = create_basic_model(dfs['train_12000'], dfs['test'], bigram=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 0.5\n",
      "==================================================\n",
      "Accuracy: 0.618\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.82      0.54      0.65       477\n",
      "           2       0.50      0.59      0.54       332\n",
      "           3       0.42      0.55      0.47       334\n",
      "           4       0.76      0.66      0.71       362\n",
      "           5       0.68      0.74      0.71       433\n",
      "\n",
      "   micro avg       0.62      0.62      0.62      1938\n",
      "   macro avg       0.63      0.62      0.62      1938\n",
      "weighted avg       0.65      0.62      0.62      1938\n",
      "\n",
      "[[258  29  18   7   2]\n",
      " [ 96 197  55  25  24]\n",
      " [ 63  69 183  52  72]\n",
      " [ 20  15  28 239  14]\n",
      " [ 40  22  50  39 321]]\n"
     ]
    }
   ],
   "source": [
    "bi_error_5000 = create_basic_model(dfs['train_5000'], dfs['test'], bigram=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 0.5\n",
      "==================================================\n",
      "Accuracy: 0.578\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.82      0.51      0.63       499\n",
      "           2       0.45      0.53      0.48       335\n",
      "           3       0.39      0.49      0.43       346\n",
      "           4       0.74      0.61      0.67       386\n",
      "           5       0.60      0.76      0.67       372\n",
      "\n",
      "   micro avg       0.58      0.58      0.58      1938\n",
      "   macro avg       0.60      0.58      0.58      1938\n",
      "weighted avg       0.62      0.58      0.58      1938\n",
      "\n",
      "[[256  27  22   6   3]\n",
      " [ 96 177  65  39  20]\n",
      " [ 72  79 170  65  53]\n",
      " [ 22  18  29 234  13]\n",
      " [ 53  34  60  42 283]]\n"
     ]
    }
   ],
   "source": [
    "bi_error_2000 = create_basic_model(dfs['train_2000'], dfs['test'], bigram=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 1.0\n",
      "==================================================\n",
      "Accuracy: 0.378\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.58      0.40      0.47       457\n",
      "           2       0.18      0.36      0.24       199\n",
      "           3       0.39      0.30      0.34       574\n",
      "           4       0.45      0.37      0.40       385\n",
      "           5       0.36      0.52      0.43       323\n",
      "\n",
      "   micro avg       0.38      0.38      0.38      1938\n",
      "   macro avg       0.39      0.39      0.37      1938\n",
      "weighted avg       0.42      0.38      0.39      1938\n",
      "\n",
      "[[181  17  72  25  19]\n",
      " [ 85  71 137  67  37]\n",
      " [ 84  54 170  76  55]\n",
      " [ 36  30  66 141  43]\n",
      " [ 71  27 129  76 169]]\n"
     ]
    }
   ],
   "source": [
    "bi_error_200 = create_basic_model(dfs['train_200'], dfs['test'], bigram=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==================================================\n",
      "The best alpha value was determined to be 0.5\n",
      "==================================================\n",
      "Accuracy: 0.253\n",
      "==================================================\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1       0.26      0.29      0.27       285\n",
      "           2       0.09      0.20      0.12       178\n",
      "           3       0.65      0.25      0.36      1161\n",
      "           4       0.15      0.21      0.17       218\n",
      "           5       0.09      0.44      0.15        96\n",
      "\n",
      "   micro avg       0.25      0.25      0.25      1938\n",
      "   macro avg       0.25      0.28      0.21      1938\n",
      "weighted avg       0.46      0.25      0.29      1938\n",
      "\n",
      "[[ 82  24 177  28   3]\n",
      " [ 58  35 235  51  18]\n",
      " [ 45  44 285  42  23]\n",
      " [ 32  32 196  46  10]\n",
      " [ 68  43 268  51  42]]\n"
     ]
    }
   ],
   "source": [
    "bi_error_20 = create_basic_model(dfs['train_20'], dfs['test'], bigram=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEPCAYAAABcA4N7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucXWV97/HPdwZChokxQiBSEoagwYrWo2QOErUQRSCTekArWii+QASDLWhVFEGqIuIRStWWSiVREbQKWkEbMOFygABVgSRWboFACE4SbuEihISBQOZ3/njWkM3O3mvWhH2bme/79dqvrMuz1/6tvZL9y3rWc1FEYGZmVk1bswMwM7PW5kRhZma5nCjMzCyXE4WZmeVyojAzs1xOFGZmlquhiULSBZLWSrqzyn5JOlfSCkm3S9q7kfGZmdmWGn1HcSEwK2d/DzAte80BvtuAmMzMLEdDE0VE3Ag8mVPkUOBHkdwMTJC0S2OiMzOzSrZpdgBldgVWl6yvybY9XF5Q0hzSXQcdHR3Tp0yZ0pAAzcxGinvvvffxiNhpsHKtlihUYVvFMUYiYh4wD6C7uzuWLFlSz7jMzEYcSb1FyrVaq6c1QOmtwWTgoSbFYmZmtF6imA8clbV+2hd4OiK2qHYyM7PGaWjVk6SLgZnARElrgK8A2wJExPnAAmA2sAJ4FjimkfGZmdmWGpooIuKIQfYHcEKDwjEzswJarerJzMxajBOFmZnlcqIwM7NcThRmZpbLicLMzHI5UZiZWS4nCjMzy+VEYWZmuZwozMwslxOFmZnlcqIwM7NcThRmZpbLicLMzHI5UZiZWa4hJ4psUqE/k9Rq06iamVkdFE4UkmZLugV4DlgFvCXbPk/SR+oUn5mZNVmhRCHpKNI0pfcAc8redx9wbO1DMzOzVlD0juI04JyIOBr4j7J9dwF71TQqMzNrGUUTRRdwTZV9zwHjaxOOmZm1mqKJYjXwtir7uoEVtQnHzMxaTdFE8QPgK9lD645smyQdAJwMfK8ewZmZWfMVbeJ6NjAFuAjYlG37LdAOzI2Ic+sQm5mZtYBCiSIiAjhB0reAA4CJwJPAdRFxbx3jMzOzJiuUKCTtB/w+Iu4H7i/b1wlMj4gb6xCfmZk1WdFnFNdTvQnsn2f7zcxsBCqaKJSzbxzwbA1iMTOzFlS16imrbppZsuk4SbPKio0F/gq4o/ahmZlZK8h7RvF24JPZcgAfAl4sK7ORNKzH52sfmpmZtYKqiSIizgHOAZD0APD+iLitUYGZmVlrKNo8dmq9AzEzs9ZUeE4JSQLeCexJejbxMhHx7zWMy8zMWkTRfhSTgOuAN5KeVwy0goqSYk4UZmYjUNHmsd8EniIN4yHSg+7dgS+R5qPYsx7BmZlZ8xWtetof+Afg4WxdEbEK+L+S2kh3EwfXIT4zM2uyoncUE4DHIqIfWAfsXLLvt8A7ah2YmZm1hqKJ4gFgl2z5LuDIkn3/hzRAYCGSZklaLmmFpFMq7N9N0vWS/kfS7ZJmFz22mZnVXtFE8WvgoGz5TOCDktZk/Ss+BfxbkYNIagfOA3pIY0cdIal8DKl/BH4eEW8DDscPyc3MmqpoP4pTS5YXSnoH8AHSJEbXRMTCgp+3D7AiIlYCSLoEOBRYVvpxbJ5a9dXAQwWPbWZmdVC4H0WpiFgCLNmKt+5KmlZ1wBpSC6pSpwNXS/ok0Am8t9KBJM0B5gBMmjSJRYsWbUU4ZmY2mK1KFAMkbQt8BPhcRLypyFsqbIuy9SOACyPim5JmAD+W9ObsQfrmN0XMA+YBdHd3x8yZM4ccv5mZDS43UUh6HWkwwCnAStIP+BOSOoATgU+THnIXnY9iTXasAZPZsmrpWGAWQET8TtJY0ox6awt+hpmZ1VDeMON/CVxJGq7jMWAH4ERJHwIuAfYAFgCHRcTvCn7eYmCapKnAg6SH1X9bVmYVabrVCyW9seTzzcysCfJaPX0VuBOYHBGvJSWK3wA3ANsB+0XE+4aQJIiIF0l3IlcBd5NaN90l6QxJh2TFTgI+Luk24GLgo9mc3WZm1gSq9hss6THg2IiYX7Jt4GH0kRFxcWNCHFx3d3csWbI1z9bNzEYvSUsjonuwcnl3FDsCj5RtG1i/d2sDMzOz4WWwVk9jJW1fofx2ZduJiGE1b/am9RtYt3ABL/T2sm1XF+N7ZtM+rrPZYZmZtZzBEkW11kw3VdjW/gpjaZhnly5l9Zzjif5+oq8PdXSw9qyzmTJvLttPn97s8MzMWkpeojimYVE00Kb1G1g953j6N2x4aVv09RHA6jnHM+3GG2jr9J2FmdmAvDmzL2pkII2ybuECor+/4r7o72fdwoVMOOywBkdlZta6ig4KOGK80NtL9PVV3Bd9fWzsXdXgiMzMWtuoSxTbdnWhjo6K+9TRwZiu3RockZlZaxt1iWJ8z2zUVvm01dbG+J6eBkdkZtbaRl2iaB/XyZR5c2nr7HzpzkIdHbR1bt5uZmabDTp6rKTtgMOAWyPivvqHVH/bT5/OtBtvYN3ChWzsXcWYrt0Y39PjJGFmVsGgiSIinpf0fdKIriMiUQC0dXa6dZOZWQFFq57uAPasZyBmZtaaik5c9BnSsN8PA1dmo8CamdkoUDRR/ArYHvgvICT9ibKZ6SJi5xrHZmZmLaBoojiPLacsNTOzUaBQooiI0+sch5mZtaiidxQASBoD/AVptrsngTsiYmM9AjMzs9ZQuMOdpJOBR4FbSVOZLgYelfT5OsVmZmYtoNAdhaRPA98Azgd+RkoYk4C/Ab4h6fmIOLduUZqZWdMUrXo6ATgrIk4r2bYcuFHSU8CnACcKM7MRqGjV0xSqz3a3CJhck2jMzKzlFE0Uq4CDquw7MNtvZmYjUNGqp3OBcyXtAPyC9IxiZ+BDwEdJVU9mZjYCFe1H8R1JzwNfAT5G6nwn4CHgExHx/fqFaGZmzVS4H0VEfC8bRXYysAvwMLAmItxj28xsBBv0GYWksZLulTQrktURcWv2p5OEmdkIN2iiiIjngAlAf/3DMTOzVlO01dNPgGPqGYiZmbWmos8oVgEflrQEWEBq9VRa7RQR8d1aB9dom9ZvYN3CBbzQ28u2XV2M75lN+zhPj2pmo5uKPGaQNFi1U0REe21CGrru7u5YsmTJKzrGs0uXsnrO8UR/P9HXhzo6UFsbU+bNZfvp02sUqZlZ65C0NCK6BytXqOopItoGeTUtSdTCpvUbWD3nePo3bCD6+gCIvj76N2zebmY2WhVt9fQ9Sfs2IqBmWLdwAdFf+aYp+vtZt3BhgyMyM2sdRVs9HQ6MrX84zfFCb+9LdxLloq+Pjb0eocTMRq+irZ6uA95dz0CaaduuLtTRUXGfOjoY07VbgyMyM2sdRRPFecAxkv5Z0nskvUnSXqWvoh8oaZak5ZJWSDqlSpkPS1om6S5JPy167K01vmc2aqv8VaitjfE9PfUOwcysZRVtHntl9udns1dpUyll64M+0JbUTko6BwJrgMWS5kfEspIy04BTgXdGxJ8k7Vwwxq3WPq6TKfPmVm311NbpJrJmNnoVTRS1qnbaB1gRESsBJF0CHAosKynzceC8iPgTQESsrdFn59p++nSm3XgD6xYuZGPvKsZ07cb4nh4nCTMb9YqOHntDjT5vV2B1yfoa4O1lZfYEkPQb0l3K6RFxZVkZJM0B5gBMmjSJRYsW1SbCiRPTC2Dx4toc08xsGKuaKCS9A7gtInI7EUjaETg4Ioo8S1CFbeU9/rYBpgEzSSPV3iTpzRHx1MveFDEPmAepw93MmTMLfLyZmQ1V3sPsm4A3DaxIape0SdLeZeVeD/y44OetIU2rOmAyaU6L8jL/FREvRMQDpLm5pxU8vpmZ1Vheoqj0v/9K24ZiMTBN0lRJY0j9M+aXlfkV2TMRSRNJVVErX+HnmpnZViraPLYmIuJF4ETgKuBu4OcRcZekMyQdkhW7CnhC0jLgeuDzEfFEI+M0M7PNCs9wVysRsYA0Am3pti+XLAebm+GamVmTDXZHUWloWc9qZ2Y2igx2R3GhpPJWTz+W9GzJujsamJmNYHmJ4qIK2+6qUvbWGsRiZmYtqGqiiAhPfWpmZo1t9WRmZsOPE4WZmeVyojAzs1xOFGZmlsuJwszMcjlRmJlZrsJDeEg6DPhr0oivY8v3R8Q+NYzLzMxaRKFEIel04MvAbaTZ6DbWMSYzM2shRe8ojgXOiogv1jMYMzNrPUWfUbwKuLaegZiZWWsqmiguAWbVMxAzM2tNRauergXOzmacuwZ4qrxANs+EmZmNMEUTxc+yP3cHjq6wP4D2WgRkZmatpWiimFrXKMzMrGUVShQR0VvvQMzMrDUNpcPdNsAHgXcBOwBPAjcBl0XEi/UJz8zMmq1oh7udgauBtwB/BB4FZgAnALdJOigiHqtXkGZm1jxFm8d+C9gReHtE7BERMyJiD+Dt2fZv1StAMzNrrqKJYjbwhYhYXLoxWz8V+KtaB2ZmZq2haKLYDnimyr5ngDG1CcfMzFpN0URxM/AFSZ2lG7P1L2T7zcxsBCra6ukk4HpgtaSrSQ+zdwYOBgTMrEt0ZmbWdIXuKCLiD8A0YB6wE3AgKVGcD0yLiNvqFqGZmTVV4X4UEfE4cEodYzEzsxbkqVDNzCxX1TsKSbcCH42IZZIWkwb+q8pToTbPpvUbWLdwAS/09rJtVxfje2bTPq5z8DeamRWQV/V0F9BXspybKKw5nl26lNVzjif6+4m+PtTRwdqzzmbKvLlsP316s8MzsxFAEcP/97+7uzuWLFnS7DAabtP6DazYf3/6N2zYYl9bZyfTbryBtk7fWZhZZZKWRkT3YOUKPaOQdIGkikONS+qSdMFQA7RXbt3CBUR/f8V90d/PuoULGxyRmY1ERR9mf5TULLaSiVSezMjq7IXeXqKvr+K+6OtjY++qBkdkZiPRUFo9VaujejPgkWObYNuuLtTRUXGfOjoY07VbgyMys5GoaqKQ9A+SVkpaSUoSvxpYL3k9BFwA/LroB0qaJWm5pBWSqvbLkHSYpJA0aP3ZaDW+ZzZqq3wJ1dbG+J6eBkdkZiNRXqunZcClpCE6PksawuPhsjIbgXuAnxf5MEntwHmknt1rgMWS5kfEsrJyrwI+BdxS5LijVfu4TqbMm7tFqye1tTFl3lw/yDazmqiaKCLiGuAaAEnPAN+PiAdf4eftA6yIiJXZcS8BDiUlpVJfA/4J+Nwr/LwRb/vp05l24w2sW7iQjb2rGNO1G+N7epwkzKxmis6Z/dUafd6uwOqS9TWkyY9eIultwJSIuEJS1UQhaQ4wB2DSpEksWrSoRiEOUxMnphfA4sX5ZUe7/n42Pf00sXEjGjOG9le/GqpU4ZnZ0ObMngEcC+wJjC3fX7Bntipse+khuaQ24NukVla5ImIeaZBCuru7Y+bMmQU+3ka7Sh0UB6rq3EHRrLKi/SgOBG4EJgPvIrVyWg/8L9JUqHcW/Lw1wJSS9cnAQyXrryK1olok6Y/AvsB8P9C2Wti0fgOr5xxP/4YNLzUrjr4++jds3m5mWyp6v30G8K9snvL0SxHxHtLdxQvAooLHWQxMkzRV0hjgcGD+wM6IeDoiJkbE7hGxO2lCpEMiYvR1u7aacwdFs61TNFHsBSwE+klVRZ0AEdELnA6cVuQgEfEicCJwFXA38POIuEvSGZIOGVroZkPjDopmW6foM4rngLaICEkPA68Dbsr2rSNVIRUSEQuABWXbvlyl7MyixzUbzEAHxUrJwh0UzaorekdxG/CGbPla4FRJB0ran1QtdUc9gjOrJXdQNNs6RRPFv7C5ddIXgQ2k6qPrSVOinlD70Mxqa6CDYltn50tDn6ijg7bOTndQNMtRtB/FgpLlByVNB14PdAD3RMTGOsVnVlPuoGg2dIX7UZSKNInFfTWOxawh2jo7mXDYYc0Ow2zYGMp8FD+rsu9iSd+vbVhmZtYqij6jOBD4RZV9lwIH1SYcMzNrNUUTxU7Ak1X2/Yn0QNvMzEagos8oeoH9SE1jy+1HGprDzMwaZNP6DaxbuIAXenvZtquL8T2zaR9Xn0YZRRPFhcBXJK0FLoqI9ZLGAUcBJwO1Gl3WzMwGUWlwy7VnnV23wS2VGjANUiiN6joP+BipP8UG0jAeyrb/XRQ5UJ10d3fHkiUeDsrMRr5N6zewYv/9Kw5i2dbZybQbbyjc3FvS0ogYdNDVov0o+oHjJJ0DvJs0YuwTwHURcW+hiMzM7BUrMrhlrZt/D6kfRUQsB5bXNAIzMyusGYNbVk0UkvYC7o+I57PlXOXzXpuZWe01Y3DLvDuKO0kTB92aLVd7BqFsX3ttQzMzs3Lje2az9qyzK/4g12twy7xE8W5gWcmymZk12cDgltWm9K3HuGV5ieJoYBVpytMAfh8R62segZmZDUmjB7ccLFGcDzxAGk58BqkayszMmqyRg1vmJYqHgZmSlpGeQ4yVtH21whHxbK2DMzOz5ssb62kecBbwNKnq6XrgmZyXmZmNQFXvKCLiDEm/Bt4I/Ag4E7i/UYGZmVlryO1wFxFLgaWSDgB+GBEPNCYsMzNrFUWH8Dim3oGYmVlryuuZ/U/AuRGxJlvOFREn1zQyMzNrCXl3FB8CfkKaa+JDgxwnSMONm5nZCJP3MHtqpWUzMxtdik6FamZmo1ShRCHpg5KOLVmfKum3kp6SdKmkCfUL0czMmqnoHcU/AuNL1v8NmEjqkLc38PUax2VmZi2i6MRFewB3AEh6NXAQ8IGI+LWkVaSEcUJ9QjQzs2YayjOKgeHP9wc2Af8vW18D7FTLoMzMrHUUTRS3AUdK6gSOA66PiOezfbsBa+sRnJmZNV/RqqcvApeThh5fT6p6GvB+4JYax2VmZi2i6BAe/y1pN2BP0jzaT5XsvgBYUY/gzMys+YreURARzwBLS7dJmhARC2oelZmZtYyi/Sj+TtLJJetvlbQGeELSUkmTi36gpFmSlktaIemUCvs/K2mZpNslXSupq+ixzcys9oo+zP4ksK5k/VzgIeDI7BhnFTmIpHbgPKAH2As4QtJeZcX+B+iOiLcAvwAGHZDQzMzqp2jV027AcgBJOwHvBA6IiEWSNgLfKXicfYAVEbEyO9YlwKHAsoECEXF9SfmbgY8UPLaZmdVB0UTxPDAmW3438CxwU7b+JFB0CI9dgdUl62uAt+eUPxZYWGmHpDnAHIBJkyaxaNGigiGYmdlQFE0UtwInZM8lPgVcGRGbsn17kKqhilCFbVFhG5I+AnSTOvht+aaIeaR5venu7o6ZM2cWDMHMzIaiaKI4CZhPGsZjNfCxkn1/A/ym4HHWAFNK1idTIclIei9wGrB/Scc+MzNrgqL9KJYBr5e0I/BkRJTeBXwOeKTg5y0GpkmaCjwIHA78bWkBSW8D5gKzIsI9vs3MmqxwPwqAiHiiwrY7hvD+FyWdCFwFtAMXRMRdks4AlkTEfOAcYBzwn5IAVkXEIUOJ08zMaqdwopC0O6kF0p7A2PL9EfHhIsfJOugtKNv25ZLl9xaNyczM6q9QopA0HbiB9HxiT+B24NXA7qTnDh7Cw8xshCra4e4c4FLgzaSWS8dGxB7Au0itltwpzsxshCqaKN4K/BToz9bHAkTEb4GvUrBntpmZDT9FE0UAG7PWTmuB0vGXVgPTah2YmZm1hqKJYhnwumz5d8BnJE3LBuw7Gbi/HsGZmVnzFW31NI/NdxFfBK4G7snWNwCH1TguMzNrEUU73P24ZPluSW8EZgAdwM3uGGdmNnINqcPdgIhYD1xT41jMzKwFVU0UkmYP5UCe6c7MbGTKu6O4gtTaqdKIr+WCNCSHmZmNMHmJYmrDojAzs5ZVNVFERG8jAzEzs9ZUtR+FpB0lXSrp4JwyB2dldq5PeGZm1mx5He4+TZq97uqcMleTqqhOqmVQZmbWOvISxYeB88smKXqZbN9c4NBaB2ZmZq0hL1F0kYbuGMzdpOHGzcxsBMpLFH3A+ALHGJeVNTOzESgvUfweKDIF6aFZWTMzG4HyEsV5wLGSjq5WQNJRwDHAd2odmJmZtYa8fhSXSfpX4IeSTgSuBFaRemHvBhwMdAPfjohfNiJYMzNrvNxBASPiJEmLSE1lPwdsl+16HvgNcGhEXFHXCM3MrKkGHT02Ii4HLpe0DbBjtvmJiHixrpGZmVlLKDzMeJYYHq1jLGZm1oKKToVqZmajlBOFmZnlcqIwM7NcThRmZpbLicLMzHI5UZiZWS4nCjMzy+VEYWZmuZwozMwslxOFmZnlcqIwM7NcThRmZpar4YlC0ixJyyWtkHRKhf3bSfpZtv8WSbs3OkYzM9usoYlCUjtp5rweYC/gCEl7lRU7FvhTRLwe+DZwdiNjNDOzl2v0HcU+wIqIWBkRG4FLSHNulzoUuChb/gVwgCQ1MEYzMytReD6KGtkVWF2yvgZ4e7UyEfGipKdJEyY9XlpI0hxgTra6XtLyrYxpYvmxh7GRci4j5TzA59KqfC5JV5FCjU4Ule4MYivKEBHzgHmvOCBpSUR0v9LjtIKRci4j5TzA59KqfC5D0+iqpzXAlJL1ycBD1cpk06++GniyIdGZmdkWGp0oFgPTJE2VNAY4HJhfVmY+cHS2fBhwXURscUdhZmaN0dCqp+yZw4nAVUA7cEFE3CXpDGBJRMwHfgD8WNIK0p3E4XUO6xVXX7WQkXIuI+U8wOfSqnwuQyD/Z93MzPK4Z7aZmeVyojAzs1yjNlEMNpRIK5A0RdL1ku6WdJekf8i27yDpGkn3ZX++JtsuSedm53S7pL1LjnV0Vv4+SUdX+8w6n0+7pP+RdEW2PjUbpuW+bNiWMdn2qsO4SDo1275c0sFNOo8Jkn4h6Z7s2swYxtfkM9nfrTslXSxp7HC5LpIukLRW0p0l22p2HSRNl3RH9p5zpfp1/K1yLudkf8dul/RLSRNK9lX8vqv9rlW7poVFxKh7kR6k3w/sAYwBbgP2anZcFeLcBdg7W34VcC9p6JN/Ak7Jtp8CnJ0tzwYWkvqi7Avckm3fAViZ/fmabPk1TTifzwI/Ba7I1n8OHJ4tnw/8Xbb898D52fLhwM+y5b2ya7UdMDW7hu1NOI+LgOOy5THAhOF4TUidWx8AOkqux0eHy3UB9gP2Bu4s2Vaz6wDcCszI3rMQ6GnwuRwEbJMtn11yLhW/b3J+16pd08LxNfIvZqu8sot/Vcn6qcCpzY6rQNz/BRwILAd2ybbtAizPlucCR5SUX57tPwKYW7L9ZeUaFPtk4FrgPcAV2T++x0v+Ibx0TUit4mZky9tk5VR+nUrLNfA8xpN+XFW2fThek4FREHbIvucrgIOH03UBdi/7ca3Jdcj23VOy/WXlGnEuZfs+APwkW674fVPldy3v31rR12iteqo0lMiuTYqlkOw2/23ALcCkiHgYIPtz56xYtfNqhfP9F+BkoD9b3xF4KiJerBDTy4ZxAQaGcWmF89gDeAz4YVaN9n1JnQzDaxIRDwL/DKwCHiZ9z0sZntdlQK2uw67Zcvn2ZvkY6a4Ghn4uef/WChmtiaLQMCGtQtI44FLg0xGxLq9ohW2Rs70hJL0PWBsRS0s3Vygag+xrheu2DamK4LsR8TZgA6mKo5qWPZes/v5QUvXFnwGdpJGdq8XVsudSwFBjb5lzknQa8CLwk4FNFYrV9VxGa6IoMpRIS5C0LSlJ/CQiLss2Pyppl2z/LsDabHu182r2+b4TOETSH0kjBr+HdIcxQWmYlvKYqg3j0uzzGIhtTUTckq3/gpQ4hts1AXgv8EBEPBYRLwCXAe9geF6XAbW6Dmuy5fLtDZU9XH8fcGRk9UYM/Vwep/o1LWS0JooiQ4k0XdbK4gfA3RHxrZJdpcOcHE16djGw/aishce+wNPZ7fdVwEGSXpP9L/KgbFtDRMSpETE5InYnfdfXRcSRwPWkYVoqnUelYVzmA4dnrW+mAtNIDxwbJiIeAVZLekO26QBgGcPsmmRWAftK2j77uzZwLsPuupSoyXXI9j0jad/suzmq5FgNIWkW8AXgkIh4tmRXte+74u9ado2qXdNiGvHAqRVfpFYQ95JaCZzW7HiqxPgu0i3i7cAfstdsUp3jtcB92Z87ZOVFmhjqfuAOoLvkWB8DVmSvY5p4TjPZ3Oppj+wv+ArgP4Htsu1js/UV2f49St5/WnZ+y6ljK5RBzuGtwJLsuvyK1FpmWF4T4KvAPcCdwI9JLWmGxXUBLiY9W3mB9L/pY2t5HYDu7Hu5H/gOZQ0YGnAuK0jPHAb+7Z8/2PdNld+1ate06MtDeJiZWa7RWvVkZmYFOVGYmVkuJwozM8vlRGFmZrmcKMzMLJcThdWdpCjwmlmDz3lE0plDfM/Y7POPe6WfP4TPHCPpZEnLJD0r6TFJv5N00hCPs72k0yW9uWD5mZKuk/S4pPXZKKMXSHptSZkhf4c28jV0KlQbtWaULHcA1wFnAr8u2b6sBp8zm809cYt6nhTf/TX4/KK+B7wf+DqpP8ZrSD2i3wd8cwjH2R74Cpv7QVQl6QDgajaPELsReBNwJKmn7iNZ0a35Dm2Ecz8Ka6hs3KpnSB2bLixQfmxEPFf3wBpE0sCwF5+OiH8r26cYwj9ISRNJAxQeERGXDFL2UqArIror7BvS59ro46onaxmSPpFVA+0t6SZJfcAns2EXvqk0uc4GSaslXSRpp7L3v6zaRNIlkv5b0mylyXnWS7qhZPiNilVPkm6W9B9KE9qslLRO0uWlVTRZuT2UJsfpk3S/pL+VdIWkK3NOczzp390j5TvKf6wl7STpB0oT2vRl38n0gbhJSQLg4pIqvNeWHzczAXi00o7Szy39DiX9eU5V4b4l7zlM0u8lPSfpIUlfl9Se8x3YMONEYa3oZ6SBEGeTqkvaSHMmnJltO4k0ecvV2Tg8eV6fve904COkQdMuLhDDfqRhFD5NmrBnBvDvAzsltZHmb5hKqso5mTSK7FsHOe6DpB/sMyUdmt1hbUFSB2l8nv1IEz79NelO7NrsTuJ5YFZW/Etsno/giSqf+3vgYEmnqGRmukH8seS4A6/5pBFzH8ziPIp0vW4CDgG+AXyKNDSIjRTNGF/Gr9H7AsaRxq/6aIV9n8j2HT/IMdqB12Vl9ynZ/giVZiY1AAAD4ElEQVRwZsn6JaS6+K6SbYdn79s9Wx+brR9XUuZm0g/uq0q2nUIa6nlg8pcPZu97S0mZqcAm4MpB4p+VHT+yY94CfAbYtqTMCUDfQJzZtu1IY/98LVufmB3j8ALf+w6kH/PIXmtIiW+PsnIv+w7L9n2QNJ/IwExp7aTxib5bVu7vgfXA+Gb/ffOrNi/fUVgr+nX5BkmHZFVCT5N+XFdku/Yc5Fj3RkRvyfrAQ/PJlQqX+F1EPFP2vnZgoGrnfwN/jIjbBwpExAOkAedyRcSVpKRyJGla1V2AbwFXltwhvZeUQNZI2kZpiOhNpB/7LZ4zFPjMJ0l3J38JnAX0AscDfyjSakrSm4ALgW/H5uchbyZ9H/85EGMW53WkuS3eONQ4rTU5UVgrellduqR3Ar8ktUz6CKkKZL9s99hBjvVU2frGGr3vtWx+RlCq0rYtRMS6iPhpRBwLdJHmen4PaZhrSHcL+5NGEy19HcHL5xwoLJL/jjTs+ztJc0e3k0YirUrSBNIouUtIw14PmJj9eW1ZjHdn27cqTms9bh5rrai8Bc4HgVWR5rAAoPSBdJM8QvohL7cTFR5U54mIkPTPpOccf06aI+FJ4DekZyTl+oYWatXPXSxpUfaZFWXPYn5CSpB/E5un0ySLEdL8BpWaNzeyybHVkROFDQcdbP4f/YAjKxVsoMXAFyS9ZaD6SWkSmb8gJ1FI2o40F0D5lLbTsj8H7qauJT2kXplVG1VS9O4ISTtHxNqybe2keQp6K78LgK+RJjTar/z9pGq2x0jPgH40WAw2fDlR2HBwDfAJSecAV5KqnQ5vbkj8ktTR7TJJXyQ9NzmdlCT6c963E+m5wIXADcA6UguuU0k/2Jdn5b4PfBxYJOlbwAOkqp4ZpOlLz4uIdZIeJs12dh+pJdQfyv7XP+A/subGl2XH2hE4DngD6U5mC1knvVOBuUBbaZNY4M6IWC/p88D3JO1AaqH2IqmhwQeA2RGxKee7sGHCicJaXkRcJulLpNY0f096oPt+4K4mxtQv6a+AecCPSAniq8AxpB//ah4Hvg30kKbXfBWpBdLlpNZGG7LjPytpf9L/6L9OSjCPklpk/bzkeB8HzibdgWxHejBe6Y7mXFIV0VdJz1f+RLojeG9EXFcl1mmkmeE+kb1KzQBujoiLJD1JSijHs7mhweXkJ0wbRtwz26xGJO0IrATOiohvNDses1rxHYXZVpJ0IvAc6X/Qk4DPZ7sualpQZnXgRGG29TaSksNupD4OtwAHRMRDTY3KrMZc9WRmZrnc4c7MzHI5UZiZWS4nCjMzy+VEYWZmuZwozMws1/8H1jPVdFF3Z/kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "error_rates = np.array([bi_error_20, bi_error_200, bi_error_2000, bi_error_5000, bi_error_12000])\n",
    "training_data_sizes = np.array([20,200,2000,5000,12000])\n",
    "fig, ax = plt.subplots()\n",
    "plt.scatter(training_data_sizes, error_rates, c='tab:red',s=50)\n",
    "ax.set(ylim=(0,1.0))\n",
    "plt.xlabel('Training Set Size', fontsize=15)\n",
    "plt.ylabel('Classification Error Rate', fontsize=15)\n",
    "plt.grid(True,axis='y')\n",
    "plt.savefig('report/working/baseline.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.74716202, 0.62229102, 0.42208462, 0.38183695, 0.32198142])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "error_rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "error_df = pd.DataFrame()\n",
    "error_df['bl_errors'] = error_rates\n",
    "error_df['datasets'] = training_data_sizes\n",
    "error_df.to_pickle('./errors.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}