index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Machine Learning Experiments with Sacred\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## About me\n",
    "\n",
    "### Hi, I am Rüdiger\n",
    "\n",
    "### GitHub: JarnoRFB\n",
    "\n",
    "<br/>\n",
    "<div style=\"float:left\">\n",
    "<div style=\"width: 35%; float:left\">\n",
    "<img src=\"http://computational-cognition.eu/files/header.jpg\"/>\n",
    "    <p><b>Neuroinformatics Group Osnabrück</b></p>\n",
    "</div>\n",
    "<div style=\"width: 55%;float:right\">\n",
    "<img src=\"http://www.sialon.eu/data2/images/63_Logo_Germany_RobertKoch.jpg\"/>\n",
    "</div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Let's try to solve a classic machine learning problem!\n",
    "* Let's take the breast cancer dataset from scikit-learn.\n",
    "* It describes 30 numeric features of the cell nuclei. \n",
    "* The target is to figure out, whether the cells are benign or a malignant turmor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from sklearn import svm, datasets, model_selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst radius</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.99         10.38          122.80     1001.0          0.11840   \n",
       "1        20.57         17.77          132.90     1326.0          0.08474   \n",
       "2        19.69         21.25          130.00     1203.0          0.10960   \n",
       "3        11.42         20.38           77.58      386.1          0.14250   \n",
       "4        20.29         14.34          135.10     1297.0          0.10030   \n",
       "\n",
       "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0           0.27760          0.3001              0.14710         0.2419   \n",
       "1           0.07864          0.0869              0.07017         0.1812   \n",
       "2           0.15990          0.1974              0.12790         0.2069   \n",
       "3           0.28390          0.2414              0.10520         0.2597   \n",
       "4           0.13280          0.1980              0.10430         0.1809   \n",
       "\n",
       "   mean fractal dimension  ...  worst radius  worst texture  worst perimeter  \\\n",
       "0                 0.07871  ...         25.38          17.33           184.60   \n",
       "1                 0.05667  ...         24.99          23.41           158.80   \n",
       "2                 0.05999  ...         23.57          25.53           152.50   \n",
       "3                 0.09744  ...         14.91          26.50            98.87   \n",
       "4                 0.05883  ...         22.54          16.67           152.20   \n",
       "\n",
       "   worst area  worst smoothness  worst compactness  worst concavity  \\\n",
       "0      2019.0            0.1622             0.6656           0.7119   \n",
       "1      1956.0            0.1238             0.1866           0.2416   \n",
       "2      1709.0            0.1444             0.4245           0.4504   \n",
       "3       567.7            0.2098             0.8663           0.6869   \n",
       "4      1575.0            0.1374             0.2050           0.4000   \n",
       "\n",
       "   worst concave points  worst symmetry  worst fractal dimension  \n",
       "0                0.2654          0.4601                  0.11890  \n",
       "1                0.1860          0.2750                  0.08902  \n",
       "2                0.2430          0.3613                  0.08758  \n",
       "3                0.2575          0.6638                  0.17300  \n",
       "4                0.1625          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = datasets.load_breast_cancer()\n",
    "data = pd.DataFrame(data=dataset.data, columns=dataset.feature_names)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Nothing easier than that! We just split the data into training and test set, and throw a standard Support Vector Machine at the problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.631578947368421"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "clf = svm.SVC(C=5, kernel=\"rbf\", gamma=0.9)\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Next we try to tune the parameters a bit to get optimal performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "However, after some time we start to forget what combinations we tried already and what the actual results where..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sacred to the rescue\n",
    "\n",
    "\n",
    "> Every experiment is sacred <br/>\n",
    "> Every experiment is great <br/>\n",
    "> If an experiment is wasted <br/>\n",
    "> God gets quite irate <br/>\n",
    "\n",
    " &mdash; Modified after *Monty Python's The Meaning of Life*\n",
    " \n",
    "[sacred](https://github.com/IDSIA/sacred) is a library that helps you to keep track of your machine learning experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sacred import Experiment\n",
    "\n",
    "ex = Experiment(\n",
    "    name=\"svm\", \n",
    "    interactive=True,  # Because we are running in IPython.\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "@ex.config\n",
    "def cfg():\n",
    "    C = 1.0\n",
    "    gamma = 0.7\n",
    "    kernel = \"rbf\"\n",
    "    seed = 42  # Add a seed to the config for more determinism.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "@ex.main\n",
    "def run(C, kernel, gamma):\n",
    "    X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "    clf = svm.SVC(C, gamma=gamma, kernel=kernel)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    return clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - svm - No observers have been added to this run\n",
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started\n",
      "INFO - svm - Result: 0.6228070175438597\n",
      "INFO - svm - Completed after 0:00:00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sacred.run.Run at 0x7f9c7e9bd128>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Modularizing using captured functions\n",
    "\n",
    "The `ex.capture` decorator injects the config into the function. We can call caputured functions without explicitly passing the arguments later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - svm - No observers have been added to this run\n",
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started\n",
      "INFO - svm - Result: 0.6228070175438597\n",
      "INFO - svm - Completed after 0:00:00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sacred.run.Run at 0x7f9c395354e0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex = Experiment('svm', interactive=True)\n",
    "\n",
    "@ex.config\n",
    "def cfg():\n",
    "    C = 1.0\n",
    "    gamma = 0.7\n",
    "    kernel = 'rbf'\n",
    "    seed = 42\n",
    "    \n",
    "@ex.capture\n",
    "def get_model(C, gamma, kernel):\n",
    "    return svm.SVC(C=C, kernel=kernel, gamma=gamma)\n",
    "    \n",
    "@ex.main\n",
    "def run():  # Now we don't have to pass parameters to run anymore.\n",
    "    X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "    clf = get_model()  # Parameter from the config are automatically injected.\n",
    "    clf.fit(X_train, y_train)\n",
    "    return clf.score(X_test, y_test)\n",
    "\n",
    "ex.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Adding Observers\n",
    "By adding observers, we can store configurations and results of experiments into a database, preferably MongoDB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load some environment variables, to match the docker environment.\n",
    "from dotenv import load_dotenv\n",
    "env_path = 'sacred_setup_protected/.env'\n",
    "load_dotenv(dotenv_path=env_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# If you use the protected setup.\n",
    "db_name = os.environ['MONGO_DATABASE']\n",
    "mongo_uri = (f'mongodb://{os.environ[\"MONGO_INITDB_ROOT_USERNAME\"]}:'\n",
    "             f'{os.environ[\"MONGO_INITDB_ROOT_PASSWORD\"]}@localhost:27017/?authMechanism=SCRAM-SHA-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# If you use the unprotected setup just running on localhost.\n",
    "db_name = \"pydata_berlin\"\n",
    "mongo_uri = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sacred.observers import MongoObserver\n",
    "\n",
    "ex = Experiment('svm', interactive=True)\n",
    "\n",
    "ex.observers.append(\n",
    "    MongoObserver.create(\n",
    "        url=mongo_uri,\n",
    "        db_name=db_name,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started run with ID \"3\"\n",
      "INFO - svm - Result: 0.6228070175438597\n",
      "INFO - svm - Completed after 0:00:00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sacred.run.Run at 0x7f9c38f37be0>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@ex.config\n",
    "def cfg():\n",
    "    C = 1.0\n",
    "    gamma = 0.7\n",
    "    kernel = 'rbf'\n",
    "    seed = 42\n",
    "    \n",
    "@ex.main\n",
    "def run(C, kernel, gamma):\n",
    "    X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "    clf = svm.SVC(C, gamma=gamma, kernel=kernel)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    return clf.score(X_test, y_test)\n",
    "\n",
    "ex.run()\n",
    "# The \"No observers have been added to this run\" warning vanishes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now we can view the result in a dashboard like [Omniboard](https://github.com/vivekratnavel/omniboard). Go to http://localhost:9000/ to see the how it looks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Recording metrics and artifacts\n",
    "We can attach metrics and even entire files as artifacts to our experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import f1_score, confusion_matrix\n",
    "\n",
    "\n",
    "def plot_confusion_matrix(confusion_matrix, class_names):\n",
    "    df_cm = pd.DataFrame(\n",
    "        confusion_matrix, index=class_names, columns=class_names,\n",
    "    )\n",
    "    fig, ax = plt.subplots()\n",
    "    heatmap = sns.heatmap(df_cm, annot=True, cmap=\"Blues\")\n",
    "    heatmap.set(ylabel='True label', xlabel='Predicted label')\n",
    "\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ex = Experiment('svm', interactive=True)\n",
    "\n",
    "ex.observers.append(MongoObserver.create(\n",
    "    url=mongo_uri,\n",
    "    db_name=db_name))\n",
    "\n",
    "@ex.config\n",
    "def cfg():\n",
    "    C = 1.0\n",
    "    gamma = 0.7\n",
    "    kernel = 'linear'  # Try out a different kernel.\n",
    "    seed = 42\n",
    "    \n",
    "@ex.capture\n",
    "def get_model(C, gamma, kernel):\n",
    "    return svm.SVC(C=C, kernel=kernel, gamma=gamma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started run with ID \"4\"\n",
      "INFO - svm - Result: 0.956140350877193\n",
      "INFO - svm - Completed after 0:00:01\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added image/png as content-type of artifact /home/jarno/projects/pydata_berlin_sacred/confusion_matrix.png.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sacred.run.Run at 0x7f9c2b39e5c0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWgAAAEKCAYAAAA/2c+EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGvdJREFUeJzt3XucXfO9//HXeyZISMSdoBqXkJ/7JYnS05a07pei9NDmuD4arYfS42hPnToIVZTzc6paFdT9rvFwrRYVjtMSSSTEpTwS1dBI3OUmSD7nj7WGbWRmrz2z1+zvzryfHusxe6+19vf7mWR85pvPWt/vUkRgZmbpaWl0AGZmtnRO0GZmiXKCNjNLlBO0mVminKDNzBLlBG1mlignaDOzRDlBm5klygnazCxRfRodQEdGXTfVUxztM3550JaNDsEStOqKrepuG/22O75wzln45MXd7q8Ij6DNzBKV7AjazKxHKb3xqhO0mRlAS2ujI/gMJ2gzMwD1SFm5Jk7QZmbgEoeZWbI8gjYzS5RH0GZmifII2swsUb6Lw8wsUS5xmJklyiUOM7NEeQRtZpYoJ2gzs0S1+iKhmVmaXIM2M0uUSxxmZonyCNrMLFEeQZuZJcojaDOzRHmqt5lZolziMDNLlEscZmaJ8gjazCxRTtBmZonyRUIzs0S5Bm1mlqgESxzpRWRm1ghS8a1qU1pF0m2Snpf0nKSdJK0m6X5JL+ZfV63WjhO0mRkgqfBWwC+A+yJiKLAN8BzwY+DBiBgCPJi/75QTtJkZ9UvQkgYCXwauAIiIDyLiHeDrwNX5aVcDB1SLyQnazAxQiwpvVWwIvA5cKelJSZdLWglYOyJm5ee8BqxdrSEnaDMzahtBSxotaWLFNrqiqT7A9sAlEbEdMJ925YyICCCqxeS7OMzMoGhtGYCIGAuM7eDwK8ArEfF4/v42sgQ9W9KgiJglaRAwp1o/HkGbmVG/GnREvAbMlLRZvuurwLPAncAR+b4jgDuqxeQRtJkZQH3nqXwfuF7S8sAM4CiyAfEtko4BXga+Wa0RJ2gzM2orcVQTEVOAYUs59NVa2nGCNjMDWlrSq/g6QZuZUd8RdL04QZuZQb1r0HXhBG1mhkfQZmbJcoI2M0tUgSncPc4J2swMj6DNzJLlBG1mlignaDOzRDlBm5mlKr387ARtZgae6m1mliyXOKyq5VrEqbtvQp9W0Sox4e/vMO6p2Wy+dn++tcMgWlvE395cyGWPzWRJ1ecx2LJq8eLFHPXtQ1hzrbX5r4suaXQ4y4b08rMTdGo+XBL87IHpLPpoCa2C/9xjE57+x1yO3flznPPAdF6b+wHf2HptvrTRajw8/a1Gh2sNcvMN1zJ4w42ZP39eo0NZZqQ4gk6v6GIs+mgJAK0tok+LWBLw0ZLgtbkfADBt1jyGbzCwkSFaA82Z/Rp/fvRh9j/wG40OZZlSryeq1FOpCVrSIUX22adJcPbem/Lrg7fg6VnzmP7mAlolNlytHwAjPj+Q1VdcrsFRWqNceP65HH/iySjBi1rNrNclaOCUgvusQgT85N4XOGHcs2y8+oqsP7AvFz/6MqOGrcuYPYew8MMlrj/3Uo8+Mp5VV1uNoZtv0ehQljlqUeGtp5RSg5a0F7A3sJ6kiyoOrQx81MnnRgOjAUYcdSpDRh5cRnhNY8GHS3h29jy2XncA9z73Omf9cToAWw7qz6CVV2hwdNYIT02ZzP88/BB/fvQRPvhgEfPnz+f0n/yIMWf/vNGhNb0Ua9BlXST8BzAR2B+YVLF/LvCvHX2o8lHmo66b2ivHiANWaGXxkmDBh0tYrlVsNag/dz0zh5VX6MN7iz6iT4vYb/O1uGNa1Se22zLouBNO4rgTTgJg0sQJ3HDNlU7OddJrEnRETAWmSrohIj4so49l1Sr9luPYnTegRVkt+vGX32XKq3M5bPtBbLveyrQIHnjhTZ6d7av3ZvWUYH4u/Ta7EZLOAD6f9yUgImKjkvttWjPfeZ9T733hM/tvnDyLGyfPakBElqodho1gh2EjGh3GMqPXjKArXEFW0pgELC65LzOzLmvphQv2vxsRvy+5DzOzbktwAF16gn5I0vnAOGBR286ImFxyv2ZmNemNI+gd86/DKvYFMLLkfs3MatLrRtARsWuZ7ZuZ1UtvvEiIpH2ALYC+bfsi4syy+zUzq0WC+bncBC3pN8CKwK7A5cDBwIQy+zQz64oUF+wvO6KdI+Jw4O2IGAPsBGxacp9mZjWTim89pewSx8L86wJJ6wJvAoNK7tPMrGa9sQZ9t6RVgPOByWR3cFxecp9mZjVLMD+XfhfHWfnL30m6G+gbEe+W2aeZWVf0xhE0knYGBrf1JYmIuKbsfs3MapFgfi79Lo5rgY2BKXyyFkcATtBmlpTeOJNwGLB5RPTKtZ3NrHmkWOIo+za7acA6JfdhZtZtvfE2uzWAZyVN4NOLJe1fcr9mZjVJcQRddoI+o+T2zczqot75WVIr2aP/Xo2IfSVdBXwFaLuT7ciImNJZG2XfZvdwme2bmdVLCRcJTwSeI3tYdpsfRsRthWOqd0SVJM2V9F67baak2yX5sVdmlgxJhbcCba0P7EM3J+aVfZHwv4EfAusB6wMnAzcANwG/LblvM7PC6pmgyXLfj4Al7fafLekpSRdKWqFaI2Un6P0j4tKImBsR70XEWGCPiLgZWLXkvs3MCqvlLg5JoyVNrNhGf9KO9gXmRMSkdl2cAgwFhgOrAf9eLaayLxIukPRNoK3mcjDwfv7a90abWTJquYsjH2yO7eDwF4H9Je1Ntg7+ypKui4hR+fFFkq4kqyh0quwR9LeBfwHmALPz16Mk9QOOL7lvM7PC6nUfdEScEhHrR8Rg4FDgTxExStKgrB8JOIBsnkinyr6LYwawXweHHy2zbzOzWvTAVO/rJa0JiGz5i+9W+0ApCVrSjyLi55J+yVJKGRFxQhn9mpl1VUsJE1UiYjwwPn9d88OyyxpBP5d/nVhS+2ZmdZXgRMKOE7SklTs6BhAR73Vy7K7869VdD83MrOc021TvZ8jKE5VRt70PYIOOPijpLjq5S8NrcZhZahJcbbTjBB0Rn+tGuxd047NmZj2uadeDlnQosFFE/Cyfwrj2Um7C/pjX4DCzZiPSS9BV74OWdDGwK9k9zAALgN8UaVzSEEm3SXpW0oy2revhmpmVo0XFtx6LqcA5O0fEseQzACPiLWD5gu1fCVwCfESW5K8BrutCnGZmparzWhx1USRBfyiphfyin6TV+ewCIB3pFxEPAoqIlyPiDLIVnszMktKsT1T5FfA7YE1JY4BvAmMKtr8oT+4vSjoeeBXo36VIzcxKVMZEle6qmqAj4hpJk4Cv5bsOiYiqc8hzJwIrAicAZwEjgSO6EqiZWZma9i4OoBX4kKzMUXiBpYh4In85DziqttDMzHpOggPo6gla0k+AbwG3k01SuUHS9RFxTiefubOzNj1RxcxS05QlDuBwYLuIWAAg6WzgSaDDBA3sBMwEbgQehwRvMDQzq5BikiqSoGe1O69Pvq8z6wC7AYeRjb7vAW6MiGe6EqSZWdmaai0OSReS1ZzfAp6R9If8/e7AEx19DiAiFgP3Afflz906DBgvaUxEXFyv4M3M6iXBa4SdjqDb7tR4hmwE3OaxIg3niXkfsuQ8GLiIrI5tZpacprqLIyKu6Gqjkq4BtgTuBcbUcFuemVlDNFWJo42kjYGzgc3JHoAIQERs2snHRgHzye6DPqHiG1f20eh0rWkzs56W4AC60EXCq4Cfki0huhfZ/cydPpE7Isp+GK2ZWV2lOIIukkhXjIg/AETE9Ig4lSxRm5ktM1TD1lOKjKDb1tOYLum7ZOtpDCg3LDOzntWaYI2jSIL+V2AlsvU0zgYGAkeXGZSZWU9LscRRZLGkx/OXc/lk0X4zs2VKgvm504kqt9P5g18PKiUiM7MGaLa1ODzjz8x6jQTzc6cTVR7syUDau/zQbRrZvSVq1eHHNzoES9DCJ7s/nmzKGrSZWW/Q6gRtZpamBO+yK56gJa0QEYvKDMbMrFFSTNBVZxJKGiHpaeDF/P02kn5ZemRmZj1IUuGtpxSZ6n0RsC/wJkBETAV2LTMoM7Oe1qLiW08pUuJoiYiX2/3WWFxSPGZmDZHgNcJCCXqmpBFASGoFvg+8UG5YZmY9q0+CGbpIgv4eWZljA2A28EC+z8xsmZFgfi60Fscc4NAeiMXMrGGabao3AJIuYylrckTE6FIiMjNrgATzc6ESxwMVr/sCBwIzywnHzKwxUrwPukiJ4+bK95KuBR4tLSIzswZo1gX729sQWLvegZiZNVKC+blQDfptPqlBtwBvAT8uMygzs56mOj1tUFJf4BFgBbIce1tEnC5pQ+AmYHVgEvAvEfFBZ211mqCVzU7Zhuw5hABLIqLTJ3qbmTWjOo6gFwEjI2KepOWARyX9HjgJuDAibpL0G+AY4JJOY+rsYJ6M742Ixfnm5Gxmy6R6TfWOzLz87XL5FsBI4LZ8/9XAAVVjKhD3FEnbFTjPzKxp1bJYkqTRkiZWbKPbtdUqaQowB7gfmA68ExEf5ae8AqxXLabOnknYJ29sO+AJSdOB+YDIfkls36U/BTOzBLUWGa7mImIsMLaT44uBbSWtAtwODO1KTJ3VoCcA2wP7d6VhM7NmUsZMwoh4R9JDwE7AKhUD3/X55NpehzpL0Mo7mF6XSM3MElavi4SS1gQ+zJNzP2A34DzgIeBgsjs5jgDuqNZWZwl6TUkndXQwIv5/TVGbmSWsjgPoQcDV+eqfLcAtEXG3pGeBmyT9FHgSuKJaQ50l6FagP9Tp5kAzs4S11CnVRcRTZNfu2u+fAYyopa3OEvSsiDizxtjMzJpSsy2WlGC4Zmbl6JPgXO/OEvRXeywKM7MGa6oRdES81ZOBmJk1UlMu2G9m1hskmJ+doM3MoNi6Fz3NCdrMDJc4zMyS5QRtZpao9NKzE7SZGeCLhGZmyVKCGdoJ2swM38VhZpYsXyQ0M0uUSxxmZolyicPMLFEeQZuZJSq99OwEbWYGQKtH0GZmaUowPztBm5kBKMEihxO0mRkeQZuZJateT/WuJydoMzM8gjYzS5anepuZJaolvfzsBG1mBr6Lw8wsWQlWOJygU3baqafwyMPjWW211Rl3x92NDscaaMjn1+La847++P2G663OWZfcw/V3T+Da847m8+uuxsv/eItRP7qCd+YubGCkzSvFEXSKCzhZ7usHHMQll17e6DAsAS++PIcvHHouXzj0XHb+1nkseP9D7nxoKicftRvjJ/yVrb5+JuMn/JWTj9q90aE2rRYV33ospp7rymq1w7DhrDxwYKPDsMTsOmIzXnrldf4+62323WVrrrvrcQCuu+tx9tt16wZH17xapMJbTym9xCGpFVi7sq+I+HvZ/Zotqw7ZYwduuW8SAGutPoDX3ngPgNfeeI+1Vh/QyNCaWnoFjpJH0JK+D8wG7gfuybcOi6mSRkuaKGniFZeNLTM0s6a0XJ9W9vnKVoy7/8mlHo/o4YCWIb1xBH0isFlEvFnk5IgYC4wFeP8j/KNm1s4e/7Q5U56fyZy35gIw5825rLPGyrz2xnuss8bKvJ7vt9r1uhE0MBN4t+Q+zHqNb+457OPyBsA9Dz/NqP12BGDUfjty9/inGhVa81MNWw8pO0HPAMZLOkXSSW1byX0uM/795JM4/FuH8vLfXmK3kV9m3O9ubXRI1kAr9l2ekTsO5Y4/Tfl43wVX3s/IHYfy9B2nseuOm3HBlfc3MMLmlmKJQ1Fi0UrS6UvbHxFjqn3WJQ5bmlWHH9/oECxBC5+8uNtZ84kZ7xbOOcM3GtgjWbrUGnSRRGxmloQEi9ClJmhJd8FnRsLvAhOBSyPi/TL7NzMrqjfOJJwBzAMuy7f3gLnApvl7M7MkSMW36m3pt5LmSJpWse8MSa9KmpJve1drp+zb7HaOiOEV7++S9EREDJf0TMl9m5kVVufx81XAxcA17fZfGBEXFG2k7BF0f0kbtL3JX/fP335Qct9mZoVJKrxVExGPAG91N6ayR9D/BjwqaTrZL6gNgeMkrQRcXXLfZmaF9dDdc8dLOpzsOty/RcTbnZ1c6gg6Iu4FhgA/4JNZhfdExPyI+O8y+zYzq0Ut81Qql6XIt9EFurgE2BjYFpgF/Fe1D5QygpY0MiL+JOmgdoc2lkREjCujXzOzLqthBF25LEUNn5n9cVfSZXSyLlGbskocXwH+BOy3lGMBOEGbWVLKvs1O0qCImJW/PRCY1tn5UFKCjojT869HldG+mVm91bMGLelGYBdgDUmvAKcDu0jalmyQ+jfg2GrtlD1RZQXgG8BgPr0e9Jll9mtmVqt6JuiIOGwpu6+otZ2y7+K4g2zm4CRgUcl9mZl1WYozCctO0OtHxJ4l92Fm1m0pPtW77Ikqf5a0Vcl9mJl1W4LLQZc+gv4n4EhJL5GVOARERPjJlmaWlgRH0GUn6L1Kbt/MrC56ciH+osqeSfgy8DlgZP56Qdl9mpl1Ra8rceRPVBkGbAZcCSwHXAd8scx+zcxqlt4AuvQSx4HAdsBkgIj4h6QBJfdpZlaz3nib3QcREZICIF/FzswsOQmWoEuvB98i6VJgFUnfAR7AT1IxswT1uhp0RFwgaTeyR11tBpwWEX4uvJklp8hC/D2t7BIHeUK+X9IawJtl92dm1hUJ5udyShySviBpvKRxkrbLH5w4DZgtyVO/zSw5vanEcTHwH8BAsnWh94qIxyQNBW4E7iupXzOzruktI2igT0T8MSJuBV6LiMcAIuL5kvozM+sW1fBfTylrBL2k4vXCdseipD7NzLosxRp0WQl6G0nvkf2joV/+mvx935L6NDPrspbekqAjorWMds3MypNehi79Njszs2bQm0ocZmZNJcH87ARtZgYeQZuZJatXTvU2M2sG6aVnJ2gzM8AlDjOzZPXGBfvNzJpDevnZCdrMDJLMz07QZmYALQkWoZ2gzcxI8yJh2c8kNDOzLvII2syMNEfQTtBmZvg2OzOzZHkEbWaWKCdoM7NEucRhZpYoj6DNzBKVYH52gjYzA5LM0E7QZmakOdVbEdHoGKwKSaMjYmyj47C0+Odi2eep3s1hdKMDsCT552IZ5wRtZpYoJ2gzs0Q5QTcH1xltafxzsYzzRUIzs0R5BG1mlign6DqQFJKuq3jfR9Lrku6u8rld2s6RtL+kH5cda0Xf20rau6f6s8+StFjSFElTJU2WtHM32jpT0tfqGZ81nieq1Md8YEtJ/SJiIbAb8GotDUTEncCdZQTXgW2BYcC9PdinfdrCiNgWQNIewDnAV7rSUEScVs/ALA0eQdfPvcA++evDgBvbDkgaIekvkp6U9GdJm7X/sKQjJV2cv95Y0mOSnpb0U0nz8v27SBov6TZJz0u6XsqmP0k6TdITkqZJGluxf7yk8yRNkPSCpC9JWh44E/jnfAT3z6X+yVgRKwNvt72R9MP87/MpSWPyfYMlPSfpMknPSPqjpH75saskHZy/3jv/+Zgk6aKKf6WdIem3+c/EDEknNOD7tBo4QdfPTcChkvoCWwOPVxx7HvhSRGwHnAb8rEpbvwB+ERFbAa+0O7Yd8ANgc2Aj4Iv5/osjYnhEbAn0A/at+EyfiBiRf+70iPggj+PmiNg2Im6u8Xu1+uiX/4J8HrgcOAtA0u7AEGAE2b90dpD05fwzQ4BfRcQWwDvANyobzH/+LgX2iogdgDXb9TkU2CNv+3RJy5XynVldOEHXSUQ8BQwmGz23LxsMBG6VNA24ENiiSnM7Abfmr29od2xCRLwSEUuAKXmfALtKelzS08DIdn2My79OqjjfGm9h/gtyKLAncE3+L5/d8+1JYDJZUh2Sf+aliJiSv17a3+dQYEZEvJS/v7Hd8XsiYlFEvAHMAdau5zdk9eUadH3dCVwA7AKsXrH/LOChiDhQ0mBgfDf6WFTxejHQJx81/RoYFhEzJZ0B9F3KZxbjv/MkRcRfJK1BNuIVcE5EXFp5Tv6z0/7vv1+NXX3m56fmYK3HeARdX78FxkTE0+32D+STi4ZHFmjnMT75p+uhBc5vS8ZvSOoPHFzgM3OBAQXOsx4gaSjQCrwJ/AE4Ov+7RNJ6ktYq2NRfgY3yZA7g6wtNzAm6jvLSw0VLOfRz4BxJT1JsxPID4CRJTwGbAO9W6fcd4DJgGtn/3E8U6OMhYHNfJGyothr0FOBm4IiIWBwRfyQrbf0lL1ndRsFfpvldRMcB90maRPaLuNOfH0uXZxImSNKKZPXJkHQocFhEfL3RcVlzkNQ/Iubl9exfAS9GxIWNjstq5/pTmnYALs7/B3sHOLrB8Vhz+Y6kI4DlyS40XlrlfEuUR9BmZolyDdrMLFFO0GZmiXKCNjNLlBO0dapixbVpkm7N7zDpaluFV++TtIqk47rQxxmSTi66v905H69nUbCvwfnsULNSOEFbNW3TkbcEPgC+W3lQmZp/jiLizog4t5NTViG7n9es13KCtlr8D7BJPnL8q6RryCbHfE7S7vmKfZPzkXbbLLg985XVJgMHtTXUbvW+tSXdrmxd5KnK1kU+F9g4H72fn5/3mRXe8v0/yVfqexT4zEqB7Un6Tt7OVEm/a/evgq9Jmpi3t29+fquk8yv6Pra7f5BmRThBWyGS+gB7AW3T2IcAv85XVZsPnAp8LSK2ByaSzYTsSzbDcT+ye7vX6aD5i4CHI2IbYHvgGeDHwPR89P7DjlZ4k7QD2XT4bYG9geEFvp1x+cp/2wDPAcdUHBuc97EP8Jv8ezgGeDcihuftf0fShgX6MesWT1SxavrlU5EhG0FfAawLvBwRj+X7v0C2/On/ZnNrWB74C9nKai9FxIsAyp46M3opfYwEDgeIiMXAu5JWbXdO5QpvAP3JEvYA4PaIWJD3UeShB1tK+ilZGaU/2fT4NrfkKwW+KGlG/j3sDmxdUZ8emPf9QoG+zLrMCdqq+fipH23yJDy/chdwf0Qc1u68T32umzpa4e0HXWjrKuCAiJgq6Uiy1QfbtJ+5FXnf34+IykTetrqcWWlc4rB6eAz4oqRNACStJGlTsgcVDJa0cX7eYR18/kHge/lnWyUN5LOr7XW0wtsjwAGS+kkaQFZOqWYAMEvZYvXfbnfsEEktecwbka0O9wfge/n5SNpU0koF+jHrFo+grdsi4vV8JHqjpBXy3adGxAuSRgP3SFpAViJZ2qpsJwJjJR1Dtkbx9/L1kf83v43t93kd+v+RrfAGMA8YFRGTJd0MTCVbgL7ISn7/SfbEm9fzr5Ux/R2YQPYIqu9GxPuSLierTU/O10d5HTig2J+OWdd5LQ4zs0S5xGFmlignaDOzRDlBm5klygnazCxRTtBmZolygjYzS5QTtJlZopygzcwS9X/TLPGnwOOEOQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "@ex.main\n",
    "def run(C, kernel, gamma, _run):\n",
    "    X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "    clf = get_model()\n",
    "    clf.fit(X_train, y_train)\n",
    "    y_pred = clf.predict(X_test)\n",
    "    \n",
    "    # Plot the confusion matrix and store it as PNG.             \n",
    "    cm_filename = 'confusion_matrix.png' \n",
    "    confusion = confusion_matrix(y_test, y_pred, labels=[False, True])\n",
    "    fig = plot_confusion_matrix(confusion, class_names=['Malignant', 'Benign'])\n",
    "    fig.savefig(cm_filename)\n",
    "    # Save the artifact.\n",
    "    _run.add_artifact(cm_filename)\n",
    "    \n",
    "    # Save the F1 score as a metric.\n",
    "    _run.log_scalar('F1', f1_score(y_test, y_pred))\n",
    "             \n",
    "    return clf.score(X_test, y_test)\n",
    "\n",
    "ex.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Retrieving results using incense\n",
    "Once we are done conducting our experiments, we can get the results back into Python using [incense](https://github.com/JarnoRFB/incense). This is especially useful for organizing results in a comprehensive way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from incense import ExperimentLoader\n",
    "\n",
    "loader = ExperimentLoader(\n",
    "    mongo_uri=mongo_uri,\n",
    "    db_name=db_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Experiment(id=1, name=svm)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp = loader.find_by_id(1)\n",
    "exp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pmap({'kernel': 'rbf', 'gamma': 0.7, 'C': 1.0, 'seed': 42})"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp.config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6228070175438597"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp.result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Experiment(id=2, name=svm)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp = loader.find_by_config_key('kernel', 'linear')[0]\n",
    "exp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbAAAAEgCAYAAADVKCZpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XlYVGX/P/D3YVRc2EQFSdFH+KooKMMmUFaooeb6RXEnd7CFKLs0s1TUpx41rdyuNPy6W1iZGq6pobZogQqY+qCITi4h4QICsgjM7w9/ThGio55znzP4fnXNdcmZmXN/Bufq7b2c+0hGo9EIIiIiC2OldgFERESPggFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWiQFGREQWqZbaBSglYkOa2iWQxiwZ4KV2CaRRDevrHvsc9XyizXpdUcrSx26L7qixAUZEJJTEAS3RGGBERHKQJLUreOIwwIiI5GD1+MOQ9HAYYEREcuAQonAMMCIiOXAIUTgGGBGRHNgDE44BRkQkB/bAhGOAERHJgT0w4RhgRERy4CpE4RhgRERy4BCicAwwIiI5cAhROAYYEZEcGGDCMcCIiORgxSFE0RhgRERy4CIO4RhgRERy4BCicAwwIiI5cBWicAwwIiI5sAcmHAOMiEgO7IEJxwAjIpIDe2DCMcCIiOTAVYjCMcCIiOTAIUThGGBERHLgEKJwDDAiIjkwwIRjgBERyYFDiMIxwIiI5MAemHAMMCIiOXAVonD8JwMRkRwkybyHGXJzcxEeHg4PDw+0a9cOhw8fxvXr1xEaGorWrVsjNDQUN27cUPgDaR8DjIhIBpIkmfUwxxtvvIGePXsiPT0daWlpaNeuHebOnYtu3bohIyMD3bp1w9y5cxX+RNrHACMikoFcAZaXl4cffvgB48aNAwDUqVMHDg4O+PbbbzFq1CgAwKhRo7B161ZFP48lYIAREclBMvPxAOfPn0eTJk0wZswY+Pj4YPz48SgsLER2djZcXFwAAE2bNkV2drYyn8OCMMCIiGRgZWVl1iMuLg7+/v6mR1xcXKXzlJWV4dixY3jllVeQkpKCBg0aVBkufJjhyJqMqxCJiGRgbqBERUUhKiqq2uebN2+O5s2bIzAwEAAQHh6OuXPnwtnZGVlZWXBxcUFWVhacnJxkqduSsQdGRCQDuebAmjZtCldXV5w+fRoA8P3336N9+/bo168f1q5dCwBYu3Yt+vfvr+jnsQTsgRERyUHGEb0lS5ZgxIgRKC0thZubG1avXo2KigoMHjwYK1euRMuWLfHVV1/J16CFYoAREclAzjkpvV6PI0eOVDn+/fffy9ZGTcAAIyKSARdViMcAIyKSgZUVlxSIxgAjIpIDO2DCMcCIiGTAIUTxGGBERDJggInHACMikgEDTDwGGBGRHJhfwjHAiIhkwFWI4jHAiIhkwCFE8RhgREQyYICJxwAjIpID80s4BhgRkQzYAxOPAUZEJAMu4hCPAVaD1LaSMK37/6CWToJOkpB0IRebj2ejvbMNhvu5QGclwXCtCCt+uYgKo9rVkprKy8sxZsQgNHFyxkeLl6ldTs3ADphwDLAa5HaFEf/Zl4mSsgroJGB6j//Bb3/kY8LTrpizLxNX8ksxsKMznnVzxMHM62qXSyr68ov1+FcrdxQWFqhdSo3BIUTx2OetYUrKKgAAOisJtawkVBiBsgojruSXAgBOZBUgoIW9miWSyv7MvoJDPx1Ev7CBapdSo8h1R2Yyn6YD7OuvvzbrGP1FkoAPerXBp+Ge+C2rAJnXbkEnSWjlWA8A0KmlPRrVr61ylaSmT+bPRfQbkyBxzkZWDDDxNP0NnjNnjlnH6C9GI/DezjOI2XwK7o3qo7l9XSz96XdE+D+FWT1bo+h2Bee/nmA//XAADR0d4dHeU+1SahwGmHianAPbtWsXdu7cicuXLyMmJsZ0/ObNm6hVq/qS4+LiEBcXBwDQdXwRrbuGK16rVt26XYFT2QXo+JQtdv43B//ekwkA8HKxgYudtcrVkVqOpx7Djwf349BPP6C0tASFhYWIfe9tzPrgQ7VLs3iSFcNJNE0G2FNPPQV/f38kJCTAz8/PdNzW1haffPJJte+LiopCVFQUACBiQ5ridWqNrbUO5RVG3Lpdgdo6CR1cbLDt5J+ws66FmyVlqGUloW97J3x74k+1SyWVvBrzFl6NeQsAcPRIEr5Yt5rhJRP2rsTTZIB5e3vD29sbw4cPR+3anK8xl0O92pjwdAtYSXfmwn79PQ+pl/MxzNcF+mZ2sJKAfWeu4VQ2V54RyY35JZ4mA+yupKQkzJw5E7///jvKyspgNBohSRLOnTundmmadDG3GNN2nqlyPP5YFuKPZalQEWmZn38n+Pl3UruMGoM9MPE0HWDjxo3DJ598Aj8/P+h0OrXLISKqFvNLPE0HmL29PV588UW1yyAieiD2wMTTdIB16dIFkydPxoABA2Bt/dfKOV9fXxWrIiKqSqdjgImm6QD79ddfAQBHjhwxHZMkCYmJiWqVRER0T+yAiafpANu/f7/aJRARmYVDiOJpOsAAYMeOHTh58iSKi4tNx2bMmKFiRUREVTG/xNN0gL388su4desW9u/fj/Hjx2PTpk3o1InLfolIe9gDE0/TeyEeOnQI69atQ8OGDREbG4vDhw/jzJmq1zkREanNykoy60Hy0XSA1at3Zwf1+vXr448//kDt2rWRlcULcolIe7iZr3iaHkLs06cPcnNzMXnyZPj6+kKSJIwfP17tsoiIqmA2iafpAJs+fToAYODAgejTpw+Ki4thb8+bMRKR9rB3JZ6mAwy4Mw9mMBhQVlZmOjZy5EgVKyIiqor5JZ6mA+yll15CZmYm9Hq9aS9ESZIYYESkOeyBiafpADty5AhOnTrFLwYRaR5XGIqn6VWIXl5euHLlitplEBE9kCSZ9yD5aLoHdvXqVbRv3x6dOnWqtJlvQkKCilUREVXFkSLxNB1gM2fOVLsEIiKzyJ1f5eXl8Pf3R7NmzbB9+3aMHj0aBw8eNK3EXrNmDfR6vbyNWhhNB9jzzz+vdglERGaRuwe2aNEitGvXDjdv3jQdmz9/PsLDw2Vtx5Jpeg7M1tYWdnZ2lR6urq4ICwvDuXPn1C6PiMhEzjmwS5cuYceOHdy44QE0HWBvvvkm5s+fj8uXL+PSpUtYsGABhg8fjqFDh2Ls2LFql0dEZGJlZWXWwxxvvvkmPvzwwyqvf++999CxY0dMnDgRJSUlSnwMi6LpAEtISMCECRNMPbGoqCh89913GDJkCG7cuKF2eUREJub2wOLi4uDv7296xMXFVTrP9u3b4eTkBD8/v0rH58yZg/T0dCQnJ+P69euYN2+eyI+nSZqeA6tfvz6++uor05jvpk2bULduXQBc8UNE2mLu/5OioqIQFRVV7fM///wzEhISsHPnThQXF+PmzZuIiIjAhg0bAADW1tYYM2YMFixYIEvdlkzTPbDPP/8c69evh5OTE5ydnbF+/Xps2LABRUVFWLp0qdrlERGZyDUHNmfOHFy6dAkGgwEbN25E165dsWHDBtOdOIxGI7Zu3QovLy+FP5H2aboH5ubmhm3btt3zuc6dOwuuhoioekqPCo0YMQI5OTkwGo3Q6/VYvny5ou1ZAk0G2Icffoi3334br7/++j2/FIsXL1ahKiKi6ukU2EoqJCQEISEhAIDExETZz2/pNBlg7dq1AwD4+/urXAkRkXk4LS+eYgH294vv7sXOzq7a5/r27QsAGDVqlKw1EREphQvLxFMswDw9PSFJEoxGo+nY3Z8lScKFCxeqfW/fvn3v+2XgXohEpDXcjF48xQLs4sWLj/zeSZMmyVgJEZHy2AMTT8gc2MaNG3Hu3Dm8++67uHTpErKzs6tcpPd33AORiCwN80s8xa8Di46Oxv79+7F+/XoAdy5Ofvnll816b0ZGBsLDw9G+fXu4ubmZHkREWqOTJLMeJB/FA+zQoUP47LPPTDtoODo6orS01Kz3jhkzBq+88gpq1aqF/fv3Y+TIkYiIiFCyXCKiRyJJklkPko/iAVa7dm1UVFSY/uKuXbtm9oaWRUVF6NatG4xGI1q2bImZM2dix44dSpZLRPRIeEdm8RSfA3vttdcwcOBA5OTkIDY2Fl999RViY2PNeq+1tTUqKirQunVrLF26FM2aNUNBQYHCFRMRPTwrppNwigfYyJEj4efnh3379gEAvv76a7P38Fq0aBFu3bqFxYsXY/r06UhMTMTatWuVLJeI6JEwv8QTsgqxvLwctWvXhiRJqKioMPt9AQEBAAAbGxusXr1aqfKIiB6bFS8EE07xAPvggw/wxRdfICwsDEajEcOHD8eIESMwderUat/Tr1+/+56TFzITkdZwCFE8xQNs3bp1SElJQf369QHcuaOoj4/PfQPs8OHDcHV1xbBhwxAYGFhpNw8iIi1ifImneIC5uLigrKzM9HNZWRlcXFzu+54rV65g7969iI+PxxdffIHevXtj2LBh8PT0VLpcIqJHwiXy4ikWYBMnToQkSXB0dISnpyd69OgBSZKwZ88e09xWdXQ6HXr27ImePXuipKQE8fHxCAkJQWxsLKKjo5UqmYjokXEKTDzFAuzuSkNPT0/07t3bdDwoKMis95eUlGDHjh2Ij4+HwWBATEwMwsLCFKmViOhxsQcmnmIBNm7cuEd+78iRI3HixAn06tULsbGxvHU2EWkeVyGKp/gcWGZmJt577z2cOnUKxcXFpuNnzpyp9j0bNmxAgwYNsGjRokp3X757K5YH3WuMiEg05pd4igfY6NGjMW3aNEyaNAm7du3C6tWrH9jVfphrxYiItIBDiOIpvhfirVu30KNHDwCAu7s73n//fezatUvpZomIhJLMfJB8FO+B3d3P0N3dHcuXL0ezZs2Qn5+vdLNERELxQmbxFA+wTz75BIWFhVi8eDHee+895OXlYdWqVUo3S0QkFPNLPMUDLDAwEABga2truqklEVFNw1WI4ikWYGFhYfed1Ny8ebNSTRMRCcchRPEUCzDumEFETxLml3iKBVi3bt2UOrVZ/m+ot6rtk/Y0DOA/qujeilKWPvY5uIxePCH3AyMiqukUvyaJqmCAERHJQMdFHMIJC7CSkhJYW1uLao6ISCjml3iK93qTkpLQoUMHtG7dGgCQlpaG119/XelmiYiEkiTJrAfJR/EAi4mJwfbt29GoUSMAgLe3N/bv3690s0REQllJ5j1IPooPIVZUVKBly5aVjul0OqWbJSISip0r8RQPMFdXVyQlJUGSJJSXl2PJkiVo06aN0s0SEQnFC5nFUzzAli1bhpiYGFy4cAHOzs544YUXsGzZMqWbJSISSsf8Ek7xAHNycsLGjRuVboaISFXsgYmneIBFRkbec+VNXFyc0k0TEQnD/BJP8QB74YUXTH8uLi7Gli1b4OrqqnSzRERCcYWheIoH2JAhQyr9/NJLL6Fz585KN0tEJBSHEMUTvpXU+fPnkZ2dLbpZIiJFMb/EUzzAGjZsaJoDq6iogKOjI+bOnat0s0REQulkSrDi4mI899xzKCkpQVlZGcLDwzFr1iycP38eQ4cOxbVr1+Dn54f169ejTp06srRpqRTdicNoNCItLQ05OTnIycnBjRs3cO7cOQwePFjJZomIhJNrJw5ra2skJiYiLS0Nqamp2L17N3755RdMmTIFEydOxNmzZ9GwYUOsXLlS+Q+lcYoGmCRJ6NWrF3Q6HXQ6HfcBI6IaS64AkyQJNjY2AIDbt2/j9u3bkCQJiYmJCA8PBwCMGjUKW7duVfLjWATF90LU6/VISUlRuhkiIlXJuZlveXk59Ho9nJycEBoaCnd3dzg4OKBWrTuzPs2bN8fly5eV/DgWQbE5sLKyMtSqVQspKSkICAiAu7s7GjRoAKPRCEmScOzYMaWaJiISztxl9HFxcZWug42KikJUVFSl1+h0OqSmpiI3NxdhYWFIT0+Xs9QaQ7EA69SpE44dO4aEhASlmiAi0gxzb2h5r8CqjoODA7p06YLDhw8jNzfX1DG4dOkSmjVr9jjl1giKBZjRaAQAuLu7K9UEEZFmyHUhc05ODmrXrg0HBwcUFRVh7969mDJlCrp06YJNmzZh6NChWLt2Lfr37y9PgxZMsQDLycnBxx9/XO3zb731llJNExEJJ9cataysLIwaNQrl5eWoqKjA4MGD0adPH7Rv3x5Dhw7FtGnT4OPjg3HjxsnToAVTLMDKy8tRUFBg6okREdVkVpAnwTp27HjPhW9ubm5ISkqSpY2aQrEAc3FxwYwZM5Q6PRGRpvAqIfEUnwMjInoScDNf8RQLsO+//16pUxMRaY65qxBJPooFmKOjo1KnJiLSHO5GL57w3eiJiGoi5pd4DDAiIhkovi8fVcEAIyKSATcrF48BRkQkA8aXeAwwIiIZyHVDSzIfA4yISAbML/EYYEREMuAcmHgMMCIiGXAVongMMCIiGbAHJh4DjIhIBtyJQzwGGBGRDDiEKB4DjIhIBhxCFI8BRkQkA8aXeAwwIiIZsAMmHgOMiEgGVuyDCccAIyKSAVchiscAIyKSAfNLPAYYEZEMOIQoHgOMiEgG7IGJxwAjIpIBA0w8BhgRkQx4PzDxGGBERDKQOAcmHAOMiEgG7ICJxwCrwWZMm4ofDh6Ao2MjbP52u9rlkEpat3TC+nljTT+3atYI/162A59vT8L6eWPR8ilH/P7HdUS8vRK5+UUqVmrZ2AMTjxso12D9/3cAln32f2qXQSrL+P1PBA2di6Chc/H08Hm4VXwbCfvTMGlMKA4knUaH/rNxIOk0Jo3prnapFs1KMu9B8mGA1WB+/gGws7dXuwzSkC6d2uL8pRxcyLqBPiEdsWHbrwCADdt+Rd8uHVWuzrJJZv5H8tH8EGJ5eTmys7NRVlZmOtaiRQsVKyKyXIN6+OGr3UcBAE6NbHHl6k0AwJWrN+HUyFbN0iwee1fiaTrAlixZglmzZsHZ2RlWVnc6i5Ik4fjx4ypXRmR5atfSoffzHTBjScI9nzcaBRdUw3AvRPE0HWCLFi3C6dOn0ahRI7NeHxcXh7i4OADAmHFRGBcZpWR5RBalR+f2SE2/iD+v5wMA/ryWj6aN7XDl6k00bWyHnP9/nB4N40s8TQeYq6sr7B9iDicqKgpRUXdCq7jsAS8mesIM7ulvGj4EgB0Hf0NE30AsWL0XEX0Dsf0ARzYeCxNMOE0HmJubG0JCQtC7d29YW1ubjr/11lsqVmU5pkx6C0eSk5CbewOhXZ/DK6+9jgEDB6ldFqmgft066Brogej3403HFqzeiw3zxmLU/wbjQtZ1RLy9SsUKLR8XaIin6QBr0aIFWrRogdLSUpSWlqpdjsWZt+BjtUsgjbhVXIrmXaZUOnY9rxC9Xl6iUkU1D6fAxNN0gMXGxqpdAhGRWRhg4mk6wPr27QvpH98Ke3t7+Pv7Y8KECahbt65KlRERVcYhRPE0fSGzm5sbbGxsEBkZicjISNjZ2cHW1hZnzpxBZGSk2uUREZlIknkPc4wdOxZOTk7w8vIyHZs5cyaaNWsGvV4PvV6PnTt3KvRJLIeme2CHDh1CcnKy6ee+ffsiICAAycnJ8PT0VLEyIqLK5Ox/jR49GtHR0Rg5cmSl4xMnTsSkSZNkbMmyaboHVlBQgAsXLph+vnDhAgoKCgAAderUUassIqKqJDMfZnjuuefg6OioSJk1iaZ7YB999BE6d+4Md3d3GI1GnD9/Hp9++ikKCwsxatQotcsjIjIRsRPH0qVLsW7dOvj7++Ojjz5Cw4YNFW9TyySjUdsbyJSUlCA9PR0A0LZtW7MXbvBCZvqnhgHRapdAGlWUsvSxz5F2wbydTH7dHW/aMQiovAHD3xkMBvTp0wcnTpwAAGRnZ6Nx48aQJAnTp09HVlYWVq16sq/d02QPLDExEV27dsXmzZsrHc/MzAQADBgwQI2yiIiqZ2YHrLrAehBnZ2fTnyMjI9GnT5+HPkdNo8kAO3jwILp27Ypt27ZVeU6SJAYYEWmO0svos7Ky4OLiAgDYsmVLpRWKTyrNDyE+Kg4h0j9xCJGqI8cQ4m+XCsx6XYfmNg98zbBhw3DgwAFcvXoVzs7OmDVrFg4cOIDU1FRIkoR//etf+Oyzz0yB9qTSdICVlJTgm2++gcFgqHQ/sBkzZjzwvQww+icGGFVHjgA7YWaAeZkRYGQeTQ4h3tW/f3/Y29vDz8+v0ma+RERa889dg0h5mg6wS5cuYffu3WqXQUT0QMwv8TR9IfPTTz+N3377Te0yiIgeSMbrmMlMmu6B/fTTT1izZg1atWoFa2trGI1GSJKE48d54z0i0himk3CaDrBdu3apXQIRkVm4G714mh5CbNmyJS5evIjExES0bNkS9evXR0VFhdplERFVIedu9GQeTQfYrFmzMG/ePMyZMwcAcPv2bURERKhcFRFRVQww8TQdYFu2bEFCQgIaNGgAAHjqqaeQn2/efmNERCJJZv5H8tH0HFidOnUgSZLp+orCwkKVKyIiujf2rsTTdA9s8ODBmDBhAnJzc7FixQq88MILvBMzEWkSl9GLp+ke2KRJk7B3717Y2dnh9OnTmD17NkJDQ9Uui4ioKqaTcJoOMAAIDQ1FaGgorl69ikaNGqldDhHRPYm4oSVVpskhxF9++QUhISEYMGAAUlJS4OXlBS8vLzg7O3NrKSLSJA4hiqfJHlh0dDT+85//IC8vD127dsWuXbsQFBSE9PR0DBs2DD179lS7RCKiyphOwmmyB1ZWVobu3btj0KBBaNq0KYKCggAAHh4eKldGRHRvXEYvniZ7YFZWf+VqvXr1Kj3HWxYQkRbxf03iaTLA0tLSYGdnB6PRiKKiItjZ2QEAjEYjiouLVa6OiKgq5pd4mgyw8vJytUsgInooHB0ST5MBRkRkaZhf4jHAiIhkwPwSjwFGRCQD9sDEY4AREcmAS+TFY4AREcmB+SUcA4yISAZWDDDhGGBERDLgEKJ4DDAiIjkwv4RjgBERyYD5JR4DjIhIBlxGLx4DjIhIBryhpXiavJ0KERHRg7AHRkQkA3bAxGOAERHJgMvoxWOAERHJgD0w8RhgREQyYH6JxwAjIpIBb2gpHgOMiEgGzC/xGGBERDJgfonHACMikgMTTDgGGBGRDLiMXjzJaDQa1S6ClBUXF4eoqCi1yyCN4feCLB23knoCxMXFqV0CaRC/F2TpGGBERGSRGGBERGSRGGBPAM5z0L3we0GWjos4iIjIIrEHRkREFokBpgGSJCEiIsL0c1lZGZo0aYI+ffrc930HDhwwvSYhIQFz585VtM6/S01Nxc6dO4W1R5XpdDro9Xp4e3vD19cXhw4deuRzzZgxA/v27ZOxOiIxeCGzBjRo0AAnTpxAUVER6tWrh71796JZs2YPdY5+/fqhX79+ClVYVWpqKo4cOYJevXoJa5P+Uq9ePaSmpgIAvvvuO0ydOhUHDx58pHPNnj1bztKIhGEPTCN69eqFHTt2AADi4+MxbNgw03NJSUkIDg6Gj48Pnn76aZw+fbrK+9esWYPo6GgAQGZmJoKCgtChQwdMmzYNNjY2AO702EJCQhAeHg4PDw+MGDECd6dAZ8+ejYCAAHh5eSEqKsp0PCQkBFOmTEGnTp3Qpk0b/PjjjygtLcWMGTPw5ZdfQq/X48svv1T0d0P3d/PmTTRs2ND08/z58xEQEICOHTsiNjYWAGAwGNCuXTtERkbC09MT3bt3R1FREQBg9OjR2LRpEwBg586d8PDwgJ+fH2JiYkw9/JkzZ2Ls2LEICQmBm5sbFi9eLPhTElXFANOIoUOHYuPGjSguLsbx48cRGBhoes7DwwM//vgjUlJSMHv2bLz77rv3Pdcbb7yBN954A7/99huaN29e6bmUlBQsXLgQp06dwrlz5/Dzzz8DAKKjo5GcnGzqCW7fvt30nrKyMiQlJWHhwoWYNWsW6tSpg9mzZ2PIkCFITU3FkCFDZPxNkDmKioqg1+vh4eGB8ePHY/r06QCAPXv2ICMjA0lJSUhNTcXRo0fxww8/AAAyMjLw2muv4eTJk3BwcMA333xT6ZzFxcWYMGECdu3ahaNHjyInJ6fS8+np6fjuu++QlJSEWbNm4fbt22I+LFE1GGAa0bFjRxgMBsTHx1cZlsvLy8OgQYPg5eWFiRMn4uTJk/c91+HDhzFo0CAAwPDhwys916lTJzRv3hxWVlbQ6/UwGAwAgP379yMwMBAdOnRAYmJipTYGDBgAAPDz8zO9ntR1dwgxPT0du3fvxsiRI2E0GrFnzx7s2bMHPj4+8PX1RXp6OjIyMgAArVq1gl6vB3Dvv8v09HS4ubmhVatWAFBpFAAAevfuDWtrazRu3BhOTk7Izs5W/oMS3QfnwDSkX79+mDRpEg4cOIBr166Zjk+fPh1dunTBli1bYDAYEBIS8shtWFtbm/6s0+lQVlaG4uJivPrqqzhy5AhcXV0xc+ZMFBcXV3nP3deTtgQHB+Pq1avIycmB0WjE1KlTMWHChEqvMRgMVf7u7w4hmute3x0iNbEHpiFjx45FbGwsOnToUOl4Xl6eaVHHmjVrHnieoKAg0/DQxo0bH/j6u2HVuHFjFBQUmOZD7sfW1hb5+fkPfB0pLz09HeXl5WjUqBF69OiBVatWoaCgAABw+fJl/Pnnn2adp23btjh37pypZ8a5TdI6BpiGNG/eHDExMVWOv/3225g6dSp8fHzM+lfvwoUL8fHHH6Njx444e/Ys7O3t7/t6BwcHREZGwsvLCz169EBAQMAD2+jSpQtOnTrFRRwquTsHptfrMWTIEKxduxY6nQ7du3fH8OHDERwcjA4dOiA8PNzsf2jUq1cPn376KXr27Ak/Pz/Y2to+8LtDpCbuxFED3bp1C/Xq1YMkSdi4cSPi4+Nu62fPAAAEqklEQVTx7bffql0WWYCCggLY2NjAaDTitddeQ+vWrTFx4kS1yyK6J86B1UBHjx5FdHQ0jEYjHBwcsGrVKrVLIguxYsUKrF27FqWlpfDx8akyl0akJeyBERGRReIcGBERWSQGGBERWSQGGBERWSQGGCnq7q7pXl5eGDRoEG7duvXI53qY3fdzc3Px6aefPnQbM2fOxIIFC8w+/nd/31PQHAaDAV5eXg9dIxHdwQAjRd3d8ujEiROoU6cOli9fXul5o9GIioqKhz5vv3798M4771T7/KMGGBFZDgYYCfPss8/i7NmzMBgMaNu2LUaOHAkvLy9cvHgRe/bsQXBwMHx9fTFo0CDTThK7d++Gh4cHfH19sXnzZtO5/r77fnZ2NsLCwuDt7Q1vb28cOnQI77zzDjIzM6HX6zF58mQA996lHQA++OADtGnTBp07d77nTv//tGLFCgQEBMDb2xsDBw6s1Kvct28f/P390aZNG9OGyOXl5Zg8ebKp7c8+++zxf5lExAAjMcrKyrBr1y7TNlkZGRl49dVXcfLkSTRo0ADvv/8+9u3bh2PHjsHf3x8ff/wxiouLERkZiW3btuHo0aO4cuXKPc8dExOD559/HmlpaTh27Bg8PT0xd+5cuLu7IzU1FfPnz692l/ajR49i48aNpht0JicnP/CzDBgwAMnJyUhLS0O7du2wcuVK03MGgwFJSUnYsWMHXn75ZRQXF2PlypWwt7dHcnIykpOTsWLFCpw/f16eXyzRE4wXMpOi7m55BNzpgY0bNw5//PEHWrZsiaCgIADAL7/8glOnTuGZZ54BAJSWliI4OBjp6elo1aoVWrduDQCIiIhAXFxclTYSExOxbt06AHfm3Ozt7XHjxo1Kr/n7Lu3AnR0nMjIykJ+fj7CwMNSvXx8AzLop6IkTJzBt2jTk5uaioKAAPXr0MD03ePBgWFlZoXXr1nBzc0N6ejr27NmD48ePm+bH8vLykJGRgTZt2pj/iySiKhhgpKi/3zn47xo0aGD6s9FoRGhoKOLj4yu95l7ve1TV7dK+cOHChz7X6NGjsXXrVnh7e2PNmjU4cOCA6TlJkiq9VpIkGI1GLFmypFLQAeCtaYgeE4cQSXVBQUH4+eefcfbsWQBAYWEhzpw5Aw8PDxgMBmRmZgJAlYC7q1u3bli2bBmAO/NNeXl5VXbLr26X9ueeew5bt25FUVER8vPzsW3btgfWm5+fDxcXF9y+fRuff/55pee+/vprVFRUIDMzE+fOnUPbtm3Ro0cPLFu2zHQDyDNnzqCwsPAhf0tE9E/sgZHqmjRpgjVr1mDYsGEoKSkBALz//vto06YN4uLi0Lt3b9SvXx/PPvvsPXdWX7RoEaKiorBy5UrodDosW7YMwcHBeOaZZ+Dl5YUXX3wR8+fPx3//+18EBwcDAGxsbLBhwwb4+vpiyJAh8Pb2hpOTk1k78f/73/9GYGAgmjRpgsDAwEo1tWjRAp06dcLNmzexfPly1K1bF+PHj4fBYICvry+MRiOaNGmCrVu3yvTbI3pycS9EIiKySBxCJCIii8QAIyIii8QAIyIii8QAIyIii8QAIyIii8QAIyIii/T/AA1zKkAtpwvKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp.artifacts['confusion_matrix.png'].render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>kernel</th>\n",
       "      <th>result</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>exp_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>rbf</td>\n",
       "      <td>0.622807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>linear</td>\n",
       "      <td>0.956140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>rbf</td>\n",
       "      <td>0.622807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>linear</td>\n",
       "      <td>0.956140</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        kernel    result\n",
       "exp_id                  \n",
       "1          rbf  0.622807\n",
       "2       linear  0.956140\n",
       "3          rbf  0.622807\n",
       "4       linear  0.956140"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader.find_all().project(on=[\"config.kernel\",\"result\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "step\n",
       "0    0.965517\n",
       "Name: F1, dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp.metrics[\"F1\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some powerful usage patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Configurations from files\n",
    "Instead of defining your configuration in a Python function you can outsource it to a JSON or YAML file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# %load config.json\n",
    "{\n",
    "  \"C\": 1.0,\n",
    "  \"gamma\": 0.7,\n",
    "  \"kernel\": \"rbf\",\n",
    "  \"seed\": 42\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started run with ID \"5\"\n",
      "INFO - svm - Result: 0.6228070175438597\n",
      "INFO - svm - Completed after 0:00:00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sacred.run.Run at 0x7f9c1bf63a58>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sacred.observers import MongoObserver\n",
    "\n",
    "ex = Experiment(\"svm\", interactive=True)\n",
    "\n",
    "ex.observers.append(\n",
    "    MongoObserver.create(\n",
    "        url=mongo_uri,\n",
    "        db_name=db_name,\n",
    "    )\n",
    ")\n",
    "\n",
    "ex.add_config(\"config.json\")  # Load the config from a file.\n",
    "\n",
    "@ex.main\n",
    "def run(C, kernel, gamma):\n",
    "    X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "    clf = svm.SVC(C, gamma=gamma, kernel=kernel)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    return clf.score(X_test, y_test)\n",
    "\n",
    "ex.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Invoking experiments from the command line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# %load experiment.py\n",
    "import os\n",
    "\n",
    "from sacred import Experiment\n",
    "from sacred.observers import MongoObserver\n",
    "from sklearn import svm, datasets, model_selection\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "\n",
    "db_name = \"pydata_berlin\"\n",
    "mongo_uri = None\n",
    "\n",
    "ex = Experiment('svm')\n",
    "\n",
    "ex.observers.append(\n",
    "    MongoObserver.create(\n",
    "        url=mongo_uri,\n",
    "        db_name=db_name,\n",
    "    )\n",
    ")\n",
    "\n",
    "     \n",
    "@ex.config\n",
    "def cfg():\n",
    "    C = 1.0\n",
    "    gamma = 0.7\n",
    "    kernel = \"rbf\"\n",
    "    seed = 42\n",
    "\n",
    "\n",
    "@ex.capture\n",
    "def get_model(C, gamma, kernel):\n",
    "    return svm.SVC(C=C, kernel=kernel, gamma=gamma)\n",
    "\n",
    "           \n",
    "@ex.automain # Use automain to enable command line integration.\n",
    "def run():\n",
    "    X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "    clf = get_model()\n",
    "    clf.fit(X_train, y_train)\n",
    "    return clf.score(X_test, y_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started run with ID \"6\"\n",
      "INFO - svm - Result: 0.6228070175438597\n",
      "INFO - svm - Completed after 0:00:00\n"
     ]
    }
   ],
   "source": [
    "!python experiment.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started run with ID \"7\"\n",
      "INFO - svm - Result: 0.956140350877193\n",
      "INFO - svm - Completed after 0:00:01\n"
     ]
    }
   ],
   "source": [
    "!python experiment.py with kernel=linear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started run with ID \"8\"\n",
      "INFO - svm - Result: 0.6228070175438597\n",
      "INFO - svm - Completed after 0:00:00\n"
     ]
    }
   ],
   "source": [
    "!python experiment.py with config.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Modularizing using dynamic imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# %load config_svm.json\n",
    "{\n",
    "    \"model\": {\n",
    "        \"path\": \"sklearn.svm.SVC\",\n",
    "        \"hyperparameters\":{\n",
    "            \"C\": 1,\n",
    "            \"gamma\": 0.7,\n",
    "            \"kernel\": \"rbf\"\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# %load experiment_dynamic.py\n",
    "import os\n",
    "from importlib import import_module  # For dynamic imports.\n",
    "\n",
    "from sacred import Experiment\n",
    "from sacred.observers import MongoObserver\n",
    "from sklearn import svm, datasets, model_selection\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "db_name = \"pydata_berlin\"\n",
    "mongo_uri = None\n",
    "\n",
    "ex = Experiment('svm')\n",
    "\n",
    "ex.observers.append(\n",
    "    MongoObserver.create(\n",
    "        url=mongo_uri,\n",
    "        db_name=db_name,\n",
    "    )\n",
    ")\n",
    "\n",
    "             \n",
    "def load(path):\n",
    "    \"\"\"Load a class, given its fully qualified path.\"\"\"\n",
    "    p, m = path.rsplit('.', 1)\n",
    "    module = import_module(p)\n",
    "    class_or_method = getattr(module, m)\n",
    "    return class_or_method\n",
    "     \n",
    "\n",
    "@ex.capture(prefix=\"model\")  # Prefix to only get the relevant subset of the config.\n",
    "def get_model(path, hyperparameters):\n",
    "    Model = load(path)  # Import model class dynamically.\n",
    "    return Model(**hyperparameters)  # Instantiate with hyperparameters.\n",
    "\n",
    "           \n",
    "@ex.automain # Use automain to enable command line integration.\n",
    "def run():\n",
    "    X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)\n",
    "    clf = get_model()\n",
    "    clf.fit(X_train, y_train)\n",
    "    return clf.score(X_test, y_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING - root - Added new config entry: \"model.hyperparameters.C\"\n",
      "WARNING - root - Added new config entry: \"model.hyperparameters.gamma\"\n",
      "WARNING - root - Added new config entry: \"model.hyperparameters.kernel\"\n",
      "WARNING - root - Added new config entry: \"model.path\"\n",
      "INFO - svm - Running command 'run'\n",
      "INFO - svm - Started run with ID \"9\"\n",
      "INFO - svm - Result: 0.6052631578947368\n",
      "INFO - svm - Completed after 0:00:00\n"
     ]
    }
   ],
   "source": [
    "!python experiment_dynamic.py --name svm with config_svm.json "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model': {'path': 'sklearn.naive_bayes.MultinomialNB',\n",
       "  'hyperparameters': {'alpha': 1}}}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# %load config_bayes.json \n",
    "{\n",
    "    \"model\": {\n",
    "        \"path\": \"sklearn.naive_bayes.MultinomialNB\",\n",
    "        \"hyperparameters\":{\n",
    "            \"alpha\": 1\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING - root - Added new config entry: \"model.hyperparameters.alpha\"\r\n",
      "WARNING - root - Added new config entry: \"model.path\"\r\n",
      "INFO - naive_bayes - Running command 'run'\r\n",
      "INFO - naive_bayes - Started run with ID \"10\"\r\n",
      "INFO - naive_bayes - Result: 0.9210526315789473\r\n",
      "INFO - naive_bayes - Completed after 0:00:00\r\n"
     ]
    }
   ],
   "source": [
    "!python experiment_dynamic.py --name naive_bayes with config_bayes.json "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model': {'path': 'sklearn.ensemble.RandomForestClassifier',\n",
       "  'hyperparameters': {'n_estimators': 20, 'criterion': 'gini'}}}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# %load config_random_forrest.json\n",
    "{\n",
    "    \"model\": {\n",
    "        \"path\": \"sklearn.ensemble.RandomForestClassifier\",\n",
    "        \"hyperparameters\":{\n",
    "            \"n_estimators\": 20,\n",
    "            \"criterion\": \"gini\"\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING - root - Added new config entry: \"model.hyperparameters.criterion\"\n",
      "WARNING - root - Added new config entry: \"model.hyperparameters.n_estimators\"\n",
      "WARNING - root - Added new config entry: \"model.path\"\n",
      "INFO - random_forrest - Running command 'run'\n",
      "INFO - random_forrest - Started run with ID \"11\"\n",
      "INFO - random_forrest - Result: 0.9649122807017544\n",
      "INFO - random_forrest - Completed after 0:00:00\n"
     ]
    }
   ],
   "source": [
    "!python experiment_dynamic.py --name random_forrest with config_random_forrest.json "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The the SciPy talk for more information https://youtu.be/qqg7RO0o1OE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQEDEQH/xAAbAAABBQEBAAAAAAAAAAAAAAAAAQIEBQYDB//EAEsQAAEDAgMDBwkGBAMHAwUAAAEAAgMEEQUSITFBkgYTFlFTYdIUFSIyVHGBkdEjQlJyorEXNWKhJENEBzNVgsHw8TRz4SU2RWOy/8QAGAEBAQEBAQAAAAAAAAAAAAAAAAECAwT/xAAfEQEBAQEAAQQDAAAAAAAAAAAAARECIQMSQWETMUL/2gAMAwEAAhEDEQA/APP0IQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgELQdD8Q7al4neFHQ/EO2peJ3hQZ9C0PQ7EO2peJ3hR0OxDtqXid4UGeQtD0NxDtqXid4UvQ3Ee2peJ3hQZ1C0XQzEe2peJ3hS9C8R7al4neFBnELR9C8R7el4neFL0KxLt6Tjd4UGbQtJ0JxLt6Tjd4UvQjEu3pON3hQZpC0vQjEu3pON3hS9B8T7ek43eFBmULTdBsT7ek43eFL0FxPt6Tjd4UGYQtP0ExTt6Pjd4UvQTFO3o+N3hQZdC1HQPFO3o+N3hS9AsU7ej43eFBlkLU9AsU9oo+N3hR0BxX2ij43eFBlkLVdAcV9oo+N3hR0AxX2ij43eFBlULVdAMV9oo+N3hR0AxX2ii43eFBlULV/w/wAV9oouN3hR/D/FfaKLjd4UGUQtX/D/ABX2ii43eFH8P8V9oouN3hQZRC1f8P8AFfaKLjd4Ufw/xX2ii43eFBlELV/w/wAV9oouN/hR/D/FfaKPjd4UGUQtX0AxX2ii43eFJ0AxX2ij43eFBlULVdAMV9oo+N3hR0BxX2ij43eFBlULU9AcV9oo+N3hR0CxT2ij43eFBlkLU9AsU9oo+N3hSdA8U7ej43eFBl0LUdBMU7ej43eFJ0FxTt6Pjd4UGYQtP0FxPt6Tjd4UnQbE+3pON3hQZlC03QfE+3pON3hSdCMS7ek43eFBmkLS9CcS7ek43eFJ0JxLt6Tjd4UGbQtJ0KxLt6Xid4UnQvEe2peJ3hQZxC0fQzEe2peJ3hSdDMR7al4neFBnULRdDcR7al4neFJ0OxDtqXid4UGeQtD0OxDtqXid4UdD8Q7al4neFBnkLQdD8Q7al4neFHQ/EO2puJ3hQbZKhKEBZCE4IBKAgBKgLJQEoCUBAAJQEoCUBAAJQEoCcAgQBKAlATgECAJQE4BKAgQBLZOAS2QNAS2TgEoCBLJbJbJbIG2S2TrJbIG2RZPsiyBtkWT7IsgZZFk+yLIGWRZPsiyBlkWT7Isg52RZPsiyDnZJZdLJLIOdkhC6WSWQc7JLLpZJZBzITSF1smkIOZCaQupCaQg5kJpC6kJpCDmQmkLoQkIQcyE0hdCE0hAxJZPITSEDEieQmlA1IU4pEDUb0pSIFSoShAqUBASoAJwQAlCAsnAIATgEAAlASgJwCBAE4BACcAgAEoCUBOAQIAnAIATgECAJ1ktktkCAJbJbJ1kDbJbJ1kWQJZLZLZLZA2yWyWyWyBtkWTrIsgbZFk6yVAyyLJ1kWQNsiydZFkDbJLJ9klkDbJLJ9klkDLJLJ9klkDLJLJ9klkHMhIQuhCaQg5kJCF0ITSEHMhNIXUhNIQcyE0hdCE0hBzITSF0ITSEHMhIQnkJpCBhSFPITSgYkKeQmoGpN6ck3oEe9sYu82F7JjqqFhs54HwUbGDlowQbWeFUmtc7Q6lBejEKXdJ+kpfONIP8AN/SVQsqQ1hYWA33rqZoBHZlg73ILnzpRj/O/SUoxWi7b9JWefZ+uxcSEGpGK0XbfpKcMUou2/SVl292xdG3NgBr1INMMUo+2/SU4YrRdt+kqnpsLqJ7HKWX2ZtFJ8yuYQJXkE7xsQWHnahH+f+kpRi9AP8/9J+ipanDpIXHQOaN6iilc7YEGlGM4f2/6T9EvnrD/AGj9J+iy76aSO2ZtrrlkQa8Y3h3tH6T9EvnvDvaP0n6LHFhCUMJ1A0CDYjHMO9o/Sfol8+4b7T+k/RY3KRuTdmlkG18/Yb7T+k/RL5+wz2kcJ+ixB03JPcg3IxzDfaRwn6J3nvDvaP0u+iwzXaJ4cEG3GN4eb2qL2/oP0R58w4f6j9J+izlPiFPFCGPiB67Bcamoon35uJwvvCDUefsN9p/Q76IOP4YNtT+h30WTzUeXTMD3hc5XwStADwLINf0hwv2r9DvolGP4YdlT+h30WUpKaje60k7b/JLVUsEXpQzNc3qvqg1fn7DR/qP0O+iaeUOFjbVfod9Fi3EBcnWKDcdIsK9q/Q76I6RYV7V+h30WCcmIPQOkWFe1Dgd9EdIsK9q/Q76Lz/5pQCdgug9A6QYWf9SOF30S+fsM9p/SfosNTwvmdZpaD3mytIcDmeLukYBusboNJ5+w32n9DvoujMWoXi7ZiR+Q/RZaTDJacl2ZriNbLgK6RhtYCyDY+c6QjSRx/wCR30TXYrRtF3SED8jvosw3FX3BLW6J8mKl4sY2lBoDjmHe0foP0SDG8OJAFRqdPVP0WRmlbIb5QD3LnFbnmfmCDd1dTDRwGaofkjBsXWUDpBhXtbeF30TOVv8AIZfzN/dYMIN95/wv2tvC76JPP2F+1t4XfRYNCK3Zx3DPam8J+iQ47hntTeE/RYZCDbnHMN9qHCfok894b7SOE/RYlCDaHGsO9pHCfokONYd7SOE/RYxCDY+ecP8AaRwn6JDjGH+0DhP0WPQg1xxeg9oHCfokOLUHtA4T9FkkqDVnFqH2gcJ+iacWoe3HCfosqhBqfO1D24+RUiCeOoYJIXZm3tdY0rS4D/Lh+coiVXiM0x51pLb7lQVDIhrEHjrurzFZuZonOte5yrPmUHbv60HMJba7U4i+wD5pADbYgUN0RlsntuSNmqsKamw95AlqH5t+WwCCtbZWNLVUsLW5oc5A2lS34ZhrYjMKx4a3aDbVQHjCS+7K8tbva4el80RZt5QRt0EZAUpmOwyjLYknQCypJqBriTR1Dahtr2aRm+Shl7oDmt6Q3INLUuOQAxStadrtoVVM6RguDdm4hdqTGZIgM5EsdvSsNB3K0opYa0OyUbBC/abbUVCw+sikYIpYw97T6NwraOnhk9I08Vzt0VScIrIJ88UbC2+52xWzPKIoQ4xjNvAN0CuwykeD9i0E/wBlBqcLfHpAyMt+RViK1jAc+0IdUMdFmDgxx2F2xEUb8GqMwFm2O8blGq8Nlp35bZh1tC07HOMbS17LnTrF1ErHYi2ZjYGMcCNXEWHxRWZbRTSAubGco3nQBPjw18pIjmgc78IfqVrI4WSRtdUwR5jqdLrm+gYJGyUznRBu1rNh7rIMnLh9TE/K+Ih1tnWpAwp8YDqmQRgi+UC5V/NWFrg0MFx9521QcRqnZi4t2DYggUUrY5xFDSiRrtCZBc7V3xHCZX1RdTsYGv1y7Mq5QYhGyZr3+iBvtdWkNVS1EnPGZ9+oaIKDzVXG+WB59yiSQyMcWvaWkbQQttzLJ5BPG97SNLX2rlW4XFWSZ3ueDa12kFBidQlDldyYMIakiV5MI1DvxJ0suEkZZKcXbvZpdBRlyQu71qYIcJqY2xtiiAI6/SCrcSwJ8AfLAQ+Ea7dQgpiku0bbpH6Jl0EulFO+QCeQsb7loKCnoImkxyNffebXWTLu9NzW2FBq6umormVswzk/iChQYyaZ+V1y3qBVDm70t/ig0ruUMDgc0TifcFVVFQ2WQuaCATvUAXXRoKDu15XQElcG3C7skc0WBsECuCItJo7/AIghz7pIj9vH+Yfug13K3+Qy/mb+6wS33K3+Qy/mb+6wKKEqEKAQhCoEIQgiVdWYjkj9beepco/KyQ7OVxqGuM7ngXF1JEvNtGa/wWbVkSYXucLPADh1b10UeOW8rQGn0gpKsKEJEqqBCEIEWlwH+XD85WaWlwH+XD85QOxwXoP+cKjpxC5+WY277rVyUcdazmZCQNtwuUvJinfbm5XMsOq90RnaiGBozRSg9y4NJzAZtFZ1mB1VI0uLczB94KrdG5qCyNThVMzm6hzi+2uh0StqMC5pshY7LfXI51vjdZ2reHuDSLOGhJUvC6MucQ/D3VEd82YhwuO5AzFK6GSQx0jcsLdmtyVWNvJILgm53LY9G8OnAexk8Nxcsva3zUWmpMLw5z3PmdJK19m6Dcgm8nMLlpoufB9JwuGEa/NT6tjaSodWkkyZbWy6aqLTY43OACLdSm1dQKylPNMZK61wCdUQ+hqZHNBlgdlft9HapjphBG1tPCDpsBAssgG43FlkBLGN1sZBoO/VNfLFVuke/Ecs5GxjfQQaCoLqudoe6ppZHg2AILVX1XnmllJjcXRss0v3EdahvraaGjhY58zww5nvYQLH91FrsZpamExieqBHqk2IRVtNj0LYWtncJ5XdTbZSonnWarNg0DmwSC0XKzMTJJ32ja57iuxbWUE4zskhftF0G4wt7TA54L3NIuY3aa9xUhtTUvDYIobseL5y69gqHDeUPM0+arqLkf5drly4xcomwT3ZHZgJIaTsQasUtSL8zVHZ6pCnUpfzQ5y4cNt1RUmMmrpJJB67Bm0Cm02NQyNYC+7naE2RDMQfmrmw5LuNterrKi447mzHJplc2ynzMDKnyh5GYtygjenvpoKyAwzC7b3b1hFZB8zZDa1upTaDCZ3StfnyRuF73XWXk3ViZ3NOZl3XOic6ZtE0xTSESt002IJbakUgML80rb6uabJalk0eWeIPDHakHcq3yoPNw4X3K2w+eKrhfFK4l17kX2IhsLzVt5iaLNHbbvCo8bwbyaFtRRmV7DfNf7q0kEJaM9K+7b2OZd6psklJKIcr3ZCMvWUV5/h9VMKhrYz9odBdalks8eGzGdnOEj0g0gn3qnw3BnCtf5S2WN8fpDKQLK4kr6OhBGhkI1uBqgy8ut9Fx+a11FLQVpLTRw32+oqzFqWngrXsgh9G17bbe5BRXTTdWsZiZ61E599BodVxqYWAZjTvhzbAd6CCCnApCLIAQdWFdgR1Lg1dWoOuZJnsm30THFA8yJ0D/t4/zD91xaYy70rhq6RiMVMeRxcMw/dBuOV38hl/M391gVv+V38hl/M391gEUIQhAIQhQCR2w2SprnBou4gDrKor2tlDnZ26JWTCxFtif5SJ5XBnqD+64TREOzM0KmNSpdLJzjz1N32UpVVPWcwXB7bg9SnRVcMugfY9R0TEtd0JEqqFQkQgFpsB/lw/OVmVpsA/lrfzlBc0o+2+Cnhut1XwPEclypzZmnejJZY+cicw/eFtQs9VcmXPuYpxc7MzVow4daRxBG1B51jODVOHEyPAc07HDYCqylxGsp5PQqC0D8TtF6dXuh8lk59gfGBctIvdeW4kYHVEjqdpZGToDuCKsajlE8H7J73uGty42v7lUVFbJPK6Vxs5xubKKdqBa2qCQyqe3YTdSYMZq6Z94n5e7aFWE2Re6C0qcarqyPm5JLjuaFAc97SdSD8l0hqjT5DCMkgFi/eVcQYlS1zWR1sDXyWtn2FBQ5zvKGu1XSqEXOv5vRt9ATdRr2KCdTVb6d2Ztib7wruHlEx8IZVxsdb1SG3Le+5Waja+Q2Yxzj3BLIySJ2WRjmOG4iyC3xPFG1jgGAhvfvXZ2HmahZPTtYS0enlfmJ+G5UtPMyKQPewPtuK1GGY3TSkNMDY5dgINrhBWUVRUU+cxZrAWNtissHllhrPtYzpqLi2qk4yYmQNmaxoc51i5n3h3rPy4nO192POmg7kGkl5VyOzRmJoGyyhwYy6Sua58jmXOmTX4KhBlma6QkWB9JxKtsOximggEctOy8erHDbf3oNvFWx8010j8n5tLqM6LCq6oLpo2GS/4tCqE8q47Boga4f1G6mU+J0U9K+RtOxj8u1o3oi1kwmkdK19O1seXRwZouNNgcVPUvkZO+/3R3LPT8oZQXc3ZhPVvS0HKB0LhLO/ML2tvRWvp3wupi2I6HQ5W2XEg0bM4qBY7njVQ6flFQ1AIBIJGqK2rAkikDS6LLcuIvZB0qZKqsb9mxjWkbXOtdZDEYaiKsLJmZXjde601LUmeYFwIB9XRJVYdR4lVTAc42paPcCUCYHTRmAlzsrzvKt/N1O9lyA8223WZOG4nTR8/I9hZGdWB+tldQVT3U7XQXuwZrDW/d70Qc15O7fladLptdhkddE3LcO6ydFLN6uNr3tyE7QV0bT5SPT+ARWffyXmL/QkBbbaVzfyWq2jRzD8VqS57PchlQ4myDKScnZ2U5ka8OcNrVVvjdGcrhYjrXogcCPVVbi9CKiECKNgffadEGKJKaVPqsMqad1nx2vssVDMEg2sd8kHFPp//AFEX5x+6eKaVx9FjinQwPZUx52kHON3eg3XK7+QS/mb+6wC3/K7+QSfmb+6wCLCJUiFAqEIVEarqhTtFhdx2BVUs0kpu9xPcutdJnqXdTdAo6B0byx2ZqlPnaYcw27LKI0gHXYhzcriCrnyaQ/uhPkOrRa1gmbksyos8NlL43Mcbluz3KaFU4c/LUgbnCytlFKkQhALUcn/5aPzlZdajk/8Ay0fnKCxkaHRm5I3iyrGYpzMhDgXi9vWtb3p/KGd9PhhfGbEvAWNdO8uuTqiPQYqtsjARKG5hp3FcZZcQbbm5ong7SQsbT4pJC8E5nNG0XU/pK0NuYnAjvQaWYyugL3TFoaLuFl55iL2OqHlnqlxsp9Zynqp2SRsDWRvFrW1VI95cblAwm5TSUt9E0oBF0l0IOsNPPUNeYYnyCMZnZRfKF0o6SqqecdTwmQQtzv7h1rbciqeOHBryD0617mj3AH6FVXJhnNyYvDv8mePldBnaSjqa6Yx0sRkeGlxaOoLhqFouRJ/+uPHXA8fsqB49I+9B0hiqTDJURRvMcRGd42DqXWrjrXSxeUQvD5WjJdurhusrbBBn5L4wwbRkP9105UvPMYRK1xaRTAgg6ghBSVGGV1JGJKmlmiYTYFzbBdYMJxGalNVFSSOhbrntuVxyummy4eDK8xvpmuylxsTrr71JxiozYThlbC3PSCLm3sBsGnvQV2DYqYT5NO0va/8AsrGbCKPyLykU8zwWmxaSRfrWTqJWvlLo4xGDsAOxb3CHPjwGGWY/4MUThJc6F1zb4oMcyhrKkuFLTTShu3K0my5QUNZVVJp4aeR8o2tA1C0WDVEsXJjFJIJHska9tnNNiNibyRlM5xKnMjhUzxeg+/pHbfX4oKKbD6ynqRTS08jZjsZa5KnMhlyspYKeYVL7hzGuve3cu+HMEXKajtLI+z7ene4OyynUBLP9oMovtkf+xQUcGHVlZznMQudzRs/W2X3qVTYPWxTB4pHVYA2NaS0H/qrHDZDzPKMA2IBI+blR4XXVQxCmhZVTsjfK0FrZCAdUGojkMNIJcQwsxc1rfmbALtTVte0iZtK/yN2ofIA0Ae5UvKutnZj0kRnlMAyEx5zlOg3LpyufMZ6erglcaWaJuTKdBbciNBPXVDJxzbY2Q7S+9guorqOSRzmOdJKRdwi10C83kqZpD6cr3e8rZ8homxUclTLtnlETfcB/5RVzR1lJMHGla5z9rmt10XOesgpJjJGSZJTYxdR9yh4RQy0ktYI8rpcrgwA9+ijYzOzpPRsFucBYJCPxXQaDyv8AwpkmjdGf6m2UeOta45g64XOtmjqZJ6WSoawh+mbVU9U2ShlGsTwdhaEGmZVh40unC4N/VuszSYzI2QMyDXqKnuxR7iWvaQPeg0MWrdoRJDzlrgEBUEeIEG7XaKXTYruOvfdBainie0B8TXAdYuoVZhEb7vhbYk7ApkNUJG3XZsgcLojNuw2Vrw0B1zs0XZmF1TS0kNcL7OpX+YFOaUVWcsCG8n5STYZ2/uvPecZ+IfNb/lsznOTkzb2u9n7rzTyIfiKixNzt/EPmjMOsKF5EPxFJ5H/X/ZBPuOtI5wa0nqCgeRn8f9knkrvxpoguJcSTvN0gU51PGW6jXrChyR5COo7FZSw1PJDmDT0h/dMSt9YLUqFeczyUgF9Ei6U/++apaO9JTvE7XO9EDVWl1BJfbKw+kNib/ivxH5rOrmLC6FX3quspc9V1n+yCetTye/lg/OVh+cqv+7LZ8li92DtMnrc45ULyp/lB/wDcasYVs+VGuEH/ANxqxxZqiOZSJ/NkIynqQR5WaXDfiuCsQ1RKkBrzYWQcE06oKEDUrQXOAAuToEi60swp6qKYsDxG4OynfZB6JSUc8FXg1PE6IRU0ZMoMgDsxad21VOCR5OUWMQ9bJQPmqU4+93KFuKvhBynSPNuta10+i5Q+S45UYi6mDxPmBjzWtfvQd+RZtyiYOuNwVHUtyzyN6nEK2hxylpcairqPDxDGxpBj5wm5O+6i4vX0ddK2SloRSnUv9MuzEoLPknNLFRYsYXZZGwh7TYGxF9xXXlDUOmw/B6mqHOufCS77t9e5U2E4p5tFUOa5wVERj22t3rpW4wyro8Pp3U+lI3K70vX1/sgtuVbmyYfhMobbNBp/ZcuSE3lT6nCJ/Sp6iMkD8LhvC4Ytj9HiGHRUzMPMboRlidzpOUf9U6kxzDcNjfLh+HysrHx5M8kuZreshPgUdRGYpXxna1xC9BwuNtTgNLh7zpU0Zy/mB/8AledvkL3EnaVfRcpTDHhYjgIdRXDjm9cHRKJ+By1GH8nsWkiOSaKQC5ANjoDtWdhrqiCsbVsfaVrs9+sq6h5T0efEGz4e58FY8PLGyWsoMdbghqZnTYbPzRIMbWTat67oLzFWtbjeE4iwFranK9zdwNx9UGMxf7RW7szs3zYqLGMdfiNTG9kfMxQgCJgN8oCs2crKV0kVZUYeX4hEzI2QPs095CDthQvNyjj3mN/7uWbw/wBHFKU//tb+6m4TjwoKuqmmp+fbUtLXtzW2lRqivpnYlHU0lGII2Oa4R5ydnegn8sf/ALhn9zf2Urk884pgtdhk/pCFnOwuP3T/AN/uqjGMTZieKuqzCWxutdmbWw71ObjlBR0U8OFUUsMlQMr3ySZrDqCCldYHYvQcPp6impsIpouaGU85MHPAOvUD7yvPmSWla8szBpBynf3KzqsekqcahxEwACEtyx5tw70Grop5o+UWKwtNmsY54HfcKPjdKG8oaCrZbJUvY743Cp6blMIsYq651N/6hmXIHbNn0UiflPTzQUbHU7iaeQSBwPVuQWOJNnrccljnc400DvRytGmgUk0VFO/MXOLBoLFVU3KelqJHvpqR8E8jgXSOkuPkudbjOaJgja1sh9axQaaLDKKMB0cIJ3XN0V+HMqY2lo5t43hZKLlDXQgBrsw/qF1Ii5V1QJ5xjXDu0QdauGWkfkc646wpeFxSylr2szNv1qhrMWdUvuG5dbqRh2PT00jWnJze+wsg3kIs0FzbLsGgj0SbdSpW4y18TcpDnO3qRHim4NCIssjmAlpKjitMUoa+9idLrtT1ImGijV9LFO5j3PLC03tfQopOWrxHycmcTpnZ+684FWz8Q+RXoXL7TktP+dn/APQXlTZHNN2usVKsWnlTPxNR5TH+JvzVYJpGm4dqhsz2ggEa7dEFn5Qzrb80c+zrbxKtdO9zcpy2H9KUVDxbRmn9KYJchIYSo9XbLEBbQdakxva+PUjVQ5oSH+gLt7lItcV2ip5XZXiNxadhA2rnzUn4D8leUDmNoo2yOAIvoTay3GVNLTywtDnsLQdlwinaXy6bQrLF3tkijbGcxDtgXKGNtPFc+sdqnVxqQ7Rkrb+5dcw6j8lXTz5n6bFz509bvmsyFWuZvf8AJLmb3/JVXPHrd80c+78Tvmqi1zM6/wCy1nJuxwptvxlef8+78bvmt3yRcX4I0k3+0dtQduUjc+Fkf1grKc3qFreUP8t/5wsxv+CqOXNpvNKUAg2QRhEepI6Brh6Tb+9Sb2Tc56wfeggS0DC05LgqDJDIw2LD8lekg7QiwPcgzhaksr2XDmSuuDY9yVtAxjHBrSC4WuUFDZIrfzQTtlHwC6Mw6KOx1cR1oKmmppquoZBAwvlebNb1pfJphU+TGM89myZN99llocCprcpKSRo0zG/yKuZ8Hvy4hma37J7ef+I0/eyDFPwysZXeRGBxqL25sanZdJV4fV0L2sq6d8LnC4DhtC2L4yz/AGit6jr+hZnGnTvxKfn3SOs9wbnJ0F919yCG2iqH0rqpsLzAw5XPA0BSjD6s0RrBA804OUybrrexUjIeSD6HL9r5KZnDv2rnydpm1fJCWlda8peG+/aEGH821IoG1xj+wc/IHX3rtNgmJw0xqJKKVsIFy8jQBXk8ZbyKiaRYirOnzRiJlfyRw084/V7w70jrqdqDOTYdVwU0VTLA9sMvqPOwpajD6mlp6eeVmWOoaXRm+0Lcz0flnIaGMayRwtkaPdtVVilOJqDk7C7Y9uU/EtQUMGCYlUUpqoqR5hAvnNhcd11xosOqq+R7KaPO5jS9wuBYBXnLGV5xY07CRFAxrGNGwaJORd/OVS0/ep3IKCmpJ6qcQ08TpJDsa0arpPhtbTVDaeamkZK+2VhGpV1yOFuUTPyP/ZMw+qfDymhnrXPcGyEXkJNhqBtQV9dgtfh8LH1UBja42GoOvUnSYDiMFH5XNTOZDa9yRex7tqssZopqflFJJNd7DKJWk7HNvdTuVVI+etjq2yXpp4xlt3DYgyVsu66cyJ8ly1hIG2ysmUjBpYEKfS08bTfnA0e5BRigqXerC879ikQYJWzC4iIHW7RaZmcaxl0gG8CwVpTNMoaZto6kGFfRSUjiHtOb3LlqvR6nD4amFzX21G3qVLUcno44Xloa42uHF2qDJoVgcPk5wR5Dcq2kwakFHnLCHAbWm+qDM+8JdFJlpSzY02UdzLbkCskcw3Y4j3FWdPi0mjZD8VU5UfFBsaHFY22JlAPvV2yqjqMocWkXGw7V5qHOFip1DXyRyxsaALvFz8UHpHKPCn4zhElFHI2Nz3NOZwuNDdY7+HFX/wAQh4CvRRsQUHnX8OKv/iEPAUfw5q/+IQ8BXoiRB53/AA6q/b4eAo/h1V+3w8BXoZTSgwA/2f1TQB5dDwFHQKq9uh4St8U0oMH0Dqh/rouApOgtUP8AWxcJW6KaVMGHHIiqa4Hy2LT+kps3Iqrk/wBbEB+UrcFMKYusH0DqfbYuAo6C1I/1sXCVuimFVGG6DVPtsXCUnQep9si4StwUwoMT0JqPbIuErRYHhz8Lw9tM+QSEOLrgW2qyKRBynjZLFlkbmaTsVfNgtPqWE6jQd6sZWc4zLrt3JzIjGPtNQN6DKS0FQ1zg2F5y9yiPa9hs9pae8Lauqqd2wG43WVLjGSc5i3IRu60FDfvTCepdSwIdGW+s3+yDmD1p1+op7WtcC3mwUhbCNObOnU4hA5jddq7Na78Vk2PmzrlI+Ke4s+64+5APksLHb3LiXl2xDgDscjIANHBBOwK/nmm1G03+Sv2VjGYZJUutz0L3wNN9dXf+FlaWaWiqo6mOziw3yneoNTUSuqXyu0zSc4WA6Xug09S0dPoHdbb/AKCqCsNRiuPtgmkdIee5sX3NzLtNjxfjkWJNp7c20NyF23S234pTj1NFXR1lNhcccrS4uJkJzXQaVsLpcUrnc7C6LybmWsbIC4W6xu2lVGHVJoeTlLUDZHWa+7W6p8OxqWhrZ6rmmyOmaQQT1lMOKvOD+b+abl5znM99fcg0nKmnZBgY5u2SSq5wW7wSq2dodyLpD+Gdw/dQ6zGpKvCqahkjA5n799thYLucchdhYoPN0eQag84dHdaC8pKkU0OCh3+6lidE4HvskxmlZSvwph1bAdD8Qs/UYm+ppKOnEQZ5MLBwOpU6txaXEeaL4svNtsbG9z1oJ2K4XFVY1KZHxxl4BDpDodAmYPQ+Q8oJKd7WZxG4Zm7CLAqZT1DK6nYKmma98bbNc5xbf3qvqMYdHjXlnk7A5jchZmuDu2oI/JqmbHygaQLGzh/ZQsR52tqXvqJHPcPRv3BWMeMRxYgyqioY48oILWuPpX3pk1fRSzMeMODWC+dolPpIJD2eW8mmSS3MtK/IHHaWowu9Zg9ZRy6iFvORk/dK41OKxOpmUtPS81Ttdmc0vJL/AIpsuLwxUksFDScxz3ruLy426ggqiU8TZdMq7eXxCCCM0URdG/M59zd46inecYOcqH+b4LSts1tzaPvCBja+Rgs3Rd4sQmYLh5A71WZx1LrHK0aO1CC7pcVkPrZnhTIaq9wXOObYCFn2TxNfcC3VqpFLW/4gOdLlt3bUGliga+xfoBuCkiKBrS2wAPWqI42LWLbDrC5vxSneNZXA96Cz820ry5sjWG5NrLicAoMhBZc9YKrTiAi1bJmG26UY+G2BaSEEDFMI8leTC17o7esdyqXMIWnGJRVMbmiWzidjgq+uhDmjLkJ23Asgp7LpSj/FRfnH7pXNsdi6Urf8TF+cfug9hGxCNyRAJClTUCFIUqQoGlNKcU0oGlNKUppQNKaU4phQNKaU4phQNKaU4ppQNKanFNQANiCVNZkeNRe4VLidUKOlEp/GAulFirZnAC90RceTx5CA0AFQajC2zbdQp8UweNE/W6DLVOBZX5g0gHW99i4zUGZvpgkDfZa51j3gKFU07pGu5kgHqdsQY6aiDP8Adusm0tA+qLhDdxG2xVpU4ZVGZpAieL+kAbK6w6i5nXmGR+7eisxLhVTTH0h7jdQpIKk3BdJfuaF6K+FkjC1zQbqkxFkdEM2QPb1XtZBkY4qixa5sh7wF1FNUOAYI3X/dXzcYoY23dFIPcAnnEn1EROHMyP65G/sgz7qCsja55gflG02VfLFKLl0ZHvU2sxHE+fJlqZA4G9tmvuVc+oc95L3kk6m6Dkfcm23rsXtLbdW9N2oOdinAXSkKTSMBd6pJOiCKGk+5dWNBK7ywFgOzTqSRROc+wCDrBGARdXFBS3cM7bt2aKPRwtIIePmr2gYbWaNAgmU1PHDGQ0Zg7Wx3LKYjDJNWyGGmDNdW7bFbEXsBZD443OzObc77IPP20tWwEuZsO9p0Rc716A6GN4LcosRYhVlTyfp3nM27e4IMgXFc3PttK0EuBNBOV5t3hU9TRviJDm9yCGZADqQgPuNDdOczrCQMaPVCAzFF0FMIQKXpM6TKghA7nSNl0hffrTcqS1kD8/egv70zLfclDLDQIFzJ2c9Z+abZGXuQPvfrXamNqmL84/dRwF2pRapi/OP3QewjYhG5IgEhQs3ymx51EXUVO1zZ3NB5zc0Hq71rnm9XIW4vpp4oGl0srGNG9xsgTRvIDZGOLhcAG9wvP6OgkxLPVVdWI4c4zyP1LidPonYlh/kEwqaCYyQWuJGn1e5dvwzc3yz7m/KaVm8B5RS1U7aWsAL3XyyaD4WWiuuPXN5uVZdBTClKaVlSFMKcUwoEKYU4ppQNKaUpTSgQpqUpED4Yo5nZJWB7SNhF1MjpYYwA1g0FgoMcgjfmK7iubex/siJrGtGwAJwaFFiqM+oBUhrrhA/KFxqGXA27dy7AoIBQQ4423yCKzetS2gAJrrh1hsStKKdl6lEqomSgtfGD7wpl1wmeLelogys+BONQS1hIvfbYK3o6UU0Ya43IXGuxPyQ/aNsNxCrHY4Xv+zaXfCyC6rcLpa2Ate1oJN7jas9Xcl4QPs7g/izK3pq7nWem8B3UkqqpzGhrWl19hugxFXQTUryHNOQbD1qPmttVxidWWPIz+kNosCFQySOe4kkaoO2cFdI6p0fqqGHEJ2c9yDs+pc5/pXI6rqRFjEkWyJvcoL3E7LLrTUzpzbn4Yj/W6yC9gqq6cNMcI163AWV1E6opwMzWF1tQHBZ+DBa+eZsprYA4WDXZr3VtFgFRcOkxR2l/Vb1oLaOsuPdtCkQTiUnQ6C97aKLQ0FPRtLRUOkJ2lwU6INDS1rtEHRuZ2txZPsCNuqblc0AMfYdVlzkEgaS30j1FA2ZrTtVbVUTJrm9yOtTC55HphoPcuRNtgCDOV1A8OL2tv3AqrkZKCQIHG3etu807m+mG3VXVMpoz6Jy31uCgymaUE5o7d10uZ1vUPzU6uIdP6DWgbySo5YL6G6CPmPUniUW1YplLRSTvbZpy31Nrq6ZyZFj9qNd/UgzBeHfdsm/BW+IYQ+kkyNBdptAvdV0tPKwAlrh7wg5hO9yblcNyUAoFsnZe5ODzls5o+CUhn3SgQMJC608bhPEbaZx+6A4Botr710gkeZ4xfTOP3QerbkiNyQlA2aQRRPkdsY0kry2qmNVUSzOLjzji70jcr1GTMY3Blg4g2vsusBDgs76l0lfkpacSHO9xyg67G/8ARen0LJtrHTu6nfi+EQSUkhL6Yc2+F1h8b313LhhtU/D3y01dFL5M7SRlvVPWm1dB5KxlXhtQ6WF3okg+k094Ck0FTPidHV09UQ6OODNzp2gjZfXXeuv8/SFFbhNFSltGwyz58zZJGajqWtglE1PHIL2e0HUWXm8EfPSsjDrZyBe116JRQGlooYC4uMbQCTvXL1+ZMXmmtqXSTOY2K7GvyF2bUH3dSR1XDY5ZWEjdm77LnNTyS1AdlYyx/wB40nMW22EJhhndTcyWRDK0Na4E62t3abFw8NldWOaXl8Vo2PyFwd7tbfFdudYclng59W67VxZSsEsksgzFz841JA0G7ZuXGjj+2kkveNpLYrjcdT/fT4KeB1mqHRP1iPN5g3Nfr6h8Uzyq7x9mebL8gfff7uq6bJFOah0mWN4B+zzPIDfhbbtTeYlu2M5eabJnDr6nW9re9PAc2pfI1xjhdcPy2cbaWvdLBKZmFxbls4jbe9lyEdRlmFmMMjrgh17aAdS7ta2NjWNFmtFgEClN3pSm71BwrH83Dm71CbVAEJ+NyGOhzD8YVIyoe6xttQaWmrLuAubdxViJso1JssbFiOSSxGt7aqyGIPlAaGuFkGmjqGneuvPN61lH1skTdb5VXzY1Ox5YXEN60G6NTGDYuCY6qYNRqvPzilXMCGPNhrtTYcaqWA5nGw3IN1JitPFpI8N96acQgmZdkjXBYCpxozMLHMuD1qEysexxMbi3TYg0mLVlM+SVpf6O624rLTzESOyPNutdmQy1TwIwLu23VhDybc9t5Klod+FougpGVErHhzZHAjvVxQ4+5hDKg3H4lHrsDnpQXNOdg2neqlzCDYhBfVlfh0wzCPO8nXaFTS5Mxy6BcbFKA46BAuYW2aozJttddE70bbT8kCZkocU02vogIJUNQYyCC8EfhKuKXHZGMyODj1ElUDDY6rpzhvoUGgixuS/pMsT1K2p8ZswZmkLHxc8512Mc63UFqcPwiSSNhdM4OIvsuiLWLFGSN9FxBPWkOJZiWl1gNpVLX0VfBKC5mdg2OjTaTCqvEWc55UxoO0EahFXT66J7PRffvCgTzzMBdHK09xUOtwaow9gc6bNGfvAWsVEbmcPTla9o3AoJjaqad55xtm2uEeTS1kgY1oDfxJ1PNGCDJlOXYrGLEqRn3mt7kESLA2gfayFx9yezDKcO0hc+283U8YxBmDQcxOxTBUsIHWdwQMpqVpYLMLB1KwhgDG2v81FEpGwaLq2pZvNigkPgje2zhdcJ6CGaPI5gI9yXytm4p7ahp3oKyTk3RvZbIbjYbqodyXeZyGvys61redb1hcZaiNv3gEGbPJZzNedLh1WXB/J2a/2V/wDmWpZVtdsIKY10plL7tsdLXVk1m3GVdgVW3bGD7iulNgVXnY8tsA4Gy17C122xUhgGmxRUtIhNuilKpeUeFS4pTs5qSzo7kMOxxVwSmkq8283YPPqKStwapFQ+nkaw+i9r22Dh1KfU4tV1MHM4fQcxE4BpLGXJ7r9S17w1ws4Bw6iEmgFhoF3vrS+bPLPtUGAYGKQNqapoM5HosP3P/lXpKUlNJXHrq9XasmEJTSUEppKyoJTSUEppKBCmlKSmlAFNKCkKBCm70pSIONSxssWV4uLrn5FC9mUNaPglrHZIQb29JcYqlotqgd5rZksQ2/XlTGUc0cnrtLPdqpPlbbesgVTb7dEHKSKUD0Y2OXB9GZ2nnoRfuCmioY77yQVIa7boggswaEA2iAKHYfGQGGFht3K0ZUsO0pTLEdiDG4ngz2OdLHHlZfZ1KPhuEyVUpBGVrd620xjljLLqMQ2E+gNN9kDqDDY4oQyQh567Lt5nbnzRvLR1JrK1rVIGIsA1KIjy4O95FpbDeCLqBPyUa8l3OjMd+VXIxGP8QSnEYyPWCKoRySpyQHyODdpACs+juH5GBsLQWCwNtVKbWRuOpXVtXHbQhBlsV5LQRRyTRuf1hoCyUsGVxynReo1E8UkZDiLFZvzRBKSXt0vtQY7mztsnRwOkNmhbF+DU3M5GMA702mwiCF1nG+8XQZLmXM9ZpHvXSOLMLt1IWxloKW3qh1+vVSqOmoYmhvNsB27NUFJgzQ22eN5PeNAtZTSw2FgRZcwaduwBcnzside9wURa5mluxchT07HZ2RBrjtsLXUWPEIwLEp/nCMjaEExzIpYyx7Q5p2gqoqeTVBLLnAewHa1p0Uo4gzrHuT21rHNvex70VmMU5NPpWCSkdJKL6i2oWfkY9ji1wII23Xo/ljetVmJUVFXXc5uST8TUGNjkcwixJVpTYk4D03WKe/BhFMLPBbfbZSxQ0hZZ7Rm6wgWPFdPSdouNTjILTzZ1CBhMV7iS2uily4bSVDAJGhrhvbogqBjMgXWPHXjU7FMGEULDYjN7yq+vwgNfemFwdyCUeUNxlsb+9V9Tisz5Lg6Dde6hvoqlpsYnXT20NSWkmIi3WgnRYxLYZG2ffr0V7R108jBnAHuKxha9rstrFSaatlgePTPeCrbrMmfpuI3ZpM5eR7lZQz3sLrHwYu0gXcp8WKtzsAN7kJbaTmRsbpCUl00lRopKaSglNJQBKaSglNJQKSmkpCU0lAEppKUlNJQISmkpSU0lAEppQUhQIUhQSmoBJvQUiBksUczMsjczb3suQoKUbIh8yu6VByFHT9n/AHKd5HT9n/croEqBjaWAbIx8yneTQ9mE8JQgYKaC1ubHzKBR04/y/wBRXQFOBQNFNB2Y+ZS+SU5/yx8yngpQUHI0FITcwi/vKTzXQl1zTtJ7yVIBTgUHAYdR9g35lOGHUY15hvzK7ApwKDiMNoxe0A126lObQUjRYQi3vK7ApQUHIUFIBbmRb3lKKCk7EfMrtdKCgjvwyhkFn0zHDvunDDaL2dvzKkXS3QcPN1Gf8hvzKb5qoC/OaVma1r63spV0t0HDzdR9g35lI7C6F4s6maR7ypN0XQRfNNB7M35lHmnD/Zm/M/VS7ougieaMPvfyZtx3n6p3mqh9nb8z9VKui6CJ5poPZxxH6o800Hs7eI/VS7ougieaaD2ZvzP1TfMuG58/kcebr1upt0XQQ/M+H+yt+Z+qXzTQezN+Z+ql3RdBC8z4dp/hWabNT9UeaMP9mb8z9VMui6CF5mw69/JWfM/VBwjDz/pWfM/VTLpLoID8Cwp5u6hiJ7wUw8nsH/4fB8lY3SXQV3R/CB/+Ph+SUYHhbSC2iiBGzap90hKBbpCUhKaSgUlISkJTSUCkppKQlISgCU0lBKaSgCUhKQlISgCU0lBSEoAlNJQUhQIUhSpqASb0IQIlTUqByUJqVA5OCZdKgeClBTAU66B6UFMBS3QdAUoKYClBQdAU4FcwUoKDoCnArndKCg6ApwK5Ap10HS6W653S3QdLpbrndLdB0ui6ZdF0HS6W653S3QPui6ZdF0HS6LrndF0HS6S6ZdF0D7oumXRdA66Lpl0XQOukum3SXQOukJTbpLoHXSXTSUl0DiU0lJdJdApKaSkJSEoFJTSUhKQlApKaSkJSXQBKQlBKaSgCUl0Ept0CkpqEiASFBSIBG9CTegELFdL8Q7Gm4XeJHS/EOxpuF3iQbZKsR0wxDsabhd4kdMMQ7Gl4XeJBuEqw/THEOxpeF3iR0xxDsaXhd4kG5S3WF6ZYj2NLwu8SXpliPY0vC7xIN1dLdYTpniPY0vC7xJemeI9jS8LvEg3gKW6wXTTEexpeF3iS9NcS7Gl4HeJBvQU66wHTXEuwpOB3iR02xLsKTgd4kHoAKW68/wCm+JdhScDvEjpvifYUnA7xIPQrpQV5704xPsKTgd4kdOcT7Ck4HeJB6HdLdeedOcT7Ck4HeJHTrE+wpOB3iQeiXS3XnXTrFOwo+B3iS9O8U7Cj4HeJB6LdF15107xTsKPgd4kdPMU7Cj4HeJB6NdLdecdPMU7Cj4HeJL08xTsKPgd4kHo10XXnPT3FOwo+B3iR09xTsKPgd4kHo10XXnPT3FOwo+B3iR09xTsKPgd4kHo10XXnPT3FOwo+B3iR09xTsKPgd4kHo10XXnPT3FOwo+B3iR09xTsKPgd4kHot0XXnPTzFOwo+B3iR08xTsKPgd4kHot0l15307xTsKPgd4kdO8U7Cj4HeJB6HdJdee9O8U7Cj4HeJJ06xPsKTgd4kHoV0l15906xPsKTgd4knTnE+wpOB3iQegkpLrz/pxifYUnA7xJOnGJ9hScDvEg9AJTbrA9N8S7Ck4HeJHTfEuwpOB3iQb0lISsF02xLsKTgd4kdNcS7Ck4HeJBu7pCVhOmmJdhS8LvEjppiPY0vC7xIN0SmkrDdM8R7Gl4XeJHTLEexpeF3iQbhIsR0yxHsaXhd4knTHEOxpeF3iQbdJdYnpjiHY0vC7xI6YYh2NLwu8SDaoWJ6YYh2NLwu8SOmGIdjTcLvEg2qN6xXS/EOxpuF3iR0vxDsabhd4kGfQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCD/2Q==\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"400\"\n",
       "            src=\"https://www.youtube.com/embed/qqg7RO0o1OE\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7f9c1bf4acc0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo(\"qqg7RO0o1OE\", width=800, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conclusion\n",
    "Don't waste time and resources on unreproducible experiments! Sacred will give you the necessary structure to find out which machine learning models are the right ones for you.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Contributing\n",
    "Issues, feature and pull requests highly welcome! Just go to https://github.com/IDSIA/sacred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thanks!\n",
    "\n",
    "## Questions?"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "livereveal": {
   "autolaunch": true,
   "scroll": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}