06-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 類別預測的任務\n",
    "\n",
    "我們先載入這個章節範例程式碼中會使用到的第三方套件模組或者其中的部分函式、功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyvizml import CreateNBAData\n",
    "import requests\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import OneHotEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 關於類別預測的任務\n",
    "\n",
    "「類別預測」是「監督式學習」的其中一種應用類型，當預測的目標向量 $y$ 屬於離散型的類別變數，那我們就能預期正在面對類別預測的任務，它更廣泛被眾人知悉的名稱為「分類器」。例如預測的目標向量 $y$ 是 `player_stats` 資料中的 `pos`，在資料類別中屬於離散型的類別 `object`；具體來說，分類器想方設法地以一組係數向量 $w$ 計算特徵矩陣 $X$ 對應目標向量中每個類別的機率，再從中挑選出最高的機率來預測類別，與迴歸模型相同的是利用係數向量建構出用來逼近 $f$ 的 $h$，不同的是迴歸模型輸出的 $\\hat{y}$ 就是預測值，而分類器所輸出的 $\\hat{p}$ 僅是類別的預測機率，需要經過一個 $argmax$ 函式轉換為 $\\hat{y}$。\n",
    "\n",
    "我們也可依 [Tom Mitchel](https://en.wikipedia.org/wiki/Tom_M._Mitchell) 對機器學習電腦程式的定義寫下分類預測的資料、任務、評估與但書，以預測 `player_stats` 資料中的 `pos` 為例：\n",
    "\n",
    "- 資料（Experience）：一定數量的球員資料\n",
    "- 任務（Task）：利用模型預測球員的是前鋒或後衛\n",
    "- 評估（Performance）：模型預測的鋒衛位置與球員實際鋒衛位置的誤分類數\n",
    "- 但書（Condition）：隨著資料觀測值筆數增加，預測誤分類數應該要減少"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating players df...\n",
      "Creating players df...\n",
      "Creating player stats df...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create_player_stats_df() 方法要對 data.nba.net 發出數百次的 HTTP 請求，等待時間會較長，要請讀者耐心等候\n",
    "cnd = CreateNBAData(season_year=2019)\n",
    "player_stats = cnd.create_player_stats_df()\n",
    "player_stats['pos'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`player_stats` 資料中的 `pos` 有 7 個不同的類別："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['G' 'C' 'C-F' 'F-C' 'F' 'F-G' 'G-F']\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "print(player_stats['pos'].unique())\n",
    "print(player_stats['pos'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們先將多元分類問題簡化為二元分類問題，鋒衛位置分作前鋒（F）與後衛（G），分別對應整數 1 與整數 0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_dict = {\n",
    "    0: 'G',\n",
    "    1: 'F'\n",
    "}\n",
    "pos = player_stats['pos'].values\n",
    "pos_binary = np.array([0 if p[0] == 'G' else 1 for p in pos])\n",
    "np.unique(pos_binary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 以 Scikit-Learn 預測器完成類別預測任務\n",
    "\n",
    "將 `apg` 與 `rpg` 當作特徵矩陣為例，特徵矩陣 $X$ 與目標向量 $y$ 之間的關聯可以這樣描述：\n",
    "\n",
    "\\begin{equation}\n",
    "\\hat{y} = 1, \\quad if \\: \\hat{p}(y=1|X; w) \\geq \\hat{p}(y=0|X; w)\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "\\hat{y} = 0, \\quad if \\: \\hat{p}(y=1|X; w) < \\hat{p}(y=0|X; w)\n",
    "\\end{equation}\n",
    "\n",
    "以 Scikit-Learn 定義好的預測器類別 `LogisticRegression` 可以計算特徵矩陣對應類別的預測機率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1.71621286]\n",
      "[[-2.75507401  1.96609808]]\n"
     ]
    }
   ],
   "source": [
    "X = player_stats[['apg', 'rpg']].values.astype(float)\n",
    "y = pos_binary\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.33, random_state=42)\n",
    "h = LogisticRegression(C=1e06) # 預測器的正規化程度\n",
    "h.fit(X_train, y_train)\n",
    "print(h.intercept_)\n",
    "print(h.coef_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這裡需要說明初始化 `LogisticRegression` 類別指定參數 `C=1e5` 的用意。參數 `C` 用來描述預測器的正規化（Regularization）程度，當 `C` 愈大表示正規化效果愈低，反之 `C` 愈小表示正規化效果愈高，`LogisticRegression` 類別預設 `C=1` 這是具有正規化效果的參數設定，由於在本章節後段我們會自訂不具備正規化效果的 `LogitReg` 類別驗證我們對演算方法的理解，為了比較最適化的 $w$ 得先將 Scikit-Learn 模型的正規化效果降到很低。有關於「正規化」的技法，將在「表現的評估」章節中介紹給讀者認識，假如讀者目前感到困惑，可待讀過表現的評估等本書後面的章節，再回來複習。\n",
    "\n",
    "`LogisticRegression` 類別的 `predict_proba` 方法輸出的數值陣列外觀為 `(m, n)`，其中 `m` 是特徵矩陣的觀測值個數，`n` 則是目標向量的獨一值，也就是類別的個數，以目前的二元分類（後衛 vs. 前鋒）問題來說，`n` 等於 2，第 0 欄是預測為類別 0 的機率 $\\hat{p}(y=0|X; w)$、第 1 欄是預測為類別 1 的機率 $\\hat{p}(y=1|X; w)$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.24174853e-02, 9.87582515e-01],\n",
       "       [9.99405251e-01, 5.94749246e-04],\n",
       "       [6.13059991e-01, 3.86940009e-01],\n",
       "       [3.53652646e-01, 6.46347354e-01],\n",
       "       [3.81597551e-01, 6.18402449e-01],\n",
       "       [3.01194136e-02, 9.69880586e-01],\n",
       "       [8.47640386e-01, 1.52359614e-01],\n",
       "       [2.47843181e-01, 7.52156819e-01],\n",
       "       [8.14691640e-01, 1.85308360e-01],\n",
       "       [4.77310176e-01, 5.22689824e-01]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_hat = h.predict_proba(X_valid)\n",
    "p_hat[:10, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "應用 `np.argmax` 函式回傳最大的欄位數，就能夠得到 $\\hat{y}$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 0, 1, 1, 1, 0, 1, 0, 1])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = np.argmax(p_hat, axis=1)\n",
    "y_pred[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最後使用 `pos_dict` 將整數對應回鋒衛位置的文字外觀。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['F', 'G', 'G', 'F', 'F', 'F', 'G', 'F', 'G', 'F']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_label = [pos_dict[i] for i in y_pred]\n",
    "y_pred_label[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "若是將 $h$ 的機率輸出在一個區間之內，例如在所有球員的場均助攻 `apg` 與場均籃板 `rpg` 均勻切割 50 個資料點，在平面上就可以對應出 2,500 個場均助攻和場均籃板的組合，每個組合都輸入 $h$ 得到一組機率組合 $\\hat{p}(y=0|X; w)$ 與 $\\hat{p}(y=1|X; w)$；假設將全部 2,500 個資料點的 $\\hat{p}(y=1|X; w)$ 視作海拔高度、場均助攻視作經度、場均籃板視作緯度，我們可以描繪出一個填滿等高線圖（Contour-filled plot）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "resolution = 50\n",
    "apg = player_stats['apg'].values.astype(float)\n",
    "rpg = player_stats['rpg'].values.astype(float)\n",
    "X1 = np.linspace(apg.min() - 0.5, apg.max() + 0.5, num=resolution).reshape(-1, 1)\n",
    "X2 = np.linspace(rpg.min() - 0.5, rpg.max() + 0.5, num=resolution).reshape(-1, 1)\n",
    "APG, RPG = np.meshgrid(X1, X2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_contour_filled(XX, YY, resolution=50):\n",
    "    PROBA = np.zeros((resolution, resolution))\n",
    "    for i in range(resolution):\n",
    "        for j in range(resolution):\n",
    "            xx_ij = XX[i, j]\n",
    "            yy_ij = YY[i, j]\n",
    "            X_plot = np.array([xx_ij, yy_ij]).reshape(1, -1)\n",
    "            z = h.predict_proba(X_plot)[0, 1]\n",
    "            PROBA[i, j] = z\n",
    "    fig, ax = plt.subplots()\n",
    "    CS = ax.contourf(XX, YY, PROBA, cmap='RdBu')\n",
    "    ax.set_title(\"Probability of being predicted as a forward\")\n",
    "    ax.set_xlabel(\"Assists per game\")\n",
    "    ax.set_ylabel(\"Rebounds per game\")\n",
    "    fig.colorbar(CS, ax=ax)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3debwcVZn/8c+XhACBsIQgA0lMAoKKIgoRUBxEFmUEARnEgCgoiv4EYRg3UAcEdERRBAXECBGQJTCgmJE4iLKEkcWETTYZQ1iSENnCkrAEkjy/P865UOl096279O3lft+v133dru2cU708Xf3UqVOKCMzMrP2t0uwGmJlZ/3BANzPrEA7oZmYdwgHdzKxDOKCbmXUIB3Qzsw7hgF4gKSS9qZfbPixp1xrL/lnSA9XWlfQNSef0rsU9buNHJc2VtFjSu6os78v+L5a0Sd9bObAkfVvShfnxG/N+DBmAemu+X5pByS8lPSPpL81uTz2SzpP0nWa3oxW1fUDPH4yX8gfx8fxir9XsdhVFxI0R8eYay/4zIj4LIGl8DqpDG9SUHwJHRMRaEXFHfxacy5zTn2UOtIh4NO/HsnrrSdpJ0ryBatcAeR+wGzAmIrZtdmOsd9o+oGcfiYi1gK2BicC3KldoYJBsJ+OAe5vdiEbxa9wn44CHI+KFnm7YyOd9IH4tdZJOCegARMR84PfA2+G1FMLhkv4O/D3P+5yk2ZIWSpomaeOKYj4saY6kpySdImmVvN2mkq6V9HRedpGkdSu2fbek+/LP1l9KWj1vW/OIrviTH5iR/z+bf3G8P7dzy8L6b5D0oqQNqpS1iqRvSXpE0hOSLpC0jqTVJC0GhgB3SXqwztNYdf9z+Z+RdH/ev6sljSssey1dk38lnSnpKkmLJN0qadPCuh+U9ICk5ySdJekGSZ+t8/xcLunSXNbtkrYqLH9Y0tcl/RV4QdJQSdtLuknSs5LukrRTYf0Jub5Fkq4BRhWWrfALSdLI/Do+lvf5Sklrkt5jG+fXaLGkjfNzf4ykB/N75DJJIwtlfzK/Lk9L+mad5x9Je0i6Q9LzSimybxeWrS7pwlzOs5JmStqwRjld7VmU35cfrbHeocA5wHvy/pyQ59f8rKjisyXpBEk/zctWlfSCpFPy9BqSXu56PiT9l6R/5Nd/hqS3Fco9T9LPJE2X9ALwAUnvyq/7IkmXAqvXe/4GtYho6z/gYWDX/Hgs6Qj0pDwdwDXASGANYGfgKdKR/GrAT4EZhbICuC6v/0bg/4DP5mVvIv0kXQ3YgBR8T6toxz25DSOBPwPfyct2AubVaPO3gQvz4/G5DUML654FfL8wfRTw3zWei88As4FNgLWAXwO/qti/N9V5Luvt/9657LcCQ0m/gm6qVjZwHvA0sG1e9yJgal42Cnge2DcvOwp4taueKm36dl6+H7Aq8BXgIWDVwnN5Z37e1wBG57o/TDpg2S1Pb5DXvxk4Nb+OOwKLaj3/wFXApcB6ue73V3s9C6/LLcCYXPbPgUvysi2Axbm+1XL9S7veA1X2eSdgy9z+dwCPA/vkZZ8H/hsYTvqC3gZYu0Y5HwM2zuV8HHgB2KjGuocA/1uYLvNZqfxs3Z2XvRd4ELi1UNZdFe/TEbnc04A7C8vOA54DdsjtXht4BDg6vwb75ffDd5ode1rxr+kN6PMOpA/0YuDZ/MKfBaxReNPtXFj3XOAHhem18ptjfGH93QvLvwj8qUa9+wB3VLTjC4XpDwMP5scrBAB6FtC3Ax4FlKdnAfvXaNOfgC8Wpt+c929oYf+6C+hV9590VHpoYdkqwIvAuMqy84fynIrn4m/58aeAmwvLBMylfkC/paLeBcA/F57LzxSWf53Cl1iedzVwMOlLaimwZmHZxdWef2AjYDmwXpU2rfB65nn3A7sUpjfqeu6B48hfaHnZmsAr1AjoVeo7DfhxfvwZ4CbgHb34rNwJ7F1j2SGsGNDLfFaKn601gJeB9YFjgG8A8/J2JwA/qVHvurmsdQrvnQsKy3cEHiO///O8m3BAr/rXKSmXfSJi3YgYFxFfjIiXCsvmFh5vTAr6AETEYtLR2+ga6z+St0HShpKmSpov6XngQgo/1+tt2xcRcSspcO4k6S2kXwrTaqy+wv7lx0OBqj/Ja6i1D+OA0/PP/GeBhaRgPJrq/lF4/CLpg93VxtfqiPQJ7e4EY3H95Xn9jastz+38WFc7c1vfRwqwGwPPxIp54uLzVTQWWBgRz3TTtmK9vynUeT+wjPTcV+7zC6T3XVWStpN0naQnJT0HfIHX32u/In1BTc2poB9IWrVGOZ+SdGehTW9n5fdsLT36rOTP3Czg/aQgfAMp8O6Q592Q2zRE0sk5FfQ86QuZinZVfmbn5/dJl1qv2aDXKQG9nuIb4THSBw+AnA9dH5hfWGds4fEb8zYA/5nL2jIi1gYOIgU0Smzbm7YWnZ/r+yRweUS8XGO9FfaP149IH+9BG2rtw1zg8/mLs+tvjYi4qQdlQzq6HtM1IUnF6e7apJTTH8OKz23xeZtLOkIvtnPNiDg5171eft2L+1jNXGCkVj5PUllfcf1/qah39UjndRZU7MNw0vuulotJX9pjI2Id4Gzyey0iXo2IEyJiC1JqY0/Sr54VKJ3f+AVwBLB+RKxLSglWvmdrKfNZqXwebiClV94FzMzTHyKl3rrODx1ISt/tCqxD+lVERbuK5S4ARuf3SZdar9mgNxgCetElwKclvVPSaqQgfWtEPFxY56uS1pM0lpQXvTTPH0FK7TwnaTTw1SrlHy5pTD75883CtmU9SfqZX9mf+0Lgo6SgfkGd7S8Bjs4n/tYi7d+lEbG0B22otf9nA8d2ncBSOtn6sR6U2+UqYEtJ++STj4cD/9TNNttI2jev/2/AElK+upoLgY9I+lA+Glxd6aT0mIh4hHQUeYKkYZLeB3ykWiERsYCUZjorPx+rStoxL34cWF/SOoVNzga+mwMpkjaQtHdedjmwp6T3SRoGnEj9z94I0q+DlyVtSwqC5HI/IGlLpd4fz5PSIMurlLEmKTA+mbf7NLmzQEllPiuVbiB9udwXEa8A1wOfBR6KiCcL+7aEdLQ/PJdbz82kg5Ij82uwL+kLwqoYVAE9Iv4I/AdwBembf1NgUsVqvwVuI+UbryLlEiHlAbcmnbC5inTCsdLFwB+AOaSTQj26+CEiXgS+C/w5/0zePs+fC9xO+oDeWKeIKaSf5DNIJw5fBr7UkzZQY/8j4jfA90k/9Z8nHe39Sw/LJiKeIp2s+wHpQ70FKcgu6aZNHweeIf1K2TciXq1R/lzSEeA3SMFsLunLt+u9fiDpvMRC4Hjqf0F+khQw/wY8QfoyISL+Rgp4c/LrtDFwOumo+g+SFpG+cLbL699L+uK6mPS+e4b6aaYvAifmco4DLiss+yfSF8TzpLTODaTXvPJ5uA/4ESkgPk46yfrnOnVWbl/ms1LpJlIuveto/D7Se3BGYZ0LSCmT+Xl5rS/mrna8QjqBfgjpNfs41T97xusn2qzFSZoCPBYRK/Wxb2c5hTIP+EREXFdl+bdJJ1sPGui2mbUbX4jRBiSNJx2lrHS5fjuS9CHgVuAl0tGz6OZIzcy6N6hSLu1I0kmk9MYpEfFQs9vTT95DSkk9Rcph71PRM8msY0iaonSh3z01lkvST5Qu4vqrpK0Ly5blnkp3SqrVu+31spxyMTNrnHwyfTGpf/1KJ6YlfZh0ruvDpPMup0fEdnnZ4kjDmpTiI3QzswaKiBmkE7q17E0K9hERtwDrStqoN3W1RQ59ldXWiqFr1uu2a2atZPiI4aXW22DEsJrLRqy2cnjSC8/WXP+VhSlm3vPks09FxEpjHfXE6hu9LZYvWdzteq8+8+i9pJ48XSZHxOQeVjeaFS+mmpfnLQBWlzSL1HXz5Ii4sl5BbRHQh665PqN2O6bZzTCzErbZ5Z2l1vvijrWHz99p/DpV5w+5rXoa+ZELX+/ZufnZV/T5StLlSxaXijkLLvviyxExsa/11TEuIuYr3WvgWkl3R0TNwfUalnKpdyJA0peVRmsrexmymbWBRgXzIbdNKxXM29R8VrxCe0yeR77SmEj3Griebnq6NfII/TzgDCou3MhXIH6QNOCUmXWIvgbzvhyVt7lpwBGSppJOij4XEQskrQe8GBFL8sHvDqQL8mpqWECPiBm5/3SlHwNfI139Z2Ztrmwgh8EZzCVdQhqhc5TSfRGOJw0FTEScDUwn9XCZTRrI7tN507cCP5e0nJRNOTlfAVzTgObQ89gW8yPirhXH2qm67mHAYQBDho+su66ZNUez8+XtICIO6GZ5kIaGqJx/E2nIhtIGLKDnEea+QUq3dCufKZ4MMGzkOHeWN2sxjcyX19JdMJ9xadVrdwaNgTxC3xSYQLoFGqTE/+2Sto2If9Td0sxaSqvlywd7IO8yYAE9Iu4G3tA1LelhYGIefc/M2kAr5ssdzF/XyG6Ll5CG7nyzpHlKN6I1szblYN76GtnLpbsTAeMbVbeZ9S/ny9tDW1wpambNM9DB3IG89xzQzawqp1jaj0dbNLOVOJi3Jx+hm9kKWvFiIQfzchzQzew1zpe3Nwd0MwN8sVAncEA3G+ScL+8cDuhmg5jz5Z3FAd1skGq1fDk4mPeVA7rZIOR8eWdyQDcbRJwv72wO6GaDhPPlnc8B3WwQcL58cHBAN+twzpcPHg7oZh3K+fLBxwHdrAM5Xz44OaCbdRjfjGLwckA36yDOl7ceSbsDpwNDgHMi4uSK5eOAKcAGwELgoIiYl5cdDHwrr/qdiDi/Xl0O6GYdwPny1iRpCHAmsBswD5gpaVpE3FdY7YfABRFxvqSdge8Bn5Q0EjgemAgEcFve9pla9fkGF2ZtzsG8pW0LzI6IORHxCjAV2LtinS2Aa/Pj6wrLPwRcExELcxC/Bti9XmUNO0KXNAXYE3giIt6e550CfAR4BXgQ+HREPNuoNph1OufLm26UpFmF6ckRMbkwPRqYW5ieB2xXUcZdwL6ktMxHgRGS1q+x7eh6jWlkyuU84AzggsK8a4BjI2KppO8DxwJfb2AbzDpWq10s1EmBfPiI4aWe399dxlMRMbGP1X0FOEPSIcAMYD6wrDcFNSygR8QMSeMr5v2hMHkLsF+j6jfrVE6xtJX5wNjC9Jg87zUR8RjpCB1JawH/GhHPSpoP7FSx7fX1KmtmDv0zwO9rLZR0mKRZkmYtX7J4AJtl1roczNvOTGAzSRMkDQMmASs80ZJGSeqKxceSerwAXA18UNJ6ktYDPpjn1dSUXi6SvgksBS6qtU7OQ00GGDZyXAxQ08xali8Waj85vXwEKRAPAaZExL2STgRmRcQ00lH49yQFKeVyeN52oaSTSF8KACdGxMJ69Q14QM95oj2BXSLCgdqsBOfL21dETAemV8w7rvD4cuDyGttO4fUj9m4NaEDPHey/Brw/Il4cyLrN2pUvFrKyGtlt8RLST4lRkuaROsgfC6wGXCMJ4JaI+EKj2mDWzpwvt55qZC+XA6rMPrdR9Zl1EufLrTd86b9Zi2m1fDk4mLcLB3SzFuJ8ufWFA7pZC3C+3PqDA7pZkzlfbv3FAd2siZwvt/7kgG7WJM6XW39zQDcbYM6XW6M4oJsNIOfLrZEc0M0GiG9GYY3mgG42AJwvt4HggG7WQM6X20DyTaLNGsTB3Aaaj9DNGsD5cmsGB3SzfuZ8uTWLA7pZP3GKxZrNOXSzfuBgbq3AR+hmfeR8ubUKB3SzPmi1wbUcyAc3B3SzXvLJT2s1DuhmPeR8ufWEpN2B04EhwDkRcXLF8h8DH8iTw4E3RMS6edky4O687NGI2KteXQ0L6JKmAHsCT0TE2/O8kcClwHjgYWD/iHimUW0w628eXMt6QtIQ4ExgN2AeMFPStIi4r2udiDi6sP6XgHcVingpIkofQXTby0XJQZKOy9NvlLRtibLPA3avmHcM8KeI2Az4U542awuNzJdXC+aPXHiZg3n72xaYHRFzIuIVYCqwd531DwAu6W1lZY7QzwKWAzsDJwKLgCuAd9fbKCJmSBpfMXtvYKf8+HzgeuDrZRtr1izOl1sNoyTNKkxPjojJhenRwNzC9Dxgu2oFSRoHTACuLcxePZe/FDg5Iq6s15gyAX27iNha0h0AEfGMpGEltqtmw4hYkB//A9iwl+WYDQjnywenDUYMq/tLq8vv4KmImNhP1U4CLo+IZYV54yJivqRNgGsl3R0RD9YqoExAfzXngQJA0gakI/Y+iYiQFLWWSzoMOAxgyPCRfa3OrMecL7d+MB8YW5gek+dVMwk4vDgjIubn/3MkXU/Kr/cpoP8E+A3wBknfBfYDvlViu2oel7RRRCyQtBHwRK0V88+WyQDDRo6rGfjNGqHV+peDg3mbmglsJmkCKZBPAg6sXEnSW4D1gJsL89YDXoyIJZJGATsAP6hXWbcBPSIuknQbsAsgYJ+IuL/8/qxgGnAwcHL+/9telmPWMM6XW3+JiKWSjgCuJnVbnBIR90o6EZgVEV1viknA1IgoHry+Ffi5pOWkDiwnF3vHVFO22+LjwI15/TUkbR0Rt9fbQNIlpBOgoyTNA44nBfLLJB0KPALsX7J+s4ZzvtwaISKmA9Mr5h1XMf3tKtvdBGzZk7q6DeiSTgIOIeVtur49gtTrpaaIOKDGol160D6zAeF8uXWCMkfo+wOb5j6UZh3H+XLrFGUC+j3AutQ5gWnWrpwvt05SJqB/D7hD0j3Akq6Z3Y0pYNbKnC+3TlQmoJ8PfJ80QEyf+5+bNZuDuXWqMgH9xYj4ScNbYjYAfDMK62RlAvqNkr5H6kNeTLnU7bZo1mqcL7dOVyagdw3luH1hXrfdFs1ahVMsNliUuVL0A92tY9aqHMxtMCl1paikPYC3Aat3zYuIExvVKLP+4Hy5DTZlrhQ9m3RbpA8A55AG5/pLg9tl1ietdrGQA7kNhDJH6O+NiHdI+mtEnCDpR8DvG90ws97yyU8brMoE9Jfy/xclbQw8DWzUuCaZ9Y7z5TbYlQnov5O0LnAKcDuph8s5DW2VWQ95cC2zcr1cTsoPr5D0O2D1iHiusc0yK6/V8uXgYG7NUeak6L5V5j0H3B0RHrDLmsr5crPXlUm5HAq8B7guT+8E3AZMkHRiRPyqQW0zq8n5crOVlQnoQ4G3RsTjAJI2BC4AtgNmAA7oNqCcLzerrkxAH9sVzLMn8ryFkl5tULvMqnK+3Ky2MgH9+nwy9L/y9L/meWsCzzasZWYVnC83q69MQD8c2Bd4X56+ALgi353a47xYwzlfblZOmW6LAVyR/8wGlPPlZuWt0oxKJR0t6V5J90i6RNLq3W9lg00j8+XVgvkjF17mYG79TtLukh6QNFvSMTXW2V/SfTkuXlyYf7Ckv+e/g7urq9Roi/1J0mjgSGCLiHhJ0mXAJOC8gW6LtS7ny60TSBoCnAnsBswDZkqaFhH3FdbZDDgW2CEinpH0hjx/JHA8MJF0hf5tedtnatVX9whd0hBJF/V1p6oYCqwhaShpJMfHGlCHtaFtdnmng7l1km2B2RExJyJeAaYCe1es8zngzK5AXbhg80PANRGxMC+7Bti9XmV1A3pELAPGSRrW8/2oWeZ84IfAo8AC4LmI+EPlepIOkzRL0qzlSxb3V/XWwnzy0zrQaGBuYXpenle0ObC5pD9LukXS7j3YdgVlUi5zgD9Lmga80DUzIk4tse1KJK1H+oaaQOr2+F+SDoqIC4vrRcRkYDLAsJHjojd1WfvwzSislYxYbWjNg4MKoyTNKkxPzrGrJ4YCm5Guwh8DzJC0ZQ/LeK2g7jyY/1YBRvSmkgq7Ag9FxJMAkn4NvBe4sO5W1rGcYrE29lRETKyzfD4wtjA9Js8rmgfcGhGvAg9J+j9SgJ9PCvLFba+v15gy3RZPAJA0PCJe7G79Eh4Ftpc0nDTW+i7ArPqbWCdyisUGgZnAZpImkAL0JODAinWuBA4AfilpFCkFM4d0IP2fOasB8EHSydOauu22KOk9ku4D/pant5J0Vvn9WVFE3ApcThpb/e7chp7+RLE252Bug0FELAWOAK4G7gcui4h7JZ0oaa+82tXA0znOXgd8NSKejoiFwEmkL4WZwIl5Xk1lUi6nkc62TssNvEvSjr3Yt9dExPGk7jg2CDlfboNJREwHplfMO67wOIB/z3+V204BppStq1Q/9IiYK6k4a1nZCsyKWm1wLQdy6yRlAvpcSe8FQtKqwFGknw5mPeKTn2aNVSagfwE4ndT/8TFSvufwRjbKOovz5WYDo0wvl6eATwxAW6wDOV9uNnDK3FN0E9IR+vak8QRuBo6OiDkNbpu1uVbLl4ODuXW2MimXi0mDy3w0T08CLiHdgs6sKufLzQZemYA+vOJG0BdK+mqjGmTtzflys+YpE9B/n8fwnUpKuXwcmJ6HdqS7ju42ePhmFGbNVSag75//f75i/iRSgK/96bRBw/lys+Yr08tlwkA0xNqX8+VmrWHA71hkncP5crPW4oBuveJ8uVnrcUC3HnO+3Kw1lbmwaAfgzoh4QdJBwNbA6RHxSMNbZy3H+XKz1lXmCP1nwFaStgK+DJwDXAC8v5ENs9bifLlZ6+v2BhfA0jxe797AGRFxJv1zKzprEw7mZu2hzBH6IknHAgcBO0paBVi1sc2yVuF8uVn7KBPQP066B96hEfEPSW8ETmlss6wVOF9u1l7KXFj0D+DUwvSjpBy6dSinWMzaU82ALmkR6dL+qiJi7Ya0yJrKwdysfdUM6BExAkDSScAC4FeASDe72KgvlUpal9Rb5u2kL43PRMTNfSnT+s43ozBrb2Vy6HtFxFaF6Z9Jugs4rtYGJZwO/E9E7CdpGDC8D2VZP3C+3Kz9lem2+IKkT0gaImkVSZ8AXuhthZLWAXYEzgWIiFci4tnelmd952Bu1jiSdpf0gKTZeSjyWuv9q6SQNDFPj5f0kqQ789/Z3dVV5gj9QNIR9emk9Mif87zemgA8CfwyX6x0G3BURKzwJSHpMOAwgCHDR/ahOqvF+XKzxpI0hHTHt92AecBMSdMi4r6K9UYARwG3VhTxYESU/qCW6eXyMOmiov4ylDR8wJci4lZJpwPHAP9RUe9kYDLAsJHjap6ctd5xvtxsQGwLzO66B7OkqaR4el/FeicB3wf6dDe4MmO5bAB8DhhfXD8iPtPLOucB8yKi65voclJAtwHii4XMBsxoYG5heh4V92OWtDUwNiKuqnJ7zwmS7gCeB74VETfWq6xMyuW3wI3AH4FlJdavK1+cNFfSmyPiAWAXVv62sgZxvtyse3rh2bq/NgtGSZpVmJ6cswvl6klX3p8KHFJl8QLgjRHxtKRtgCslvS0inq9VXtmbRH+9bANL+hJwUe7hMgf4dD+XbxWcLzdriKciYmKd5fOBsYXpMXlelxGk7tvXSwL4J2CapL0iYhawBCAibpP0ILA5UPwCWUGZgP47SR+OiOkl1i0lIu4E6j0J1o98MwqzppkJbCZpAimQT6LQqSQingNGdU1Luh74SkTMyunuhRGxTNImwGakA+CaygT0o4BvSFoCvEq6uCh8pWh7cL7crHkiYqmkI4CrgSHAlIi4V9KJwKyIqJfX2RE4UdKrwHLgCxGxsF59ZXq5eKjcNuV8uVnz5ezG9Ip5VS/MjIidCo+vAK7oSV1lernsWKPiGT2pyAaO8+Vmg1OZlEuxG83qpH6VtwE7N6RF1ifOl5sNXmVSLh8pTksaC5zWsBZZrzlfbja4lTlCrzQPeGt/N8T6xvlyMyuTQ/8pr4+LvgrwTuD2RjbKynO+3My6lDlCL3ZiXwpcEhF/blB7rAcczM2sqEwO/fx8RefmedYDjW2SleF8uZlVKpNy2Qk4H3iYdFHRWEkHu9ti8zhfbmbVlEm5/Aj4YB5IC0mbA5cA2zSyYbYyp1jMrJ4ydyxatSuYA0TE/wGrNq5JVo2DuZl1p9RJUUnnABfm6U9QZ7Qv63++GYWZlVEmoP8/4HDgyDx9I3BWw1pkK3C+3MzKKtPLZYmkM4A/kUb8eiAiXml4y8zB3Mx6pEwvlz2As4EHSb1cJkj6fET8vtGNG6ycLzez3ijby+UDETEbQNKmwFWAA3oDOF9uZr1VJqAv6grm2RxgUYPaM6j5YiEz64uaAV3SvvnhLEnTgctIY7p8jHRbJetHzpebWV/VO0IvDpv7OPD+/PhJ0rjo1g+cLzez/lIzoEfEpweyIYOR8+Vm1p/K9HLZHPgZsGFEvF3SO4C9IuI7falY0hDSBUrzI2LPvpTVjpwvN7P+Vuak6C9It6H7OUBE/FXSxUCfAjpwFHA/sHYfy2k7zpebWSOUCejDI+IvkorzlvalUkljgD2A7wL/3pey2onz5WbWSGUC+lO573kASNoPWNDHek8DvgaMqLWCpMOAwwCGDB/Zx+qazzdvNrNGKxPQDwcmA2+RNB94iDRAV69I2hN4IiJuy2OtVxURk3O9DBs5Lmqt1w6cLzcbvCTtDpwODAHOiYiTK5Z/gRRnlwGLgcMi4r687Fjg0LzsyIi4ul5dZcZymQPsKmlN0nC7LwKTgEd6uF9ddgD2kvRhUvfHtSVdGBEH9bK8luZ8udnglTt/nAnsBswDZkqa1hWws4sj4uy8/l7AqcDukrYgxdq3ARsDf5S0eUQsq1VfzfHQJa0t6VhJZ0jajRTIDwZmA/v3dgcj4tiIGBMR43Njr+3EYL7NLu90MDezbYHZETEnD2o4Fdi7uEJEPF+YXJOc3s7rTY2IJRHxECn2bluvsnpH6L8CngFuBj4HfJM0ONdHI+LO8vsz+DhfbtbeXlm4sFTqEhglqXh/iMk5XdxlNDC3MD0P2K6yEEmHkzqIDAN2Lmx7S8W2o+s1pl5A3yQitsyVnUM6EfrGiHi5XoE9ERHXA9f3V3mtwPlys0HlqYiY2NdCIuJM4ExJBwLfImVDeqxeQH+1UNkySfP6M5h3IqdYzKzCfGBsYXpMnlfLVNKFnL3Ztu49RbeS9Hz+WwS8o+uxpOfrbDcoOZibWRUzgc0kTZA0jHTecIUPtaTNCpN7AH/Pj6cBkyStJmkCsBnwl3qV1RvLZUgvGj/o+GIhM6slIpZKOgK4mtRtcUpE3CvpRGBWRCbhfiwAAAwiSURBVEwDjpC0Kykr8gw53ZLXuwy4j3Qx5+H1erhAuX7oVoPz5WbWnYiYDkyvmHdc4fFRdbb9LumK+lIc0HvJKRYzazX1cuhWg4O5mbUiH6H3gPPlZtbKHNBL8s0ozKzVOaCX4JOfZtYOHNC74Xy5mbULB/QanC83s3bjgF6F8+Vm1o4c0Cs4X25m7coBvcD5cjNrZw7oOF9uZp1h0Ad034zCzDrFoA7ozpebWScZtAHd+XIz6zSDLqA7X25mnWpQBXTny82skw14QJc0FrgA2BAI0l2yT290vc6Xm1mna8YR+lLgyxFxu6QRwG2SromI+xpVofPlZjYYDPgNLiJiQUTcnh8vAu4HRjeqPgdzMxssmppDlzQeeBdwa5VlhwGHAQwZPrLHZTtfbmaDTdMCuqS1gCuAf4uI5yuXR8RkYDLAsJHjoidlO19uZoNRUwK6pFVJwfyiiPh1f5btFIuZDVYDnkOXJOBc4P6IOLU/y3YwN7NWI2l3SQ9Imi3pmCrLd5R0u6SlkvarWLZM0p35r/b421kzjtB3AD4J3C3pzjzvGxExvbcF+mIhM2tFkoYAZwK7AfOAmZKmVfTqexQ4BPhKlSJeiojSAW7AA3pE/C+g/irP+XIza2HbArMjYg6ApKnA3sBrAT0iHs7Llve1sra+UtTB3MwaYdHCl8p+lkdJmlWYnpw7dHQZDcwtTM8DtutBU1bP5S8FTo6IK+ut3LYB3flyM2sBT0XExAaWPy4i5kvaBLhW0t0R8WCtldsuoDtfbmZtZD4wtjA9Js8rJSLm5/9zJF1Pum6nMwK6b95sZm1mJrCZpAmkQD4JOLDMhpLWA16MiCWSRpE6lPyg3jZtE9CdLzezdhMRSyUdAVwNDAGmRMS9kk4EZkXENEnvBn4DrAd8RNIJEfE24K3Az/PJ0lVIOfS6Y161RUAfPmJ4qfWcYjGzVpO7ZE+vmHdc4fFMUiqmcrubgC17UteAX1jUKA7mZjbYtcURej3Ol5uZJW0d0J0vNzN7XdsGdKdYzMxW1JY5dAdzM7OVtdURum9GYWZWW9sEdOfLzczqa4uAvsGIYVXn+6jczOx1bZlDBwdzM7NKbRnQHczNzFbWFimXIufLzcyqa5uA7qNyM7P62iLlMmK16t87DuZmZq9ri4BejYO5mdmKmpJykbQ7cDppfOBzIuLksts6X25mVt2AH6FLGgKcCfwLsAVwgKQtymzrYG5mVlszjtC3BWZHxBwASVOBvYG6d+JwisXMrL5m5NBHA3ML0/PyvBVIOkzSLEmznp7/aNWCHMzNzF7XsidFI2JyREyMiImj1l17peUO5mZmK2pGymU+MLYwPSbPK8X5cjOz6poR0GcCm0maQArkk4ADy2zoo3Izs9oGPKBHxFJJRwBXk7otTomIe7vbzsHczKy+puTQI2J6RGweEZtGxHe7W/+VhQvrLncwN7NWJWl3SQ9Imi3pmCrLV5N0aV5+q6TxhWXH5vkPSPpQd3W17EnRshzMzaxVlbzu5lDgmYh4E/Bj4Pt52y1IKem3AbsDZ+XyamqbwbkqOZCbWRsoc93N3sC38+PLgTMkKc+fGhFLgIckzc7l3VyrsrYI6Pc8+exTm599xSM93GwU8FQj2tME3pfW5H1pTW/uawGPLFty9WefeWBUiVVXlzSrMD05IiYXpqtdd7NdRRmvrZPPMT4HrJ/n31Kx7UrX7BS1RUCPiA16uo2kWRExsRHtGWjel9bkfWlNFQG2VyJi9/5oy0Br+xy6mVkLK3PdzWvrSBoKrAM8XXLbFTigm5k1zmvX3UgaRjrJWTkw1TTg4Px4P+DaiIg8f1LuBTMB2Az4S73K2iLl0kuTu1+lbXhfWpP3pTW1zL7Uuu5G0onArIiYBpwL/Cqf9FxICvrk9S4jnUBdChweEcvq1af0RWBmZu3OKRczsw7hgG5m1iE6LqB3d5ltu5A0VtJ1ku6TdK+ko5rdpr6SNETSHZJ+1+y29IWkdSVdLulvku6X9J5mt6m3JB2d31/3SLpE0urNblNPSJoi6QlJ9xTmjZR0jaS/5//rNbONA6mjAnpfbm/XgpYCX46ILYDtgcPbeF+6HAXc3+xG9IPTgf+JiLcAW9Gm+yRpNHAkMDEi3k46aTepua3qsfNIl8UXHQP8KSI2A/6UpweFjgroFC6zjYhXgK7LbNtORCyIiNvz40WkoFH3KrFWJmkMsAdwTrPb0heS1gF2JPVMICJeiYhnm9uqPhkKrJH7Pw8HHmtye3okImaQeoYU7Q2cnx+fD+wzoI1qok4L6KVub9du8uhr7wJubW5L+uQ04GvA8mY3pI8mAE8Cv8zpo3MkrdnsRvVGRMwHfgg8CiwAnouIPzS3Vf1iw4hYkB//A9iwmY0ZSJ0W0DuOpLWAK4B/i4jnm92e3pC0J/BERNzW7Lb0g6HA1sDPIuJdwAu06U/6nFvem/QltTGwpqSDmtuq/pUv0Bk0fbM7LaD36fZ2rUbSqqRgflFE/LrZ7emDHYC9JD1MSoPtLOnC5jap1+YB8yKi69fS5aQA3452BR6KiCcj4lXg18B7m9ym/vC4pI0A8v8nmtyeAdNpAb3MZbZtIQ+feS5wf0Sc2uz29EVEHBsRYyJiPOk1uTYi2vJIMCL+AcyV1DWi3y6sOBRqO3kU2F7S8Px+24U2PcFboXgp/cHAb5vYlgHVUZf+9/b2di1qB+CTwN2S7szzvhER05vYJku+BFyUDxrmAJ9ucnt6JSJulXQ5cDupV9UdtNBl82VIugTYCRglaR5wPHAycJmkQ4FHgP2b18KB5Uv/zcw6RKelXMzMBi0HdDOzDuGAbmbWIRzQzcw6hAO6mVmHcEC3qiTtIykkvaWX258oadduym/3wcbMWooDutVyAPC/+X+PRcRxEfHHOqvsQxoRc0DkkTjNOpoDuq0kjx/zPuBQCsOpStpI0gxJd+bxs/85j3F+Xp6+W9LRed3zJO2XH5+cx3X/q6QfSnovsBdwSi5rU0lHFtaZWqVNh0j6raTr8zjXxxeWHSTpL7msn3cFb0mLJf1I0l3AeyrKe3eu605Jp3SNpy1pvKQbJd2e/96b5+8k6Ybchjl5nz6R671b0qZ5vQ0kXSFpZv7boT9fG7O6IsJ//lvhD/gEcG5+fBOwTX78ZeCb+fEQYASwDXBNYdt18//zSHcwXx94gNcvYltheWG7x4DViutUtOkQ0oiA6wNrAPcAE4G3Av8NrJrXOwv4VH4cwP419vEe4D358cnAPfnxcGD1/Hgz0o18IV2N+CywEbAaaYygE/Kyo4DT8uOLgfflx28kDd3Q9NfUf4Pjr6Mu/bd+cwDpJg6QBtM6ALiNNFbOlDxo2JURcaekOcAmkn4KXAVUDr/6HPAycG6+U1GtuxX9lXQ5/ZXAlTXWuSYingaQ9GvSr4ilpC+VmWk4Etbg9cGYlpEGN1uBpHWBERFxc551MbBnfrwqcIakd+btNy9sOjPysKySHizs693AB/LjXYEtclsA1pa0VkQsrrFPZv3GAd1WIGkksDOwpaQgHYmHpK9GxAxJO5JuVHGepFMj4gJJWwEfAr5AGjfjM13lRRpfZ1vSwE/7AUfk8ivtQbpxxEeAb0raMiKWVqxTOU5FAALOj4hjq5T5ckQs69ETAEcDj5PuRLQK6cuoy5LC4+WF6eW8/llaBdg+IorbmQ0I59Ct0n7AryJiXESMj4ixwEPAP0saBzweEb8g3Xloa0mjgFUi4grgW1QMJZvz8etEGlTsaFKgBFhEStkgaRVgbERcB3wdWAdYq0rbdlO6X+QapJOqfybdYmw/SW/IZY3M7awp0h2GFknaLs8q3nZtHWBBRCwnDY7W05OpfyAN3kVuzzt7uL1Zr/kI3SodAHy/Yt4Vef4twFclvQosBj5FuiPUL3NQBqg8Uh4B/Fbp5sMC/j3Pnwr8QtKRpIB6rtLt3QT8JKrf1u0vuS1jgAsjYhaApG8Bf8hteBU4nDTKXj2H5vqXAzeQUkOQcvBXSPoU8D+kG1j0xJHAmZL+Svp8zSD9cjFrOI+2aG1B0iGkmxkf0U/lvZbXlnQMsFFEHNUfZZs1i4/QbbDaQ9KxpM/AI6ReNGZtzUfoZmYdwidFzcw6hAO6mVmHcEA3M+sQDuhmZh3CAd3MrEP8f9dYJrRAoJosAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_contour_filled(APG, RPG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "從填滿等高線圖中我們可以看出場均籃板與被 $h$ 預測為前鋒的機率有正向的相關性、場均助攻與被 $h$ 預測為前鋒的機率有反向的相關性，這一點與場上籃球員的分工相符，擔任中鋒與大前鋒的球員比較擅長爭搶籃板球、擔任後衛的球員比較擅長傳球助攻；而圖中左下到右上有一段帶狀區間，被稱作「決策邊界」（Decision boundary），決策邊界能夠隨著門檻（Threshold）設定往左上角或右下角移動，預設的門檻多半設定在 50%，亦即由 $h$ 預測出的 $\\hat{p}$ 如果大於 50%，就輸出 $\\hat{y} = 1$，否則輸出 $\\hat{y} = 0$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_decision_boundary(XX, YY, x, y, target_vector, pos_dict, h, resolution=50):\n",
    "    Y_hat = np.zeros((resolution, resolution))\n",
    "    for i in range(resolution):\n",
    "        for j in range(resolution):\n",
    "            xx_ij = XX[i, j]\n",
    "            yy_ij = YY[i, j]\n",
    "            X_plot = np.array([xx_ij, yy_ij]).reshape(1, -1)\n",
    "            z = h.predict(X_plot)\n",
    "            Y_hat[i, j] = z\n",
    "    fig, ax = plt.subplots()\n",
    "    CS = ax.contourf(XX, YY, Y_hat, alpha=0.2, cmap='RdBu')\n",
    "    colors = ['red', 'blue']\n",
    "    unique_categories = np.unique(target_vector)\n",
    "    for color, i in zip(colors, unique_categories):\n",
    "        xi = x[target_vector == i]\n",
    "        yi = y[target_vector == i]\n",
    "        ax.scatter(xi, yi, c=color, edgecolor='k', label=\"{}\".format(pos_dict[i]), alpha=0.6)\n",
    "    ax.set_title(\"Decision boundary of Forwards vs. Guards\")\n",
    "    ax.set_xlabel(\"Assists per game\")\n",
    "    ax.set_ylabel(\"Rebounds per game\")\n",
    "    ax.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOydd3hcxdW437NNbdVlySq25Y5xB1NDwGBKgFAS+BIINi2E0EICfAnw5RdawpfwJYGQEFoKxNjYIRBiUxIMGEJJcDAuyN2SZcuWLcnqu6u2ZX5/3Ct5JWullbSrVZn3efbR7t17Z86dvZozc+acM6KUQqPRaDQaAEusBdBoNBrN0EErBY1Go9F0oJWCRqPRaDrQSkGj0Wg0HWiloNFoNJoOtFLQaDQaTQdaKQwDROTvInJNGOe5RWRSFOrfKyJnR7rcfsjxgIgsi7UcACLyExGpFpGKWMsSCURkoYgciLUcQwURKRQRJSK2WMsy2GilECHMjrNZRFwiUi8i/xKRm0RkwG2slDpfKfWnMM5zKqX2DLQ+Tc+IyHjgLuBYpdTYbr5fKCIBU0m3v14bfEmHLyIyVURWishhEWkUkd0i8hsRKYi1bCMdrRQiy0VKqWRgAvAz4G7gD7EVSQMgItYIFjceqFFKVfVwzkFTSbe/LuprJRGWObjcIT36FZEpwDrgIDBfKZUCfAEoAU6LQn1Duj0GG60UooBSqkEptRr4OnCNiMwCEJE4EfmFiJSJSKWIPC0iCe3XicglIrLJHBmViMiXzOPvi8gN5vspIvJPEWkwzRd/Drpemf9QiEiqiCw1R1r7ROT/tc9aRORaEfnIlKVOREpF5PxebusEEdlmnv+ciMQH1fstESkWkVoRWS0ieebxo6bgXe6lRzlEZKJ5ry4ReRvIChZIRP4iIhVmW3wgIjODvnteRJ4SkTdFxAPcaba5Neicr4rI5u5uNlT7mWa0t4E8cwbwfC/t1rXcGWYb1IvIVhG5uAeZrzfPa//dficiVUHnvyAi3zPfXyci28222iMi3w46b6GIHBCRu8Uwdz0nIglmfXUisg04oYucd4tIuVneThFZ1M29nGS2f3CbfkVEPjffnygi683nuVJEHg2zmR4APlZK3amUOgCglKpSSv1KKbXSLPtaEfmoizzBz/+FIrLRrHu/iDwQdF77c/lNESkD1oqI1XwOq0VkD3Bhl7KvNdvVZT6nV4V5L8MPpZR+ReAF7AXO7uZ4GXCz+f4xYDWQASQDrwE/Nb87EWgAzsFQ1vnAMeZ37wM3mO9XAD80z4kHTguqSwFTzPdLgVVmPYXALuCb5nfXAl7gW4AVuBljVCY93NsWYJwp+8fAT8zvzgKqgeOAOOA3wAfmd4WmTLagsoLvpUc5gH8Dj5rlng64gGVBZV1v3l8c8CtgU9B3z5vt+YWgttoGnB90zqvAXSHuuaf2Wwgc6OFZ6PZ7wA4UA/8DOMy2cwHTe5C5DDje/H4nsAeYEfRszTffXwhMBgQ4A2gCjguSxwc8YrZVAsZM9kPz9xxn/r4HzPOnA/uBvKDfcXKIey0Bzgn6/BfgnqDfb4n53gmcHOb/UgVwbS/nXAt81OVY8PO/EJhttuMcoBK4tMtzuRRIMtvjJmAHR57x98xzbOY5jUG/Uy4wM9Z9TrReMRdgpLwIrRQ+wejEBfAE/3MBpwCl5vtngMdClP0+RzrSpcCzQEE35ylgCkYH24Zh827/7tvA++b7a4HioO8SzWvH9nBvNwV9vgAoMd//Afi/oO+cGB19IeEphW7lwDDR+ICkoO9fJEgpdJExzbw21fz8PLC0yzl3A8vN9xkYHWduN2X11n4L6V0pBID6oNfXgC9idHiWoHNXAA/0IPMLwJ1mm+wE/g+jA5tolmsJIcPfgO8GydMGxAd9vwf4UtDnGzmiFKYAVcDZgL2X5/4nwB/N98kYz/gE8/MHwINAVh//l3xdZLvNvFc38LugZyekUuimzF9h/n8FPZeTgr5fS+dn/Fw6K4V64DIgoS/3Mhxf2nwUffKBWmAMRqf3mWkSqAf+YR4HY4RSEkZ5P8BQMP8xzQ/Xd3NOFsaodF/QsX2mLO10eM0opZrMt84e6t3fpaw8831ecD1KKTdQ06WungglRx5Qp5TydKkXMOztIvIzMcxsjRiKCzqbmIJlBlgGXCQiSRid9IdKqUPdyBRO+/XGQaVUWtDrJfOe9iulAj2U21Xmf2J06qdjdLLvY8wEzjDlDwCIyPki8olpwqvHUNzBbXFYKdUS9DmPo39TAJRSxcD3MMw4VWIs+ObRPS8CXxWROOCrwAalVHtZ3wSmATtE5FMR+XKIMrpSgzEab5fnCaVUGkbHbg+nANO09Z5p/mvAUKRZXU4Lvv+e2sODYQq+CTgkIm+IyDFh3suwQyuFKCIiJ2D8w3+EYWJpxph2tncUqUqp9o54P8b0v0eUUhVKqW8ppfIwRq9PtttRg6jGGK1PCDo2HigfwO2M61LWQfP9weB6zA4306yrvUNPDLr2KG+dEBwC0s3ygutt5xvAJRij2VSM0R8YCrOdTimAlVLlGCaNrwJLMEbh3RGN9gOjrcZJZ4+0ruV2TVv8T4wZxkLz/UcY5qUzzM+YHfIrwC+AHLMDfZMe2gKjfbv+pkdOVupFpdRpGG2gMExPR6GU2obRgZ6P8Zu8GPTdbqXUlUC2ef3LXX7PULyL8Rv1hIeg50pEuj5XL2KYascppVKBp+ncHtC5TXprj7eUUudgKKsdwO96kW/YopVCFBCRFHNUtBLD3FFkjuh+BzwmItnmefkicp552R+A60Rkkbmgmd/daERE/kuOuOXVYTzYwSNPlFJ+4CXgYRFJFpEJGCaIgfj43yoiBSKSgWEOa1/gXmHKPc/snP4XWKeU2quUOozR4S02R/bXE4biM+9hH7AeeFBEHCJyGhDswZMMtGKMKhPNesNhKcZsazbw1xB1R6P9wPCoaQJ+ICJ2EVmIcU8rQ12glNqNMZhYDPxTKdWIYR+/DFMpYKxPxAGHAZ8Yi/Xn9iLLS8C9IpJuPk/faf9CRKaLyFnm79li1h8IUQ4YHfB3MWYzfwkqZ7GIjDGf/XrzcE/ltPMA8EUReVRE8s2ysoAZQedsBmaaz128eU0wyUCtUqpFRE7EUFg98RJwu/mMpwP3BN1HjhhOIEkYz5w7zPsYlmilEFleExEXxqj/hxiLpNcFfX83xkLjJ6bJ4x2MRT2UUv8xz30MY7Hxn3QeqbZzArBORNwYI6Hvqu5jE76DMZragzG6fBH44wDu7UVgjVleCYYtGaXUO8CPMEaqhzA6/SuCrvsW8H2Mznsm8K8+1PkN4CQM89v9GB16O0sxRqjlGAvIn4RZ5qsY7fpqkLmqOyLdfiil2jCUwPkYs5EngauVUjt6ufSfGC6w+4M+C7DBLNcF3I7RsdVhtNvqXsp8EKP9SjF+1+BZUxzGQnQ1hnkvG7i3h7JWYMxc1iqlqoOOfwnYaj6rjwNXKKWaoSPQ8ovdFaaU2oXxuxcAm83/qY8xZlo/CjrnIYz/od0Yv1EwtwAPmdfeh9E2PfE74C0MZbOBzgMGC8ag4CDGs3gGhlPEiKTdy0OjGTWISAnwbVOhaTSaIPRMQTOqEJHLMExua2Mti0YzFNGRfJpRg4i8DxyL4Ts/Ym3CGs1A0OYjjUaj0XSgzUcajUaj6WBYmI/SMzJVXv643k/UaDSaEUabP0BAKeLsNhwWhSXgQ5TC1+pFbD3H8m3evq1aKTWmx5O6MCyUQl7+OFasejvWYmg0Gs2gsr+uiSavn8n56RQmBUhorMZdXIJf2bCNCRVkfoQx82fv6/WkLkTNfCQifxSRKhHZ0s13d5lZCruGnWs0Gs2oZ39dEzurXHjafEwaE0eBtwa1c1ufFEJ/ieZM4XngCToHHCEi4zCiLcuiWLdGo9EMS0LNDrzKhm3M+Kibd6JWvlLqAxEp7OarxzDSDKyKVt0ajUYz3GhXBkopJmfHm7ODQ7g9rqjPDoIZ1DUFEbkEKFdKbRbpmpvqqHNvxEjnS27eyN2Br2jzRla/spbyA1XkF2Rz8WVnMXvu/FiLpdGMelTAD60NEPBGvS5/QJFjBbvDgs2ikCYPVT4fxCVBYkr7WSGvtyOkBQRrL/1qOAyaUhCRRIzNRXpL1AWAUupZjH0DmDl73ogMpijavJEnH3sbZ/JicnImU19XwpOPLeOWO9CKQaOJNa0NZGWkkZaeTm+D2P7S5gsQUBDnsBJnAeX3IV4vKmDGVtocvZahlKKuvp766moy1cDlHMw4hckYG4NsFpG9GMmuNnST8nbUsPqVtTiTF5OcMg2xWElOmYYzeTGrX9EZGDSamBPwDqpCsPh90NpqKASbIyyFACAipKel4T0qO3r/GLSZglKqCCPbIgCmYljQJaviqKL8QBU5OZ0zSTudkyk/0NN+8BqNZrCIhkLoThn4m1qPGIfCVAbBRFLOaLqkrsDY0GS6GJuGfzNadQ1X8guycbs7b7bmdpeQX5Ad4gqNRjOc6VYhtLYaX/ZhdhBNoqYUlFJXKqVylVJ2pVSBUuoPXb4vHM2zBICLLzsLt2sZrsZdqIAfV+Mu3K5lXHzZWbEWTaPRRJA2X4AWbwC73UpKvJUE5UM1eQyF0IsyqKysZMm11zJ95rGc9IVT+eKZC/nb6ug5bw6LiOaRyuy587nlDlj9yooO76OrbzhHLzJrNCOE9pkB9DA76AGlFJdf8XWWXHUVLzz/PAD7ysp4/Y03oiazVgoxZvbc+VoJaDQjgM8+/ZS3li2lcu9ecgoLOevKxcw7/oTu1w7CNBO99/77OBwObrzhWx3HJowfz603R2/jN50lVaPRaAbIZ59+yhsPPcgV1TU8PjaPrx2u5u8/eYgtG9fjUD5oaQ57dhDMtu3bmT93XpSk7h6tFDQajWaAvLVsKYtTUpiakorFYmFGejrXp6WydvnSfrmZhuL2O77H8SedxClfPC1Ckh+NVgoajUYzQA6V7mWiMxkFWC1gQTElKYmDe/YYJ/RTGRw7YwYbN2/q+Pzrx37FW2+8QXV19Hx0tFLQaDSaAdDmC5A9oZBStwuHVYyFWr+f3Y2NjC2cNKDZwZkLF9LS0sIzv3u241hTc/PAhe4BrRQ0Go2mH7S7mQaU4vwlS1je2MDO2lq8Xi87Gxp4weXi/MWLB1SHiPDyyj/zwUcfMe3YGZx6+hf55o3f4uEf/yRCd3E02vtIo9Fo+kjXILTTTjwe+733sOLFF6ko28/YCYVc/L07WLBgwYDrys3NZfmflvZ+YoTQSkGj0WjCpLsEdrR48QcCLDj+eBacdEqsRRwwWiloNBpNGIRKUaFgSKSniBRaKWg0Gk0P9JjAbgQpg3b0QrNGo9GEoNcEdiMQPVPQaDSaLuyvayLHqrDbR8fsIBitFDQajcYkeJ9ku9MyamYHwWiloNFoNBxRCJPz0ylMClBRXm6kt4aYKYP4ZCezZs7s+Pzyn1+icMKEqNaplYJGoxnVBM8OJmfHU+CtQe08RCA+yTghhrODhIQE1n+yblDr1EphiFG0eSOrX1nbsb/CxZedpVNrazRRouvsIKGxGndxCV5lg8SUPimET9evZ+myNezdV0HhhLFcvfhcTohA8Npgo72PhhBFmzfy5GNvU193JTk5v6a+7kqefOxtijZvjLVoGs2IYn9dEzurXHjafEwaE2fODrbhLi7Br2zYxuT1qbxP16/nwR+/QU3tFeTnPUFN7RU8+OM3+HT9+gHJ2dzczIKTT2LBySdx+RVfH1BZ4aJnCkOI1a+sxZm8mOSUaQDm38WsfmWFni1oNBGip9mBbcz4fnWKS5etISV1CWmp0wHMv0tYumzlgGYL2nw0yik/UEVOzuROx5zOyZQfqIqRRBrNyCHU2oHb4+rX7CCYvfsqyM+b0ulYSvIU9u6rGKjYg07UzEci8kcRqRKRLUHHfi4iO0TkcxF5VUTSolX/cCS/IBu3u6TTMbe7hPyC7BhJpNGMDIJnB2dPT+NYqxvZswuvuxnJGj8ghQBQOGEsja7iTscaXcUUThg7oHJjQTTXFJ4HvtTl2NvALKXUHGAXcG8U6x92XHzZWbhdy3A17kIF/Lgad+F2LePiy86KtWgazbCkfe0gNzuFRdNSmeGvGtDaQSiuXnwujQ0vUN+wk0DAT33DThobXuDqxedGpPzBJGrmI6XUByJS2OXYmqCPnwCXR6v+4cjsufO55Q5Y/cqKDu+jq284R68naDT9INTagX8AawehOGHBAu7/ESxdtrLD++iO2y8csPdRXdXhCEkYPrFcU7ge+HOoL0XkRuBGgNy8gsGSKSyi6TY6e+78YaUEtAutZqhxtDKowr2hhIYoKINgTliwYFi6oHYlJi6pIvJDwAcsD3WOUupZpdQCpdSC9IzMwROuF7Tb6BF0W2iGEvvrmnqZHUTGVDTSGfSZgohcC3wZWKSUUoNd/0DRbqNH0G2hGUzaO/yeiPTsQCmFiPTz6sEjkl3poCoFEfkS8APgDKVU02DWHSm02+gRdFtoBovgGcDEzMRuz1EtbiByswM7Ql19PelpaUNaMSilqKuvx05kZIyaUhCRFcBCIEtEDgD3Y3gbxQFvm438iVLqpmjJEA3yC7KpryvpGB3D6HUb7Utb6LUHTX/obn2AHsYc7uLIrR2kBYT66moOVw/+Ym9fsSOkBYRI6IVoeh9d2c3hP0SrvsHi4svO4snHlgGLcTon43aX4HYt4+obzom1aINOuG3RvvbgTF5MTs5k6utKePKxZdxyB1oxaELS0/pAaCK3dmAVIVMN3RnCUURIVB3R3Ee02+gRwm0Lvfag6Qux8h7SGOj27QfDzW00moTTFnrtQRMOXdNQaO+h2KCVgibq6HUYTW90ayrasFnPDmKATp2tiTo6fYcmFJFOYa0ZOFoBazqIloeQXofRdEc0UlhrBo5udw0QfQ8hvQ6jaSeaKaw1A0crhSFGrPz5tYeQZjDQs4Ohj/4NhhCx9OfXHkKaaKJnB8MHvdA8hAgerYvFSnLKNJzJi1n9ytqo1603+NFEi2hvcKOJLHqmEGEGYv7pz2g9UuYmHamtiTTdBaE17azSs4Mhjp4pRJCBppLu62g9kqmrDQ+hc0hLX0Fl5e2kpa/glju0h5Cmf4RcO9CzgyGPnilEkIEu1vZ1tB7pxWHtIaQZKDpFxfBH/0YRZKCLtX3159eLw5qhwv46IxO+3uBm+KOVQgSJRDqHvozWdfoIzVBAzw5GFnpNIYIMdjoHnT5CE0u6pqjQs4ORgQyHHTFnzp6nVqx6O9ZihMVgB5/1pz694Y1moOi9kIcHY+bP/kwptaAv12ilMMoIDpALXszWnkaacOgahDZOmrFWHiKg3UyHJP1RCtrcN8rQ6Sw0/UWnqBgd6N9xlNFfjyVtchq96BQVowu90DzK6E86i0gGyWmGFzpFxegjakpBRP4oIlUisiXoWIaIvC0iu82/6dGqX9M9/fFYimVOJk1saPcsys1O0ZvfjDJ6VQpisFhE7jM/jxeRE8Mo+3ngS12O3QO8q5SaCrxrftYMIv1JZ1F+oAqnUwfJjRa6rh3o2cHoIpw1hSeBAHAW8BDgAl4BTujpIqXUByJS2OXwJcBC8/2fgPeBu8MVVhMZ+prOQgfJjQ5CBaG5Qc8ORhHhmI9OUkrdCrQAKKXqAEc/68tRSh0y31cAOf0sRzOI6CC5kU9PcQd6djC6CGem4BURK6AARGQMxsxhQCillIiEDJIQkRuBGwFy8woGWp1mAOg9lkcuOkWFpivh/Oa/Bl4FskXkYeBy4P/1s75KEclVSh0SkVwgpFFaKfUs8CwYwWv9rC+mjCQ3Tp1BNfoM5vPSrgwAHZWs6USvSkEptVxEPgMWAQJcqpTa3s/6VgPXAD8z/67qZzlDnlhurakZfgzm86JnB5qeCNcltRL4EPgXkCAix/V2gYisAP4NTBeRAyLyTQxlcI6I7AbONj+PSLQbp6YvDMbz0jWBXYG3Rs8ONEfR66BARH4MXAuUYK4rmH97XGVUSl0Z4qtFfZBv2KL3OtD0hWg/L6EWkvXsQNOVcJ6FrwGTlVJt0RZmJKHdODV9IVrPi05Roekr4ZiPtgBp0RZkpKHdODV9IRrPi05RoekPvabOFpEFGAvCW4DW9uNKqYujK9oRhmvq7JHkfaSJPpF6XnR6a0070Uqd/SfgEaCICMQnjCa0G6emL0TiedHprTUDJZxnpEkp9euoS6LRaPpNV2Xgq9drB5r+EY5S+FBEfooRYxBsPtoQNak0wwZtIos93c4O9ujZgaZ/hPO8tP+Hnxx0rFeX1NHCaO4UizZv5JEHl1JXl463zcK+Ug/btyzl7vt1gN5g0FMQGujZgaZ/hBPRfOZgCDIcGe1Ry889s5yD5enEJ9xEQuIUvN5iDpY/zXPPLOfRJ0f+/ceSnhLYaWWgGQhhzSxF5EJgJhDffkwp9VC0hBoujPb9jrcW7SU+/rc4HNMBjL/q22wtujXGkoXPcJvp6RQVmmgTTkTz00AicCbwe4yEeP+JslzDgu6iUNta6/ngvX9Tfv3dEe1khmTnpSxA1+CqbPP40KfrTG/vnlXcdesvGDs2henHThkabWyiE9hpBotw/ntPVUpdDdQppR4ETgGm9XLNqKDrfsc11esp2vwyNtutEd3LeKjukTxzzmRaWt7D29aAUgpvWwMtLe8xc87k3i8eAgTP9GprN7KvtAjUQzQ23jlk2hg6zw4WTUtlhr8K/4Z1NOzep4PQNBEnHKXQbP5tEpE8wAvkRk+k4UPXKNTiXS8BpzF1+imIxYrXm0HZvhP479se5uH7ftnvDmaoJte77ttfI69gLci7NDV9APIueQVrue7bX4upXOESvM1o2d412OxLSEg8nqamtiHRxt0lsNP7JGuiTTgmyNdFJA34ObABw/Po91GVapjQdfMZn7eU2fPuIiMzi9qaarYVlWOzLUQF1pgjz/4tQg/V5Hqz587n7vvoYta6bMiYXHojON+Qx1NBQsIUfN5GkpISgNi2sQ5C08SKcLyPfmy+fUVEXgfilVIN0RVr+BAchfrwfb+kvq4ByGFf6UHs9qkoKkhy5g5oEXooJ9cbzlHbF192Fk8+tgxYTGJiDk0eYyY39Zh8IDZtrBPYaWJNOAvNX+3mWANQpJTSeaCDCO5k3G43dtshfL5lTJ1+IdD/kWdwuU7nZNzuEtyuZVx9wzmRvYFRRvBMz5mylaamHRROvI309PSOhHSD2cYjdXawYcsWVr72L8rKaxifn8kVF53KcbNmxVosTQjCSYj3Bsbi8nvmoYXAZ8BE4CGl1AvRFBCGV0K8di+hD9auw2afxZRpXyMzy8hH5WrcRVr6Cn740F39LndIeR+NMGLVxiM5gd2GLVv42VP/IsV5FcnOSbjce2h0L+eem7ViGAyilRDPBsxQSlUCiEgOsBQ4CfgAiLpSGE60m1MuvszwGHI4UlAB/4BH98PZTDNciEUbj9TZQTsrXzMUQmryVADz71WsfG25VgpDlHCeuXHtCsGkyjxWKyLeKMk17Om6CJ1fkM3VN5wzqJ3OcJhdDAcZo8FoSWBXVl5Dbs6kTseSnZMoK6+Jar0btxTx7murqCovJzs/n0UXXcL8WbOjWudIIRyl8L65wPwX8/Nl5rEkoD5qko0ABjryHEiHORxScAwHGaPBaEpgNz4/k9r6PR0zBQCXew/j8zOjVufGLUWseeoJFjuTmZyTQ0l9PcueegJuvk0rhjAIJ07hVuA5YJ75WgrcqpTy6LxI0WOgAWtDNbYhmOEgYyRpjzvIzU45KghtJM0OgrniolNpdC+nwbWbgPLT4NpNo3s5V1x0atTqfPe1VSx2JjMtORmrCNOSk1nsTObd11ZFrc6RRDguqQp4xXxpBomB5lUaqrENwQwHGSPFaE1gd9ysWdxzM6x8bXmH99EtS6K7yFxVXs7knJxOxyY7nVSVl0etzpFETGaqInIHcANGIFwRcJ1SqiUWsgxV+tNhBpubKg4exNu2inETjngUD8TvPhq2/6EcfxEpdAI7QzEM5qJydn4+JfX1TEtO7jhW4naTnZ8/aDIMZwY9c5mI5AO3AwuUUrMAK3DFYMsx1OmaVwl67jC7mpuyxtzDzu1/Zf++vw54I/ho5V6Kxmb1Q4XgFBWjbXYQaxZddAnL3C52uVz4lWKXy8Uyt4tFF10Sa9GGBT0OVETECixVSl0VhXoTTO+lROBghMsf9vQ1YK2rualg/EkAVB/+GXbHuwPyfopWivCh4KEVDfTsILbMnzUbbr6NFUHeR+cuuUYvModJj8+nUsovIhNExKGUaotEhUqpchH5BVCGkWxvjVJqTdfzRORG4EaA3LyCSFQ9rOhrh9mduSm/YAE2ex5P/vGRAckSTdv/SIq/0Ckqhg7zZ83WSqCfhDNo2QN8LCKrAU/7QaXUo/2pUETSgUswIqLrgb+IyGKl1LLg85RSzwLPghHR3J+6hhL9scn3pcOMpn0+3LJHa8wBjPwgNM3oIZw1hRLgdfPc5KBXfzkbKFVKHVZKeYG/AtHzTxsCDMZ+CNG0z4dT9lDd8yHa6PTWmpFGOC6pDwKISKJSqikCdZYBJ4tIIob5aBGwPgLlDgmCR8vx8RbAy9bPy8w8SI1H7bMwc3YhYKelJUB+QTYz5+Sz9fPyPo+2o2mfD6fsSKw7DLeZhp4daEYi4WRJPQX4A+AExovIXODbSqlb+lOhUmqdiLyMsTeDD9iIaSYa7gRH6NrtqWz6bB3wEf7A5SRZz2D7lmXkj2ugfH8WNttCWluWs+mzFOA0Zs87ib17PuatN/7KMcd+h/yCBX2O8I2mfb63sge67jCcopv12oFmJBOO+ehXwHlADYBSajNw+kAqVUrdr5Q6Rik1Sym1RCnVOpDyhgrBo+WyvZUkJJxNfMJNBHxbEcnFZl9C8c43sdunghzG7/MSn3ATCQlnU7a3kurDO4iP/z6HqxKGXYRvX11ouzJcopuDZwdnT0/jWKsb2bMLr7tZb42pGRGENcNVSu0XkeBD/uiIM/wINnmU7CrlmJk3AeDxNJOQkAI4sdk9eDyraGvdSVPTevz+B3HENWCzO7HbpwAWPJ5moIL4hC5hvh8AACAASURBVHl4PJ90lD9cInwHuufDUI9u7s7NtGlnlZ4dRBC978LQIBylsF9ETgWUiNiB7wLboyvW8KCryWNf6b0UbVrHnPmnkpSUQFtrI4oK4hPiaWneSkB9CbEcR0D5gY+IT4jH6y1GGGtuATkWj3sTSc6EjjqiHeEbKTv+QNc0hnJ0c88pKvTaQSQI3nchN2cStfV7+NlTy7nnZrRiGGTCeZ5vAh4H8jGCzN7CSJI36um6uDpl2tco2vQSu3cKU6adSNGmd4CPcDi82O3/hd0OBeNzOVDmBiahAs/S2vY0cBpTpp9Ek+cYaqp/TuGk70RkD4beiLQdfyBrGkNxdzkdhDZ46H0Xhg7heB9VA5GOaB4RdDV5ZGYtYPbcADu234fX+w/mHW94H332n92kZwQonDSOjMwsUlKr2btnP3W1Ozj+xKnA27S0vEXhpGwuvHQhWz//B+UHlkY9wjdakcr9YahFN4/WBHaxIlb7LmiOJhzvo0kYM4WTMRLY/Ru4Qym1J8qyDXm6M3k44tI4/cxTOm25+fB9v6S+LonklCwAMjKzsNtrSUs/vdutOS+9PPqyw9Cz40fTeypcM5meHcSGWOy7AHoznu4Ix/voReAlIBfIw9hsZ0U0hRouhBswNlQTvw3UY2i4EE5gXdcgND07GFxise9C+2Y8V9bX8+ucHK6sr2fNU0+wcUtR1OocDoSjFBKVUi8opXzmaxkQH23BhgOGyeMc0tJXUFl5O2npK7jljqNNHuGeN9gMVWUVaXpzd+3OzbR98xvtZjo4GPsunEpG2nIOVX6PjLTl3HNzdL2P9GY83RPObPjvInIPsBLDfPR14E0RyQBQStVGUb4hT7gmj6GY+G2o2fGjRSgzWUnpIXZWuaIShKbdK/vOYO+7oDfj6Z5wlMLXzL/f7nL8CgwlMQnNsGUoKqtI093aT1XtLjLHZkYlRYV2rxwe6M14uicc76OJgyGIZvAYbjmGBkqwu6s4JtDYWILHs4wbbj47KikqtHvl8GDRRZew7KknWIwxQyhxu1nmdnHukmtiLVpM0c4UYRKqI+16PJyEdrHslIdTjqFI0W4me/HFFzhYXsWUqfncuPgCTps6HnfxrognsNPulcMDvRlP92ilEAahOtLzvryTt14v6zi+d8+qXhPaRbpT7quCGUqxCYPF/romHLlTuObOyUzOjmecNGOtPBQ1z6JYuVdq+o7ejOdoBn2P5uFIKO+Vpb9/o9PxcBLaRTLxW3/2MCg/UIXTOXRiE6JNLBLYxcK9UqOJFOEEr30B2KSU8ojIYuA44HGl1L6oSzdECOW9UlPdyNz5R457PL0ntItkwFh/Rv3RzDE0lNYqQgWhuSHqcQeGeyWsfG15h/fRLUu095FmeBCO+egpYK65j8JdwO+BpcAZ0RRsKBGqI83MSsHtPnI8Kan3hHZ96ZR762T7o2B6yjEUbqfe3XlAv8xi0VAkvaWoGAyb6WC7V2o0kSIc85FPKaUw9lV+Qin1Wwa2HeewI1SQ19U3XNjpeNaYY2hp+TljsptDBoOFGzAWjmmoPxHJoQLpgLBMUaHkeu6Z5X02i0V6C8/2qOTc7BQWTUtlhr9KB6FpNH0knEGTS0TuBRYDp4uIBbBHV6yhRU9BXpOnbuw4Xjgpm7nHTWTtmp+xc3sjSU47uXlOnnq8hvyCtR2j4HACxrqahoK38Dz9zFO4+LKz+p1ZtLvYhIfv+2VYpqhQJqvPN93Oqaf1bdYSqUXv/XXGLrE6gZ1GM3DCUQpfB74BfFMpVSEi44GfR1esoUE4po3gDrZo80YeeegVWluvRySZ6sNluBo/4bgF11Bfl9ZhTgmnzjVvfkB6xsUUTqoGYFtROTbbQlRgjTmiXsYtd5zDLXecE1ZEcnd7R7e0WDvuK1xTVKjzUJZOpjTofdYSifUVncAu+uikcaOLcILXKoBHgz6XYawpjGj64zr63DMvcfDAOSQkLMLn82KR42htSWfXzhc55bRfAYt57plHaW7K7rZcOGKXT8+w0eSxsK2oHKutGbt9DooKkpy5nUbUP3zorl5H1aH2jp499/IOZZWQ6A+rUw+1JjJzzmTcrr7NWgay6N11n2Q9O4gO7UnjFjuTmZyTQ0l9PcueegJuvk0rhhFKyDUFEXGJSGOo12AKGQv64zq69fMS4uPPxO5IxecLYLVlYbGeSV3tbsAYBW8t2ovffyHFu1r4+MP1FO9qwe+/kNWvrO1U54SJ5wF/A6C2ph6lDuHzvsD4wnM7ygp3RB1q7+iyfe903BfYB5Tx9bpvf63PSf/6m5BPJ7AbPHTSuNFHyJmCUioZQER+DBwCXgAEY8Od3IFUKiJpGF5MszDyJ12vlPr3QMqMND2ZNkKalSQAVAEZ2GxWAn4vIodRKgAYo2Bvm4/SEjsOxxQSElJoa22ktGQHLc3FOFNSsNvq2bj+f/F4KrDaLKjAswT8GxHLacyY9TUysxZQU72e4l0v4fOW8vB9v+zVSyjYFBW8d7THU9FxX5WVgbBMUb2tifRlLaCvCfm6zg6ikaJC0xmdNG70EY7J9WKl1Nygz0+JyGbgvgHU+zjwD6XU5SLiABIHUFZUCGXaiI/3hzQrzZxdyKbPngG5CaezgNqaz1DqRTKzJnWMguMTLPi8VuyOVADsjlS8Xitut5us7CQ2ffYy8Qk3kZAwBa+3mNa2p5kzfzwORzwORwo1h/9D0eaXgdOYPe8u6usaujVrBZuMOpuiBJ/X2Ds6KWlsx33lF2THJONruGWFcjONdIoKTWd00rjRRzguqR4RuUpErCJiEZGrAE9/KxSRVOB04A8ASqk2pVR9f8uLFqFMG2APaVa67ttXkZdfh8jTBAJ3kZr2J5KSt5OVc7jDnJKRlUZr65NUVdzFoYO3UFVxF62tT5LotGM4dZ2GMBawmH9PIzVtTIdpZsf2+7BYcomLX8/2rQ9TvOtlfL55R5m1QpmilPJSV/tbDh28hbK96/nH6zdSvPuBIbuHQtfNb4zZwTa9djBILLroEpa5XexyufArxS6Xi2VuF4suuiTWommiRDgDrG9gjOwfxzD1fGwe6y8TgcPAc2ZA3GfAd5VSnRSNiNwI3AiQm1cwgOr6RyjTxlOPrwxpVpo9dz53349pWvKQX5DFxZfd0zlxXtlBPO4AVtu5GBvZ1RIIvEF2dgYtLQGSnBkcKHuXQAAsFigYbxxvH1Fvv/x6Kg82IHItCQlTaG0tZu+epTS3bOskU7D5KzNrAXnjtlO880Hcrl1AATbbVVitUwgEKqg6tIKS3TuHXO4jPTuIPTpp3OgjHO+jvRiBa5Gs8zjgO0qpdSLyOHAP8KMu9T4LPAswc/Y8FcH6w6Y700Z+wdoePWa6uybYlBOf4MXl+hJKJZCRMQmrNY7mZj/wNrU1ZZTtLcViuRibbQyBwGHK9q4myXmgo6wmtxeRr+BwTAfA4ZiOz/sVmtybu8h5xPxVU72eg/v3kJh4Py3Nz2Cx3IDFYiMtYzzx8afT5JnE0t8/wKWXXxHJ5us3wcpAtTXptYMYo5PGjS7CyX00BvgWUBh8vlLq+n7WeQA4oJRaZ35+GUMpDAv6EzAWbMrx+xvJyjqRxsZq6uvXk5uXw5TpJ9HS8hb7SvcjsgmL5XQgC5EK4N/s2FbcsaDsdDpxNfrxtjVgs6cY6wPKj9PpPCoWoari58D32Vf6FnApAIpGbPYTUMqN21VCfPwY4hPmUVM9NBzKjpod+N16dqDRDCLh/I+tAj4E3gH8A63QDIDbLyLTlVI7gUXAtt6uGyr0ZwvLYFNOUtJYWlsPMCZ7Gs3N+5m/YA6uxl1k52Sz7mMHjrirCPhX4vfvQ6kERK4EVdcRsJaVnUZ8gpfDVcV4PM0kJSWQP86LMzmt0wK4210C8iR+/6PU1e4mPeMCCieN47N1OXi9O7DaZuLzNdHScpj6uveA5h49maJNLBPYaTSaI4SjFBKVUndHuN7vAMtNz6M9wHURLj+q9NX7JtiUM77wXLZveQGf91ISkxxBeZTO4R+vv0NrC8Qn3EdL8waUmkwgsA+rNaMjYM3vfxSr9Q2mTFuM0znTnKm8DHi7SRlxC2npKzj3glzq65JITsliyvQL2LL5GQKBb2Cx+Kg+/B6od5g17wHq6wpjsuHOUEhgp9FoDML5f3tdRC5QSr0ZqUqVUpuABZEqb6hz8WVn8ciDP2fblnS8bc0EAo34/f9NSspY0tKndMw0vr54IX985mFo/SF+fyuwF6WeZtKUIwFrlZXWbuMJnnp8ZacYh6SksYyfcDblB6q4+btXdJi8CicupKXlAKUl/43X6yYubh7TZ3ydiZMXmdIO3oY7OkWFRjP0COf/7rvA/4hIK+DFCGBTSqmUqEo2TOkux1BVVS3lB8BmOw8kD1iH4YDVmdvuvBt4hD8v+y5tbc1YrTOZPPWrzJprrPP3lAIiPt7fKcahtbWYos1PM+94P7Pnzue8L+9k6e/vpqa6kcysFO65/xuseXMjOTm/RizWjnLCiZQeaLrrdmUA6AR2Gs0QIxzvo1GVJnsghMox5HBYsdtvBCB/XDXl+xuxWH5KY2OA+rqkTiab2+68m9vuvDuorBmogL9jQfsLZ4zvNniura0euOCoGAcwUlG/9XoZk6c+wtz5xnrDW6+Hn+8o1D32ZztRPTvQaIY24Xgfnd7dcaXUB5EXZ3gT7GW0cf3nJCScjWIWtTW3kpY2g8bGajZvfJr4uG+RkjaTpqbPSE6ZT3cmm1AL2qFSaldVvkVmViYBtYXmZkVSUkKHV1OoFNV+/2/6nMSuv+muu0tRkdDo17ODGKMzoGq6Es7A7PtB7+OBEzECzoZmCOwgE2xKKdlVyjEzbwLolGMo4Ifami1YbcehAk0oNY/amgoys4Sa6vXsK32LulpDxwabYrpb0A4Onqutqe5IqW21PE3A7wTg2Nn5ZGRmdXg1hcrjFG6+o2D6k+461EJyJGYHulPrPzoDqqY7wjEfXRT8WUTGAb+KmkTDiK6mlH2l91K0aR1z5p9KUlICba1GjiGrbSx+398IBGxYrMZoHry0tm5j+5Z9wKWkZ1xwlCmpO4I9mfaVHsRun4qigvTMqfh9fwMuZe+e/djttR2j/tWvhA64G4gnVdeyuhLtBHa6UxsYwRlQASMDKrDitVW6/UYx/RmkHQBmRFqQoUJfFlFXv7K2Iw22x7Meq+U4fN732b1TmDLtRIo2vQN8hCPui9gSM/G47ychwYfP90uSk2/D7dpASuo3ASicNI7klCx6M8UEB8+53W7stkP4fMuYMWsxAPtKX6eudi1p6ad3GvX3Z4e23urvqayeUlR8XtnIytf+1bGp/RUX9W1T+w1btrDytX/xzw/WscDmp3FK/JG0zvStUxvNMw2dAVXTHeGsKfwGI+cRGAn05gEboilUrOjrIurObcWU7VO0tWXh98djtY4jEPBwuOonuF1O7HY/ilaaGj/C7sjgmFlfZ9LkJR2prxvq15OYdB2Fk8aRkZkF9G6KCV5rsMg6xDKrI6U2gMORQlq6jx8+dFe314RrJuqubdqVZUKiH7//N1RWBo4qq7fZweeVtfzsqX+R4ryK3JxJ1Nbv4WdPLeeemwlLMWzYsqXjemtgIT5Vx0PbX+G+GbUsyMzoU6c22mcaOgOqpjvCmSmsD3rvA1YopT6Okjwxpa+LqNXVh3G7fTjs+dhsdnxeN62tk4mPz2DecQ92pLieNfcYSnYXsa1oLaUlXpzOMaRntDF9xvG4XW3sKz3Iti0lJCUlMCa7mQkTDVNMqC00E+KFNKqYPdbDzop9NDf5O3kotY/aB+o62k5XZdleT9dNdMJJYLfy96+T4ryK1OSpAObfq1j52vKwlMLK1/7VcX2i04W0xZNqu5xlZb9jQWZGnzq10W4+WXTRJSx76gkWY8wQStxulrldnLvkmliLpokhvabOVkr9CViBsbi8GfhPtIWKFeUHqoz9hoPoaeTe3OTGImuAUiCAz7cVeAd/wE7ZvnfMmIGzqThUj9WSh8VyNq3NLwEfgvIwaUoaO7b9Bo9bER9/Mh63Yse23zBzTn5HR1xfdyV2+31s+uwcNn2WQnPTJez9bAH7P/NxZ0Ym9xXW01R6ByW7r+m021nw9Tk5vzbTZBjuqX2lt13o2tNb52an9Jreuqy8hmTnpE7lJzsnUVZeE5YswdfnTyhkt7cNxRhKPf4+p3WuKi9nstPZ6dhoMp/MnzWbc2++jRVpadxeWcmKtDTOHSWzJE1owjEfLQT+BOzFCFwbJyLXjESX1J4WUbsbdTvinFgsx9PauhKftwKFwuE4C4vVhcdTQULCFMBCZUUjmZmnkpL2BZqb3+OkU/8PV+Mu1q65m+kzllB9+A08nj+Q5BzLhIlfZevnRWz9vLxb99bSnY8zK34m+z1p3PDv7ZyXl8x3C5PZNCmVky87i9WvrOWpx1dScfAgmVlLup31AH2aQfTkcdTXBHbj8zOprd/TMVMAcLn3MD4/M6zfKPj6rIwMOHYWW3d/iEir0an1Ia2zNp/oDKiaownHfPRL4FwzeR0iMg1j5nB8NAWLBaEWUUMFjBWMc1Ja8jEpqTdht0/hcOVreL3vkZ5+CvEJe2hsfJVmz2ZaW3bQ0PApcfHHkpIytsMNdf++QzQ3/RMjz6Axaauvhzdf/QdtbT4cjuNIcFbSUN9EXLyFJGcuTc37KPZasFtuIqCqqWl1s3zPy7hrPmFDWWaHjDu2/RVX4+ckJo3vWG9wOiezc1sxT5b5+xR81p2yrKrdRVJmGrnZKR1BaB+++CbvvrWWqspKciZO6nbR9oqLTuVnTy0HriLZOQmXew+N7uXcsuTUsH6jrtfb7TXkjd/IPTff26fFatDmE83odjQIRTg7r9nbFQKAUmoXxhZhIw5jQfbozeeDR+3B5pPUtNyOndaam24lyfkqdvs2xk2YSGpaAQ11L9HmPYu4uF/R1noejfUvY7P72b7lDZo8X8buuI/DhxdSU52EyLXU1lxI8fa3iPeNIccxndbmZuqqC7BY5uPzjqeu5l/4VQPCFQhjSbHbSHWMQ+QSSiodnWRMTZ2OyFco27um4/7c7hLcbnePpqDu6LoLXWX1dlyuF7jq2gs71g4+fP1N1jy/nG+0efnNuPFcWV/PmqeeYOOWok5lHTdrFvfcfCoZacs5VPk9MtKWc8/N4XsfDfT6YLT5ZHTT7mhwZX09v87JCfnMjjbCWmgWkd8Dy8zPV9F58XlE0VvAWDvtwV9333910E5r45g552S2fv4PPnjv3yQ5F9LW9hfa2mpRKpmEhNM5VP53EhPvN8tIpr5+Ej6fjcOVP0QF5iJMJNHWhMKHVT4BZuDzOrDaDqDUP4EWXIEMElQJx6baaWhro0Jl4Q04Oq2HTJiYx9bPy2hoKEEF/BzYv4p9e/+Ex11LIPAXJkw8r9MMIhyPpxdffIE9+6uYMjWf+xZfwGlTC3BvWEeDsrH2w3UsycwKa9H2uFmzOjrxdvfS/3v6tU7uqT2N4IKvD6Y/oz5tPhm9jHZHg1CEoxRuBm4Fbjc/fwg8GTWJhiB9CdiaPHU6l15+Bdsvv54D+1wI12CzjQdKaWt7EZ+vjKwxAQonjWPzhnpEWoAvoNQbBNSXibOsxuOrI86aQWbcBbh8f6HNt5vsMVNRgeOoqfmYhLRNJNmz2eX3EQiU4WEDPr+X/3zyJlOnn0JGZhYZmVlMnFxC9eFadu++kqqKOAon/g9VlQ6aPBa2b/kbM2YZW3X2lu9of10TjtwpXHvXlJAJ7Prj8x7sXhrsnvqVc4spW/P3PrmKjnb30mgyUk0sQzlOI5ZtHo73USvwBPAgcD/wW/PYqKGr+aR9D4SZc/JDevjUVjfidp8PHI/NlofVciJKXUZCvJ2p05PIyMzC5w1gtUwiPr6RJGchDvtUlPoqPuXHaU/DZknFabue1OTrOeW0h5g9bwFfXLiAKdOKmDonl3HHJtPYuhW/fxHHzHyA5qYP+Xzjv6iprsTVuAur9Q0efOROZhw7k7nzf0TB+JMonDTOvKtL2Vf6Vse9XHxZ91lLgheSF01LZYa/Cv+GdTTs3odkje/wLMrOz6fE7e50bW+LtsHupRaxkpo8lRTnVTz3wpsdI7iOoDRnMu++tipkWcGjvnCv0fTOSDax9OeZHQxi3ebheB9dCDwNlGB4H00UkW8rpf4ebeGGCt2lnr76hgs7rTWA4eGzvyyZW677Hxobm0G9BEoRF78IRRMiDqw2f0cSOqtNaGndTSCwivTUr2KzQX1tI3ZfgMzmPLb5n6bV8kWOnXUKB8rWsbf0CbLHtpKdDX7/o+zYvpeExFs7ZgepqRkU73qJHVuf5vSzTuoIKgs2f2VkZnHsbNi7Zz91tWvx+/eTkGjnqcdXkl+wtsMTqa8pKvqzaFtWXkNuztHuqVU1TexNaeOhT0o40AIF8fDVsTbWNW7mByFGTpEY9Y3UEfFAePe1Vczz+Xm5eDcVHg9jk5KYlzWGd0eAiWWoOhrE2qwVrvfRmUqpYgARmQy8AYwapRAq9bTLvYOpU4/Y8feUvEDxrs+Bn2CRZgLKgdf7JF7vI9jtx5CYNIeExIyOJHRxceuAQlpb5lJbXYndfhinYwe2wCHK5EUyHM20OT6jsXEVrsYAEwqvoWDcJR1eUTk5HiZPvaBjPwRXQwOe+hY8TYfZ9p9/M3NOHrPnzj/K/JWRmYXdXovfP5XmpuyjPJEu+2YL46bP6jEIrSvzZ82Gm29jRVCnGuwe2l2HG8o91Wb3cd+WeCbabmSWPZdDLeX8dNdfmJJWya9DmIYG6l46msxPfVF+W7dvR1VWcrXdxpSEBIpbW1m6dy/bmpsHWerI09szGw3CaftYm7XCUQqudoVgsgdwRUmeIUmoSOfKirs77UdQsvNvWOSHOOKm0Na2kdYWC/B9RN7EaruYJs8jTJ3u7FjMnjlnJY/+9H2SU04mPmEe1ZXvY1FreXD+GL4xeQoAuxsb+NbueibN/3WP9e8teZcdm1eRJF9hQsL5ZLtq+cNP/wyEdrVNSLR3ui9L3CRUyxX8Y9Vyfvrz4/ucwC7Uom2oDnf+uefz6pqj3VMDykcTl+KQXEQseANjaJFLaWz7Q8gcRwMd9cV6dDZY9FX5eZs8fEVgusMBGH+/4vOxuckz2KJHhcF0NAi37WMdPxNSKYjIV82360XkTeAljBxI/wV8OgiyxYTugtRCBW85nU4OlR/ZZtPlPojNWkBychL19YlYLDkEAk4Mj143cBalJcu585Z7AS9bPy/D4RiD1fYcLS0BrPiYk3Yym+rW8g2znknOZGqq9zNt/tH1JzrtHaaoPTsNhWCzWJiaEk9W/CS83q/w7P8+zunzxpGXmEC9/9dUVioS4oW8xCo+/k856RnVjJ2UgcOZhi+gGJs9A1d1Pcda3Xz4uhF3cLiufkDmlJAd7rYtfOXcWTz3wh1U1TSRnZnIdUsu4JFt8RyTeQzFnmaafV4alGJB3DgqfUce164jp4GO+mI9OhsswlV+7SPa2soqqnxeSgOKwqQkGn1e/ErhTHKGqGFwGU4mv3DbPtZmrZ5mCsEpsyuBM8z3hzH2VRhxdM3xs3fPKu669Rf4vB72ld7LlGlG4rmizU+xZ/er+P21gBelvIiMQak2vIF/Ul8/jZbmWsQyGQvFQCIiftLSL8Hv/5R/f+SgpWUDAb9gs1mJT/Cw4KR7OLzXT3qLh30eI7ZgQ3U1T++qwdXq6+RZ1J5Qz+d1kz2nDr//UTxNn1HgOJd82pA6RY0ISW0J+Anwq5yx7HG7eNFVypgvX8Lh11fxjeRknBmJlLr2UL7ZQ8oxsynIy8bfuI3xyfaOuIMlmVkDNqeE6nC3bt+O/8B+npucwuS5ecbDv+bvJCVZafNVMT9rIgAbq6vxtJVREPTUdTdyCh71tXcWK55+MqzOItajs8EiHOUXPKK1jRlDWkM9pW4XJX4f6Wlp+PLzmTKhcJAlP5rhZvILd+ARC7NWMCGVglLqukGRYAgRbCYyoo6LgIdwxLlpbqqhaNNLJCa9zP59pYj8P5SaAWwBnsViOY9AoASlltPc9H0s1rEE/KXAi6SkHk9y8vEoKqiuOkhbWwpG8tnxBALleNz/R9GmnzN3/m/ZtvF15iQG+PTwYR7YDDVcy7SZeewrfYfPNyrGT4ynrPQd4DRmz7sLq7UBt2sZ2RlWxtTvodA+DrvFiqe5ib3+g6Ql+rFaLExNSeUbwA9//zQPT53G1JRULi1o4ZfbV5FvvZSaij1Ykqupr/odN109m7Uffhx23EFvhOpwvU0eFufmHlXHzlwvRbv/DHyd3PhxOGw1bGn6Kw9OjsOvVK8jp/50FrEenQ0W4Si/4BHteRMm8MZ2D5fa7QQSE4mfMpWX3S7ODTO/VDQZbia/vgw8Yhk/06tLqohME5F3RWSL+XmOiPy/gVYsIlYR2Sgirw+0rEgRnBCvbO8aAoELaWqyUnPYTVz8WKzWBRwo+wcW660kJJwGBIAzgXvx+/+GzeYELkGp5/H7foRSdyMylsaGsdTVvkeT57d4vS4slruxWE/BYrEgMgeRH1BfV4bNXotKX8Mmpbjso/1sb0kndeI4Jk4+l9nzvkZC4kfs2PoACYlfZM78U8nMyqGtrZGyfS1U1bkp8q5mY1MpFR4Xe30H2KGWc6BZMf2vmzjtzc2sqzqMp/ow2Y4E6pu9nJqfy8MLFJMT/0h15R2kWJ7i+1fP5oQvnhnRZHGLLrqEZW4Xu1wu/Ep1JK5zJjm7rSMrLo477ziTw87neafhXtoyXmTJdfPZO2tWWJHH7762ii/7/bQU72b9xx/TUrybL/v9Pbqn9je6eeOWIn7x05/wg9tu5hc//cmQd9UM9VsEJxEM/u0XZGZy4YxjeT0xif+uqx9SUd/DLaFhOG0/FAhnofl3GFtyPgOglPpcRF4EfjLAur8LbAdSBljOwwoeNgAAIABJREFUgAheQ6g4eBCfdz0F40+ivm4nTZ7zERlPXPw4vG1FuF3/RikrVvkYRRqQDcQBxwEu/P4DwC3AGOKkglb1HoFAOcgmfL6xeH01BAJtwHSsNGO12rHb7bS2TiUQ8OH3P0piUja5Ux/AvaEauy2HA/uXkZyaSmbWAjIy5rP27XM58WTD46imej2fb/wb3pb/IuA9CRsfUcIKqnBjxYWHmdjUHeQ4jqPRu437N/+KMSk2ShpdzBs7hmS74pRUJykFjaTPmMD3H7qzo10iaU4JNR1+97VVIeu48pJLufKSS/v1mxbv2IGt4hBTHA5SEhJobGtjR2kpxc0tvcrZl85uuJkvoGfTRLvJraSkmDf37eXUqdPIzMhgQWYmKQ4HvrQ0/vveAY8HI8ZgmfwitW4Ra7NQuISjFBKVUv8RkeBjvoFUKiIFwIXAw8CdvZweNbquIXjbVrFj228A8HpdKKUQAYdjJ27XWgKBJcAZ+P3jUK2rgJMxFMMWIJlAIAnYiJBBQDZhVT9CkQiyAaUyEUlC5LsotRm/fyI2uxMRPxbrBsbmZZKZlYvVeqWRj8jZQltrLjb7Esr2ruyIPM7MSunwONq9YzU+94UkW8aQainjOBawk3HMiP8DO9rS8fq+RxMzsIgNm8zEx3dosP2ENbSQ11BJglLscDezzNPMeTfc0qltIm1OCdXhRsNk4/a4sYqQajdSdKXa7Vi9Xtwedy9X9o3hZr5op7vfIljBnX/MMbxcVIT6fDMnz55DvcMxJE1pg2Hyi7TiHw5pVcJRCtVmbIICEJHLgUMDrPdXwA+A5FAniMiNwI0AuXkFA6yue7q6mo6bYDhcVR/+GYHAQSyW5SQn30Bzy7soLkUsEB9vobW1kUDgAiyW1wgEbMAziJyNUg3AbxFuxRuoQMhF2EUgECA1bSLNzTW0tdUD/wvcTmvLFHzWYhxxT3HjbZez5s2NHV5OEybmsa1oNzbbJDzuQx2Rx1ffcCFvvW54HNXXlJIi2cA+xlm9pASEmSqbza1Qp+wkyjG4LRYO+XxYrDaSs0/D22Lh4u/cxAt/fJ7KisPkTJzEed2MfAZjVBOtOuyJSbzqcpPc1sYUu51ir5dXlXG8KwMZBUbaYymWnjSdFVwyltlzeKm4mKd3bOek088YkiPawXhGh6viHwjhKIVbgWeBY0SkHGNHmav6W6GIfBmoUkp9Zu7V0C1KqWfNepk5e54Kdd5A6M7VtGDcJdgd73L8ifPY8nkhB/b9HrdrPVbrWThTssnIKMThCFC8+wA+7+vEJ7xBXJwXkcM01NWiuAzFO8AuFG+hyAHs1NXuwmbxkOCYTZu/HqV+SSDgIj4hmZyxDiZPnU5+QTl796yi+vAOPJ4K/H4f9fUVKHWIbUW3MzUvwI43N3a4l6rAduJsH3BsWiE2VwIBrxevdz8q0Eyq1YaLPTji/j975x0eR3X27ftM2SqtyqrYlqxiSbbcCw4QWgrgkGAw7aWFkrwkJCSENBxC+suX0CGNEggkoQQMoRiHanoJLTY2ttwlq1iyrLJq23dn5nx/rCRkWZILtuUy93VxSVrtzJxZ4fOcec7z/H6zyR4zBociCQc/JDNNp8KbwdXX3bTTz2ekvoO9NXnti2tMnTyZ6W43i9rb+rtwZxQUIkpKdrjGp1kF7o30Rd99rlm3DmdrC98rKeGIgsIRx7IvgsfgADfX72d2djZXtbQcUCmjwezrlffhUqo8kF3RPtospTwJyAUqSZWmHvcprnkscLoQog5YBHxRCPHwyIfsGwoK8wiFarZ7rU8cbuqMApq2vIzbcwnetC+hqjqRUCNZ2V6mzpjFF0+eyLlfPY0Pql7nreXv8OayZ8jIcgGlKPwIOAtYDHiAIzGMdGKJv+Mw68nTf41X/xPjCm7mlPnPUj7xRpY8+RpTZxSwYd1ThEOnIvgV3d0nY5ljKZmwEFf0fCI1aVyi6fxYVZgcqeW4Wdnku/+NrrThTPOyzdpGi/IcR+arnDnBRdD6E5qrBo0k4eCHRMM38a2zP7PTJrSR2B+6LJ/2GieetoCVmso55RX8/tjjOKe8gpWausOG3qfVS/q0G4cD73NmKMTVgLOujq7OzmHHsq8+/wNVB2i0ORw/l5Ga13yknhIKgGeAV3p//jGwCvjnnlxQSnktcG3vNT4PXC2lvGhPzvVpOf3sL3LTdXextupEEgk/DkeArOxXueQbZ7PkydeYNPks2tueI5FcjWluwOW6jI5AD/6cjdt5IfehKAqa9g6mmQ5yJXAKKT+iblQxDkWeTsi8G58jD5esw9EbkwdKVxeOv5St9a0EQw0IZSbe9Am0Nf2NEzLOAs7nX/X3cfPcUi4EbjctMgqaMTvvImAJPJkxEolGmvxF+OfM5uw5cV54/gfU1ybRtCTpriS3/q2eB595l69f/JU92sjd24/TQ616h7rG/O4ubvjNLxk3Ziymy4UOWLHYkCvl2dOms2HeKVzz0AP0BDrw+bOZP0RaYU9XgQPHrLg93G6aqC0tu52+GHifLZEws91uQoZBdX0d/uzsIceyr9IZh0tJ7u5yOH4uI6WPHgI6gfeAbwI/JyWId6aUcuV+GNv+QYaBtxFEAXfvz6nUUuH4Bf37DH1uaSkRuSm4PckdROQcjgyczklEo/cgZUvqfMwHJuNWBZY0iJp/RhVvcER2CRvN1H5939PJhrWbkK1Hc0KGi/8mTSzpoaPHQdJqocqMMNZS2Gq2sI5WckomoCWTzP/173jvyceQtbVEHVkkmYkqBXUtBtu2xDh+1m20BrayasOLmJHj+Iw/n2Soldt/n5LAqCwr+2SCc7lIAuowky2kJtIuXeP6FZ8IpJ1UVLRHj9PDpW9qQ2HKysv73xfo6ECrrSXbtLh0/HieWLmC4yC1CTpEmmVF1Wrql77ITWXllM3s/Ye89EVWlJV/ajmBHcbcN0l8yqa+MV4v1fE4E3UH0XBk2LHsSSDblXTT3sjPH0zdxbvKwVIxtDcZKShMkFJOB+g12WkGiqSUI9f17QZSyjeAN/bW+XaXJU++xtiChUyc/IlPQrBnI0uefHQHETl/zlwcDh+muYVoJGtIO8usbNjWvB5FuR3YAmSTesiysOTxGNZ75LskpekvELcW4PSWDthAPpkbP3yPMaKdDMcENCVCazSGQhCFCQSjE+iR/+FYt0FF3OK1lR8hjjiSaTPnkFFUyccrV/DsQ+8yNusiMp0FvPHWc4Rj75GXZ7Kp4XWytP/FIcZSE17PMTmlwHnc9dd7mF/s4qK0dDJ1nfdXruAd4Jzp08kcJqdtulw8sXIF33a5+gXS/rJ6Neas4T2eh2O4Ve81LS3UhEL9rzfV16EKQVmGj1caGvi2y8UYIahuqGfG7Dk7rJT3pZzA3lypDwxK84qKeWjdWs4wDBweT38qavBYdjeQ7c6+yafJzx+M5bm7ysFQMbQ3GSkoJPu+kVKaQojGvRkQDgSG2mhOxLt46/X3yM/PpaHhcnStHCHc6A43WVmd5I1xYZqnUr0xRji8DFWpxTC6ufq7NxDs6UCIc5Gyh9R2TSup7Ze3SFiZwN1M8BpcWNrNH+vuRPdNIDOrol/iemxaFKPnGboT5yAtF5b8EEW8j5BfICq24RD/QZJkcyjCC93dvP/Ki3z3pGOZfsFltNW3oyW/RN3KZiLhagLBNLzp/0tNw1NEYwGy9VKEUAkZFgBjXeNZ1tzDRVNzmZiezqoVH3GS2800KVnU0MDPhphsIeXDehwwRggUBGOE4Djg5T34/Idb9eoeLw+Hgv2T9YbuHtaoKvOLinlw3VrK3W4URP+KevBKeV/ICfStgt9auhQtK4svFRcz1+8f9ty7wsCgNDs7m4biEn5VX4cv3Ud5ZuYO/QN9T3O3tLawkHG7FMj2V/XM4Vilc6gyUlCYKYTo6f1eAO7enwUgpZSj2nS2Nxj8NBBoX8bqj5/A7fkuWdn51G1eRHd4Lh7PGHRHGMSr1Nduoqdbx+EoR4iNtLVVYRoXIJR2TPNOkEWo6hhMs5SUCN5G4Dk010ocZjlrutfxSHsb//ODb7PgnAu3G8+kKdOY5arm3bb7iJjd5LsrSJpJOpIPkeXKxSEn82HiA+6PbeVcRUPTVC6LRbj9LzezMm0y3vBnmKi78HnTeDMSYltngq6edzFNg63m9fi0k8lwpGr3m2NbcKrJ/o7QaDiCz+0mDcm2cCqFNtRkZ8ViHD19BtUN9UTDEdxeD0eXV/BSbOT1wlCpheFWvVMnT+bE0xb0T9ZbMzO42J/DXL+fpb1pljFC4PZ6+o8ZuFLe2Wp6d9McA1fBp2dlokTCLF63FiZPYa7fv/ea+kpKWPi97+8gTDc4XXUXYpf3MfZX9czhWKVzqDKS9pG6PwcyGgyWlK7e+DhwHBWTPkt97b2k+a4knTE4nNXMnvtFgj2TeP8/l+F2q+iODDq2PYdpXAyMQREfYSGQuDBNR+8VMhHCh6p6KMv4Pvm0UOSt4lsVfh559hmqKiqZNnNO/3g+e/Z5LP6/n2MmVFyqwJIabu+xFHqm4Ol6gdrwcySkg7cZTxFNFDhcVBoJLuvu5sKOjRjqJkJZk8kQggIXbIlUIWUZWRnXEuh4l5i5hByjhDVNaWy2nibL3cnzH37AMRUTcXs91AVDfBwJs8myuH7FR1Tm5JLXW8Y5sNv1A03r73YF2BgMkpeXN+xku6JqNQ/cfANZnV0oyQTh+noeWLeWo8/+H2558l9kre0imkzg1h10ZmVy6aBVb35JKU/U1rFkq0F9xMficIDznT1cNGfOkGmWkdJCu5rmGHgvW7c19+tABUpKaVy7hjOAZ+vr8X3Kxq6dpSaGWoF/B3h0F7uL91fX7+EiKHg4sNOS1EOZlCH9yWRmPUpLy1UYySqmzzqKbH8O4fA2dL0cTfcRDqcMRdLSyrAskPJpEokNxGONwDiEqE8Z3QgT+BeptFE+sA0pH8EyE+TTAjzDhaW+lDhdejrvPfnYduORCDaII1jFd5HuX9ApT6bD3EQo+TRrQjUk5C/I4DG6uY0brOMIR2JEAgGOsyzG0UahtZjqwFoae7pojDWTpr2DU50I1ODWdVzKTGqsRXQaf2C6o4oflZfxdjTKf1Z9jKHrvNXVyevJJL/1ZXBqOMxTG9ZTOGXqdmWQ11VW9h/TGgj0T8qFU6YOWyr58D/+RtbWrXxbWtzpdvNtaZG1dSsvPPtvvAiOR3KBhOOReBFsqKne7lzzgiE+aC3gv4lv0KH9kmDGz7hbO5IrAx1DavGMpGO0K2Wog8s+L+zqZlVdHcsCAfzZ2RROmYrl8fBaZ+c+1wL6tPo++0tv52DR9bHZObvSvHZI02d4A/C7X91GV2c3kI/XO4ZQzxrCkQwsq4f33nkfI9mIaZrE48sJhT7ENAMIcTWqegZO53iMpIZkBqn2iztISWBMw5JvkB65mYsm5TAnJ1VVMyEtnUBjA3fcfiNPPvwqobCBUCxycn9CprsI6Qzh9KQTTjpoa/0dmrgFlXJ6pIXGFFxcyXNGLb9VG6kSgjFScq2+mvuMJt7rciEdHmZm/C+dvhkAlHsySdey2dCi8kS6RW0YHli/ns9mZfGqYbC8oYEjsrL4rKZhmBa618tVJSW8sHYNjWvX7LTbdSTbxro1VdzpcvUbtRRLyXmxGN/8aDmXjhu33VPHpGCQax56gJvKyvtXnW+1w3TvhbR4JzBzdurJqjs4m+zMf3L1tZfv0t95Q031Lu8JDF6dT8rwMTEc5rmGeub6/fizswnoOidkZvaX0O6qRPfu8mlX4PureuZwrNI5VDnsg8JABvYtdHXmEA79AVU7CZ9vNm2tq0G+zphxZ7Jtax2Ik3A4JpBIvEwy+UckfiyrldSW6y9R1WlY5hrgHhQSLPQYsCVMR2Ym2f4cNoeC1LS1s+Lud9CYgypM4okNNDW14013cUxuHj3JBOutdFqlSppSTpaighR0myYRKqjDyY9Mkw3AGT4fvmSSOxxdXGGaqP7prIw6KS8uoXrtmpQwXLIRvxVB6elhlqLwgITTFYWJwEeqyu3HHY86QOPKlJIHeifLnXW7/unmG3ewbbxj00Y+qN6EDIV4SNc5w+djuhCEOzuZICUu4PNSsnntGpgytb82vyfQQdnMT1bHdWGTsa7x1IU/cftKT5tAQ1NgyJQVsF2K6Jm6Op566UWuqqwccU9g4Gby6VmZBEpK8WdnU1BcQv2aKmq6e7aT7i4+5th9XnEzVCrsruZmYqbJT668YpcC0f6qnjncqnQOVeygMJjevgXTaEXXDYT4K+GQiUP/DB7vBXR3Pk1W1pWEIxnE408hRD3wUywzBtQDy1GUu7AsgSryEXImTu0NFtPDGQi2bq4moOs8EgxSVRMEeSSKegUKZSR4HsEb1AbjHOHPxa1oFGgBqkmgsxEHU4lIMAEvm3ARYybQAUzUdYq8abzc1UmNqjJ9UhbprcvQ9bm4PC6aInVYLOYragdCChqEYKyu9ovFoSjblYHC9ivSna1WB9s2ZluS40MhQg4HX/fnYLa38VRHB92KwmzgHSnJdLloAyp0vb9hqyYUwufP3m4sJV6VuvAW3N4J/dcLhjbjdokhJ+VWt4cfDVjpr29vY6HLhWhro2CYPYGBE7yWlYUSCdM4IFjVlJbS0d7OVQM2d/dHxc3gFbjichFDslBVD7nST5sDg8N6T2EwS558DY/3GzgcF2GaF+B0/RRfxvUoqoPc/Jvxpn2eWKwNRYuiKA9gmU/hdJ2M212Kpgl0RwHwJYRwo4pbcCgLEGIF4z1gZeWwIODlpBqFeW+u45VOjUjcgcSBlEGEUNE5EjiGJMt4vbmT99s3Ekz+ixytgZj1Z3rMNQStBE7WYXIHZ9HIVCH4DrAoEOC+QDt/SCYZU1DIOL+fS88uIzvznzjT76SFO/hacScTnQ7eNC3uNgzmeb1sSCR4WoIvN2/YnPCu5IvTvGmYUtKdTCIl1PR0M1cIVKeTiZWVONPTmacoPBKP87ai8IbHywWTKnnIMGiWklAo3H/e+Rdfut31TsiBjbHHyMi1sKRJd3ATPaF/kkXLkPsDdWuqtsvDbwuHmeVyEw1Hht0TGJgi+1JxMYt7j22oq2VjMMizqsqPf/P/uPmOu7n62l8we9r0/abnP3vadK6+9hfcfMfd5Pj9LBw7bo+lOWxsdob9pDCADWurad6aKjd1uooxkoJgogYpE4RCTxMNf0zS2EDLtvsRXI5l1ZFMHIeqbiYnLwOXy0Mg4CHU8xckX0cVPiZmFJCmO7hzvZuEnIOkhWBYZ13VPCAbU04kbv4rNQAxHin9QCOqughpCTo6m0lXBKd6/8tr8atpN1TyiPMttYnPOl34kkkCyST/BToMg8vdbk4uLEx1+i59gQuuuJLZ117enxr5oKGOHEDG4/yho4Ncl4uZhYWIadO3KwPdISe8k3xxeWUlhttFdVsb0XCEoGXhychkfHpaar9g5iwa6mpZ3dRERlY2F5SXM9fvZ1lGBvdUV1OlCI4aWJtfVj6gVLOYH31lGivWPktD0wMUFfj5zsXHsOgvK4Ysg1RguyeNMV4vK8Nh3N6USurAPYG+9Nejf7mr/1xz/X6YPIVn6+t5rbOTEzIzKT7m2P69g77O74aaaq6tr+fc3nsBdkhF7e3uXrv002ZfYweFAYRCodSK3ZFBenqMzo5wr6eCpLvzcRBXoGmtJBNfQJKJohQgZRuGkUsy0cCkyePo7nqXsQWzqZj4Y7asfAEnS/i4o424nI/ClcATSE4F4kATEg8W55KwnsSS8wDI1Cfx+azz2dbWTr0yh3EZD+ORLdzo6uKVaJSvmSaaopMJFDscBFUVr2lyY17esJ2+ff8tmjKV139/GwvT05nlcrMyFuWWpia+cO75I+aEd5YvPvG0BTx79x1cVF7B1LQ0nv/wA/4ejXJuUTHwyUR8xIQyXNEIPocDU0p8DgeuovH8bIgKosHXu2BQIctrw2zClk2dtl3zW2VOLrcEAlxVUjKsnefgDd2BxjInnragP7U0sPP7h0XFRBvqeXz1Kqzp08l0OPf5XoNd+mmzr7GDwgA8aTrBnqdJJNJxOMtxOpcSDP4d0+zC7S7Bk7aZzkAnQpmFIlqxZC6K+CuqeinBYARd72Bc4Wvk5ceIxa4jq6ydtq0Bgu2ZgAtBGIsWBHOQ9ACNpGnVhI3xGLIKQR4e11pyfJ/htUAH2bpOZfpkEtLLuZNTFT/vJ5PUJBJcIiX5ySQfKQqPCEGG2025ro/Y6QvQuHYNZ02q5Ln2Nu7vrRI6q7iE1WvXwIIzh/1sRupB6Hs9kEhy5Zo1mOEwqtdLTkZm/+TfNxGfd8WVACM6f+3q6nq4foQdrlFSwhe+ciovrF3DA8M86YzU2zBw72C7zu/uLr41YyZi00Z+tX49nx1QibWv9hoOR4E2m/2LHRQGMHnKVNyu6bS3LaKzczXxhANfxjVEoxGy/RUYxsPouopltWGYGtIqQioSw7gNmayiZtN4LvnGqZxxzvms/ngFN1/3JIo4h5Q4RAUmi4AIkrVAKaDx2RyVTcF3aTOqGT8uiEufj4xNoqdrGw5dJ2w2U5Gu9lf8XLBpE4GtW7m3I0BSSjTTRDgcTE9P58NwBBEJE7QsVq34iFhuLnnFJQA7VNZ8rreyBlJVRq/uRFBtqJXvhnmnUL/0xf4V9BtVq3kmFsP0eCgSgmokfzZNrCE6b0dy/trV1fXOyiB3OG6EoDfSuQamlgZ3fvuzs/nKkUfx4oBKrIHv72NvpXjs0k+bfY0dFAaQ6nB+mfKJF7FpgySizwfA7XEgRMoa07Suwkj+iZSC+JFYlg94E1/GqZRVfJWXnn2YsooV/OOex4g0zmWqO49aAph4SXkT3Qs8DHwRhQQ+VzsFxhKOr8ziiCPn8If7n2eiK51ih4dQooHa+DOcV5iSoaoJhdjS3Mycnm5+4HJRAayJxbgzkeCj7m7aEgmuAI7357A+HOZPgQBf+PKp2024Q1bW7CT9MKxw3YB+gvffe4+yWIxrhOA5w+A8afGX7m56gNvuuHunn/2erq73ZhnkcOcamLJxez30JBJsk5IxvXsUuyuzsa/GaWOzNzjsg8Lqj1ew5MnXaGps7TXXsXhv6VU01QdwOGbiTksnGYVodDlpaZN7S0/PJWUnEUCIMqQ8G8P4T6+G0kUsefJRNq5aR7GYz/ruOEIIkFuAQlLdztOB6/BqXahJNyU5US489zu88c67/HZShLfa/86mZIwss5Ofu4LUdmWyMZjDw6EgVrCH76sq07TUn26m28134nEuDoeZm5vLUt3B46axXVpoh8qadWs5g1RlTUDXd5p+GG5zc2A/QbCzg88IAYbB+kSCCPA/Lhe/XFO1S3+HA9nacmDKZkxRMa+sXpVSk62o2G2Zjb095kNRrtpmdDmsg8Lqj1dw1+9f7pfBrq9dRvPa3/GryZKnIy7ea+sk3jOFNH8RlrWRQOBpLCuAqo5F0+eQTHShqMegawpG8lngE8McK9nDhmAAt3Y8DqUeaZZi8i6wBU24SfechkddRMnsQk4670JmT5vO44/9iwWFBZw1PtVAFujooKGulvs7OzF6K3NevuxrjFM+qSRWVZUShwM1FuOq40/Yofns1UHNZ0NV1uws/TDcyndgP4FlWchkknYhmKwolJkmq7u7SQw4ZiRGy9pyV9ihV2DWbJLAA7EYeXn5O3x+e5Li2ZP02aEsV20zehzWQWHJk6+Rln5Rv0pqqBXKXefxbtsDCAw08S4uMZGeDoukVYOmbsCkAk3zowgLl9sJxJFWIy5XbuocvYY5zTUOIj1vYcmJIFUETThZRpa6gGPzZ7A6UkvFtAoW/r/r+sczeGIcqnQyPTePl1tbOEUIMhSFbsviZcNAdbt3uflsYGXNroiqDbfynX/xpTy89EUuAqSiskkmeAE4Q3fgEYKElDhc7l36W3zaDdSBE+S/QiHmA9TV0eXxMjE7+1Nv9O5uymZ3378n6TNbrtpmX3BYB4XBfgqxcAiHyOelrUFihoVKOj3JXxMyOtG0WWRknE8iWUM89giWdSEKSQzzRZCvUDltwXaGORvWbiLWWYkZvwfLakYjiZdzQBlDR6yBWPQxtq7dwE+v/mG/25nicvHT2s2UJ5LDqoae983LufvG63kuHk9NukLQoOt86cKLeHht1bCT6qeZcEda+fb1E7yiKmQ5HJwHFCBZDryQlo6vdzN7IMOmPAZcw3S50N0eHv3LXby6C2mRPbG2PJDYk/SZ3bNgsy84rIPCYD8FS9X4sLUBnz4ZRYvQGosj+Q2q8g90x62Ewhvw50yjYlIpmzY8TDTyLmnpgkQkyJoVH5Lm1Tj7ohPJLJqEKyMNoaeTqZyKGY4wwaqllleIJzfQ2h3i/6XF8EsIDXA7IxTk962tjHW7OUooBJC8ithuzJPKyikuKOSEbc34EwkCDgdvjRnLvM99Hj73+T1uPhuKXbVx7Httel0d69vbuK631PWInFwcg6S3q9evp6dlG5cWl7CgcMfUTl9p6u6mRfbE2vJAYk/SZ3bPgs2+4LAOCoP9FMLJVmLybaZ6j6Yu9DZSnoAlJiCUPJDNwFigkdKyE8nJHU9nRwst64OUZ13JWHcxzdF6Xnj4MbSMxyh0d6I6nmec81xWJzVk3E2laKFQbGahw4E3ZrFG1TgpO6vf7Ww+8FOvl+e8Xo7pVQOdFAzuYDW5sKyMibNm9d/HUb3v6ZNfGIrdTWfs7sTc3+BVXrH908hpC7Y7VzjYgwIsrq+jyONhrt+/x3aaA9kTa8sDiT1Jn9k9Czb7gsM6KKT8FGDJk4/S1NiKotZQOetHdHRm0N71Jg7XTCxlK/FYHoh78KVfhmEY/WmiQGMd5a7vUOidQNKU5DlLQJzH+089xBFj4KS8AH+t+y3NcZO4jPJ92cZyJP5Egs2qimUabDAsHgy7eS4uWK0m+bFP6Xd3QqZ/AAAgAElEQVQ+g6GtJjN1nVUrPup3PhtTVDxkymBXVvqLnnmaZx96gJ5AB6rXS/bYsfidzu2MZWDnE/NIKaZbb/ht/yT/QSTKsW4P6ckEi3qlqHfFTjMzkeCDt97kJ8Pcy4mnLeCum2/kxLVr8CeS+KTFtaaFf5C15YHKnmxO2z0LNvuC/R4UhBDjgQdJudBI4F4p5R/39ziGwut10xF4j472HuJGC4p1J2nppzNm7AkUlRxN9ca/YySrMM0y3J4kjdsSCFcWmoiS7XCi6yrj9WI2tnaTyPHwQX0D10oLKcMEgKeAHuA/pIpS3zMt/txZgkOcQY5jHCG5jYWdTzIlp75/TIPTAYrLxRvL/kuZYSAsi7ZAO+9u3sx7QnDK547lC+eezzXf/d4OK/3l9XX8+eofEs/L77e73FBTzeu/v43fuFwUuJy83drCK60tnDFtOl1d3awKhvpX87DzfPXgp5GPqqr4yQ33snTpejqy3FxcnOyv8y/X9f7gN1Sd//L6Oly9OkpSVWkKh5iW7uOGEZ5awkjeRhAV4Ha4KMjK5NKF1xw0k+Se9B/YPQs2e5vRUEk1gB9LKacARwPfFUJMGYVx9JekdnVeQH7+n7DMedRtXkE0ehoO52OY5gl0dT6GplfjcPgoKnZxxQ/OIhrJQlV/hNt9NN0JN1VdKmEriVOaNHdvIifdQShmcFwsxvGWhVcoLABuBCp1nRUOBx1C8JrMIWKdxhaZx/h0Hx7vBBrlfBqM7GHVSJu7u3kmEkE3TQqSSToMg1XAr6Xkj/E4a/5+Pzfd+eftUjBdnZ046+q4GpgZCvW7ov3rr/ey0OVirsfDtnCYU3SdH2gaL9ZUMynDx5kCljYMH6BG4qOqKm68+106ur6KP+sm6iNf57p1Ttoys9mUTLIiGiXf4x3yHgunTOVP69cjw2GOdrlo7uzgtUiEL+XlDasM+uq/n2Hh2HHcfNRR/Pm447n5qKNYOHacrR5qY7Ob7PcnBSllM9Dc+31QCLEOKADW7u+xLHnyNUzzVKo3xgiHl9HRXoWmLQQSCFGDw5lNMnEU1Rt+h2kUcck3TmXNqiZa20uoWnkH4fAmpHUXDnEhbzVZ5Dq20mY+TCYWLy8PUaPkEDI7KENSJwQTACyLmV4v14XDvI+LLF8Z0WictkAAh8tJ9oTPUtv9DFe1tKTUOAdV4IQat3CMx8N1kQiNhsEM4CzALwSfdTj4OfD9xxcxY/qM/hRMU30dFbpOmqZxfyTcnwp6pa2VWWPHARAxDDJUlTQJbbEYBcUlrPjvhzzb1sZrjY24dB1Hfj4/+r/fDvt5DkxXrdxmkp7zGzLSKygs6aBxbZQxnMGSzr/z1ZIS/lxXRzwtrb//YuBqd7A+0ybg2owMMrs6ScmDDJ1W69I1rl+Rcn7LVDUqpcVb0RiA3dRlY7OLjOqeghCiBJgNfDDE7y4HLgcYO65wj86/+IlFPHjfcwTae/Dn+Pp1ifoYKJXtdvswzPuByTj0tWRmTaS9dS3SmozES0tjBn/43SPEog1YxslkOS4mwzWBtsgSLHk3hmwhwwrSboxnjHEJFcKBabVym7mYM1nOAlVlqZRskJLxPh+XTCijrikJsVbmZpfi0zR6DIOPtq1CTzPoCcapXbWKgmQCh2XRvEzwxosvEAyHeV1KLgDeAG4FaoF2KTFNk+mGQTQeZ+u2ZpYbSY4sHE80HCGuCF7v7GSTZXH9io84qagIqaqsjEWZ6/Hg0TS6TZNGKcl1uejq6WFLJIJPSkqSBqphsL6ujut+/UuOnTt3h0l2cLrqlPW1dAaDpHk6yMnOhinTaKyrYU1nlOzPVPKtK78/YiXRgsJCzho/HoDrV3xEQTxOe28lUaCjg3c3baTGMLj1ht9y4mkLMF0unli5gm+7XBQqClWBdh6Rkkp/DhfYTV02NrvMqAUFIUQa8CTwAyllz+DfSynvJSUUxNTps+Tunn/xE4u4/YY3cLl+Q0bmLELBldx+wy0A/YFhoFQ2gKaOwzBqsawknYF1GEkdgYYmjscTO5lg9EESNOIXZ+A0xhM0k+QppxCzxuMSPyZfqni0MwkZfnRHkkgsF53TeIIGgmYb6wFTVdkcCrG0sxPdtFDjj5BQL0ZqJSQSDSihv2LEQ6xr1SDp5Ch6mK07eNFIMjeRQJGSAuAlQABtQDHQDiTjcVZZFm5d52J/DreuWcO5tbV4urpYkkzyoabxO7+f1wMBvloXoZt8Tm+NcqayhdMVwSrD4G1F4azKyby5popXpOS7DgenKwrr43HulZKVzc1DTrKDK4amZjioCwdoqq8jJzubnOxsdD3AEZkn7NRXeXCp5byiYu5ZvYrj3W5aAwHe75WZuG76dDJ7xxJKJPkKMEYIoqEQ5YrC5yyLV4Xd1GVjszuMivOaEEInFRD+KaV8al9c48H7nsPlWojHOxdF0fB45+JyLeTB+57rf48nTUfKp0kkNiClidc7DcltSLYQi3UiUIEnSRfH4lPLyVa/hoUbXVhAhLhhYlhhVExykYQMnSzDT3ciTkhKuoGp5JGPiy+SSnzkAl/s7maOZTFFU7kpfRN68Ebqe64lGPodmUo6Ul5BMnk1RVzJaxzB35JJvqMonKooaMAkReF8wAHcB2witWP/kWVxPTCvtJQijwe3orA0HGYRgmW9HcbPh6Pc31NBmfwOx2q/oli5iqfNGfzKMHnG6WS1x8uDpskdiQTnORyc43BgGQbTFIUrFYVQPDZkTn+wC9lFRR4suZju7o3buaWdf9oxO/3bDXZ68zkcdI4bx9JJlXxt/TrecXs4d/oMjuytjrooLZ2Oxi0cPX0G1Q4H78fjNGgax2RlY5kmYDd12djsKqNRfSSA+4F1Usrb99V1Au09ZGR+Ussfj8UIh8cTaG/mR9+5GtBp2xYiEv2QaPu/sSwdTXOQnZ2HYfyVnu4IijiRTP1k3GYlqhDoMheBIEtZzzYzQhyBwGSCaKBcMwBoMFpJWgW0WSbTVJW42cp44pymaczUNL6VTHJGbi5HS8nCYJDx6W7udZlUO3q4oVlFF2cQtfIIk8BPPlFOZyUNlNNJklQUFw4H+YZBj2UxV1W5K5nkY8CracwrLeWX02dw/YqP+FV6Ou1GalI8QlFY1dPNuSEXlcqZFDuL2BqPM8VVRI55Pm2OHh47eRYbg0EezcykdtNGTpCpBzRpWShCIVNa6L3aSoMn2R1NarL5WkkjtwUeprnlP/1uaXOmTdvp326oUstLe/cdfnLlFdyQn7+dxlNZWhoW0OVwMKO3v2NCn5KpMyWzMXCT3BaRs7EZntFIHx0LXAysFkKs7H3tZ1LK5/fmRfw5PkLBlXi8c4nHUi5qllWH7vCycrkPOA5v2nmEmmsRYiUO51dBQnfXb5k5yaAuFqY77qIjsRR4HGFmoOEmSw3RZj7NDBYQxU8zrQTkEs42txFWVV6Si7HkWWgylzgttCr/5geikzSXmzGmgZCSjYbJC+EQbdEo34vHmQdUWBabzAKSVgYVLpVGQyEsLTRyieLifcMgRwikolAjJbqmUanrHO/zsSYWI9Pp5C/TpvdPytvCYfI0jbDXA8D7wSBvKwo90kEaeUQsCwCHUBirjqUqtR/bP9mXTJ3GCx+vZL4QpAlBq2XygpSU9cpWDK5EGqqRaqWmcNNv9lyEbvCexa03/JbaYSwwSwa4rY2kZGqLyNnYjMxoVB+9A4O0G/YBl3zj1N49hIWEw+OxrDqk/D1ebyFO17cRjKF565s4nadhmkdiGI/gdP4AYV5BoP5nXJZtckfzMpJ8H5VKBOtJ8kf8Wgff0UMsjd5Fk6mjEaWSdt4XCvkOnUnKGtapATqSKt1mmKvT4xxnuWg3DN60LDy6zlOdHZwjBF/QdRoMg6dNk1c1jYCIo9CKixIKNY0NyQQ+WiklxkPAEVJSOr6Imu6ulJmNptEpFDrHjeOrZ/9PvzhdWVoaLt3B69EIx06cxMrubl5obORCIXhKJAhZ2zASY1AcDhLSotVsJt+V+tz6JvsLTlvAA7/5Fea2ZrJkqlzsPx4P3502bcgO4X3ZSDVwIr+icjLvr161gwXmRYPc1oZTMh3YSAf2foONzWAO2Y7mM845n8aGeh57+EpCwSiq6iYtfTw93c04YkHS07MwkhbetFx0cjCNbnRFIihgS9jiD2EvOl8mkx7iLGeskHjkF6mLr2VqtsYsn+DBQAsXJ5PMdThZLgTTsrJ4JRrl5VnlnPe1/+WJm2/E6uxgRThMQyLB+04XBQ6ded3dJBUFoSicommUKwqL/Tl4olGaQi/wsXEalpVLhBa2sYQs2oipKvUeD/FEnLK5nyGdlIiet6CA03vTH4tIGd/0BDqQHjedmVlU6jrvd3ZyXloa4WiUea4gT8SXUK6eiVsUsDHRyFYWc22Zb7vJfva06fCb6/rTLIrLhYvh5aJh3zVSDd7EPnYIC8xh3dYGYYvI2diMzCEbFFZ/vII1qwRHH3s/q1f+kUC7m1j0QjTtOQxD0NGxGVWVmEYARABFzSceayNuVKGKGShSRXIWQdaRwVaOQMEU+WySTmo6WvmiLwNVKMzQdRoSCTp1nWqHg6PLK3gpFmP2tOksPepobn58EclIBJxOMsaNw9i2DTUnl6QQdLe3IZ1OpqWl8Q/TYEZWJuerNTwWvpfqpEqmiHKlEmCKM430jAweTBqszc3jxlt/33+fffnxe2656ROhuZmzqAmFuKV5K382TdZ2dvKVrEymzJzF8dnZTNpcy73Vf2NF1GJMfiZTx6p86PRQN6hn4EDplh08kQ9lgbmr2CJyNjYjc8gGhYFeCULREeJzKEo5khOQ1hNYch7eNJ1w+FlgOf7cC2ltXg68Tba2gHDyA6ADmEoPQZwySA3bcBNnlZRk9XSToahsQpDUdTL8OcyYPYeNwSB5eXkseuZpNv/rMe5wu5mVlc3KWJRbmrdCxUT82dn9JvB6IkGjlIxxu5lXVMzjq1dxod/k9e4OLkok0BSVonQfuQ4HZxoGH0c+0UXamdDcQsbxaGYmJ8ybh7erC3/vRHjJhFKOzk1tKO/upDoa7M2J3BaRs7EZmVEpSd0fNDW2kpaW8kowTQtvmgfDuI5Y9B6gGk3cRTT0C7LSb8OpvkRX67eR8mYc4gt4Hcfi4hiS/BODVkxgNctZxuPEcFJv+nlcwuumwW1CYPl8WIbBxmCQu5qbaQ4EuP+mGzg7kaBYUdEUwVyPh4UuF1u3NvWXW44pKuaVaJS/xGKcVFS0XenlcgkBIchOSyfH6aQ7mcSUkjTvJ2WfA9MqiUiUI9weLta0fmmKvrTI4BLPoaQlDmT25vhnT5vOvCuu5NHMTK5qaeHRzEzm2ZvMNjb9HLJPCgO9ElTFoiu0FE37FqqWCYl/Y8lFpKsWs6IRMp1dfHd2JZd93ENzMosOKYkxHRWIcxcWb1LFOEo4lwR5KFaANvNp0pWP+XJODg8bBpsUQZlpEgiHuWDDerpjMT6jKFQHAuD3k+tyMsvlRnR3pyalYTZELx2wIZpRX0dHWxtNvWqoRkEB5cUl/fc4MK0yktDcwa6mubfHf6CkxWxsDkQO2aAw0CsBVCx5NNLyIORKkBuRXIvD6sAjt9IUXcwD9fUsnDSOa6ruxJQ/QNFKSTcdeMRWxlkxSvkyTjGGdRKEzENjAS2yicmmybPxOGf/8MesfPstzu/u4kS3m+WaRtg0GWck2dTdRa4rn5WxKD5/9i5NSieetoBne/0JpvamOZ7o9SfoY2BapaC4hE1r10AyuZ3QXF9a5GCfCA/28dvYHCwcsumj6TNn86X5RdRsuoaWlvdR1dfwpL2MaTwB8gtY0sU2U+fN+Hji8nTe79S5ZEIJN0wLoPFjFMeFdMrvEbLWsRo/z1HGa1JSoChIRaGWPOqlk+e8Xs6aVEnj2jXUrKniCy4XGbrOV3w+HiYlld0Ri7EsEuGWWIz5u2GBubM0x8C0SmZWFvGSEm4FPk5Ls9MiNjY2e8Qh+6Sw+uMVvPRsA2OLryeWiNDVHqW7YxFSbsYgHSgDignJBCuTLrJ7PwqfrnHUuARboyFWRyYjuAZYikkhAYK8b22lTNfI17rIdjn42exZmFLyalMTFtAKZAPH+nwA3NPdzdumSbamk11RxEcvL+W/b7+FDli9vsx9Hs2Du2t3tjreIa1SXDKi0JyNjY3Nzjhkg8IjjyxF6hfg9JahKv8lHE4guBT4IaACGgIfEg2TOsKmwlNbtvDUhvVcW1nJguWd6FyD4AgECkmew2I+IXrIVLZQK5dwTVlKSK8vd2+6XNyzcgXfBsp1nRyXCx0oKZ3AFIfORWnpdCXiPLFyBccBlcUlrF+/rt+jOXMPumvttIqNjc3e5JAMCls6I2xtaqW0eDI+p8J/mjpxiElI2UOSLuAx4JtISjGpA14mYiX4/fp1fLOwkCMLxxP+oAc3M0EIEmIOLgRx63lMlrBRifDNkjS+WlrJh41bUt4Aeflk5+XRmJHJXxIJotEoPZZkq5TI2s3E3B56yst5paGBb7tcjBGCD6o3cZLP1+/R/LPZc+zuWhsbm1HlkAoKWzojRJImUkrKJ+TTsvlfrGleQyyxAZ2pKEwC0oATgMVAHZCPymRcqpur9DBr2gMsCwRwiyRJ+TFO5iKlRNfmYiFRxdNcc81lPPvQAzy7dCmmabCgpJQfVVSkGsa8HqKlpbS0tuJsbeF3EyowauvIlxYPr1vL+kSCbzldbA6HiIRCJKWkMC2tv2LI7q61sbEZTQ6ZoNAXEMoKsijxWnTO8POzfz+F0/FjBJdh0ILgDwicSP4DXAZkoNCJyR+Z4IowKcPHxHCY5xrqOTs9wMM9NyHlNcBM4uYKktaNTC5KUL/0RW4qKydsWSiRCIs7Aqzo6NiuYSzH7+eCsWNTTWptbYxNJLhY07gyGOTdSITPKQoxTUMYBlUdHWT6cwC7u9bGxmZ0OeiDwuBgYHQFkBua+eCdjUwt+wYt3SqBro9JGioql2LyBJKtpByTLRR8qDQTiIR5IubCEQrzeJeTQm8pBWIZW+XlSFzowiDf1YQv7CLWsIWe8nLMSJRj3R7SkwkWNdQz1+/fbqXf10PQVy46QdOIGyZvIimwLAp8Pt4MhXhTSuZIa0ihORsbG5v9yUEdFAYHBHdPO6HNNSSlRkNHnLSMifR0byFTiSB0SUfSgcVydDR8Ipuw9OJGYZY4EimqOSIRZmFsChH9HIRWRjKjCW90EdmsoFyBKyeU4Wlr608FTVJVphvJIRvGgP4eAn92NkyZyhubNhIC5vj9LBYKXaaB4veTlHB/NMqmIfyKbWxsbPYnB2VQ2DEYtBL6qIZuqQEaWu443C7BxpWvMN1dSonTAYbBeqWRFjPB0drXyVPH8k48Tqkl8dJCtprk5WgahcqZdGZPpmxSJY1rFcY4L6U90sr/eYLQ3kZU0xgrBBdrGv+QMDmZHLZhbKDGTkDX+W9REbMnVTJXVblwgI5Pn7HNwaBDZGNjc2hz0AWFIZ8OqmswZSoY9JFFCxnGU3R1fgVHMovuZBNCLsZBDe8ai3BwLiHGEacZL0/zXW+QRcZYiv2VdJgmTfV1VOgO0rUi1nbCbL+bkGFQZUk2JZNM0DQ6TYO2nFzu2FxDcyjMMkUwf+BKfwhpBsAWZLOxsTlgOWiCwkhPB1pu0Q43Emlr45uigfeppV4qeGUPKklaGI9FiC7+jcBHFJU4EV4CErpJXbwRd1Yl0XAYn9tNT7KRfBdUJ5NM1B3IaJTC3lTQGsOgob2NS6fPYEFhYWqCX/oiK8rK+/sHhkwFHcQ6RDY2Noc2B0VQiBnmLj0dDCQUDjHd6eDrWfB8Qx1/ZhYOTidOEVlk4OFpoownQymlVG4jJ3o3Z6V38LOehygY9xNcMo2mSB0Wi/lmWQYPbQ1yhmHg8Hj6U0ET3R5+pKq77eJlN5zZ2NgcqBwUQcGpa5w4MWOnTwcD0T1e7gx0YoTgTbMUjTG4iGGho1FJmPOJ83eKRSkekU+VqfOdLBf/NzbMvbHbcKbn0BKpY2GxlwWFJfwzHuWHtbUkw2FyFIX5F19K4OWltouXjY3NIcVBERQc0kBuWEsoHBzx6WAgWXl5rKzPweArdJKHk2zCPIaKgy4qyaCIGGHGI2iytlHoVpgxew5TpeTDlhZuvuP6flezp9evpycQ4MpBaSLT7aEmFLJdvGxsbA4ZRiUoCCFOAf5ISoToPinljSMeEI+RDEV3+nQwkE7yqeqJIHkaiBEjA5iBl3U4mE4EcJNGi9xGkmf4sjsEDD2pB1pbmKY7KPJ4UIXoTxP92TRTpvEMvWncF1T69g4Git3Z2NjYHIjsd+lsIYQK3Al8GZgCXCCEmDLSMRKxS08HA3nh9VeQmMAtwFvAb4B6wqxjjFgH4nYylf/QJO7kG74aKlRlO0evPqvLC7q6+IUl+Za0eG7dWpYFAkAqCFix2LDy1gOP/1N+Phd0dbH07jtYUbV6tz8zGxsbm/3FaDwpHAlUSyk3AwghFgELgLXDHSA0fbcvYlrpwNXAUb2vHA1cBVzO5ILF3FmewVz/MSwLBHi8upq/G0mOGtA8dusNv+23uoyleftlKvo6lwe6mg21+h9olQm7vgltY2NjM5qMRlAoALYM+LmRT2bufoQQlwOXAxSOHbsHl9GACUAC0AETqABUioqc+Bw6ppT4HA5cReP52SC56oFWlwNlKprD4V2Soxh4fB/2JrSNjc2BzgHrvCalvFdKOVdKOdefmbUHZzCADUA30NL7dQNg7JJxe15BATWh1D6DPzs71ZsgBJuE2CVXs4HH92FvQtvY2BzojMaTQhMwfsDPhb2v7VX8GR0Eum8Cfk4qY/UhcBP+jI5d9kgeSqbiF7togDP4eLtz2cbG5mBASCn37wWF0ICNwImkgsF/gQullGuGO2bWlKnylUce2+1rVX7+cwS6s0mlj5L4MzpY/8abu3z8p60esquPbGxsRpPc2dOXSynn7s4x+z0oAAghvgL8gVRJ6t+klL8b6f17GhRsbGxsDmf2JCiMSp+ClPJ54PnRuLaNjY2NzfAcsBvNNjY2Njb7Hzso2NjY2Nj0Myp7CruLEKINqP8Up8gB2vfScA5GDuf7P5zvHez7P9zvf5KUMn3nb/uEg0IQT0qZ+2mOF0Is293NlkOJw/n+D+d7B/v+7fsXy3b3GDt9ZGNjY2PTjx0UbGxsbGz6OVyCwr2jPYBR5nC+/8P53sG+f/v+d5ODYqPZxsbGxmb/cLg8KdjY2NjY7AJ2ULCxsbGx6eeQDgpCiFOEEBuEENVCiJ+O9nj2J0KI8UKI14UQa4UQa4QQ3x/tMY0GQghVCLFCCPHsaI9lfyOEyBRCPCGEWC+EWCeE+Oxoj2l/IYT4Ye//91VCiEeFEK7RHtO+RgjxNyFEqxCiasBr2UKIl4UQm3q/7tSH4JANCnti+3mIYQA/llJOIWU7993D7P77+D6wbrQHMUr8EXhRSlkJzOQw+RyEEAWkbBbnSimnkRLePH90R7Vf+AdwyqDXfgq8KqWsAF7t/XlEDtmgwADbTyllAuiz/TwskFI2Syk/6v0+SGpCOKwcfoQQhcCpwH2jPZb9jRAiAzgBuB9ASpmQUnaN7qj2Kxrg7pXq9wBbR3k8+xwp5VtAx6CXFwAP9H7/AHDGzs5zKAeFoWw/D6tJsQ8hRAkwG/hgdEey3/kD8BPAGu2BjAKlQBvw99702X1CCO9oD2p/IKVsAm4FGoBmoFtKuXR0RzVq5Espm3u/3wbkj/RmOLSDgg0ghEgDngR+IKXsGe3x7C+EEPOBVinl8tEeyyihAXOAu6WUs4Ewu5A6OBTozZsvIBUYxwFeIcRFozuq0Uem+g922oNwKAeF/WL7eSAjhNBJBYR/SimfGu3x7GeOBU4XQtSRSh1+UQjx8OgOab/SCDRKKfueDp8gFSQOB04CaqWUbVLKJPAUcMwoj2m0aBFCjAXo/dq6swMO5aDwX6BCCFEqhHCQ2mhaMspj2m8IIQSpfPI6KeXtoz2e/Y2U8lopZaGUsoTU3/41KeVhs1qUUm4DtgghJvW+dCKwdhSHtD9pAI4WQnh6/x2cyGGyyT4ES4A+Y/hLgWd2dsBBoZK6J0gpDSHElcBLfGL7OawP9CHIscDFwGohxMre137W63pnc3jwPeCfvYuizcDXR3k8+wUp5QdCiCeAj0hV4a3gMJC7EEI8CnweyBFCNAK/Bm4EHhdCXEbKfuDcnZ7HlrmwsbGxsenjUE4f2djY2NjsJnZQsLGxsbHpxw4KNjY2Njb92EHBxsbGxqYfOyjY2NjY2PRjBwWbAxohxBlCCCmEqNzD468TQpy0k/MfjkKBNjZDYgcFmwOdC4B3er/uNlLKX0kpXxnhLWeQUtHdL/Sq99rYHLDYQcHmgKVXt+k44DIGSB8LIcYKId4SQqzs1cs/vtc34R+9P68WQvyw973/EEKc0/v9jb3+EquEELcKIY4BTgdu6T1XmRDiqgHvWTTEmL4mhHhGCPFGr0b9rwf87iIhxIe957qnLwAIIUJCiNuEEB8Dnx10vs/0XmulEOKWPi18IUSJEP+/vfMJsbIKw/jvmRCbcBrRCoTMKAoKJMmo7B8WRQtrNwSDMEhthETRkBTdtKtNhFobmbI/SJuJiQhCF6FRhqMROpsWGW4KF4KhC0Wdp8X7eufzMle5MTDT9P7gcN97vnPOd7778X3nnu/c+zz6QdIvmZ7K/NWSDmUfTuUxrc39npR0f5a7U9KIpLFMT0/nuSnmMLYrVZqVCVgLDGf8E7Ay47eAHRnfAvQBK4GDjboL83UfMAAsBn5j8g+b121v1PsTmN8s09andYTy5mKgFxgHHgMeAr4B5mW5j4ChjA281uEYx4FVGb8LjGd8G3Brxg8AxzJeDZwDlgDzCT2vd3LbJuCDjPcDz2R8DyF3MuPntNLsT3NW5qKYEwwSRjEQonaDwIXMOtAAAAJHSURBVHFC1+rjFPwbtf2rpFPAfZJ2A98C7VLJfwMXgeF0YevkxHaCkIYYBUY7lDlo+yyApK+I2cwVYmAaC7kdepkUH7tKCBNeh6SFQJ/tI5m1H3gl43nAHkkrsv6DjapjTjlkSb83jvUk8HzGLwIPZ18Abpe0wPaFDsdUFMAc1j4q/ttIWgS8ACyXZGJGYElbbR+W9BxhoLNP0vu2P5P0CPAysJ7QeHn9WnsOLazHCXG0AWBDtt/OGsKc5lVgh6Tltq+0lWnXhjEg4FPb26do86Ltq119ALAZOEM4pvUQA9o1LjXiicb7CSav6R7gSdvNekVxU2pNoZitDACf215m+17bS4E/gGclLQPO2N5LuKo9KukOoMf2CLCTNpnoXJ/odwgCbiZutgDnicdPSOoBltr+Hngb6AcWTNG3lxTet73EQvWPhNXhgKS7sq1F2c+OOJzQzkt6IrOalpH9wF+2Jwhhw24XqA8Qgnhkf1Z0Wb/4n1IzhWK2Mgi815Y3kvk/A1slXQYuAEOEq94neWMHaP/G3gd8rTBwF7Al878E9kraSNyUhxVWlgJ2eWoLy6PZl7uBL2wfA5C0EziQfbgMvEkoU96IN3L/E8Ah4jEXxJrEiKQh4DvCJKcbNgIfSjpBXOeHiRlUUdyQUkktii6QtI4whN8wTe21nvNL2gYssb1pOtouin9DzRSKYmZZI2k7cS2eJn7dVBQzRs0UiqIoiha10FwURVG0qEGhKIqiaFGDQlEURdGiBoWiKIqiRQ0KRVEURYt/AKwIAri3TPukAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_boundary(APG, RPG, apg, rpg, y, pos_dict, h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再將特徵矩陣 $X$ 描繪在依據 $h$ 所得的決策邊界，就可以觀察到誤分類觀測值：藍色的資料點落在紅色的決策邊界（真實位置為前鋒、預測位置為後衛）、紅色的資料點落在藍色的決策邊界（真實位置為後衛、預測位置為前鋒）。\n",
    "以 `LogisticRegression` 類別找出 $h$ 的最關鍵方法，與迴歸模型相同是呼叫 `fit()` 方法，究竟它是如何決定 `X_train` 與 `y_train` 之間的關聯？接下來我們試圖理解它。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 羅吉斯迴歸\n",
    "\n",
    "羅吉斯迴歸（Logistic Regression）分類器在機器學習領域中扮演著承先啟後的橋樑，能夠協助我們由數值預測過渡至類別預測的任務、再過渡至自動尋找特徵的深度學習。從前述例子中得知，欲得到對於特徵矩陣 $X$ 的類別預測 $\\hat{y}$，必須先得到類別預測機率 $\\hat{p}$，羅吉斯迴歸分類器透過 Sigmoid 函式（亦稱 S 函式、Logistic 函式），這裡使用 $\\sigma$ 表示。\n",
    "\n",
    "\\begin{equation}\n",
    "\\hat{p} = \\sigma(Xw) = \\frac{1}{1 + e^{-Xw}}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return(1 / (1 + np.exp(-x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_sigmoid():\n",
    "    x = np.linspace(-6, 6, 100)\n",
    "    y = sigmoid(x)\n",
    "    fig = plt.figure()\n",
    "    ax = plt.axes()\n",
    "    ax.plot(x, y)\n",
    "    ax.axvline(0, color = 'black')\n",
    "    ax.axhline(y = 0, ls = ':', color = 'k', alpha = 0.5)\n",
    "    ax.axhline(y = 0.5, ls = ':', color = 'k', alpha = 0.5)\n",
    "    ax.axhline(y = 1, ls = ':', color = 'k', alpha = 0.5)\n",
    "    ax.set_yticks([0.0, 0.5, 1.0])\n",
    "    ax.set_ylim(-0.1, 1.1)\n",
    "    ax.set_title(\"Sigmoid function\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deXhU5d3/8feXrJCwB2RJZBcEFRXUutaltu76tFq16q9Y+9hF7fL4a+tW29ra2tZWbbW1rdVTxCIWKArRiFFEEVGIhgiEQIjBEAlZCCEh22Tmfv7IOE+KAQIkOZnJ53VduZh7zsnc3xPOfHJyn3vOMeccIiIS/fr4XYCIiHQOBbqISIxQoIuIxAgFuohIjFCgi4jECAW6iEiMUKBLtzGz68xsaU/r18xeN7Ov72OZmdlTZlZtZu92XZXt9v2SmX21O/uU6Gaahy6dyczOAH4DTAOCQD7wPefcal8L2w8zex2Y45x7op1lZwJzgcnOuT1dWMNPgYnOueu7qg+JffF+FyCxw8wGAEuAbwHPAYnAmUCTn3UdpjFAcVeGuUhn0ZCLdKajAJxzc51zQedcg3NuqXMuD8DMZpnZik9WNrPPm1mBmdWY2Z/MbPknQx/hdd8ys4fMbJeZFZnZaeHnS8ysvO1whJkNNLPZZlZhZlvN7B4z67OPfs83s43hfh8FrL2NMbObgCeAU82szsx+tvdrhddzZjYx/Ngzs8fMLNPMas3sHTOb0GbdaWb2ipntNLMdZnaXmV0A3AVcHe5nbXjdyFCQmfUJb9PW8LbPNrOB4WVjwzV81cw+MrNKM7v7kP8XJWop0KUzbQKCZvYPM7vQzAbva0UzSwPmA3cCQ4EC4LS9VjsFyAsv/yfwLHASMBG4HnjUzFLD6/4RGAiMBz4L/D/gxn30uxC4B0gDtgCnt1ejc+7vwDeBt51zqc65nxzoBxB2DfAzYDBQCNwf7rs/kA1kAaPC2/Gqcy4L+CUwL9zP9HZec1b465zwNqYCj+61zhnAZOA84F4zO7qD9UqMUKBLp3HO7aY1VBzwN6DCzF4wsyPaWf0iYL1zbqFzrgX4A1C21zofOueecs4FgXlABnCfc67JObcUaAYmmlkcrSF6p3Ou1jlXDPwOuGE//c53zgWAh9vp93D92zn3bni7ngGODz9/CVDmnPudc64xXOs7HXzN64DfO+eKnHN1tP4ivMbM2g6b/iz8V9FaYC3Q3i8GiWEKdOlUzrl859ws51w6cAytR6IPt7PqKKCkzfc5YNte6+xo87ghvN7ez6XSeqSdAGxts2wrMLqD/Za0s97haPsLoj5cI7T+QtpyiK85ik9vXzzQ9pflvvqVXkKBLl3GObcR8GgN9r1tB9I/aZiZtW0fpEogQOsJzE8cCZTuo9+MvfrNaGe9fdkD9Gvz/SMO4ntLaB0uac+Bppt9zKe3r4X//KUnvZwCXTqNmU0xs9vNLD3czgCuBVa1s3omcKyZXREeNrgFOJhwjAgPyTwH3G9m/c1sDPA/wJx99DvNzL4Y7vc7B9nv2vD3H29mycBPD+J7lwAjzex7ZpYUrvWU8LIdwNhPTuS2Yy7wfTMbFz5v8MmYe8tB9C8xToEunamW1hOZ75jZHlqDfB1w+94rOucqgatonbNeBUwF1nDoUxxvo/XouQhYQetJ1Cf30+8D4X4nAW91tBPn3CbgPlpPbm4O99XR760FzgcupXV4ZDOtJzkB/hX+t8rM3mvn258EngbeAD4EGmndZpEIfbBIeoTwkek24Drn3DK/6xGJRjpCF9+Y2RfMbJCZJdE6D9tof3hGRDpAgS5+OpXWWR+VtA5DXOGca/C3JJHopSEXEZEYoSN0EZEY4dvFudLS0tzYsWP96l6kXQUFBQBMnjzZ50pE2peTk1PpnBvW3jLfAn3s2LGsWbPGr+5F2nX22WcD8Prrr/tah8i+mNnWfS3TkIuISIxQoIuIxAgFuohIjFCgi4jECAW6iEiMUKCLiMQIBbqISIxQoIuIxAgFuohIjFCgi4jECAW6iEiMUKCLiMQIBbqISIw4YKCb2ZNmVm5m6/ax3MzsD2ZWaGZ5ZnZi55cpIiIH0pEjdA+4YD/LL6T1zumTgJuBPx9+WSIicrAOGOjOuTeAnftZ5XJgtmu1ChhkZiMP9LpVVVXk5uYCEAwG8TyPvLw8AAKBAJ7nsW5d6x8FjY2NeJ5Hfn4+APX19XieF7kZQV1dHZ7nUVhYCEBNTQ2e51FUVARAdXU1nudRXFwMQGVlJZ7nUVJSAkB5eTme51FaWgpAWVkZnudRVlYGQGlpKZ7nUV5eDkBJSQme51FZWQlAcXExnudRXV0NQFFREZ7nUVNTA0BhYSGe51FXVwe03kTB8zzq6+sByM/Px/M8GhsbAVi3bh2e5xEIBADIy8vD8zyCwSAAubm5eJ4X+Vnm5OQwe/bsSHv16tXMmTMn0l61ahVz586NtFeuXMm8efMi7RUrVjB//vxIe/ny5SxcuDDSXrZsGYsWLYq0s7OzWbx4caS9dOlSMjMzI+2srCyysrIi7czMTJYuXRppL168mOzs7Eh70aJFLFu2LNJeuHAhy5cvj7Tnz5/PihUrIu158+axcuXKSHvu3LmsWvV/95aeM2cOq1evjrRnz55NTk5OpO153j73vVAoRFlZmfY97XtA9+57Hc29/emMG1yMBkratLeFn9u+94pmdjOtR/EMHTq0E7oWEekcgWCIPU0thOqaKCyvpb45yOYdtWxvTqJu3XYaAkFWFlaSWh3HB4EtNLWEWLX2Y/qXwos7UmluCbH2na30L4YhG/vQHAxR8OYWUgqCDHi3iUDIUbxyE/3WBbm7z3AunHZEp29Dh24SbWZjgSXOuWPaWbYEeMA5tyLcfhX4kXNuv7cjmjlzptMdi6Sn0R2LoltTS5CqumZ27mmmur713131gdavhmZqGgLsbmhhd2OAusYW6prCX40tNAdDB91ffB8jMb5P61dcHxLi+pAU3/pvfFzrsoQ+rY/j4/qQ0MeIjzO+csoYPntUu3eROyAzy3HOzWy3nkN6xf9UCmS0aaeHnxMR6RTOOar2NLOtuoHtuxrYXtNI2e5GdoS/ymubqKhtoraxZZ+vkZoUz8C+CfRPjmdAcgKjBiWTmhRP/+QEUpLiSU2Ko19iPP0S4+iXFE+/hDj6Joa/EuJITogjOaEPSfFxJMW3Bnd8XM+aKNgZgf4CcKuZPQucAtQ45z413CIisj/OOSpqmygsr2NL5R6KK/ewtWoPW6vq2VbdQEMg+B/rJ8b3YcSAZI4YkMTRIwZw1qQkhqYkMjQ1iSEpiQxNTWRwvwQG90tkQN8EEnpY+HaFAwa6mc0FzgbSzGwb8BMgAcA59zjwInARUAjUAzd2VbEiEhsamoNs2L6bDdt3s3H7bgrKainYUfsfR9jJCX0YMySFcWkpnHXUMNIH9yV9cD9GDkxm1KC+DO6XgJn5uBU9zwED3Tl37QGWO+CWTqtIRGJKKOQoqtzDe1uree+janJLdrG5vI5gqPX8Xf/keI4eMYDLjx/FxGGpTBzenwnDUziifzJ9+iiwD0ZnDLmIiEQ459hSUcdbhVWsKqrinQ93snNPMwAD+yYwPWMQnzv6CI5NH8i0UQMYPaivjrQ7iQJdRA5bQ3OQNzZX8HpBBW9sqqB0VwMAowf15ZzJwzll3BBOHDOY8WkpOuruQgp0ETkktY0BXtmwg6x1ZbyxuYLGQIjUpHhOnziUW86ZyJmT0sgY0s/vMnsVBbqIdFggGOK1jeU8n1vKq/nlNLWEGDkwmatnZvD5aSM4edyQXjGbpKdSoIvIARVV1DFvTQkLcrZRWddMWmoi1558JJdOH8WJRw7SGHgPoUAXkXaFQo7lmyt46q1i3thUQVwf47wpw7n6pAw+e9SwHvehGlGgi8hemltCLHq/lMff2EJRxR6G90/i9vOP4uqTMxjeP9nv8mQ/FOgiArReB+W51SU8vryI0l0NTB05gEeuOZ4LjxlJYryOxqOBAl2klwuGHIveL+X3r2yidFcDM8YM5hdXHMPZk4dpbDzKKNBFerE3N1fwiyX5FOyo5ZjRA/jVF4/lzElpCvIopUAX6YVKdtZzf2Y+WevLOHJIPx79yglcdMxIfegnyinQRXqRlmCIv75ZxCPZm+ljxg++MJmbzhhHckKc36VJJ1Cgi/QS6z+u4UcL8lhXupsLpo3g3kunMmpQX7/Lkk6kQBeJcS3BEI8t28IfX9vMoH6J/Pm6E7nw2APe9leikAJdJIZtq67ne8/msmZrNZcfP4qfXTaNQf0S/S5LuogCXSRGvfTBdn64IA/n4OGrj+eKE0b7XZJ0MQW6SIxpCYb47csF/OWNIo7PGMQfrz1BVz3sJRToIjGkqq6JW//5Pm8XVXHDZ8bw40um6lOevYgCXSRGbN5Ry43eaipqm3jwqulcOSPd75KkmynQRWLAW4WVfHNODskJcTz3jVOZnjHI75LEBwp0kSj3rzUl3LnwAyYMS+Xvs2aSPljj5b2VAl0kij3xZhG/yMznzElpPHbdiQxITvC7JPGRAl0kCjnneOiVTfzhtUIuOnYED199gk5+igJdJNo457hvyQaeequYq2dm8MsvHkucLqolKNBFokrbMP/a6eP48SVH61K3EqG/0USihHOOX76Yz1NvFXPj6WMV5vIpCnSRKPGblwv425sf8tVTx3DvJVMV5vIpCnSRKPDXN7bw59e38JVTjuSnl01TmEu7FOgiPdz8nG388sWNXHzcSH5++TEKc9knBbpID/Zq/g5+tCCP0ycO5fdfnq7ZLLJfCnSRHipv2y5u+ed7TB05gL/cMJOkeN0mTvZPgS7SA22vaeDr/1jD0JQknpx1EqlJmmEsB6a9RKSH2dPUwk3eGuqbgyz41ikM65/kd0kSJXSELtKDhEKO7z6by8ay3Tz6lROYPKK/3yVJFFGgi/Qgj7y6mez8Hdx7yVTOnjzc73IkyijQRXqI7A07eOTVzXzpxHS+etpYv8uRKKRAF+kBiirq+P68XI4ZPYD7/0tzzeXQKNBFfFbf3MI3ns4hPs54/PoZJCdoeqIcGs1yEfHZvc+vp7Cijqe/doruNiSHRUfoIj5akLON+TnbuO3cSZwxKc3vciTKKdBFfFJYXsc9i9ZxyrghfPe8SX6XIzFAgS7ig8ZAkFv/+R59E+P4w7Un6Bot0ik0hi7igwde2sjGslq8G0/iiAHJfpcjMUJH6CLd7I1NFXgri5l12lh9eEg6lQJdpBtV72nm//9rLZOGp3LHhVP8LkdijIZcRLqJc447F35AdX0zT914kuabS6fTEbpIN1mUW0rW+jL+5/zJTBs10O9yJAYp0EW6QfnuRn76wgZmjBnMzWeN97sciVEKdJEu5pzjrn9/QGMgyG+vPE5TFKXLKNBFutii3FKy88v5wRcmM35Yqt/lSAxToIt0obZDLTeePs7vciTGKdBFutBPF6+nIRDkNxpqkW6gQBfpIkvXl/HiB2V897xJTNBQi3QDBbpIF6htDHDv8+uZMqK/ZrVIt9EHi0S6wG+yCthR28jjN8wgIU7HTdI9tKeJdLKcrdXMeWcrN542juMzBvldjvQiCnSRThQIhrj73x8wckAyt3/+KL/LkV5GQy4inch7q5iNZbX85YYZpCTp7SXdS0foIp3k410NPJS9ifOmDOfzU4/wuxzphRToIp3kvsUbCDnHTy+bhpnmnEv3U6CLdIJlBeVkrS/jtnMnkTGkn9/lSC+lQBc5TE0tQX72wnrGD0vhv8/UnHPxj87aiBymJ978kOKqemZ/7WQS43WMJP7R3idyGD7e1cCjrxVywbQRnHXUML/LkV5OgS5yGO5/MZ+Qc9xzydF+lyKiQBc5VCsLK8nM284t50wkfbBOhIr/FOgih6AlGOJnizeQMaSvLr4lPYYCXeQQ/PPdjyjYUcvdF00lOSHO73JEAAW6yEGr3tPM75Zu4rQJQ/nCNH0iVHoOBbrIQXo4e1Pr9c4vnapPhEqPokAXOQgFZbXMeecjrv/MGKaMGOB3OSL/QYEu0kHOOX6+ZAOpSfF8/3O6NK70PAp0kQ56bWM5Kwor+d7nJjE4JdHvckQ+RYEu0gHNLSHuz8xn/LAUrv/MGL/LEWmXAl2kA+as2kpR5R7uufho3SNUeiztmSIHUL2nmYezN3HmpDTOmTzc73JE9kmBLnIAj7y6mbqmFu65WNMUpWdToIvsR2F5HU+v2sq1Jx/J5BH9/S5HZL8U6CL78cBL+fRNiOP752uaovR8CnSRfVhZWEl2fjm3nDORtNQkv8sROSAFukg7giHHLzLzGT2oLzeePtbvckQ6RIEu0o4F721jw/bd/OjCKbqaokSNDgW6mV1gZgVmVmhmd7SzfJaZVZhZbvjr651fqkj3CIYcD75cwPEZg7j0uJF+lyPSYQe8SbSZxQGPAecD24DVZvaCc27DXqvOc87d2gU1inSr7TWNtNQ28efrT9Q0RYkqHTlCPxkodM4VOeeagWeByw+346qqKnJzcwEIBoN4nkdeXh4AgUAAz/NYt24dAI2NjXieR35+PgD19fV4nkdBQQEAdXV1eJ5HYWEhADU1NXieR1FREQDV1dV4nkdxcTEAlZWVeJ5HSUkJAOXl5XieR2lpKQBlZWV4nkdZWRkApaWleJ5HeXk5ACUlJXieR2VlJQDFxcV4nkd1dTUARUVFeJ5HTU0NAIWFhXieR11dHQAFBQV4nkd9fT0A+fn5eJ5HY2MjAOvWrcPzPAKBAAB5eXl4nkcwGAQgNzcXz/MiP8ucnBxmz54daa9evZo5c+ZE2qtWrWLu3LmR9sqVK5k3b16kvWLFCubPnx9pL1++nIULF0bay5YtY9GiRZF2dnY2ixcvjrSXLl1KZmZmpJ2VlUVWVlaknZmZydKlSyPtxYsXk52dHWkvWrSIZcuWRdoLFy5k+fLlkfb8+fNZsWJFpD1v3jxWrlwZac+dO5dVq1ZF2nPmzGH16tWR9uzZs8nJyYm0Pc/b577X2NzCR9s+5jOD9zBjzBDte9r3um3f62ju7U9HAn00UNKmvS383N6+ZGZ5ZjbfzDLaeyEzu9nM1pjZmtra2g50LdK9tu1qAODrZ4zzuRKRg2fOuf2vYHYlcIFz7uvh9g3AKW2HV8xsKFDnnGsys28AVzvnzt3f686cOdOtWbPmsDdApLOsK61h5qlnMGpQX4ry3vW7HJF2mVmOc25me8s6coReCrQ94k4PPxfhnKtyzjWFm08AMw6lUBG/OOe4PzOf+Lg+jBrU1+9yRA5JRwJ9NTDJzMaZWSJwDfBC2xXMrO1UgMuA/M4rUaTrZeeX83ZRFemD+xLfRydCJTodcJaLc67FzG4FXgbigCedc+vN7D5gjXPuBeA7ZnYZ0ALsBGZ1Yc0inSoQDPGrF/OZMCyFlv7JfpcjcsgOGOgAzrkXgRf3eu7eNo/vBO7s3NJEuscz4WudPzlrJvctPvD6Ij2VPikqvVpNfYCHX93MGRN1rXOJfgp06dX++NpmahoC3H3x0foQkUQ9Bbr0Wh9W7uEfbxfz5RkZHD1ygN/liBw2Bbr0Wr98MZ/EuD7c/gVd61xigwJdeqWVhZW8smEH3z5nIsM1s0VihAJdep1gyHHfkg2MHtSXm/QRf4khCnTpdZ5bU8LGslruvEjXOpfYokCXXmV3Y4DfLS1g5pjBXHysrnUusUWBLr3Ko68VUlnXzI8vmappihJzFOjSaxRV1PHUWx9y1Yx0pmcM8rsckU6nQJde4/7MfJLi4/jBBZP9LkWkSyjQpVd4vaCcVzeWc9u5mqYosUuBLjEvEAzx8yUbGDu0H7NOH+t3OSJdRoEuMc97q5gtFXu45+KpJMVrmqLELgW6xLTy3Y088upmzp0ynM9NPcLvckS6lAJdYtoDL22kuSXEvZdM9bsUkS6nQJeYtbp4JwvfL+Xms8YzNi3F73JEupwCXWJSMOT4yfPrGTUwmW+fM8HvckS6hQJdYtLst4vZsH0391wylX6JHbrTokjUU6BLzNmxu5HfLd3EWUcN48JjRvhdjki3UaBLzPn5kg00B0Pcd9k0Xa9FehUFusSUNzdXsCRvO7ecPVEnQqXXUaBLzGgMBPnxonWMS0vhm2eP97sckW6ns0USMx5bVkhxVT1P33SyPhEqvZKO0CUmbCzbzZ9f38IXTxjNmZOG+V2OiC8U6BL1giHHHQs+YEDfBO7RJ0KlF1OgS9R7+u1ickt28eNLjmZISqLf5Yj4RoEuUa10VwO/fbmAs44axhXHj/a7HBFfKdAlajnnuGNBHg64/4pjNOdcej0FukStZ1eX8ObmSu68cAoZQ/r5XY6I7xToEpVKdzVwf2Y+p44fynWnjPG7HJEeQYEuUeeToZaQc/zmyuPo00dDLSKgQJco9Mw7H2moRaQdCnSJKlsq6vhF5gbOnJSmoRaRvSjQJWoEgiG+Py+X5IQ4HrxquoZaRPaia7lI1Pjjq5vJ21bDn647kSMGJPtdjkiPoyN0iQprinfy6LJCrpyRzkXHjvS7HJEeSYEuPV71nma+M/d90gf34yeX6lotIvuiIRfp0Zxz/GD+WirqmljwrdPon5zgd0kiPZaO0KVH+/uKD8nOL+eui47muPRBfpcj0qMp0KXHyi3Zxa+zNnL+1COYddpYv8sR6fEU6NIjVdY18a05OQzvn8xvrzxOF94S6QCNoUuP0xIMces/32PnnmYWfOs0BvXTNc5FOkKBLj3OAy9tZFXRTn531XSOGT3Q73JEooaGXKRHeT63lCdWfMhXTx3Dl2ak+12OSFRRoEuPkbO1mh/Mz+PksUO4+2LNNxc5WAp06RFKdtZz8+w1jByYzOM3zCAxXrumyMHSu0Z8t7sxwNe81QSCIZ6cdZJu9CxyiHRSVHzV1BLk23Pe48PKPcz+2slMGJbqd0kiUUuBLr4Jhhz/89xaVhRW8uBV0zltYprfJYlENQ25iC+cc/xs8Xoy87Zz54VTuFIzWkQOmwJdfPHIq5uZ/fZWbj5rPN/47AS/yxGJCQp06XaPLSvk4ezNXDkjnTsumOJ3OSIxQ4Eu3erx5Vv47csFXHH8KH79peN0GzmRTqRAl27ztzeKeOCljVw6fRQPXjWdOIW5SKfSLBfpcs45HsrezB9e3czFx43koS9PJz5OxxIinU2BLl0qFHLct2QD3spivjwznV/+17EKc5EuokCXLtPcEuKOBXksfL+Um84Yxz0XH63rmot0IQW6dIma+gDfmLOGVUU7uf38o7j13IkKc5EupkCXTvdRVT2zvHfZtrOBh66ezn+doA8NiXQHBbp0qhWbK7lt7ns44OmbTuaU8UP9Lkmk11CgS6dwzvHn5Vt48OUCJg5P5S83zGRcWorfZYn0Kgp0OWw19QF+uGAtL6/fwSXHjeTXXzqOlCTtWiLdTe86OSyriqr4/rxcKmqbuOfio7npjHE6+SniEwW6HJKmliB/eHUzf3p9C2OHprDw26dxXPogv8sS6dUU6HLQ3v+omh8tyGPTjjq+PDOdn1w6TUMsIj2A3oXSYXVNLTz0yiaefOtDRgxI5slZMzl3yhF+lyUiYQp0OSDnHM/nfsyvXspnx+4mvnLKkdx54RT6Jyf4XZqItKFAl/3KLdnFLzPzebd4J8eOHsifrpvBjDGD/S5LRNqhQJd2FZbX8eDLBWStL2NoSiK/+uKxfHlmhi55K9KDKdDlP2zeUctjywp5Ye3H9E2I43ufm8TXzxxPqk56ivR4epcKAO99VM1flxeRtb6MvglxfO30cXzz7AmkpSb5XZqIdJACvRdrbgnx0rrtPPlWMWtLdtE/OZ7bzp3IjaePY0hKot/lichBUqD3QoXltcxbXcKC90rZuaeZ8Wkp3Hf5NL50Yrrmk4tEMb17e4ny2kaWrN3O82s/Zm3JLuL7GOdPPYKrT8rgrEnDdLNmkRigQI9hJTvreWXDDpZuKOPdD3cScjBt1ADuumgKXzwxXePjIjFGgR5DmltCrNm6k+WbKlheUMHGsloAJh/Rn1vPmchlx49i4vD+PlcpIl1FgR7FGgNB8rbV8O6HVawq2smarTtpDIRIiDNmjhnCXRdN4fNTRzBW1yUX6RUU6FEiGHJsqajjg201fFBaw/sfVbP+4920hBwAU0b059qTj+S0CWmcNmGoTm6K9EJ61/cwzjk+rmmksLyOwvI6Csp2U1BWS8GOWhoDIQD6JsRxbPpA/vus8cw4cjAzxgxmsKYZivR6CnQfNDQH+bimgW3VDZTsrKekup6Pqur5sHIPW6vqaQgEI+sOSUmMHH0fM2ogx6UPZPywVH0EX0Q+RYHeSYIhx676Zqrrm6msa6aqrpnKuiYq65oo393EjtpGymoa2V7TSE1D4D++NyHOyBjSj7FDUzh1wlAmDEtl4vBUJgxLJS01UXcAEpEO6VCgm9kFwCNAHPCEc+6BvZYnAbOBGUAVcLVzrrhzS+18zjmagyEam0M0BILUN7dQ3xykIRBkT1Pr47qmFvY0tVDX2EJtUwu1jQF2N7SwuzHA7oYAuxoC7KoPsLsxgHOf7qOPQVpqEkcMSGb0oL7MHDuYkQP7MnJgMumD+5ExpC/D+yfriFtEDtsBA93M4oDHgPOBbcBqM3vBObehzWo3AdXOuYlmdg3wa+Dqrij4naIqlm+qoCXkCARDtAQdLaEQzS2t/waCrY+bgyGaW4I0t4TCj0M0tYRoCoRoagnSGAjR2BJsN4T3JSm+D/2T4xnQN4EByQkM7JfI2LQUBvVtfTw0JZHBKYkM6ZdIWv9E0lKTGNwvUWEtIt2iTwfWORkodM4VOeeagWeBy/da53LgH+HH84Hz7ADjBFVVVeTm5gIQDAbxPI+8vDwAAoEAnuexbt06ABobG/E8j/z8fPK21fCX1zbwx8f/xjNZb7Mk72Nezt3KC/96hrffX8/mHXVs3V7BB8ueZ/eOUvomxjHAmgjkL2N8cj1nTkrj3LF9mVqXw/XTUvjBFybz3dOH84WEfO45ZziPXz+Dhy4dw6y0D3n6K0fx5g/PIfOmqdwzaQdvfmcGa+45n398eQJXpGzm95eN55FrTuDG6akMKV3JZUcP5LLpoxgVt5tVWQtJCPUPA/gAAAbkSURBVDYS18coLCzE8zzq6uoAKCgowPM86uvrAcjPz8fzPBobGwFYt24dnucRCLQOzeTl5eF5HsFg69h6bm4unudFfpY5OTnMnj070l69ejVz5syJtFetWsXcuXMj7ZUrVzJv3rxIe8WKFcyfPz/SXr58OQsXLoy0ly1bxqJFiyLt7OxsFi9eHGkvXbqUzMzMSDsrK4usrKxIOzMzk6VLl0baixcvJjs7O9JetGgRy5Yti7QXLlzI8uXLI+358+ezYsWKSHvevHmsXLky0p47dy6rVq2KtOfMmcPq1asj7dmzZ5OTkxNpe563z30vFApRVlbW7r4HUF9fj+d5FBQUAFBXV4fneRQWFgJQU1OD53kUFRUBUF1djed5FBcXA1BZWYnneZSUlABQXl6O53mUlpYCUFZWhud5lJWVAVBaWorneZSXlwNQUlKC53lUVlYCUFxcjOd5VFdXA1BUVITnedTU1ABo34uifW9/uQf/t+/tT0eGXEYDJW3a24BT9rWOc67FzGqAoUBl25XM7GbgZoChQ4d2oOtP+++zxnPdzBE899xuTj31VCZPnkxdXR3z51dxxhmfYeLEidTU1PDvf+/grLNOZPz48VRXV/N86jbOPnsaY8eOpbKykiWhLZz32QlkZGRQXl7Oi1VpnH/MKEaPHkFZGdRsTmXS8P6MGNKP0oZE4uM68rtPRMQ/5g4w5mBmVwIXOOe+Hm7fAJzinLu1zTrrwutsC7e3hNepbO81AWbOnOnWrFnTCZsg0nnOPvtsAF5//XVf6xDZFzPLcc7NbG9ZRw47S4GMNu308HPtrmNm8cBAWk+OiohIN+lIoK8GJpnZODNLBK4BXthrnReAr4YfXwm85g506C8iIp3qgGPo4THxW4GXaZ22+KRzbr2Z3Qescc69APwdeNrMCoGdtIa+iIh0ow7NQ3fOvQi8uNdz97Z53Ahc1bmliYjIwdDUDRGRGKFAFxGJEQp0EZEYoUAXEYkRCnQRkRihQBcRiREKdBGRGKFAFxGJEQp0EZEYoUAXEYkRCnQRkRihQBcRiREHvMFFl3VsVgFsPcRvT2OvuyFFMW1LzxMr2wHalp7qcLZljHNuWHsLfAv0w2Fma/Z1x45oo23peWJlO0Db0lN11bZoyEVEJEYo0EVEYkS0Bvpf/S6gE2lbep5Y2Q7QtvRUXbItUTmGLiIinxatR+giIrIXBbqISIyI6kA3s9vMbKOZrTez3/hdz+Eys9vNzJlZmt+1HAoz+234/yPPzP5tZoP8rulgmdkFZlZgZoVmdoff9RwqM8sws2VmtiH8/viu3zUdDjOLM7P3zWyJ37UcDjMbZGbzw++TfDM7tTNfP2oD3czOAS4HpjvnpgEP+lzSYTGzDODzwEd+13IYXgGOcc4dB2wC7vS5noNiZnHAY8CFwFTgWjOb6m9Vh6wFuN05NxX4DHBLFG8LwHeBfL+L6ASPAFnOuSnAdDp5m6I20IFvAQ8455oAnHPlPtdzuB4CfghE7Vlq59xS51xLuLkKSPeznkNwMlDonCtyzjUDz9J60BB1nHPbnXPvhR/X0hoco/2t6tCYWTpwMfCE37UcDjMbCJwF/B3AOdfsnNvVmX1Ec6AfBZxpZu+Y2XIzO8nvgg6VmV0OlDrn1vpdSyf6GvCS30UcpNFASZv2NqI0BNsys7HACcA7/lZyyB6m9WAn5Hchh2kcUAE8FR4+esLMUjqzg/jOfLHOZmbZwIh2Ft1Na+1DaP1z8iTgOTMb73roPMwDbMtdtA639Hj72w7n3PPhde6m9U/+Z7qzNvk0M0sFFgDfc87t9rueg2VmlwDlzrkcMzvb73oOUzxwInCbc+4dM3sEuAP4cWd20GM55z63r2Vm9i1gYTjA3zWzEK0XvKnorvoOxr62xcyOpfU391ozg9ZhivfM7GTnXFk3ltgh+/s/ATCzWcAlwHk99ZfrfpQCGW3a6eHnopKZJdAa5s845xb6Xc8hOh24zMwuApKBAWY2xzl3vc91HYptwDbn3Cd/Kc2nNdA7TTQPuSwCzgEws6OARKLwSmzOuQ+cc8Odc2Odc2Np/U8/sSeG+YGY2QW0/ml8mXOu3u96DsFqYJKZjTOzROAa4AWfazok1np08Hcg3zn3e7/rOVTOuTudc+nh98Y1wGtRGuaE39MlZjY5/NR5wIbO7KNHH6EfwJPAk2a2DmgGvhqFR4Sx5lEgCXgl/NfGKufcN/0tqeOccy1mdivwMhAHPOmcW+9zWYfqdOAG4AMzyw0/d5dz7kUfaxK4DXgmfMBQBNzYmS+uj/6LiMSIaB5yERGRNhToIiIxQoEuIhIjFOgiIjFCgS4iEiMU6CIiMUKBLiISI/4X3CeMKi2dYcoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_sigmoid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sigmoid 函式將迴歸模型的輸出 $Xw$ 映射至 $[0, 1]$ 之間後我們就能獲得 $\\hat{p}$，並依據門檻值獲得 $\\hat{y}$。\n",
    "\n",
    "\\begin{equation}\n",
    "\\hat{y} = 1, \\quad if \\: \\hat{p} \\geq 0.5\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "\\hat{y} = 0, \\quad if \\: \\hat{p} < 0.5\n",
    "\\end{equation}\n",
    "\n",
    "截至於此，資料（Experiment）與任務（Task）已經被定義妥善，特徵矩陣 $X$ 外觀 `(m, n)`、目標向量 $y$ 外觀 `(m,)`、係數向量 $w$ 外觀 `(n,)`，通過將 $X$ 輸入 $h$ 來預測 $\\hat{y}$，$h$ 的組成則可以拆解成 Sigmoid 函式 $\\sigma$ 以及門檻值比較兩道程序，為了方便，我們將門檻值比較表示為階躍函式（Step function） $\\chi$。\n",
    "\n",
    "\\begin{align}\n",
    "\\hat{y} &= h(X; w) \\\\\n",
    "&= \\chi(\\sigma(Xw))\n",
    "\\end{align}\n",
    "\n",
    "其中，\n",
    "\n",
    "\\begin{equation}\n",
    "\\sigma(x) = \\frac{1}{1 + exp(-x)}\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "\\chi(z) = 1, \\quad if \\: z \\geq 0.5\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "\\chi(z) = 0, \\quad if \\: z < 0.5\n",
    "\\end{equation}\n",
    "\n",
    "接下來還需要定義評估（Performance），在數值預測任務中評估 $h$ 性能的方法是計算 $\\hat{y}$ 與 $y$ 之間的均方誤差（Mean squared error），但是在類別預測任務中則是計算 $\\hat{y}$ 與 $y$ 之間的誤分類觀測值個數，當誤分類數愈低，分類器的表現愈好。\n",
    "\n",
    "\\begin{equation}\n",
    "Min. \\sum_i \\mid \\hat{y}_i^{(train)} \\neq y_i^{(train)} \\mid\n",
    "\\end{equation}\n",
    "\n",
    "羅吉斯迴歸使用交叉熵（Cross-entropy）函式作為量測 $J(w)$，這個函式的組成有兩個部分。\n",
    "\n",
    "\\begin{equation}\n",
    "J(w) = -\\frac{1}{m}log(\\sigma(Xw)), \\quad if \\: y = 1\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "J(w) = -\\frac{1}{m}log(1-\\sigma(Xw)), \\quad if \\: y = 0\n",
    "\\end{equation}\n",
    "\n",
    "設計以交叉熵函式的巧妙之處在於讓誤分類的成本趨近無限大，亦即當真實的類別 $y$ 為 1，$\\sigma(Xw)$ 若離 0 比較近，預測為類別 0 的機率較高，則成本將趨近無限大；而當真實的類別 $y$ 為 0，$\\sigma(Xw)$ 若離 1 比較近，預測為類別 1 的機率較高則成本將趨近無限大。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_cross_entropy():\n",
    "    epsilon = 1e-5\n",
    "    h = np.linspace(epsilon, 1-epsilon) # 利用微小值 epsilon 避免 log(0) 的錯誤\n",
    "    y1 = -np.log(h)\n",
    "    y2 = -np.log(1 - h)\n",
    "    fig, ax = plt.subplots(1, 2, figsize = (8, 4))\n",
    "    ax[0].plot(h, y1)\n",
    "    ax[0].set_title(\"$y=1$\\n$-\\log(\\sigma(Xw))$\")\n",
    "    ax[0].set_xticks([0, 1])\n",
    "    ax[0].set_xticklabels([0, 1])\n",
    "    ax[0].set_xlabel(\"$\\sigma(Xw)$\")\n",
    "    ax[1].plot(h, y2)\n",
    "    ax[1].set_title(\"$y=0$\\n$-\\log(1-\\sigma(Xw))$\")\n",
    "    ax[1].set_xticks([0, 1])\n",
    "    ax[1].set_xticklabels([0, 1])\n",
    "    ax[1].set_xlabel(\"$\\sigma(Xw)$\")\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deZxcdZnv8c/T1VVdVb2nu9NZOhsEAiHsYREEBWREjOLCAO6iDjozuF1fzmXujOPMiDPO3NGrqCNyBXEbwIVRRFS4KCDIlgAxCxDIRvbupNd0V3VVV/3uH6eq0+mkSXe61lPf9+tVr6o6deqcp15JP/Wc5/c7p8w5h4iIiIifVBU7ABEREZFcU4EjIiIivqMCR0RERHxHBY6IiIj4jgocERER8R0VOCIiIuI7KnBERETEd1TgiIiIiO+owJGSYGbXm9lKMxs2s9uLHY+I+I+ZzTCz/zazQTPbambvLnZMkj/VxQ5AJGMncCPwRiBS5FhExJ++CSSAduA04Fdmtto5t664YUk+qIMjk2ZmdWaWMrPZY5YtM7NdZlY/nW075+52zv0c2DftQEWkbOUrz5hZLfBO4HPOuf3OuUeBe4D3TT9qKUUqcGTSnHP7gReAM8Ys/hLwL865gewCM7vXzHonuN1b6LhFpHzkMc8cD4w45zaMWbYaOCkfn0OKT0NUMlVP4yWeX5nZhcBS4B1jV3DOrShGYCLiG/nIM3VA/7hlfcC0us9SutTBkanKJh6Af8dr9yaKGI+I+E8+8sx+oGHcsgZg4DDrig+owJGpeho4w8zeCYSB/xq/gpn92sz2T3D7dcEjFpFyk488swGoNrPjxiw7FdAEY58y51yxY5AyYmY1eG3eXcBfOudyUrCYWTXekOnngQ7gL/DGy0dysX0RKR95zDN3Ag74CN5ZVPcB5+ksKn9SB0emxDk3DKwBtuQq6WT8PRADbgDem3n89zncvoiUiTzmmb/CuwxFJ3AHXvGk4san1MGRKTGzEPAycJVz7olixyMi/qM8I7mgDo5M1eeBx5R0RCSPlGdk2lTgyKSY2Rlm1gdcCHy82PGIiP8oz0guaYhKREREfEcdHBEREfEdFTgiIiLiOypwRERExHdU4IiIiIjvqMARERER31GBIyIiIr6jAqfCmdkWM3tDjrb1r2b2qXytP4XtPmVmJx1pmYhMTzHzRz7lIxblpcJTgSM5YWZtwPuBbx/t+mZ2rJkNmdnsMcveY2Y7zWzeFML5D+CfJ7FMRErABPngejNbaWbDZnZ7sWJRXipfKnAkVz4I3Oecix3t+s65jcAvgU8BmNlrgG8AVzjntk0hlnuAi8xs1hGWiUhp+CCH5o+dwI3AbcWMRXmpfKnAkVFmdqKZPWRmvWa2zszeOua1M8zsWTMbMLOfmNldZnbjmLe/CXh43PaCZvbFTBs7aWYuc/vT4dbP+Dfgo2a2DLgb+Khz7unM9vab2fzM449ltjUr8/yzZnYrgHMuDqwC3pjd6OGWiUju5Dp/OOfuds79HNiX4zhfLS8dNhaUl8qSChwBvD96vKOU+4GZeL8D8yMzW5L5Zd//Bm4HZgB3AG8ft4mTgRfHLbsRuAS4AGgCHsxs520TrI9z7hngKeBJ4FvOuR+PebkHqDMzA/4a79eGmzLPPwrcNGbd54FTx23+cMtEZJrylD/y5dXy0mFjUV4qT9XFDkBKxrlAHfAl51wa+J2Z3Qu8C/gd3v+Vm5z342V3m9lT497fBAxkn5hZPfAJ4JRsG9fMfgZc7ZzbZGYHrT/mfVVACkjjHTWN1ZuJ8c/wkkh2v5cBO5xzq8esOwDMPvjth10mItOX0/yRL0fKSxPForxUntTB8alMq9hNcHv0MG+ZA2zLJKesrcDczGs73MG/zDp+7LkHqB/z/EJgk3PupTHLmoHdE6yf9WW85PAS8J7D7KMObyz8a0B/Zpt/xcFHSWS23TuJZSIyTgnkj3zFeqS8NFEsyktlSAWOTznnXu+cswlurz3MW3YC8zJHKlnzgR3ALmBupuWaNf7sgT8Bx4953ob3hw9A5r1vB+6dYH3M7KOZda7AO0r67Lh99gLLgVnOuYfwEsmpwDLg5+PiORFYPYllIjJOCeSPfMV6pLx0SCzKS+VLBY5kPQkMAX+TmYT3euAtwJ3A43jt2evNrNrMrgDOHvf++4DXjXm+FjjDzE4zswjwr4AD7jrc+uZdS+NfgBXOuU7gp0AIL6lk9QCfBr6eed4PfBL4tnMuNWZbYeBM4IFXWyYiOZPr/EFm3TAQAAJmFjaz6U6rOFJeOigW5aXypgJHAHDOJfAS0puAvcB/Au93zr2Qee0dwIfxjlbei3fEMzxmE98HLs8kDZxzK4Ev4iWLTcAs4HLnXHL8+mZ2Al4ifJ9zbm3m/SngK8D/HLOPHryx/P/KPO/Haxv/33Ef5y3AQ865nUdYJiI5kOv8kfH3QAy4IfOeWGbZdOI8Ul4aG8uJKC+VNTt4WFRkcszsSeBm59x3xyz7F6DTOffVSW5jSutPMbYPZ5PSRMtEpDhykT/yKR+xKC8VngocmRQzex3eqZN78SbZ3Qwc45zbVdTARKTkKX9IMeg0cZmsJcCPgVq81u6VSk4iMknKH1JwR+zgmNltwAq8dt2yzLL/jTd2mAA2Atc653Sam4gcNeUaEcmlyUwyvh3vgkVjPQAsc86dAmwA/jbHcYlI5bkd5RoRyZEjDlE55x4xs4Xjlt0/5ukTwJWT2Vlra6tbuHDhEdcTkfK3atWqvc65tsmur1wjIkdjolyTizk4H+LgawgcxMyuA64DmD9/PitXrszBLkWk1JnZ1hxvUrlGRA4xUa6Z1nVwzOzvgBHgRxOt45y7xTm33Dm3vK1t0gdzIiKjlGtEZKqOuoNjZh/EmxB4idO55iKSJ8o1InI0jqrAMbPLgL8BXuecG8ptSCIiHuUaETlaRxyiMrM78H5LZImZbTezDwPfwPsF1AfM7DkzuznPcYqIzynXiEguTeYsqncdZvGteYhFRCqYco2I5FJJ/thmOu0YHB4hldZwu4iISKWIJ1PEk6kjrzgJJVng/PJPOznp87/llW4NuYuIiFSKf/rlOi7899/nZFslWeCEgwEAhhIjRY5ERERECmUokSISCuRkWyVZ4EQyBU6u2lQiIiJS+mKJ1GgNMF2lWeBkqrdYIl3kSERERKRQYsnU6CjOdJVmgZP5cDF1cERERCpGPOnzDk5YBY6IiEjFiSX9Pgcn8+HiCRU4IiIilSLm90nGUXVwREREKk48mfb3EFW2ehtSB0dERKRiDCVG/F3g1FR7YamDIyIiUjl8PwfHzIgEA7oOjoiISIVIpx3xZNrfp4mDN0wV0xCViIhIRRge8a595+shKvA+oIaoREREKkP2Oz8SzE1pUrIFTjhYpQJHRESkQmS/86Oh6pxsr2QLnGioWkNUIiIiFSL7nR/28yRjyAxRqcARERGpCNnvfN/PwQmHNAdHRESkUhyYg+PzAicSrNJp4iIiIhVitMAJ+XySsc6iEhERqRyjc3B838HRdXBEREQqRrxShqjC6uCIiIhUjIKfJm5mt5lZp5mtHbNshpk9YGYvZe6bcxLNGFF1cEQqSrFyjYiUhmKcRXU7cNm4ZTcADzrnjgMezDzPqUgwwEjakUylc71pESlNt1OEXCMipSHbwQkXapKxc+4RoHvc4iuA72Uefw94W06iGSM7yUjDVCKVoVi5RkRKQyyRosogFCjuWVTtzrldmce7gfaJVjSz68xspZmt7OrqmvQOsj+XHtcwlUgly3uuEZHSEEumiAQDmFlOtjftMsk55wD3Kq/f4pxb7pxb3tbWNuntRtTBEZEx8pVrRKQ0xJKp0eZGLhxtgbPHzGYDZO47cxZRhgocEaEAuUZESkM8URoFzj3ABzKPPwD8IjfhHJD9sa0hDVGJVLK85xoRKQ3ZIapcmcxp4ncAjwNLzGy7mX0Y+BJwqZm9BLwh8zynokHNwRGpJMXKNSJSGnJd4BzxajrOuXdN8NIlOYviMLJtKg1RiVSGYuUaESkNQ4lUzn6mAUr4SsaagyMiIlI54iUyyTjvRq+DoyEqERER34slCjwHp1hGr4OjDo6IiIjvFXyScbFoiEpERKRyVNwQlU4TFxER8b+KGaIKVBk11VXq4IiIiPicc65krmRcEJFQQNfBERER8bnhkTRpR2WcJg7ePBx1cERERPwte0JRRQxRQbbASRc7DBEREcmjbDOjYoaowsGAroMjIiLic9nv+milFDiRUIBYcqTYYYiIiEgeZTs4lTUHRx0cERERX6u8OTghzcERERHxu+w17ypmDk4kGNBPNYiIiPhcdrSmcjo4GqISERHxvcqbgxPSdXBERET8Ll6Rp4mrwBEREfG10dPEK6aDEwyQGEmTSrtihyIiIiJ5kj2hqGI6OJGQF566OCIiIv6V/Z6vqc5dWVLiBU41gCYai4iI+FgsMUIkGMDMcrbN0i5wMmNxOlVcRETEv2LJVE6Hp6BMChwNUYmIiPhXLJHO6TVwYJoFjpl92szWmdlaM7vDzMK5CgzGzMHREJVIRct3rhGR4oonU4SDue25HPXWzGwu8AlguXNuGRAArslVYHDggj9DKnBEKlYhco2IFFcsmSKamXebK9Mtl6qBiJlVA1Fg5/RDOkBzcEQkI6+5RkSKK5ZIlc4QlXNuB/AfwCvALqDPOXf/+PXM7DozW2lmK7u6uqa0j+yEI83BEalchcg1IlJcsWSKcKlMMjazZuAKYBEwB6g1s/eOX885d4tzbrlzbnlbW9uU9hEN6jRxkUpXiFwjIsXldXBKZA4O8AZgs3OuyzmXBO4GzstNWJ6wLvQnIgXINSJSXLFkCQ1R4bWLzzWzqHlX5rkEeD43YXk0B0dEKECuEZHiKqnr4DjnngR+CjwDrMls65YcxQUcOItKQ1QilasQuUZEiiueSBEJ5vYsqmltzTn3eeDzOYrlEMFAFcGAMaQOjkhFy3euEZHi8jo4pTMHpyDCwYA6OCIiIj6VTKUZSbuSmoNTEJFgQHNwREREfCp7IlG40gqcaCigs6hERER8KjtKUzKTjAtFQ1QiIiL+NVrgVFoHJ6IOjoiIiG9lv+Mrr8BRB0dERMS3RgucShuiigTVwREREfGreKUOUYU1RCUiIuJbFd3BiWuISkRExJcqdg6OThMXERHxr6FEhV4HR3NwRERE/CteqUNU4WCAeDJNOu2KHYqIiIjkWEVfBwcgPqIujoiIiN9U7Byc7AfWtXBERET8J5ZMUVNdRVWV5XS75VPgaB6OiIiI78QTqZzPv4EyKHDC2SEqFTgiIiK+E0umcj48BWVQ4ERHh6jSRY5EREREcm0oUaEFTrZtpSEqERER/4knUzm/Bg6UQYGT/dBDiZEiRyIiIiK5FkumiFbiHJxs20pzcERERPwnVqmTjDVEJSIi4l+xZLoyh6gimmQsIiLiW/FSPIvKzJrM7Kdm9oKZPW9mr8lVYFm6Do6IFCLXiEhxxPJ0FlX1NN//NeA3zrkrzSwERHMQ00Eiug6OiBQg14hIcQwlRvIyB+eoCxwzawQuBD4I4JxLAInchHVAMGAEqkxnUYlUqELlGhEpjngJzsFZBHQB3zWzZ83sO2ZWO34lM7vOzFaa2cqurq4p78TMiAQDmoMjUrkKkmtEpPBGUmkSqXTJnSZeDZwBfMs5dzowCNwwfiXn3C3OueXOueVtbW1HtaNwMKA5OCKVq2C5RkQKKz7iNS9KbZLxdmC7c+7JzPOf4iWhnIuEqjQHR6RyFSzXiEhhxRLed3u4lDo4zrndwDYzW5JZdAmwPidRjeMNUanAEalEhcw1IlJY2eZFKZ5F9XHgR5mzGjYB104/pENFNEQlUukKkmtEpLCGEiVa4DjnngOW5yiWCUVCKnBEKlmhco2IFFb2uz0Syv11h0v+SsagISoRERE/io12cKY7oHSo8ihw1MERERHxndE5OKU0ybiQwurgiIiI+E4sj5OMy6LAiQQDOk1cRETEZ2J5nGRcNgWOhqhERET8JfvdHq7YScaZOTjOuWKHIiIiIjmiDk4ogHMwPKLfoxIREfELzcHJfHBNNBYREfGPWDJFKFBFdaBSh6iyBY7m4YiIiPhGLJEiHMxPKVIeBU5IBY6IiIjfxJOpvFwDB8qkwAlriEpERMR3YslUXubfQJkUONkPr2vhiIiI+Ic3RFXJBY6GqERERHwnVulDVNkOzpCGqERERHwjlqj0IaqQhqhERET8JpZMEVUHR5OMRURE/CSWrPQ5OLoOjoiIiO/ENUSlAkdERMRvKn6ScU21F2ZcQ1QiIiK+UfHXwTEzIsGAzqISERHxiXTaEU+mK3sODkA0FNAQlYiIiE/ER7zv9Io+iwq8n2tQgSMiIuIP2TOjS3YOjpkFzOxZM7s3FwFNJBIK6Do4IhWsULlGRAoj27Qo5SGqTwLP52A7ryoSDOg6OCKVrSC5RkQKI9u0KMlJxmbWAbwZ+E5uwplYRENUIhWrkLlGRAojlkgDJVrgAF8F/gZIT7SCmV1nZivNbGVXV9dR7ygcChBLTrgbEfG3guUaESmMbNOi5ObgmNkKoNM5t+rV1nPO3eKcW+6cW97W1na0u6M2FGDvwDDOuaPehoiUn0LnGhEpjK6BYaA0z6I6H3irmW0B7gQuNrMf5iSqw7j4hJns6I3x2Mv78rULESlNBc01IlIYd63cRlt9DSfNaczL9o+6wHHO/a1zrsM5txC4Bvidc+69OYtsnLecOofWuhDffWxzvnYhIiWo0LlGRPLv5c4BHtnQxfvPXUCoOj9XrCmr6+C8+5wFPPhCJ5v3DhY7HBERETlK331sC6HqKt59zvy87SMnBY5z7iHn3IpcbOvVvPfc+QQDxu3q4ohUpELlGhHJn96hBD97ZjtvO20OLXU1edtP2XRwAGbWh3nLKXP4yart9MWSxQ5HREREpujOp7cRT6a59vxFed1PWRU4ANeev4ihRIqfrNxW7FBERERkCpKpNN/74xbOO7aFE2c35HVfZVfgnNzRyNkLZ3D7H7eQSuuUcRERkXLx23W72dUXz3v3BsqwwAG49vyFbO+J8cD6PcUORURERCbpu49tYUFLlItPmJn3fZVlgXPp0nbmNkW4TZONRUREysJz23pZtbWHD563kECV5X1/ZVngVAeq+MB5C3hqczdrd/QVOxwRERE5gu8+tpm6mmquPLOjIPsrywIH4Oqz5lNXU80/3rOOkZR+o0pERKRUPbFpH79cvZNrzppHfThYkH2WbYHTGAnyxbcvY+XWHr7ywIZihyMiIiKHsW//MJ+881kWttTyqUuPL9h+y7bAAbjitLlcc9Y8/vOhjTy8Qb8eLCIiUkrSacenf7yanqEk33j3GdTVVBds32Vd4AB8/i0nsaS9nv9x13Ps6Y8XOxwRERHJuPmRjTyyoYt/WLGUpXPye92b8cq+wImEAnzzPaczlEjxiTue1bVxRERESsDKLd18+f4NvPmU2bwnj785NZGyL3AAFs+s5wtvW8aTm7v52oMvFTscERGRitYzmODjdzxLR3OEL73jZMzyf1r4eL4ocACuPLODd57Rwdd/9xI/W7W92OGIiIhUpIF4ko/+YBX79if4xrvOKNhZU+MVbrZPAdz4tmXs7o/xmZ+sZiiZ4n3nLih2SCIiIhWjdyjBB257inU7+/k/V5/GyR2NRYvFNx0c8Obj3PqBs7jkhJl87udr+fbDG4sdkoiISEXoGhjmmlue4PldA3zrvWfyllPnFDUeXxU4AOFggJvfdyZvPmU2//rrF/jKAxtwThOPRURE8mVnb4yrv/04W/cNcesHl3Pp0vZih+SvIaqsYKCKm645nWgwwE0PvkR/LMnfvflEggHf1XMiIiJFtWHPANd+92n6Y0m+/+GzOWvhjGKHBPi0wAEIVBn/9s5TaIgEufXRzfxpey83vet0OpqjxQ5NRESk7DnnuPPpbfzTL9dRV1PNj/7iHE7paCp2WKN83dKoqjI+t2Ip33j36by0Zz+Xf+0P/GbtrmKHJSIiUtYG4kk+cedz/O3da1i+YAb3ffKCkipuwOcFTtaKU+bwq09cwMLWWj72w2f4h1+sJZ5MFTssERGRsrNmex8rvv4o963ZxWffuITvf+hsZtaHix3WISqiwAGY3xLlpx87j4+8dhHff3wrl331Ef1+lYiIyCTtHx7hxnvX87b/fIzkSJo7rzuXv75oMVVVhb+I32RUTIEDEKqu4u9XLOWHHz4HM+MDtz3FX/5wFTt7Y8UOTUREpCQ557hn9U4u+fJD3PrYZq5a3sF9n7ygZCYTT+SoJxmb2Tzg+0A74IBbnHNfy1Vg+fTa41r5zacu4Dt/2MzXf/cSD73YxccvWcyHzl9EOBgodngiMkY55xqRcrdhzwD/eM86/rhxH8vmNnDze8/k9PnNxQ5rUqZzFtUI8Bnn3DNmVg+sMrMHnHPrcxRbXtVUB/jrixbz1lPn8IV71/Pvv3mR7/1xC9dftJirzppHTbUKHZESUda5RqQcbezaz00PvsQ9q3dSX1PNF962jHefPZ9AiQ5HHc5RFzjOuV3ArszjATN7HpgLlFXSmTcjyi3vX87jG/fxlQde5HO/WMfND2/i+osXc+WZHbp2jkiR+SXXiJSDrfsGuenBl/nvZ7dTUx3gY687lusuOIbm2lCxQ5syy8VVfs1sIfAIsMw51z/uteuA6wDmz59/5tatW6e9v3xxzvHoy3v58v0beG5bL3ObIlx7/kKuOmseDUX6sTCRcmVmq5xzy3O8zYX4INeIlJrV23q59dHN/GrNLqqrjPedu4CPvf5YWutqih3aEU2Ua6Zd4JhZHfAw8EXn3N2vtu7y5cvdypUrp7W/QnDO8fsXO7n54U08tbmb2lCAq86ax7XnLWJ+iy4UKDIZuS5w/JhrRIoplXbcv243tz66mZVbe6irqebqs+bx0QuPYWZD6Z32PZGJcs20rmRsZkHgZ8CPjpRwyomZcfEJ7Vx8Qjtrtvdx66Ob+MHjW/neH7dw0ZKZXH3WPC46YaaGr0QKxK+5RqQYdvXF+OnK7dy1chvbe2LMmxHhcyuWctXyDup9NFoxnbOoDLgVeN4595XchVRaTu5o5KvXnM4NbzqRHzyxhR+v3M6DL3TSWlfDlWd2cPVZ81jUWlvsMEV8q1JyjUg+JVNpHny+k7uefoWHN3SRdnD+4hb+7vIT+bOTZpXV5OHJOuohKjN7LfAHYA2Qziz+X865+yZ6jx/axiOpNL9/sYu7nt7G71/sJJV2nDavibeeOocVp8wuq7aeSD7laoiqUnONyHSl046nt3Rzz+qd3LdmFz1DSdobavjzM+dx1fJ5vplykfMhKufco4D/Sr4jqA5UcenSdi5d2k5nf5y7n93BPc/t5J/vXc+Nv1rPuce08JZT53Dp0vaymJwlUuoqNdeIHI102vHc9l5+vWYX9/5pF7v64kSCAd6wtJ23nz6HC49ro7pCplfk5CyqyfLzUdXLnQPc89xO7lm9ky37hjCD5QuaeeNJs3jjSbOYN8MflbLIZOXjLKrJ8nOuERkvMZLmiU37+O263Tywfg+dA8NUVxmvO76Nt57mHXBHQ9OaclvS8nYW1VRUQtJxzrF+Vz/3r9vDb9ft5oXdAwAsaa/n9Se0cdGSmZy5oFkTlMX3VOCI5E9nf5yHNnTx0Iud/OGlvQzER4gEA7zu+DbeuKydi5e00xj1z4ThV5OXs6jkUGbGSXMaOWlOI5++9Hi27hvkgfV7+N0Lndz26Ga+/fAm6muqOX9xKxcc38prF7cyf0YUbx6liIjIoeLJFM9s7eGxjXt5eEMXa3d4l4Fqb6jh8mWzecPSdi44rlU/NzSGCpw8W9BSy0cuOIaPXHAMA/Ekj728j4c3dPLQi138Zt1uADqaI7x2cSuvObaFc49poV0TlUVEKtpIKs3anf08sWkfj728l6c2dzM8kiZQZZwxv4nPvnEJFy2ZyYmz63WAPAEVOAVUHw5y2bJZXLZsFs45Nu0d5LGX9/LYy3v51Zpd3Pn0NgAWtkQ5Z1EL5xwzg7MWzqCjOaL/wCIiPhZPpli7o48nN3fz5OZuVm3pZjCRArwpDu85ZwHnL27h7EUzfHWtmnxSgVMkZsaxbXUc21bH+1+zkJFUmvW7+nlyUzdPbt7Hr9fu4q6VXsHT3lDDmQuaOWN+M8sXzmDp7AZC1ZrDIyJSrroGhlm1tYdnXulh5ZZu1u7oJ5HyroJwfHsd7zijg3OOmcHZi2Yws15d/aOhAqdEVAeqOKWjiVM6mviLC48hnXa8sHuAVVu7Wbm1h1Vbe7hvjTekFQpUsXROA6fNa+K0eU2cOq+JBTOiVPnwQk0iIuVucHiEdTv7Wb2tl+cytx29McDL5yd3NHLt+Qs5fX4zZy1spkWXGMkJFTglqqrKWDqngaVzGnjfaxYCsKc/zqqtPaN/IHc9vY3b/7gFgPpwNcvmNHJyRyPL5jaybE4DC1tqVfSIiBTQ/uERnt/Vz5rtfazd0ceaHX1s7NpPOnPCckdzhNPmN2UKmiaWzW2kploTg/NBBU4ZaW8Ic/nJs7n85NmANwntpc79rN7Wy5od3h/T7Y9tGW1z1oYCnDjbK5JOmtPACbMaOL69nkhIf0wiItPhnGN3f5wXdg+wfme/d9vVz5Z9g2SvvjKzvoaT5zZy+cmzOXluI6fOa6KtXt2ZQlGBU8aqA1WcOLuBE2c3cE1mWWIkzYY9A6N/bOt29nH3Mzv4/uNbATCDhS21nDCrniWz6lnSXs9x7fUsbIlWzNUtRUSmoi+W5KU9A7y4Z4ANuwd4fvcAL+4eoC+WHF1n/owoS2c38I7T57J0TgPL5jbqjNgiU4HjM6HqKm+Iam7j6LJ02vFK9xAv7B7ghd39vLBrgOd39fObdbtHjzRCgSqOaavl+PZ6Fs+sY/HMOo6bWceCllpNaBaRitA9mODlzv283LmflzoHvPs9+9ndHx9dpzYUYMmset58ymxOnFXPCbMbWDKrngad2VRyVOBUgKoqY2FrLQtba7ls2azR5UOJETZ2DvLinoHRo5NnXunhntU7R9cJVBkLZkQ5pq2OY9tqObatjmPaalnUWrW/wvcAAAwGSURBVMuM2pBOXxeRspJMpdnWPcSmrkE27d3Pxs7Mfdcg3YOJ0fUiwQDHzqzlvGNbOH5WPce313HczHrmNkU0t7FMqMCpYNFQNSd3eBOTxxpKjLCpa3D0CGZT1yAbu/bzyIau0fk94E1sPiZTOC1sqWVha5QFLbUsaqmlKRpU8SMiRTGSSrO9J8aWfYNs3TfE5r2DbNk3yJa9g2zriZFKH/iJota6EMe01vFnS9s5tq2Oxe11LG6rUyHjAypw5BDRUPUhw1wAqbRje4935LN57+Bo0li5xev6jP1Zs4ZwNQtaapk/I8r8ligLZkSZPyPKvBlRZjeGNd9HRKalP55kW/cQ27pjvNLtFTKvdHu3HT0xRsYUMZFggEWttZw0t5EVp8xhUWsti9pqOba1rmJ+r6kSqcCRSQtUGQtaalnQUstF416LJ1Ns7xliy94htnYPsWXvIK90D3k/PLp+N8mUO2g7c5rCzGuOMq85SkdzhLnNEToyj9sbwgR05CRS0QaHR9jRG2N7zxDbe2KZm1fQbOsZoncoedD6TdEgC2ZEOXluIytOmZ3pKteyoCVKW12NOsoVSAWO5EQ4GGDxzHoWz6w/5LVU2rGrL8Yr+4bYNiZBvdI9xO9e7KRrYPig9aurjFmNYeY2Rbxbc4Q5Td5tblOY2Y0Ramv0X1ekXKXTjr37h9nRG2NXX5ydvV4Bs7M3xo7MbXwBE6quoqMpQseMKKfOa/QOkGZ4B0nzW6I0RtSJkYPpW0LyLlBlme5M9LCvx5MpL6n1eIXPjp7Y6PPHN+1jT3+cMd1mABojQWY3hr1bU4TZDd79rIYwszLLVQSJFF467dg7OMzuvji7+uLs7ouzsy/mPe/1Hu/pjx/U1QWIhgKjBzSnzWs6qKvb0RyhtbZGc2JkSvQNIEUXDgZGf5frcJKpNHv646NHejt6Y+zqjbOrzzv6W72976CzH7Lqa6ppbwwzqyFMe0OYWY01zGoIMzP7vCFMa11I84FEJmn/8Ah7+uPs6Y/T2T/M7n6vgNnTH2d3f5w9fXE6B4YPmv8CEAxY5sAjwpkLmpndeKAbOyfTqW2IVGsYSXJKBY6UvGCg6lU7QOB1gbJF0IEjx5iXgPuH2bhxL50DwwedPQHehQ9bamuYWV/DzIYa2uvDzGyooa3eW9ZWH87c1xAO6grQ4j/OOXqHknQODNM1MEzngFekdPYfeNw1MExnf3z0163Hqquppr2hhlmNYc49toVZDV4HdVaj11Ftb6xR90WKQgWO+EI4GBidAD2RVNqxb9BL3Lv74uwZiLOnf5iuzH3nQJz1O/vZu3/4kCEx8E6Lb6uroTVT8LTV1dBaF6KtvobWusytvoaW2pCKISkq5xx9sSR79yfYu98rUPbuz9wGEnSNWzZ+uAi8ISOv8A9z0pwGLloyk/aGGtobvIOA9gav+K/XBe6kRKnAkYoRqDJm1oeZWR8+5BT4sbKFkHc0692P3jJfDOt39rN3YJiB4ZHDbqO+pprWeq8AmlEboqWuhtbMvfc8RGvmcXM0pLPG5IhiiRT7BofZtz8x5j7Bvv0HHu/df+D1wxUtgSqjpdYrytvqa1gyq360WJ/ZkL0P01ZfQ53msEmZ0/9gkXHGFkInHWHdeDKVOQpOsDdzRLxvMHHQ0fHmvd61grqHEgddKyjLDJoiQa/wqc0UPbUhWmq94mjsrbk2xIxoSD+YWubSaa/D0j2UoGfQK07G3ndnHw8l2Lffex5LHjo8BFBTXZXpIIZoz3RbWrIdxUwhnX3cHA1pqEgqxrQKHDO7DPgaEAC+45z7Uk6iEikT4WDgiPODslJpR/fol9ewd5858u4e83xj1366t3hfbocbKvP2W0Vz1PvCaq4NHngcDdKUWdYU9YqhpsyyhnD5TuIs5VyTGEnTG0vQO5SkZzBBz1CS3qGx9wm6B73H3UPeer2v8m8bCQYOKmoXt9V5j+tCtNYe6AC21NbQUhciGgqU7b+rSD4ddYFjZgHgm8ClwHbgaTO7xzm3PlfBifhJoMpGhwbg0OsFjTf2KL87c2Sf/bLMHtlnv0B39vbTM5SgL5Y8bJcou//GSNAreCJeUdQYDdIUyRZBwczrIZoiwdF168PBog6hFSrXxJMp+mJJ+mLJ0SKkN5akbyg5WsD0xjLLhw6sc7iJt1mh6qrRInNGbYgTZzXQFA3Sku3G1YZoioYOPFd3TiRnptPBORt42Tm3CcDM7gSuAFTgiORAVZXRnPniO7Ztcu9JZYqinqFM8TN48JdyT+a+L5Zkd3+cF3YP0BdLsn+CuUTgDaHV11TTGA3ypXecwvmLW3P0CSctb7nmU3c+y+Ob9tE7lGR4JD3heoEq84q+qFcYzmoIs6S93uuWRYM01YZGi8amaND7d4sGiQTVXREplukUOHOBbWOebwfOGb+SmV0HXAcwf/78aexORI4kUGWjQxtTkUylRzsXfTGvMBrtZsSS9GeKpObo1LabI3nLNYtn1lFTHaApGqQhMqaLlelqZbtYdTXlO7wnUqnyPsnYOXcLcAvA8uXLJ2iei0gxBQNVo5NRy9XR5JrrLz4urzGJSPFM5xKuO4B5Y553ZJaJiOSSco2ITNl0CpyngePMbJGZhYBrgHtyE5aIyCjlGhGZsqMeonLOjZjZ9cBv8U7dvM05ty5nkYmIoFwjIkdnWnNwnHP3AfflKBYRkcNSrhGRqdLPKIuIiIjvqMARERER31GBIyIiIr5jbqLruudjZ2ZdwNZJrt4K7M1jOCIydVP5u1zgnJvkNZhzS7lGpKxN9W/ysLmmoAXOVJjZSufc8mLHISIH+PHv0o+fSaSc5epvUkNUIiIi4jsqcERERMR3SrnAuaXYAYjIIfz4d+nHzyRSznLyN1myc3BEREREjlYpd3BEREREjooKHBEREfGdkixwzOwyM3vRzF42sxuKHY9IJTOz28ys08zWFjuWXFOuESkduc41JVfgmFkA+CbwJmAp8C4zW1rcqEQq2u3AZcUOIteUa0RKzu3kMNeUXIEDnA287Jzb5JxLAHcCVxQ5JpGK5Zx7BOgudhx5oFwjUkJynWtKscCZC2wb83x7ZpmISC4p14j4WCkWOCIiIiLTUooFzg5g3pjnHZllIiK5pFwj4mOlWOA8DRxnZovMLARcA9xT5JhExH+Ua0R8rOQKHOfcCHA98FvgeeDHzrl1xY1KpHKZ2R3A48ASM9tuZh8udky5oFwjUlpynWv0Uw0iIiLiOyXXwRERERGZLhU4IiIi4jsqcERERMR3VOCIiIiI76jAEREREd9RgSMiIiK+owJHREREfEcFjuSEmUXM7GEzC2Sef8zMvjXm9RvN7AeT2E7IzB4xs+p8xisi5Ud5RqZCBY7kyoeAu51zqczz7wMrzKzJzFYAbwauO9JGnHMJ4EHg6rxFKiLlSnlGJk0FjkyJmZ2aOfJZb2ZpM3Nm9s/Ae4BfZNdzzg0BdwBfBG4CrnTOxcxsoZk9ktnWGZn3t5pZwMzWmFkU+HlmeyJSgZRnJBfUnpNJM7MwcBfwfufcU2b2BSCMl1yuc85tGfeW2/B+4+cK59zGzLJeoC7z+OPAE0ATcB7w/5xzQ2a2Fjgrrx9GREqS8ozkijo4MhVvAJ5xzj2Vef4nYAbQgpdQxvsHoIuDC+l+IGpmrcBs4DGgGa+t/C2ATPs5YWb1+fgQIlLSlGckJ1TgyFQsA9aMeX4G8AwQwzvCGmVmn8ksuwr4ZHa5cy4NOOAjwK3AAHAqEHDObRiziRognvuPICIlTnlGckJDVDIV+4CLAczseOAdwHnOuZ7M2HbYORc3s4uBa4HXOOcGzKzBzE5zzj2X2U4aeCtwIdABfAa4IbsTM2sB9jrnkoX7aCJSIpRnJCfUwZGpuAOoy4xd3wK8yzm3L/Pa/cBrzWw+8B3gz51zA5nXvgZ8asx2ksCvnXMjZFrJwL1jXr8I+FX+PoaIlDDlGckJc84VOwbxATM7A/i0c+59OdjW3cAN41rJIlLhlGdkKtTBkZxwzj0D/D57Aa6jZWYh4OdKOiIynvKMTIU6OCIiIuI76uCIiIiI76jAEREREd9RgSMiIiK+owJHREREfEcFjoiIiPiOChwRERHxnf8PAUtY01JCwjoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_cross_entropy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "將 $y$ 與 $1-y$ 加入 $J(w)$，可以把兩個情境（$y=0$ 或 $y=1$）的成本函數合而為一。\n",
    "\n",
    "\\begin{equation}\n",
    "J(w) = \\frac{1}{m}(-ylog(\\sigma(Xw)) - (1-y)log(1-\\sigma(Xw)))\n",
    "\\end{equation}\n",
    "\n",
    "當 $y=1$ 時，$J(w)$ 只剩下前項；當 $y=0$ 時，$J(w)$ 只剩下後項，巧妙的交叉熵函式特性依然被具體地留存下來，合而為一的優點在於便利接下來應用「梯度遞減」演算方法。我們希望可以運用在迴歸模型中提到的「梯度遞減」演算方法找到一組係數向量 $w$，這組係數向量能夠讓 $J(w)$ 儘可能降低，根據梯度遞減的演算方法，下一步需要求解 $J(w)$ 關於 $w$ 的偏微分。\n",
    "\n",
    "\\begin{equation}\n",
    "w := w - \\alpha \\frac{\\partial J}{\\partial w}\n",
    "\\end{equation}\n",
    "\n",
    "想順利求解 $J(w)$ 關於 $w$ 的偏微分之前，得具備三個先修知識：\n",
    "\n",
    "1. 連鎖法則（Chain rule）\n",
    "2. $e^{x}$ 關於 $x$ 的微分\n",
    "3. $log(x)$ 關於 $x$ 的微分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們發現 $J(w)$ 函式是一個由多個不同函式複合而成的，先是結合了 Sigmoid 函式 $\\sigma$；再來是 $log$ 函式，欲求解複合函式偏微分，就得仰賴連鎖法則。假設 $f$ 和 $g$ 為兩個關於 $x$ 的可微函式，其複合函式關於 $x$ 的微分：\n",
    "\n",
    "\\begin{align}\n",
    "(f\\circ g)(x) &= f(g(x)) \\\\\n",
    "(f\\circ g)'(x) &= f'(g(x))g'(x)\n",
    "\\end{align}\n",
    "\n",
    "$e^{x}$ 關於 $x$ 的微分：\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{d}{dx}e^{x} = e^{x}\n",
    "\\end{equation}\n",
    "\n",
    "$log(x)$ 關於 $x$ 的微分：\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{d}{dx}log(x) = \\frac{1}{x}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "具備先修知識以後，接下來要推導 $J(w)$ 關於 $w$ 的偏微分：\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial}{\\partial w}J &= \\frac{\\partial}{\\partial w} (-ylog(\\sigma(Xw)) - (1-y)log(1-\\sigma(Xw))) \\\\\n",
    "&= -y\\frac{\\partial}{\\partial w}log(\\sigma(Xw)) - (1-y)\\frac{\\partial}{\\partial w}(log(1-\\sigma(Xw)))\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "將前式拆解成兩部分，首先計算 $log(\\sigma(Xw))$ 關於 $w$ 的微分：\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial}{\\partial w}log(\\sigma(Xw)) &= \\frac{\\partial}{\\partial w}log(\\sigma(Xw)) \\cdot \\frac{\\partial}{\\partial w}(\\sigma(Xw)) \\\\\n",
    "&= \\frac{1}{\\sigma(Xw)} \\cdot \\sigma'(Xw) \\cdot \\frac{\\partial}{\\partial w}Xw \\\\\n",
    "&= \\frac{1}{\\sigma(Xw)} \\cdot \\sigma'(Xw) \\cdot X\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再接著計算 $log(1-\\sigma(Xw))$ 關於 $w$ 的微分：\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial}{\\partial w}log(1-\\sigma(Xw)) &= \\frac{\\partial}{\\partial w}log(1-\\sigma(Xw)) \\cdot \\frac{\\partial}{\\partial w}(1-\\sigma(Xw)) \\\\\n",
    "&=\\frac{1}{1-\\sigma(Xw)} \\cdot (-\\sigma'(Xw) \\cdot \\frac{\\partial}{\\partial w}Xw) \\\\\n",
    "&=\\frac{1}{1-\\sigma(Xw)} \\cdot (-\\sigma'(Xw) \\cdot X)\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "兩部分都得先計算 $\\sigma'(Xw)$ 也就是 Sigmoid 函式關於 $w$ 的微分，才能夠繼續推導。\n",
    "\n",
    "\\begin{align}\n",
    "\\sigma'(Xw) &= \\frac{\\partial}{\\partial w} \\frac{1}{1 + e^{-Xw}} = \\frac{\\partial}{\\partial w} (1 + e^{-Xw})^{-1} \\\\\n",
    "&= \\frac{-\\frac{\\partial}{\\partial w}(1 + e^{-Xw})}{(1 + e^{-Xw})^2}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分子部分我們得先推導 $e^{-x}$ 關於 $x$ 的微分。\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{d}{dx}e^{-x} = \\frac{d}{dx}\\frac{1}{e^x} = \\frac{-\\frac{d}{dx} e^x}{(e^x)^2} = \\frac{-e^x}{(e^x)^2} = \\frac{-1}{e^x} = -e^{-x}\n",
    "\\end{equation}\n",
    "\n",
    "於是 $\\sigma'(Xw)$ 就可以寫成：\n",
    "\n",
    "\\begin{align}\n",
    "\\sigma'(Xw) &= \\frac{-\\frac{\\partial}{\\partial w}e^{-Xw}}{(1 + e^{-Xw})^2} = \\frac{e^{-Xw}}{(1 + e^{-Xw})^2} \\\\\n",
    "&= \\frac{e^{-Xw}}{(1 + e^{-Xw}) \\cdot (1 + e^{-Xw})}\n",
    "\\end{align}\n",
    "\n",
    "接下來的推導有些狡猾，需要在分子設計一個 `+1-1`。\n",
    "\n",
    "\\begin{align}\n",
    "\\sigma'(Xw) &= \\frac{e^{-Xw}}{(1 + e^{-Xw}) \\cdot (1 + e^{-Xw})} \\\\\n",
    "&= \\frac{1}{1 + e^{-Xw}} \\cdot \\frac{e^{-Xw} + 1 - 1}{1 + e^{-Xw}} = \\frac{1}{1 + e^{-Xw}} \\cdot ( \\frac{1 + e^{-Xw}}{1 + e^{-Xw}} - \\frac{1}{1 + e^{-Xw}}) \\\\\n",
    "&=\\frac{1}{1 + e^{-Xw}} \\cdot ( 1 - \\frac{1}{1 + e^{-Xw}}) \\\\\n",
    "&=\\sigma(Xw)(1-\\sigma(Xw))\n",
    "\\end{align}\n",
    "\n",
    "推導出 $\\sigma'(Xw)$，再回去計算未完的兩部分。\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial}{\\partial w}log(\\sigma(Xw)) &= \\frac{1}{\\sigma(Xw)} \\cdot \\sigma'(Xw) \\cdot X \\\\\n",
    "&= \\frac{1}{\\sigma(Xw)}\\sigma(Xw)(1-\\sigma(Xw))X \\\\\n",
    "&= (1-\\sigma(Xw))X\n",
    "\\end{align}\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial}{\\partial w}log(1-\\sigma(Xw)) &= \\frac{1}{1-\\sigma(Xw)} \\cdot (-\\sigma'(Xw)) \\cdot X\\\\\n",
    "&=\\frac{1}{1-\\sigma(Xw)}(-(\\sigma(Xw)(1-\\sigma(Xw)))X) \\\\\n",
    "&=-\\sigma(Xw)X\n",
    "\\end{align}\n",
    "\n",
    "最後回到 $J(w)$ 關於 $w$ 的偏微分。\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial J}{\\partial w} &= \\frac{1}{m}(-y(1-\\sigma(Xw))X - (1-y)(-\\sigma(Xw)X)) \\\\\n",
    "&=\\frac{1}{m}(-X^Ty + y\\sigma(Xw)X + X^T\\sigma(Xw) - y\\sigma(Xw)X) \\\\\n",
    "&=\\frac{1}{m}(-X^Ty + X^T\\sigma(Xw)) \\\\\n",
    "&=\\frac{1}{m}(X^T(\\sigma(Xw) - y))\n",
    "\\end{align}\n",
    "\n",
    "我們終於將「梯度」的公式推導完畢，與迴歸模型相同，在迭代過程中 $w$ 更新的方向性取決於梯度正負號，如果梯度為正，$w$ 會向左更新（減小）；如果梯度為負，$w$ 會向右更新（增大）。\n",
    "\n",
    "\\begin{equation}\n",
    "w := w - \\alpha \\frac{1}{m}(X^T(\\sigma(Xw) - y))\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自訂羅吉斯迴歸類別 LogitReg\n",
    "\n",
    "我們可以依據羅吉斯迴歸的定義自訂預測器類別，檢視迭代後的 $w$ 是否與 Scikit-Learn 相近、交叉熵是否隨著迭代而下降。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LogitReg:\n",
    "    \"\"\"\n",
    "    This class defines the vanilla descent algorithm for logistic regression.\n",
    "    Args:\n",
    "        fit_intercept (bool): Whether to add intercept for this model.\n",
    "    \"\"\"\n",
    "    def __init__(self, fit_intercept=True):\n",
    "        self._fit_intercept = fit_intercept\n",
    "    def sigmoid(self, X):\n",
    "        \"\"\"\n",
    "        This function returns the Sigmoid output as a probability given certain model weights.\n",
    "        \"\"\"\n",
    "        X_w = np.dot(X, self._w)\n",
    "        p_hat = 1 / (1 + np.exp(-X_w))\n",
    "        return p_hat\n",
    "    def find_gradient(self):\n",
    "        \"\"\"\n",
    "        This function returns the gradient given certain model weights.\n",
    "        \"\"\"\n",
    "        m = self._m\n",
    "        p_hat = self.sigmoid(self._X_train)\n",
    "        X_train_T = np.transpose(self._X_train)\n",
    "        gradient = (1/m) * np.dot(X_train_T, p_hat - self._y_train)\n",
    "        return gradient\n",
    "    def cross_entropy(self, epsilon=1e-06):\n",
    "        \"\"\"\n",
    "        This function returns the cross entropy given certain model weights.\n",
    "        \"\"\"\n",
    "        m = self._m\n",
    "        p_hat = self.sigmoid(self._X_train)\n",
    "        cost_y1 = -np.dot(self._y_train, np.log(p_hat + epsilon))\n",
    "        cost_y0 = -np.dot(1 - self._y_train, np.log(1 - p_hat + epsilon))\n",
    "        cross_entropy = (cost_y1 + cost_y0) / m\n",
    "        return cross_entropy\n",
    "    def fit(self, X_train, y_train, epochs=10000, learning_rate=0.001):\n",
    "        \"\"\"\n",
    "        This function uses vanilla gradient descent to solve for weights of this model.\n",
    "        Args:\n",
    "            X_train (ndarray): 2d-array for feature matrix of training data.\n",
    "            y_train (ndarray): 1d-array for target vector of training data.\n",
    "            epochs (int): The number of iterations to update the model weights.\n",
    "            learning_rate (float): The learning rate of gradient descent.\n",
    "        \"\"\"\n",
    "        self._X_train = X_train.copy()\n",
    "        self._y_train = y_train.copy()\n",
    "        m = self._X_train.shape[0]\n",
    "        self._m = m\n",
    "        if self._fit_intercept:\n",
    "            X0 = np.ones((self._m, 1), dtype=float)\n",
    "            self._X_train = np.concatenate([X0, self._X_train], axis=1)\n",
    "        n = self._X_train.shape[1]\n",
    "        self._w = np.random.rand(n)\n",
    "        n_prints = 10\n",
    "        print_iter = epochs // n_prints\n",
    "        for i in range(epochs):\n",
    "            cross_entropy = self.cross_entropy()\n",
    "            gradient = self.find_gradient()\n",
    "            if i % print_iter == 0:\n",
    "                print(\"epoch: {:6} - loss: {:.6f}\".format(i, cross_entropy))\n",
    "            self._w -= learning_rate*gradient\n",
    "        w_ravel = self._w.ravel().copy()\n",
    "        self.intercept_ = w_ravel[0]\n",
    "        self.coef_ = w_ravel[1:].reshape(1, -1)\n",
    "    def predict_proba(self, X_test):\n",
    "        \"\"\"\n",
    "        This function returns predicted probability with weights of this model.\n",
    "        Args:\n",
    "            X_test (ndarray): 2d-array for feature matrix of test data.\n",
    "        \"\"\"\n",
    "        m = X_test.shape[0]\n",
    "        if self._fit_intercept:\n",
    "            X0 = np.ones((m, 1), dtype=float)\n",
    "            self._X_test = np.concatenate([X0, X_test], axis=1)\n",
    "        p_hat_1 = self.sigmoid(self._X_test).reshape(-1, 1)\n",
    "        p_hat_0 = 1 - p_hat_1\n",
    "        proba = np.concatenate([p_hat_0, p_hat_1], axis=1)\n",
    "        return proba\n",
    "    def predict(self, X_test):\n",
    "        \"\"\"\n",
    "        This function returns predicted label with weights of this model.\n",
    "        Args:\n",
    "            X_test (ndarray): 2d-array for feature matrix of test data.\n",
    "        \"\"\"\n",
    "        proba = self.predict_proba(X_test)\n",
    "        y_pred = np.argmax(proba, axis=1)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:      0 - loss: 0.904725\n",
      "epoch:  10000 - loss: 0.275947\n",
      "epoch:  20000 - loss: 0.274739\n",
      "epoch:  30000 - loss: 0.274683\n",
      "epoch:  40000 - loss: 0.274680\n",
      "epoch:  50000 - loss: 0.274680\n",
      "epoch:  60000 - loss: 0.274680\n",
      "epoch:  70000 - loss: 0.274680\n",
      "epoch:  80000 - loss: 0.274680\n",
      "epoch:  90000 - loss: 0.274680\n",
      "-1.7162124454268828\n",
      "[[-2.75507425  1.96609802]]\n"
     ]
    }
   ],
   "source": [
    "h = LogitReg()\n",
    "h.fit(X_train, y_train, 100000, 0.01)\n",
    "print(h.intercept_)\n",
    "print(h.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['F', 'G', 'G', 'F', 'F', 'F', 'G', 'F', 'G', 'F']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = h.predict(X_valid)\n",
    "y_pred_label = [pos_dict[i] for i in y_pred]\n",
    "y_pred_label[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "比對 $w$ 與前十筆預測值可以驗證自行定義的 `LogitReg` 類別與 Scikit-Learn 求解的邏輯相近。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二元分類延伸至多元分類：One versus rest\n",
    "\n",
    "暸解羅吉斯回歸如何進行二元分類預測任務後，最後將問題還原回本來的多元分類問題，原始資料集中球員的鋒衛位置不只分作前鋒（Forward, F）與後衛（Guard, G）亦有中鋒（Center, C），以及能夠勝任兩個位置的搖擺人（F-G、G-F）等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['C' 'C-F' 'F' 'F-C' 'F-G' 'G' 'G-F']\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "pos = player_stats['pos'].values\n",
    "print(np.unique(pos))\n",
    "print(np.unique(pos).size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "將二元分類延伸至多元分類的技巧是直觀的，例如要面對的類別預測任務 $h$ 的目標從輸出 $\\hat{y} = \\{0, 1\\}$ 成為了 $\\hat{y} = \\{0, 1, ..., 5, 6\\}$，要依據場均助攻、場均籃板將球員分類為 7 個鋒衛位置其中之一。這時可以採取一種 One versus rest（亦稱 One versus all）的技巧，操作方式是訓練 7 個羅吉斯迴歸分類器，每個鋒衛位置一個，輸出預測的類別機率，再以 $argmax$ 函式決定分類預測。\n",
    "\n",
    "\\begin{align}\n",
    "\\hat{p}_{C} = \\hat{p}(y=0|X; w) = 1 - \\hat{p}(y \\neq 0|X; w) \\\\\n",
    "\\hat{p}_{C-F} = \\hat{p}(y=1|X; w) = 1 - \\hat{p}(y \\neq 1|X; w)  \\\\\n",
    "\\hat{p}_{F} = \\hat{p}(y=2|X; w) = 1 - \\hat{p}(y \\neq 2|X; w)  \\\\\n",
    "\\hat{p}_{F-C} = \\hat{p}(y=3|X; w) = 1 - \\hat{p}(y \\neq 3|X; w)  \\\\\n",
    "\\hat{p}_{F-G} = \\hat{p}(y=4|X; w) = 1 - \\hat{p}(y \\neq 4|X; w)  \\\\\n",
    "\\hat{p}_{G} = \\hat{p}(y=5|X; w) = 1 - \\hat{p}(y \\neq 5|X; w)  \\\\\n",
    "\\hat{p}_{G-F} = \\hat{p}(y=6|X; w) = 1 - \\hat{p}(y \\neq 6|X; w)  \\\\\n",
    "\\hat{p} = argmax (\\hat{p}_{C}, \\hat{p}_{C-F}, \\hat{p}_{F}, \\hat{p}_{F-C}, \\hat{p}_{F-G}, \\hat{p}_{G}, \\hat{p}_{G-F})\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 'G', 1: 'C', 2: 'C-F', 3: 'F-C', 4: 'F', 5: 'F-G', 6: 'G-F'}\n",
      "{'G': 0, 'C': 1, 'C-F': 2, 'F-C': 3, 'F': 4, 'F-G': 5, 'G-F': 6}\n",
      "[0 1 2 3 4 5 6]\n"
     ]
    }
   ],
   "source": [
    "unique_pos = player_stats['pos'].unique()\n",
    "pos_dict = {i: p for i, p in enumerate(unique_pos)}\n",
    "pos_dict_reversed = {v: k for k, v in pos_dict.items()}\n",
    "pos_multiple = player_stats['pos'].map(pos_dict_reversed)\n",
    "print(pos_dict)\n",
    "print(pos_dict_reversed)\n",
    "print(np.unique(pos_multiple))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 Scikit-Learn 定義好的 `LogisticRegression` 類別只需要在初始化時加入參數 `multi_class='ovr'` 就能面對多元分類問題。這時`predict_proba` 方法輸出的數值陣列外觀為 `(m, n)`，其中 `m` 是特徵矩陣的觀測值個數，`n` 則是目標向量的獨一值，也就是類別的個數，以目前的多元分類（C、C-F、F、F-C、F-G、G、G-F）問題來說，`n` 等於 7，第 0 欄是預測為類別 0（C）的機率 $\\hat{p}(y=0|X; w)$、第 1 欄是預測為類別 1（C-F）的機率 $\\hat{p}(y=1|X; w)$、第 6 欄是預測為類別 6（G-F）的機率 $\\hat{p}(y=6|X; w)$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00948771, 0.10618846, 0.10813516, 0.17693642, 0.43670401,\n",
       "        0.06700586, 0.09554238],\n",
       "       [0.77748262, 0.00158668, 0.00195679, 0.00939342, 0.06109545,\n",
       "        0.03020358, 0.11828145],\n",
       "       [0.36334642, 0.01957374, 0.03931401, 0.08027655, 0.28359635,\n",
       "        0.04937886, 0.16451407],\n",
       "       [0.20955361, 0.02755559, 0.06565499, 0.11284797, 0.35253646,\n",
       "        0.05151994, 0.18033144],\n",
       "       [0.23437795, 0.03348503, 0.05061705, 0.10472033, 0.35048298,\n",
       "        0.06369875, 0.16261792],\n",
       "       [0.02200364, 0.08355364, 0.10016853, 0.16809991, 0.44064353,\n",
       "        0.06784798, 0.11768277],\n",
       "       [0.4660762 , 0.00868968, 0.03326444, 0.06041734, 0.22125811,\n",
       "        0.03232719, 0.17796704],\n",
       "       [0.15965737, 0.05316157, 0.0510323 , 0.11778835, 0.39030225,\n",
       "        0.08309005, 0.14496812],\n",
       "       [0.45630382, 0.01006995, 0.0336024 , 0.06276895, 0.22902161,\n",
       "        0.03480357, 0.1734297 ],\n",
       "       [0.2820003 , 0.02435055, 0.05048142, 0.09636016, 0.3221153 ,\n",
       "        0.05246425, 0.17222802]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = player_stats[['apg', 'rpg']].values.astype(float)\n",
    "y = pos_multiple\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.33, random_state=42)\n",
    "h = LogisticRegression(C=1e5, multi_class='ovr')\n",
    "h.fit(X_train, y_train)\n",
    "p_hat = h.predict_proba(X_valid)\n",
    "p_hat[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "應用 `np.argmax` 函式回傳最大的欄位數，就能夠得到 $\\hat{y}$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 0, 4, 4, 4, 0, 4, 0, 4])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = np.argmax(p_hat, axis=1)\n",
    "y_pred[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最後使用 `pos_dict_reversed` 將整數對應回鋒衛位置的文字外觀。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['F', 'G', 'G', 'F', 'F', 'F', 'G', 'F', 'G', 'F']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_label = [pos_dict[i] for i in y_pred]\n",
    "y_pred_label[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二元分類延伸至多元分類：Softmax 函式\n",
    "\n",
    "除了以 One versus rest 技巧將羅吉斯回歸延伸為多元分類，另一種更常見的方法則是引入 Softmax 函式替代原本所使用的 Sigmoid 函式，我們可以將 Softmax 函式視為一種泛化的 Sigmoid 函式。首先對特徵矩陣 $X$ 分類 $k$ 計算一個得分 $s_k$，然後通過 Softmax 函式來轉換為每個類別的概率。\n",
    "\n",
    "\\begin{equation}\n",
    "s_k(X) = Xw^{(k)}\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "\\hat{P}_k = \\sigma(s(X))_k = \\frac{e^{s_k(X)}}{\\sum_{j=1}^K e^{s_j(X)}}\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "\\hat{y} = \\underset{k}{argmax} \\hat{P}_k\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "J = -\\frac{1}{m}\\sum_{k=1}^{K}y_klog(\\hat{P}_k)\n",
    "\\end{equation}\n",
    "\n",
    "其中 $K$ 表示多元分類的類別數、$\\sigma(s(X))_k$ 則是每個觀測值為 $k$ 分類的機率，$J$ 則是泛化形式的交叉熵，當 `K=2` 的時候，由於 $P_1 = 1 - P_0$ 就是針對二元分類形式的交叉熵。\n",
    "\n",
    "使用 Scikit-Learn 定義好的 `LogisticRegression` 類別只需要在初始化時加入參數 `multi_class='multinomial'` 就能夠以 Softmax 函式面對多元分類問題。這時`predict_proba` 方法輸出的數值陣列外觀為 `(m, K)`，其中 `m` 是特徵矩陣的觀測值個數，`K` 則是類別的個數，以相同的多元分類（C、C-F、F、F-C、F-G、G、G-F）問題來說，`K` 等於 7，第 0 欄是預測為類別 0（C）的機率 $\\hat{P}_0$、第 1 欄是預測為類別 1（C-F）的機率 $\\hat{P}_1$、第 6 欄是預測為類別 6（G-F）的機率 $\\hat{P}_6$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.46834231e-03, 1.10515982e-01, 1.34322740e-01, 2.30095688e-01,\n",
       "        4.75943125e-01, 3.57092827e-02, 1.19448412e-02],\n",
       "       [9.71344012e-01, 1.12987543e-07, 1.20665619e-07, 1.77849834e-06,\n",
       "        9.62067511e-05, 6.15985327e-04, 2.79417833e-02],\n",
       "       [3.73022411e-01, 7.39121345e-03, 1.52304099e-02, 4.24652094e-02,\n",
       "        2.53139760e-01, 6.77690959e-02, 2.40981900e-01],\n",
       "       [1.64560622e-01, 2.07609657e-02, 5.25290319e-02, 1.05772840e-01,\n",
       "        4.28180466e-01, 6.39325187e-02, 1.64263556e-01],\n",
       "       [1.77783413e-01, 1.72639048e-02, 2.77647975e-02, 7.76848869e-02,\n",
       "        4.03888738e-01, 9.67847960e-02, 1.98829464e-01],\n",
       "       [5.05878698e-03, 8.19709980e-02, 1.14680545e-01, 2.09123648e-01,\n",
       "        5.16685164e-01, 4.72562965e-02, 2.52245619e-02],\n",
       "       [6.74798205e-01, 1.66114300e-03, 6.16676574e-03, 1.43567047e-02,\n",
       "        9.06955187e-02, 2.20323031e-02, 1.90289360e-01],\n",
       "       [9.19952935e-02, 2.49220140e-02, 2.62448611e-02, 8.77398931e-02,\n",
       "        4.71541151e-01, 1.34074507e-01, 1.63482281e-01],\n",
       "       [6.23519221e-01, 2.26463195e-03, 7.40805921e-03, 1.80366615e-02,\n",
       "        1.13589028e-01, 2.85344995e-02, 2.06647899e-01],\n",
       "       [2.55642516e-01, 1.31288239e-02, 2.84139768e-02, 6.95853231e-02,\n",
       "        3.48894985e-01, 7.28649579e-02, 2.11469417e-01]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = LogisticRegression(C=1e5, multi_class='multinomial')\n",
    "h.fit(X_train, y_train)\n",
    "p_hat = h.predict_proba(X_valid)\n",
    "p_hat[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "應用 `np.argmax` 函式回傳最大的欄位數，就能夠得到 $\\hat{y}$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 0, 4, 4, 4, 0, 4, 0, 4])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = np.argmax(p_hat, axis=1)\n",
    "y_pred[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最後使用 `pos_dict_reversed` 將整數對應回鋒衛位置的文字外觀。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['F', 'G', 'G', 'F', 'F', 'F', 'G', 'F', 'G', 'F']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_label = [pos_dict[i] for i in y_pred]\n",
    "y_pred_label[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 兩種表示類別向量的形式\n",
    "\n",
    "截至目前為止我們表示類別目標向量 $y$ 的形式稱為標籤編碼（Label encoder），將類別變數的獨一值用 0 到 `n_classes - 1` 的整數表示，可以使用 Scikit-Learn 中的 `LabelEncoder` 轉換。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['G' 'C' 'C-F' 'C-F' 'F-C' 'G' 'G' 'C' 'F' 'F-G']\n",
      "[5 0 1 1 3 5 5 0 2 4]\n"
     ]
    }
   ],
   "source": [
    "le = LabelEncoder()\n",
    "pos = player_stats['pos'].values\n",
    "pos_le = le.fit_transform(pos)\n",
    "print(pos[:10])\n",
    "print(pos_le[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外一種表示目標向量 $y$ 的方法稱為獨熱編碼（One-hot encoder），將類別變數的獨一值用 `(m, n_classes)` 的稀疏矩陣表示，用 `1` 標註是該類，其餘欄位則用 `0` 標註不是該類，可以使用 Scikit-Learn 中的 `OneHotEncoder` 轉換。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['G' 'C' 'C-F' 'C-F' 'F-C' 'G' 'G' 'C' 'F' 'F-G']\n",
      "[[0. 0. 0. 0. 0. 1. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 1. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 1. 0.]\n",
      " [0. 0. 0. 0. 0. 1. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 1. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 1. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "ohe = OneHotEncoder()\n",
    "pos_ohe = ohe.fit_transform(pos.reshape(-1, 1)).toarray()\n",
    "print(pos[:10])\n",
    "print(pos_ohe[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "兩種表示形式中，標籤編碼適合應用於具有量值層級意義、有排列順序的類別變數（例如冷熱可以對應溫度、快慢可以對應速度）與二元分類的情境；獨熱編碼適合應用於一般無排列順序的類別變數以及多元分類的情境，特別是使用 Softmax 函式常會搭配獨熱編碼的矩陣型態。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 延伸閱讀\n",
    "\n",
    "1. Machine Learning Basics. In: Ian Goodfellow ,Yoshua Bengio, and Aaron Courville, Deep Learning (https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618/)\n",
    "2. Training Models. In: Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1492032646/)\n",
    "3. Sigmoid function (https://en.wikipedia.org/wiki/Sigmoid_function)\n",
    "4. Derivative of the Sigmoid function (https://towardsdatascience.com/derivative-of-the-sigmoid-function-536880cf918e)\n",
    "5. Step function (https://en.wikipedia.org/wiki/Step_function)\n",
    "6. Log Loss (http://wiki.fast.ai/index.php/Log_Loss)\n",
    "7. Cross entropy (https://en.wikipedia.org/wiki/Cross_entropy)\n",
    "8. Multiclass classification (https://en.wikipedia.org/wiki/Multiclass_classification)\n",
    "9. Softmax function (https://en.wikipedia.org/wiki/Softmax_function)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python Machine Learning",
   "language": "python",
   "name": "pyml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}