02_TrainLanguageModel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8dbbf92f",
   "metadata": {},
   "source": [
    "## Before training the classifier we will train a language model to prepare the enocoder that has relatively better understanding of the language corpus that we are working with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "86c67495",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from fastai.text.all import *\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f5dfc14d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the language model data\n",
    "fastai_language_model_data = torch.load(\"..\\lm_data.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d705f517",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the batch size to 16 (default is 64 but my machine does not have a lot of VRAM, sad life)\n",
    "fastai_language_model_data.bs = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f536f022",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos xxmaj may xxmaj god xxup shield all truck drivers xxunk goods school staff providing breakfast lunch to kids grocery store employees stocking shelves and all first responders including my team and i during this time and the next few weeks xxbos xxmaj grocery store workers , truckers , warehouse workers , farmers , teachers , physicians , nurses , refuse collectors , delivery drivers , gas station attendants , pharmacy workers</td>\n",
       "      <td>xxmaj may xxmaj god xxup shield all truck drivers xxunk goods school staff providing breakfast lunch to kids grocery store employees stocking shelves and all first responders including my team and i during this time and the next few weeks xxbos xxmaj grocery store workers , truckers , warehouse workers , farmers , teachers , physicians , nurses , refuse collectors , delivery drivers , gas station attendants , pharmacy workers ,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxmaj mask xxmaj scam xxunk xxmaj lab xxmaj scam 1 xxrep 3 0 xxmaj cr xxup covid xxmaj test xxmaj kit xxmaj scam 2 xxrep 3 0 xxmaj cr xxup xxunk xxmaj hand xxmaj sanitizer xxmaj scam 5 xxrep 3 0 xxmaj cr xxmaj corona xxmaj relief xxmaj scam 2 xxrep 4 0 xxmaj cr xxmaj xxunk xxmaj gandhi xxmaj virus xxmaj research xxmaj centre 3 xxrep 4 0 xxmaj cr xxbos</td>\n",
       "      <td>mask xxmaj scam xxunk xxmaj lab xxmaj scam 1 xxrep 3 0 xxmaj cr xxup covid xxmaj test xxmaj kit xxmaj scam 2 xxrep 3 0 xxmaj cr xxup xxunk xxmaj hand xxmaj sanitizer xxmaj scam 5 xxrep 3 0 xxmaj cr xxmaj corona xxmaj relief xxmaj scam 2 xxrep 4 0 xxmaj cr xxmaj xxunk xxmaj gandhi xxmaj virus xxmaj research xxmaj centre 3 xxrep 4 0 xxmaj cr xxbos xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>xxmaj fry 's xxmaj food xxmaj and xxmaj drug https : / / t.co / xxunk xxbos xxmaj as part of the consumer # coronavirus response , we also saw early hoarding of important xxunk and medical supplies like hand sanitizer , sterilizing wipes , and ( most importantly ) surgical masks . xxmaj we should note that these hoarding behaviors were global . xxbos a plunge in prices has # oilandgas</td>\n",
       "      <td>fry 's xxmaj food xxmaj and xxmaj drug https : / / t.co / xxunk xxbos xxmaj as part of the consumer # coronavirus response , we also saw early hoarding of important xxunk and medical supplies like hand sanitizer , sterilizing wipes , and ( most importantly ) surgical masks . xxmaj we should note that these hoarding behaviors were global . xxbos a plunge in prices has # oilandgas xxunk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fastai_language_model_data.show_batch(max_n=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cdf82430",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Premare the model\n",
    "lm_learner = language_model_learner(fastai_language_model_data,\n",
    "                                    AWD_LSTM,\n",
    "                                    metrics=accuracy,\n",
    "                                    drop_mult=0.2\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "700e9ede",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(valley=0.005248074419796467)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAubklEQVR4nO3dd3gc5bn38e+t3tdFsiVbNu4FdyPTTHHiHEyPA4QQCAGHEhMCSU7CC5w0kpOQcgIJgQOmk+SYGHBMC8QhEMCmGeTee5Otblll1aX7/WNXRsiSLNmanVnt/bkuXdLOzM78di3vrWfmmecRVcUYY0zkinI7gDHGGHdZITDGmAhnhcAYYyKcFQJjjIlwVgiMMSbCWSEwxpgIF+N2gO5KT0/XYcOGuR3DGGPCysqVK0tUNaO9dWFXCIYNG0Zubq7bMYwxJqyIyN6O1tmpIWOMiXBWCIwxJsJZITDGmAgXdtcI2tPQ0EBeXh61tbVuR3FNQkIC2dnZxMbGuh3FGBNmHC0EItIHeAKYCCjwDVX9sNV6H/B/wNBglt+p6tPdPU5eXh6pqakMGzYMEemR7OFEVSktLSUvL4/hw4e7HccYE2acPjX0ALBUVccBU4DNbdbfCmxS1SnALOA+EYnr7kFqa2vp379/RBYBABGhf//+Ed0iMsYcP8cKgYikAecATwKoar2qHm6zmQKpEvgETwEOAY3HebzjD9sLRPrrN6a3e3NTITuKKh3Zt5MtghFAMfC0iKwWkSdEJLnNNg8B44GDwHrgO6ra7GAmT0hJSQFgz549TJw40eU0xphw8K2Fq/jbqgOO7NvJQhADTAceUdVpgB+4q802c4A1wCBgKvBQsCXxGSJys4jkikhucXHxiSdb9zz8fiLc0yfwfd3zJ75PY4xxSF1jE/VNzaTEO3NZ18lCkAfkqeqK4OPFBApDa/OAJRqwA9gNjGu7I1V9TFVzVDUnI6PdO6S7bt3z8OrtUL4f0MD3V28/oWJw55138vDDDx95fM899/Czn/2M2bNnM336dCZNmsTLL7/c6T6ampq44447mDFjBpMnT+bRRx8F4Nprr/3Mc6+55hpeeeWV485qjAk//romgPArBKpaAOwXkbHBRbOBTW022xdcjogMBMYCu5zKBMBbP4eGms8ua6gJLD9OV111Fc8999yRx88//zzz5s3jxRdfZNWqVbz99tt8//vfp7NpQZ988kl8Ph+ffPIJn3zyCY8//ji7d+/mxhtv5OmnAx2pysvL+eCDD7jwwguPO6sxJvxU1QYunSY7VAicvo/gNmBhsCfQLmCeiMwHUNUFwH8Dz4jIekCAO1W1xNFE5XndW94F06ZNo6ioiIMHD1JcXEzfvn3Jysrie9/7HsuWLSMqKooDBw5QWFhIZmZmu/t44403WLduHYsXLw7EKS9n+/btnHfeedx6660UFRWxZMkSLr/8cmJiesXtH8aYLqqqCxQCp1oEjn6iqOoaIKfN4gWt1h8EznMyw1F82cHTQu0sPwFXXHEFixcvpqCggKuuuoqFCxdSXFzMypUriY2NZdiwYZ1271RVHnzwQebMmXPUumuvvZaFCxeyaNEinnrqqRPKaYwJP04XgsgbYmL2TyA28bPLYhMDy0/AVVddxaJFi1i8eDFXXHEF5eXlDBgwgNjYWN5++2327u1w4D8A5syZwyOPPEJDQwMA27Ztw+/3A3D99dfzhz/8AYAJEyacUE5jTPjxtxSChDBsEXjS5CsD39/6eeB0kC87UARalh+nCRMmUFlZyeDBg8nKyuKaa67hkksuIScnh6lTpzJu3FHXwD/jxhtvZM+ePUyfPh1VJSMjg5deegmAgQMHMn78eObOnXtCGY0x4anySIsg2pH9S2cXML0oJydH285HsHnzZsaPH+9SIudVV1czadIkVq1ahc/n63C73v4+GBOp/vrxPu5esp6P7p5Npi/huPYhIitVte2peiASTw2FmTfffJNx48Zx2223dVoEjDG916e9hpxpEUTeqaEw84UvfIF9+/a5HcMY46KWi8XJcXax2BhjIlJVXSPJcdFERTkzplivKQThdq2jp0X66zemN/PXNTp2Mxn0kkKQkJBAaWlpxH4YtsxHkJBwfBeRjDHeVlnX6FjXUegl1wiys7PJy8ujRwakC1MtM5QZY3off12jYzeTQS8pBLGxsTYzlzGm16qqdbYQ9IpTQ8YY05tV2TUCY4yJbFV1jaRaITDGmMhlvYaMMSbCVTnca8gKgTHGeFhdYxMNTWoXi40xJlK1jDNkhcAYYyJUy3zFdo3AGGMiVGVdYLIqaxEYY0yEamkRWCEwxpgIVdXSIrBeQ8YYE5mqjrQInJmUBqwQGGOMp33aayjWsWNYITDGGA/z1zk7TSU4XAhEpI+ILBaRLSKyWUTOaGebWSKyRkQ2isi7TuYxxphwU+nwNJXg/DDUDwBLVfUKEYkDklqvFJE+wMPA+aq6T0QGOJzHGGPCit/haSrBwUIgImnAOcD1AKpaD9S32exqYImq7gtuU+RUHmOMCUdVtc6OMwTOnhoaARQDT4vIahF5QkSS22wzBugrIu+IyEoR+Xp7OxKRm0UkV0RyI3kWMmNM5Kmqd3bkUXC2EMQA04FHVHUa4AfuamebU4CLgDnAj0VkTNsdqepjqpqjqjkZGRkORjbGGG/xOzwXAThbCPKAPFVdEXy8mEBhaLvNUlX1q2oJsAyY4mAmY4wJK1W1YdwiUNUCYL+IjA0umg1sarPZy8DZIhIjIknAacBmpzIZY0y4qXJ44npwvtfQbcDCYI+hXcA8EZkPoKoLVHWziCwF1gHNwBOqusHhTMYYEzbCvhCo6hogp83iBW22+R/gf5zMYYwx4crv8OxkYHcWG2OMp1U5PF8xWCEwxhjPCsU0lWCFwBhjPCsU01SCFQJjjPGsUExKA1YIjDHGs1qmqbRrBMYYE6FaWgSp1mvIGGMiU5W1CIwxJrJV2TUCY4yJbNZryBhjIlzLNJV2Z7ExxkSolmkqk2Kdm68YrBAYY4xn+YMDzjk5TSVYITDGGM8KzEXgbGsArBAYY4xnVdU7PwQ1WCEwxhjPqqq1QmCMMREtFHMRgBUCY4zxrKq6RpLjrBAYY0zEqrIWgTHGRLZQzFcMVgiMMcaTVPXIfQROs0JgjDEeVNfYTEOTOj7yKERQIXh7SxFn//bflFbVuR3FGGOOqWWcIafnIgCHC4GI9BGRxSKyRUQ2i8gZHWw3Q0SaROQKp7L0S45j/6Ea3ttR4tQhjDGmx1QFC0Fv6DX0ALBUVccBU4DNbTcQkWjgN8A/nQwycbCPvkmxLNtmhcAY431HCkE4nxoSkTTgHOBJAFWtV9XD7Wx6G/A3oMipLADRUcLMUeks316Mqjp5KGOMOWHl1YHZyXyJsY4fy8kWwQigGHhaRFaLyBMiktx6AxEZDHwJWOBgjiPOGZ1BUWUdWwsrQ3E4Y4w5bmXBQtA3ObwLQQwwHXhEVacBfuCuNtv8AbhTVZs625GI3CwiuSKSW1xcfNyBzh6TDsByOz1kjPG4sup6APolxTl+LCcLQR6Qp6orgo8XEygMreUAi0RkD3AF8LCIzG27I1V9TFVzVDUnIyPjuANl+RIZPSCFZduPv5gYY0wolPkDhaBPOBcCVS0A9ovI2OCi2cCmNtsMV9VhqjqMQKH4lqq+5FQmgHPGZLBi9yFqGzpthBhjjKvKqhtIjosmLsb5Xv5OH+E2YKGIrAOmAveKyHwRme/wcTt09uh06hub+Xj3IbciGGPMMR2urg9JawAC5/Edo6prCJz+aa3dC8Oqer2TWVqcNrw/cTFRLNtWzDljjv80kzHGOKmsup5+yaEpBBFzZ3GLxLhoTh3Wj+Xb7YKxMca7yqob6JPkfI8hiMBCAIHTQ1sLKymsqHU7ijHGtKusup6+ITo1FKGFIHBK6M3NhS4nMcaY9pX56+lrLQLnjM9KZVxmKn/+YK/dZWyM8ZzGpmYqahtDdrE4IguBiHDDWcPZWlhpg9AZYzynvCZwV7FdLHbYpVMHkZ4Sz5Pv7XY7ijHGfEbLXcV2sdhh8THRXHfGSbyztZjtNvaQMcZDjowzZKeGnHfN6ScRHxPFU+9bq8AY4x0tw0tYIQiBfslxXDY9myWrDtjMZcYYzzgcbBHYqaEQueGsYdQ1NrNwxT63oxhjDNBq5FG7WBwaowakctaodJ7P3U9zs3UlNca471B1PXHRUSTFRYfkeBFfCAAumz6YvLIacveWuR3FGGM47A8MLyEiITmeFQJgzoRMkuKieXF1nttRjDEmpMNLgBUCIDA59PkTMvn7unybp8AY47rDIRxwDqwQHHHZ9Gwqaxt5a3OR21GMMRHuUAiHoAYrBEecMbI/A9PiWbLKTg8ZY9wVyklpwArBEdFRwtxpg3lnWzEldk+BMcYlqsrh6oaQjTwKVgg+47Jp2TQ1K6+uPeh2FGNMhKqsa6SxWe1isVvGZqZyclYaz31y/PcUNDcrDU3NPZzMGBMpDvtDe1cxWCE4yk3nDGdLQSXP5e7v8nNKqupYuGIvtz67ihm/fJOpP3uD92wqTGPMcTgU4ruKwQrBUeZOHcypw/vx639s6dL4Q39fd5DZ973LD1/cwMo9ZZw7JoPsvknc+OdP+GhXaQgSG2N6k0+HoLZC4BoR4RdzJ+Kva+Q3S7d0uF15dQPfWbSabz+7mmHpybx2+1l8ePfnuf8rU1l402lk903iG898Qu6eQyFMb4wJd4erW0Ye9dipIRFJFpGo4M9jRORSEQldyhAbMzCVG84ezvO5ee1+kG/Or+DCPy7n7+vy+d4XxvC3+WcwYZDvyO3g6SnxPHvjaWSmJXD905/w639sYeXesi5fd7DpM42JXGX+0M5FABDTxe2WAWeLSF/gLSAX+ApwTWdPEpE+wBPARECBb6jqh63WXwPcGXxYBdyiqmu78wKccvvnR/PqmoP814vruf/KqUwc7APgX5sK+c6i1aQmxPC3W85k6pA+7T5/QFoCz950Onf+bR1PLN/Fgnd3kp4Sz/kTB3LJ5EHMGNaPqCihuVnZXepnXd5hVu09zKp9ZWwtqCQ1IYYBqQkMSIvn8unZfHHqoG6PO6KqrNxbRnlNA2MGpjK4TyJRUZ/uo6lZWb2vjH9vKeL9HSXUNjQTHSXERAsp8TFkpMaTnhLP5Gwfl0we9JnnGmOcUVZdjwikJYbub23pyl+fIrJKVaeLyG1Aoqr+VkRWq+q0YzzvT8ByVX1CROKAJFU93Gr9mcBmVS0TkQuAe1T1tM72mZOTo7m5uV14aSfu7a1FfPMvK6lvbGbCoDSmDOnDXz/ex+TBPh77eg4D0xK6tJ/ymgbe2VrEGxsLeWtLIbUNzWT5EhjUJ5HN+RVU1weGtUiOi2bq0D5MGOTDX9dIUWUdO4uq2FXi5+zR6fxy7iSG9k865vFUlXe3FfPgv3ewstVAeslx0WT1SaRZlaZmpcxfT0VtIzFRwvST+tIvKY7GZqWxuZmKmgZKquoprqyjpqGJGcP68qvLJjFqQOrxvZldUFnbwLbCSrYUVFLmr2dkRgpjM1M5qX8y0VaETIT40UvreW1dPqt/cl6P7ldEVqpqTrvrulgIVgPfAn4P3KCqG0VkvapO6uQ5acBaYIR24SDB1sYGVR3c2XahLAQQOF/38pqDPJ+7n40HK7h4cha/+/IUEmKPb3hYf10jb24u5NW1+VTUNHDyoDQmDEpj4mAfYwamHvWB19SsLFyxl98u3UpjczNnjUrHX9dERW0DTc1KZrCgpCfHUeqvp7Cijt0lVews9jPIl8Ats0YyPiuNbYVVbCuspLCiNvBXf5SQGBfDzFH9OXt0Br4O/vpQVV5Ymce9r2/GX9fIdWcMY3xWGv1S4khPjmdAWqDV0N4HdUNTM4erGzjkr6ekqo6SqjqKKurIK6tmf1kNeWXVVNU2UtfYTH1jM5V1je1miI+JYmi/JE7qn8zQfknUNzVRUhnYpxJoQvdLjiUhNpqq2kYqahupqmugur6J6vomaoKFNjpKiI4SGpubqalvpq6hiaT4aM4Y0Z+Zo9I5a3Q6Wb7E4/p3Naan3PrsKjYfrODfP5jVo/vtiUJwLvB94H1V/Y2IjAC+q6q3d/KcqcBjwCZgCrAS+I6q+jvY/gfAOFW9sZ11NwM3AwwdOvSUvXv3HjOzEw5X1+NLDN3QsK3ll9dw7+tb2FZQiS8xltSEGEQgv7yW/PJaDvnr6ZsUy8C0BDJ9CVwwMZMvTcsmLqZn+gOUVtXxi9c28+LqA0eti44SMlLiiY+NorEp0KKormvq8IM9NT6G7H5JZPdNxJcYS3xMFPEx0fRPiWNcZipjM1PplxzHjqIqthRUsr2wkj2l1ewrrWbfoWriY6NIT4knPSWOKBEO+es5XN1AdX0jqQmxpCXGkhofQ1J8NElx0STERiMIzao0NiuxUUJ8bDSJsdGUVNXxwc4SSqoCF+gmDEpjzoRMvjB+IOmpn56jjYuOIjEumrjoKFf+/U3kuOaJj6ipb2LJt2b26H5PuBC02VkUkKKqFcfYLgf4CJipqitE5AGgQlV/3M62nwMeBs5S1U77XIa6RRAumps1JOfwK2sbKK2qp9RfR0lVPUWVdRSW11JQUUtjUzMx0VHERAkJsdH0S46jb1IsfZPjgh/c8WSkxJOWGOOpD1NVZWthJe9sLeZfmwpZta+Mjv5bxEQJA1LjmTKkD1OH9OGk/knsLPazKb+CXcV+MtPiGZuZxrjMVNJT4omNFuJiouiXHMeQvkl2ncUc04UPLCfLl8CT18/o0f12Vgi6dLFYRJ4F5gNNBP6y94nI/ar6P508LQ/IU9UVwceLgbva2fdkAheULzhWETAdC9UHTGpCLKkJsQxLTw7J8UJBRBiXmca4zDTmnzuSospa3ttecuTajQINjc3UNDThr2skr6yGNfsP848NBUf2MaRfIiMzUsgvr2X59hIa2+khlhQXzeiBqYwbGGj1tHylp8SH6qWaMFBWXc/4rLSQHrOrvYZOVtWKYC+f1wn09FkJdFgIVLVARPaLyFhV3QrMJnCa6AgRGQosAa5V1W3H9QqM6WEDUhO4bHr2Mbcrqaojr6yGERnJpCV8eo2lvrGZ3SV+ymsaaGhqpr6pmaKKWrYUVLIlv5I3NhV85s71wX0SmTq0D9OG9GHW2AGMGpDiyOsy4aGsup5+yaHtnd/VQhAbvG9gLvCQqjaISFfOKd0GLAz2GNoFzBOR+QCqugD4CdAfeDh4qqCxo6aLMV7TcrqrrbiYKMZmdty7SlUpqapna0Elm/MrWJN3mDX7DvPaunx+8dpmxmelccmULOZOHcygPnbxOpLUNjRR29Ac0ruKoeuF4FFgD4FeQMtE5CSg02sEAKq6Bmj7wb6g1fobgaMuDhvTm4kIGanxZKTGc9bo9CPL88trWLqhgFfXHuS3S7dy3xvbuHhyFjefM4IJg3wuJjahUnbkrmIPFgJV/SPwx1aL9gYv8BpjekiWL5F5M4czb+Zw9h+q5k8f7OGvH+/j5TUHOWtUOt/63EjOGNHfUxfaTc865A/98BLQ9SEmfCJyv4jkBr/uA3rP1UJjPGZIvyR+dPHJfHD3bO48fxxbCyu5+vEVXPbIB/xrUyGNNtR5r3S4umUIag+2CICngA3AlcHH1wJPA5c5EcoYE+BLjOWWWSOZN3MYL6zM49F3d3LTn3NJT4ljzoRMLpqcRc5J/XrsfhHjrjIXhqCGrheCkap6eavHPxORNQ7kMca0IyE2mmtPP4mrZgzhzU2F/H19PktWHWDhin3ExURxclYaU7J9nDchk5mj0o+9Q+NJZdUtA855s9dQjYicparvAYjITKDGuVjGmPbERkdxwaQsLpiURXV9I8u2FbNybxlr88p5PjePP324l7NGpXPXBeOODJRowkdxZR0i0NejLYL5wJ9FpOU3qwy4zplIxpiuSIqL4fyJWZw/MQuAusYm/u+jfTz47+1c/OB7XDZ9MD8fvomU9+6F8jzwZcPsn8DkK4+xZ+OWwvLa4B3poT3V19VeQ2uBKcGB5AjeXPZdYJ2D2Ywx3RAfE80NZw3nilOyeeSdnRS+/2eiNz0OBM47U74fXg0OD2bFwJPyK2rJ8nVtVOOe1K2yo6oVrcYY+k8H8hhjTpAvMZa7LhjHb/q8RGJLEWjRUANv/dydYOaYCstruzy8fU86kfaHdWY2xsPiqg62v6I8L7RBTJfll9d4v0XQhs2naIyX+dofL6kueVCIg5iuqK4PzKXhuRaBiFSKSEU7X5WA/TYZ42WzfwKxnx2rqJZ47q74EktbjZxqvKGgvBbAey0CVU1V1bR2vlJVtas9jowxbph8JVzyR/ANAQR8Q2i86A/szrqQW59dxWvr8t1OaFopqAgUgkwXWgT2YW5Mbzb5ys/0EEoB/jK5kXlPf8zti1bTpMqlU6xx7wUtLYJMr7UIjDG9T0p8DM/MO5VTTurLdxet5oVWcyMY9xxpEVghMMaEQnJ8DM/Mm8GZI9O5Y/E67n19M03tzKpmQqegvJa0hBiS4kJ/osYKgTERKikuhqfnzeC6M07isWW7uP7pjykPjnVjQq+gvNaV1gBYITAmosVGR/GzL07k15dN4qNdpXzpkfcpCp6iMKFVUFFLps+dGemsEBhjuOrUoSy88XQKymv56uMfUVxZ53akiFNQXktm2tFTn4aCFQJjDACnDu/H09fP4MDhGr72xIojs2UZ5zU0NVNcVWctAmOM+04b0Z8nr5vBnlI/X3tiBRW1ds0gFIor61B15x4CsEJgjGlj5qh0Hr32FLYVVnLrwlU02LSYjst38a5isEJgjGnHrLEDuPdLk1i+vYSfvrIRVeta6qTC4AV6N8YZAocLgYj0EZHFIrJFRDaLyBlt1ouI/FFEdojIOhGZ7mQeY0zXXTljCLfMGsmzK/bxxPLdbsfp1dxuETh958IDwFJVvUJE4oCkNusvAEYHv04DHgl+N8Z4wB3njWVfaTX3/mMzg/smcuGkLLcj9UqFFbXExUTRJ8RzFbdwrBAEZzM7B7geQFXroe0sGXwR+LMG2p0fBVsQWapqo2EZ4wFRUcJ9V06hsKKW7yxaTVJcNLPGDnA7Vq+TXx6YmUzEnWlenDw1NAIoBp4WkdUi8oSIJLfZZjDQeqCTvOCyzxCRm0UkV0Ryi4uLnUtsjDlKQmw0T14/g9EDUpn/fyv5ePchtyP1Om7NTNbCyUIQA0wHHlHVaYAfuKvNNu2Vv6OuSqnqY6qao6o5GRkZPZ/UGNMpX2Isf77hVAb1SeQbz3zChgPlbkfqVfIr3JmZrIWThSAPyFPVFcHHiwkUhrbbDGn1OBvoYH49Y4yb0lPiWXjjafgSY7npz7l2w1kPUVUKy+tcu4cAHCwEqloA7BeRscFFs4FNbTZ7Bfh6sPfQ6UC5XR8wxruyfIks+NoplFbV893n1tBsI5aesEP+euqbml0bcA6cv4/gNmChiKwDpgL3ish8EZkfXP86sAvYATwOfMvhPMaYEzQp28dPLz2ZZduKeejtHW7HCXtuzkzWwtHuo6q6Bshps3hBq/UK3OpkBmNMz7v61KF8svsQv39zG9OH9uWs0eluRwpbbs5M1sLuLDbGdJuI8MsvTWJURgrffW61jVZ6AtycmayFFQJjzHFJjo/hf6+ZTmVtI3csXmvDUByngvJaogQyUtwZghqsEBhjTsCYgan88KLxvLO1mGc+2ON2nLBUUF5LRmo8MdHufRxbITDGnJBrTz+J2eMG8Kt/bGFzfoXbccLO/rJqBvVxZx6CFlYIjDEnRET47RWT8SXGcvtfV1Pb0OR2pLCyq9jPyIwUVzNYITDGnLD+KfHc9+UpbC+q4jdLt7gdJ2xU1jZQVFnHiIy2o++ElhUCY0yPOGdMBtefOYyn39/De9tL3I4TFnaX+AEYkW4tAmNML3Hn+eMYmZHMD15YS3m1TXN5LLuKA4VgpLUIjDG9RWJcNH/4yjRKqur40csb3I7jebuKq4gSGNq/7VQtoWWFwBjToyZl+/juF0bz6tqDvLT6gNtxPG1nsZ8h/ZKIj4l2NYcVAmNMj5t/7khmDOvLj17awN5Sv9txPGtncRUj0t09LQRWCIwxDoiJjuIPV00jSuD2v66mvrHZ7Uie09ys7Cn1M8LlrqNghcAY45DBfRL5zeWTWZtXzv3/2uZ2HM85WF5DbUOz611HwQqBMcZBF0zK4qunDmXBuzutS2kbn/YYshaBMaaX+8nFJzNqQIp1KW1jV3EVgLUIjDG9X2JcNPdfOYXiqjrueXWj23E8Y1eJn9T4GFdHHW1hhcAY47jJ2X349udG8eLqA/xjvc1GC8EeQxnJiIjbUawQGGNC49ufH8WkwT7+68X1NpENgWsEXugxBFYIjDEhEhsdxf1XTsFf38TdS9ZF9EQ21fWN5JfXeuIeArBCYIwJodEDU/l/c8by5uYi/m/FPrfjuKalx5C1CIwxEekbM4dz9uh0fvH3TWwvrHQ7jit2tYw66oEeQ2CFwBgTYlFRwn1XTiElPobbF62JyIlsdhVXIQLDI+HUkIjsEZH1IrJGRHLbWe8TkVdFZK2IbBSReU7mMcZ4w4DUBP7ny5PZnF/Bb5dudTtOyO0q9jO4TyIJse4ONtciFC2Cz6nqVFXNaWfdrcAmVZ0CzALuE5G4EGQyxrjs8+MGct0ZJ/HU+7tZsavU7TghtaukyjPXB8D9U0MKpEqgI20KcAhodDeSMSZU7rpgPEP6JXL3kvURc4qoqVnZWeR3fTKa1pwuBAq8ISIrReTmdtY/BIwHDgLrge+oqg1TaEyESIyL5t4vTWJXiZ+H/r3D7TghsafUT01DEydnpbkd5QinC8FMVZ0OXADcKiLntFk/B1gDDAKmAg+JyFHvjojcLCK5IpJbXFzscGRjTCidPTqDy6dns+DdnWwpqHA7juM2HQy8xpMHRUghUNWDwe9FwIvAqW02mQcs0YAdwG5gXDv7eUxVc1Q1JyMjw8nIxhgX/Oii8fgSY7nzb+tpau7dN5ptyq8gNloYPSDV7ShHOFYIRCRZRFJbfgbOA9pOYroPmB3cZiAwFtjlVCZjjDf1TY7jJ5eczNr9h/nTB3vcjuOoTQcrGDUglbgYty/RfsrJJAOB90RkLfAx8JqqLhWR+SIyP7jNfwNnish64C3gTlW1QcuNiUCXThnErLEZ/O6NreSVVbsdxzEbD1YwwUOnhQBinNqxqu4CprSzfEGrnw8SaCkYYyKciPCLuRM57/fL+PFLG3jq+hmeGJmzJxVV1lJSVeepC8XgfvdRY4w5IrtvEt8/byxvby3m7+t633DVXrxQDFYIjDEec/2Zw5ic7eNnr27kcHW923F61Kb8QCEYby0CY4zpWHSU8OvLJlNW3cDP/77J7Tg9atPBCrL7JuJLjHU7ymdYITDGeM7Jg9K4ddZIlqw6wD83Frgdp8ds8uCFYrBCYIzxqG9/fjQTBqXxX0vWU1IV/jOa+esa2V3q5+Qsn9tRjmKFwBjjSXExUdx/5VQqaxv54Yvrw35Gsy0Flah670IxWCEwxnjY2MxUvn/eGP65sZAlqw64HeeEtFwotkJgjDHddOPZI5gxrC/3vLKR/YfC90azTQcr8CXGMsiX4HaUo1ghMMZ4WnSUcP+VUwH43nNraGwKzwGKN+UHLhR78SY5KwTGGM8b0i+J/547kdy9ZTz8zk6343RbY1MzW/IrPHdHcQsrBMaYsDB32mDmTh3EA29tZ9W+MrfjdMuWgkrqGpuZMNgKgTHGnJCfz51Ili+B7yxaTWkYdSn916ZCRAJzL3iRFQJjTNhIS4jlwa9Oo6iijm888wn+uvCY2fafGwuYMawf6SnxbkdplxUCY0xYmTa0Lw9dPZ31B8q5ZeEq6hu9ffF4T4mfLQWVzJmQ6XaUDlkhMMaEnf84eSC/umwSy7YVc+ff1tHs4VnNWobImDNhoMtJOubYfATGGOOkr8wYSnFlHb97YxtpCTHcc+kET3bNXLqxgEmDfWT3TXI7SoesEBhjwtatnxtFeU0Djy/fTUJsNHddMM5TxaCgvJbV+w5zx5yxbkfplBUCY0zYEhH+68Lx1DQ08eiyXSTERvO9/xjjdqwj3tjUclrIu9cHwAqBMSbMiQg/v3QitQ3NPPDWdlLiY7jpnBFuxwJg6YYCRmYkM2pAittROmWFwBgT9qKihN9cPpnq+kZ++fpmBqTF88Wpg13NVOavZ8XuQ8w/1xtFqTPWa8gY0yu0jEl02vB+/OCFtby/o8TVPK9vyKepWT1/WgisEBhjepGE2Gge+3oOI9JT+OZfVrLhQLkrOZqalSeX72bi4DQmDfbeRDRtWSEwxvQqvsRYnvnGDNISYrjuqY/ZUVQV8gxvbCxgV4mfW84d5aleTB1xtBCIyB4RWS8ia0Qkt4NtZgXXbxSRd53MY4yJDFm+RP5y42mICNc88RF7S/0hO7aqsuDdnQzrn8T5E71/WghC0yL4nKpOVdWctitEpA/wMHCpqk4AvhyCPMaYCDAyI4WFN55GfWMzVz++gryy0Exq8+GuUtbmlXPTOSOIjvJ+awDcPzV0NbBEVfcBqGqRy3mMMb3I2MxU/nLDaVTWNnD14ytCMsPZI+/sJD0lnsunZzt+rJ7idCFQ4A0RWSkiN7ezfgzQV0TeCW7z9fZ2IiI3i0iuiOQWFxc7GtgY07tMHOzjLzecRnlNA19e8CE7iiodO9aGA+Us317CN84aRkJstGPH6WlOF4KZqjoduAC4VUTOabM+BjgFuAiYA/xYRI66LVBVH1PVHFXNycjw5njexhjvmjKkD89983SaVPnygg9Zn9fzvYmam5XfLN1CSnwM15x2Uo/v30mOFgJVPRj8XgS8CJzaZpM8YKmq+lW1BFgGTHEykzEmMo3LTOOFb55BUlwMX338Iz7cWdqj+3902S6Wby/h7gvH4UuM7dF9O82xQiAiySKS2vIzcB6woc1mLwNni0iMiCQBpwGbncpkjIlsw9KTWXzLGWT6ErjuqY9ZuiG/R/a7cu8hfvfGVi6alMXVpw7tkX2GkpMtgoHAeyKyFvgYeE1Vl4rIfBGZD6Cqm4GlwLrgNk+oattiYYwxPSbLl8gL3zyDCYPT+NbCVTy7Yt8J7e9wdT23/3UNg/ok8KvLJ4XFfQNtiap3J3RoT05OjubmtntLgjHGdFl1fSO3LlzF21uLuWhyFrfOGsXJg7o3ufzeUj93/W09uXsPsXj+mUwZ0seZsD1ARFa2140fbNA5Y0yESoqL4bGv5/DAm9t55oM9vLYun8+PG8C1p5/EGSP7d9rrp7iyjgf/vZ1nV+wjJlr45dxJni4Cx2ItAmNMxCuvaeAvH+7hqff3cMhfT0JsFGeOTOfcMRnMHNWfkRkpiAhbCyr504d7eHHVAeqbmvnKjCF8Z/ZoBqYluP0SjqmzFoEVAmOMCaptaGLF7kO8vaWIt7cWsbc0cAPawLR4Mn2JrN1/mPiYKL44dRDzzx3JiAxvzzPQmhUCY4w5DvtKq3l/Zwnv7yhhT6mfiyYN4qoZQ+ibHOd2tG6zawTGGHMchvZPYmj/oXw1DLuEdofbYw0ZY4xxmRUCY4yJcFYIjDEmwlkhMMaYCGeFwBhjIpwVAmOMiXBWCIwxJsJZITDGmAgXdncWi0gxsDf40AeUd/Jz22WxQEk3D9l6H11Z13ZZVzO2fE/vZsZQ5WtZZu+ht/KFQ0av5zuRjJ0t89p7eJKqtj/Fo6qG7RfwWGc/t10G5J7IMbqyru2yrmZs9b1bGUOVz95Db+YLh4xez3ciGY+R1VPvYWdf4X5q6NVj/NzR+uM9RlfWtV3W1Yxez3esY3XG3sNjH6czx3qe1zN6PV9H67uS8VjLusPp97BDYXdq6ESISK52MOiSV3g9o9fzgfczej0feD+j1/NBeGRsEe4tgu56zO0AXeD1jF7PB97P6PV84P2MXs8H4ZERiLAWgTHGmKNFWovAGGNMG1YIjDEmwlkhMMaYCGeFIEhEzhaRBSLyhIh84Hae9ohIlIj8UkQeFJHr3M7TlojMEpHlwfdxltt52iMiySKyUkQudjtLe0RkfPD9Wywit7idpz0iMldEHheRl0XkPLfztCUiI0TkSRFZ7HaWFsHfuz8F37dr3M7TVq8oBCLylIgUiciGNsvPF5GtIrJDRO7qbB+qulxV5wN/B/7kxYzAF4HBQAOQ58F8ClQBCR7NB3An8HxPZuvJjKq6Ofh7eCXQ410PeyjjS6p6E3A98BUP5tulqjf0ZK72dDPrZcDi4Pt2qdPZuq07d7559Qs4B5gObGi1LBrYCYwA4oC1wMnAJAIf9q2/BrR63vNAmhczAncB3ww+d7EH80UFnzcQWOjBfF8AriLwAXaxF/+Ng8+5FPgAuNqrGYPPuw+Y7uF8Pfp/5ASz3g1MDW7zrJO5juerV0xer6rLRGRYm8WnAjtUdReAiCwCvqiqvwLaPS0gIkOBclWt8GJGEckD6oMPm7yWr5UyIN5r+UTkc0Aygf+YNSLyuqo2eyljcD+vAK+IyGvAsz2Vr6cyiogAvwb+oaqrvJYvVLqTlUALORtYgwfPxPSKQtCBwcD+Vo/zgNOO8ZwbgKcdS3S07mZcAjwoImcDy5wMFtStfCJyGTAH6AM85GiygG7lU9UfAojI9UBJTxaBTnT3PZxF4DRCPPC6k8Fa6e7v4W0EWlc+ERmlqgucDEf338P+wC+BaSJyd7BghEpHWf8IPCQiF3H8Q1A4pjcXAmlnWad3z6nqTx3K0pFuZVTVagLFKlS6m28JgWIVKt3+NwZQ1Wd6PkqHuvsevgO841SYDnQ34x8JfLCFSnfzlQLznYvTqXazqqofmBfqMF3luSZKD8oDhrR6nA0cdClLR7ye0fKdOMt44ryer7VwynpEby4EnwCjRWS4iMQRuEj4isuZ2vJ6Rst34izjifN6vtbCKeun3L5a3RNfwF+BfD7tVnlDcPmFwDYCV/F/aBktn2X0dkav5wvXrMf6skHnjDEmwvXmU0PGGGO6wAqBMcZEOCsExhgT4awQGGNMhLNCYIwxEc4KgTHGRDgrBKZXEJGqEB+vR+askMAcDuUislpEtojI77rwnLkicnJPHN8YsEJgTLtEpNNxuFT1zB483HJVnQZMAy4WkZnH2H4ugRFUjekRvXnQORPhRGQk8L9ABlAN3KSqW0TkEuBHBMaLLwWuUdVCEbkHGAQMA0pEZBswlMDY8kOBP2hgwDVEpEpVU4Kjhd4DlAATgZXA11RVReRC4P7gulXACFXtcNhkVa0RkTUERrBERG4Cbg7m3AFcC0wlMF/BuSLyI+Dy4NOPep3H+76ZyGMtAtObPQbcpqqnAD8AHg4ufw84PfhX+CLg/7V6zikExrq/Ovh4HIGhtU8Ffioise0cZxrwXQJ/pY8AZopIAvAocIGqnkXgQ7pTItIXGM2nQ4wvUdUZqjoF2ExgCIMPCIxdc4eqTlXVnZ28TmO6xFoEplcSkRTgTOCFwDwqwKeT5WQDz4lIFoG/tne3euorqlrT6vFrqloH1IlIEYHZ19pOw/mxquYFj7uGQIuiCtilqi37/iuBv+7bc7aIrAPGAr9W1YLg8oki8gsC8zukAP/s5us0pkusEJjeKgo4rKpT21n3IHC/qr7S6tROC3+bbeta/dxE+/9n2tumvXHpO7JcVS8WkTHAeyLyoqquAZ4B5qrq2uBkOrPaeW5nr9OYLrFTQ6ZX0sB0o7tF5MsQmF5RRKYEV/uAA8Gfr3MowhZgRKupDI85ybuqbgN+BdwZXJQK5AdPR13TatPK4LpjvU5jusQKgektkkQkr9XXfxL48LxBRNYCGwnMHQuBFsALIrKcwIXcHhc8vfQtYKmIvAcUAuVdeOoC4BwRGQ78GFgB/ItAYWmxCLgj2OV0JB2/TmO6xIahNsYhIpKiqlXByd7/F9iuqr93O5cxbVmLwBjn3BS8eLyRwOmoR92NY0z7rEVgjDERzloExhgT4awQGGNMhLNCYIwxEc4KgTHGRDgrBMYYE+GsEBhjTIT7/+uMnCFpYh2BAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Finding the best learning rate\n",
    "lm_learner.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "75b8c623",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.182218</td>\n",
       "      <td>4.014528</td>\n",
       "      <td>0.319493</td>\n",
       "      <td>03:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.805860</td>\n",
       "      <td>3.787229</td>\n",
       "      <td>0.336370</td>\n",
       "      <td>03:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.623776</td>\n",
       "      <td>3.723620</td>\n",
       "      <td>0.342193</td>\n",
       "      <td>03:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.494596</td>\n",
       "      <td>3.703640</td>\n",
       "      <td>0.344256</td>\n",
       "      <td>03:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.428021</td>\n",
       "      <td>3.702196</td>\n",
       "      <td>0.344270</td>\n",
       "      <td>03:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Training just the last few layers for 5 epoch\n",
    "lm_learner.fit_one_cycle(5, 0.005)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "6da6ac86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(valley=0.0002290867705596611)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnYklEQVR4nO3deZRcdZ338fe3uqu700s6S3d2QhIIBMKWGAMYQRQfGCMio4xmiCgoIMpRxxkZZRa386jznDMzMoAQcUE9wgAT2RR0FGQVBZKQBEIwZE+nk96S9L5WfZ8/qrptmu5Odbpv3arqz+ucOl11l7qfqq6ub/9+997fNXdHREQEIBJ2ABERyRwqCiIi0kdFQURE+qgoiIhIHxUFERHpo6IgIiJ98sMOMFIVFRU+b968sGOIiGSV9evX17t75dGWy7qiMG/ePNatWxd2DBGRrGJme1JZTt1HIiLSR0VBRET6qCiIiEifrNunMJju7m6qqqro6OgIO0poioqKmDNnDtFoNOwoIpLFcqIoVFVVUVZWxrx58zCzsOOknbvT0NBAVVUV8+fPDzuOiGSxnOg+6ujoYOrUqeOyIACYGVOnTh3XLSURGRs5URSAcVsQeo331y+S6373Wg3ba1sC307OFIVsUlpaCsDu3bs57bTTQk4jIpnO3fns3etZu74q8G2Nz6Kw+X747mnw9UmJn5vvDzuRiMiQDrd10x1zpk8sDHxb468obL4ffvl5aNwHeOLnLz8/qsLw5S9/mdtvv73v8de//nW+8Y1vcOGFF7J06VJOP/10Hn744WGfIxaLceONN/L2t7+dM844g+9///sAXHnllW9ad/Xq1TzyyCPHnFVEss/BxsT+whkTiwLf1vgrCk98E7rb3zytuz0x/RitWrWK++67r+/x/fffz9VXX82DDz7Ihg0bePLJJ/mHf/gHhrv06Y9+9CPKy8t56aWXeOmll/jBD37Arl27uOaaa7jrrrsAaGxs5Pnnn2flypXHnFVEsk9Nc6IoTEtDUciJQ1JHpHGIPrmhpqdgyZIl1NbWUl1dTV1dHZMnT2bmzJl88Ytf5JlnniESibB//35qamqYMWPGoM/x29/+ls2bN7N27dpEnMZG3njjDS666CJuuOEGamtreeCBB/jwhz9Mfv74+7WJjGe1TYmikI7uo/H37VI+J9l1NMj0Ubj88stZu3YtBw8eZNWqVdx9993U1dWxfv16otEo8+bNG/aQUXfn1ltv5eKLL37LvCuvvJK7776be++9lx//+Mejyiki2aemqROAyjLtUxh7F34VohPePC06ITF9FFatWsW9997L2rVrufzyy2lsbGTatGlEo1GefPJJ9uwZfoDCiy++mDvuuIPu7m4Atm3bRmtrKwBXXXUVN998MwCLFy8eVU4RyT41TR1MKSmgMD8v8G2Nv5bCGR9J/Hzim4kuo/I5iYLQO/0YLV68mObmZmbPns3MmTNZvXo1H/jAB1i2bBlnnXUWixYtGnb9a665ht27d7N06VLcncrKSh566CEApk+fzimnnMJll102qowikp1qmjqZloZWAoANt/MzEy1btswHXk9h69atnHLKKSElCl5bWxunn346GzZsoLy8fMjlcv19EBmvLr3tOSYXF/DTTy4/5ucws/Xuvuxoy42/7qMs8/jjj7No0SI+97nPDVsQRCR31TR1pGUnM4zH7qMs8973vpe9e/eGHUNEQhKLO3XNnUxPw+GooJaCiEhGa2jpJO7pOUcBcqgoZNu+kbE23l+/SK7qPRx1epp2NOdEUSgqKqKhoWHcfjH2Xk+hqCg9/0mISPrU9J24lp6/70D3KZjZJOCHwGmAA5909z/2m38B8DCwKznpAXcf8XgTc+bMoaqqirq6utFGzlq9V14TkdzSO8RFThQF4L+A37j75WZWABQPssyz7n7JaDYSjUZ1xTERyUk1TZ2YQUVpQVq2F1hRMLOJwPnAVQDu3gV0BbU9EZFcVNvUQUVpIfl56entD3IrC4A64C4ze9nMfmhmJYMsd66ZbTKzX5vZoGM4mNl1ZrbOzNaN5y4iERl/0nmOAgRbFPKBpcAd7r4EaAW+MmCZDcDx7n4mcCvw0GBP5O53uvsyd19WWVkZYGQRkcxS09TJ9LL0HUQSZFGoAqrc/YXk47UkikQfd29y95bk/ceAqJlVBJhJRCSr1DZ3pO0cBQiwKLj7QWCfmZ2cnHQh8Fr/ZcxshiWvOG9my5N5GoLKJCKSTbpjcepbutJyxbVeQR999Dng7uSRRzuBq83segB3XwNcDnzGzHqAdmCVj9eTDUREBqhtTp64lsZ9CoEWBXffCAwclW9Nv/m3AbcFmUFEJFul+8Q1yJEzmkVEclHvZTin5cjRRyIiMgp94x6ppSAiIjVNHeRHjCnF6TmbGVQUREQyVu9lOCMRS9s2VRRERDJUus9RABUFEZGMle4hLkBFQUQkY9U0pe8ynL1UFEREMlBHd4zG9m4VBRERgdrk4ajT0nQZzl4qCiIiGSjdV1zrpaIgIpKBwhjiAlQUREQy0l/OZlb3kYjIuHewsZ2C/AjlE6Jp3a6KgohIBtrd0Ma8qcUkLzmTNioKIiIZaFd9K/MrBrusfbBUFEREMkws7uxpaGV+RWnat62iICKSYfYfbqc75ixQS0FERHbWtwAwv1JFQURk3NtV3wqgfQoiIpIoCmVF+UwtSd/FdXoFWhTMbJKZrTWz181sq5mdO2C+mdktZrbdzDab2dIg84iIZINd9a0sqChJ++GoEHxL4b+A37j7IuBMYOuA+e8DFiZv1wF3BJxHRCTj7awL53BUCLAomNlE4HzgRwDu3uXuRwYs9kHgZ57wJ2CSmc0MKpOISKbr6I5R3dgeyuGoEGxLYQFQB9xlZi+b2Q/NbGDpmw3s6/e4KjntTczsOjNbZ2br6urqgkssIhKyPQ1tuIdz5BEEWxTygaXAHe6+BGgFvjJgmcE6zPwtE9zvdPdl7r6ssrJy7JOKiGSIXcnDUcM4RwGCLQpVQJW7v5B8vJZEkRi4zHH9Hs8BqgPMJCKS0XYmD0edl2tFwd0PAvvM7OTkpAuB1wYs9gjw8eRRSOcAje5+IKhMIiKZblddK9PKCiktzA9l+0Fv9XPA3WZWAOwErjaz6wHcfQ3wGLAS2A60AVcHnEdEJKOFNRBer0CLgrtvBJYNmLym33wHbggyg4hINtlZ38rFi6eHtn2d0SwikiGOtHVxqLUr1JaCioKISIb4y5hH4ZyjACoKIiIZI8yB8HqpKIiIZIhd9a1EDOZOKQ4tg4qCiEiG2FnfynFTiinID++rWUVBRCRD7ApxILxeKgoiIhnA3UM/RwFUFEREMsLBpg7au2OhjXnUS0VBRCQDbD3QBMDJMyaGmkNFQUQkA7xWnSgKp8wsCzWHioKISAbYUt3E8VOLKSuKhppDRUFEJANsqW5i8axwu45ARUFEJHRNHd3sPdTG4lnlYUdRURARCdvW5P6EU9VSEBGRLcmisHimioKIyLi3pbqJitJCpk0sCjuKioKISNi2VDdmxE5mUFEQEQlVZ0+M7bUtKgoiIgJv1LTQE/eM2MkMKgoiIqHaUt0IkBGHowLkB/nkZrYbaAZiQI+7Lxsw/wLgYWBXctID7v7NIDOJiGSSLdVNlBbmc3yIF9bpL9CikPRud68fZv6z7n5JGnKIiGSc16qbOGVmGZGIhR0FUPeRiEho4nFn64GmjOk6guCLggO/NbP1ZnbdEMuca2abzOzXZrZ4sAXM7DozW2dm6+rq6oJLKyKSRrsbWmntinFqBpy01ivo7qMV7l5tZtOA35nZ6+7+TL/5G4Dj3b3FzFYCDwELBz6Ju98J3AmwbNkyDziziEhabMmg4S16BdpScPfq5M9a4EFg+YD5Te7ekrz/GBA1s4ogM4mIZIrXDjQRzTNOmh7uNRT6C6womFmJmZX13gcuAl4dsMwMM7Pk/eXJPA1BZRIRySSv7m9k4bQyCvIzZ/dukN1H04EHk9/5+cA97v4bM7sewN3XAJcDnzGzHqAdWOXu6h4SkZwXjzsb9x3hA2fOCjvKmwRWFNx9J3DmINPX9Lt/G3BbUBlERDLV9roWmjt6WDp3cthR3iRz2iwiIuPIhj2HAVg6d1K4QQZQURARCcHLe48wqTjK/IqSsKO8iYqCiEgINuw9zJLjJpHc75oxVBRERNKssb2bN2pbMm5/AqgoiIik3cZ9RwBYeryKgojIuLdhz2EiBmceNynsKG+hoiAikmYb9h7mpOlllBamY6DqkVFREBFJo96T1jKx6whUFERE0mpH8qS1JRnYdQQqCiIiabVhb/KktWxuKSQHt4sk759kZpeaWTTYaCIiuWfDnsRJawsy7KS1Xqm2FJ4BisxsNvAEcDXwk6BCiYjkqkw9aa1XqkXB3L0N+BBwq7v/NXBqcLFERHJPJp+01ivlomBm5wKrgUeT0zLvWCoRkQz2cobvT4DUi8LfATcBD7r7FjNbADwZWCoRkRz0xx0NFORFWJJhI6P2l9J/++7+NPA0QHKHc727fz7IYCIiuea57fUsPX4SxQWZ29GS6tFH95jZxORlNV8D/mxmNwYbTUQkdxxq7WJLdRMrTsjsy9Cn2n10qrs3AZcBjwFzgSuDCiUikmue31EPwIqFuVEUosnzEi4DHnb3bkDXUhYRSdEftjdQVpjPGbPLw44yrFSLwveB3UAJ8IyZHQ80BRVKRCTX/GF7PeecMJX8vMweSCKldO5+i7vPdveVnrAHePfR1jOz3Wb2ipltNLN1g8w3M7vFzLab2WYzW3oMr0FEJKPtO9TG3kNtrDhhathRjiqlXeBmVg58DTg/Oelp4JtAYwqrv9vd64eY9z5gYfJ2NnBH8qeISM74w/bEV+A7M3x/AqTeffRjoBn4SPLWBNw1Btv/IPCzZOvjT8AkM5s5Bs8rIpIxnttez/SJhZxQWRp2lKNKtSic4O5fc/edyds3gAUprOfAb81svZldN8j82cC+fo+rktPexMyuM7N1Zraurq4uxcgiIuGLx53ndzSw4oSKjB3vqL9Ui0K7mb2z94GZrQDaU1hvhbsvJdFNdIOZnT9g/mDv0FuOanL3O919mbsvq6ysTDGyiEj4Xj/YzKHWLlacmPldR5D6+EXXAz9L7lsAOAx84mgruXt18metmT0ILCcx4mqvKuC4fo/nANUpZhIRyXi9+xOypSikevTRJnc/EzgDOMPdlwDvGW6d5DUYynrvAxcBrw5Y7BHg48mjkM4BGt39wEhfhIhIpnpuez0nVJYwo7wo7CgpGdEBs+7elDyzGeDvj7L4dOA5M9sEvAg86u6/MbPrzez65DKPATuB7cAPgM+OJI+ISCZr6ezhjzsbOP+k7On2Hs2oTMPuMXH3ncCZg0xf0+++AzeMIoOISMZ6YmsNXT1xVp6ePQdVjubUOg1zISIyjEc3H2D6xELelsEX1Rlo2JaCmTUz+Je/ARMCSSQikgOaO7p5alsdq8+eSySS+Yei9hq2KLh7WbqCiIjkkie21tLVE+eSM7Kn6whG130kIiJDePSVA8yYWMSS47Kn6whUFERExlxzRzdP/7mOlafPzKquI1BREBEZc49vraErFuf9WdZ1BCoKIiJj7tHNB5hVXsSS4yaFHWXEVBRERMZQU0c3z2yr531Z2HUEKgoiImPqd1uyt+sIVBRERMbUfS/tY97UYs6aMynsKMdERUFEZIxsq2nmxd2HuCLLTljrT0VBRGSM3PPCXgryIlz+tuOOvnCGUlEQERkDbV09/GJDFStPn8GUkoKw4xwzFQURkTHwq00HaO7o4Yqzjw87yqioKIiIjIG7X9jDwmmlvH1edg1rMZCKgojIKL1S1cimqkZWnz0Xs+zcwdxLRUFEZJTueXEPRdEIf710TthRRk1FQURkFBrbunl4YzWXnjmL8gnRsOOMmoqCiMgo/PyFPbR1xfjEO+aFHWVMqCiIiByjju4Yd/1hF+efVMniWeVhxxkTgRcFM8szs5fN7FeDzLvAzBrNbGPy9tWg84iIjJVfbKiivqWL69+1IOwoY2bYy3GOkS8AW4GJQ8x/1t0vSUMOEZExE4s7P3hmJ2fOKefcBVPDjjNmAm0pmNkc4P3AD4PcjohIuv3vloPsbmjj0+86IesPQ+0v6O6jm4F/BOLDLHOumW0ys1+b2eKA84iIjJq7s+bpHcyvKOHixTPCjjOmAisKZnYJUOvu64dZbANwvLufCdwKPDTEc11nZuvMbF1dXd3YhxURGYE/7mhgc1Uj1563gLwsHQ11KEG2FFYAl5rZbuBe4D1m9vP+C7h7k7u3JO8/BkTNrGLgE7n7ne6+zN2XVVZWBhhZROTobn9qBxWlhXxo6eywo4y5wIqCu9/k7nPcfR6wCvi9u3+s/zJmNsOSnXFmtjyZpyGoTCIio7V+zyGe217Pp89fQFE0L+w4Yy4dRx+9iZldD+Dua4DLgc+YWQ/QDqxyd093JhGRVN38+BtMLSlg9Tlzw44SiLQUBXd/CngqeX9Nv+m3AbelI4OIyGht2HuYZ9+o56b3LaK4IO3/U6eFzmgWEUnRfz3+BlNKCrjy3Oy+ZsJwVBRERFKwcd8Rnt5Wx7XnLcjZVgKoKIiIpOSWJ95gUnE0p1sJoKIgInJUL+0+xO9fr+Xa8xZQWpi7rQRQURARGVZnT4yv/GIzcyZP4OoV88KOE7jcLnkiIqN0+5M72FHXyk8/uTyn9yX0UktBRGQIb9Q0c/tT27nsrFm866TxMZqCioKIyCDicecrD7xCSWE+/3rJqWHHSRsVBRGRQdz94l7W7znMv7z/VKaWFoYdJ21UFEREBthW08y3H93KeQsr+HAODno3HBUFEZF+Wjt7+OzdGygpzOc//ubMnLqATipUFEREktydf3rwFXbWtXDL357FtIlFYUdKOxUFEZGku1/Yy8Mbq/n7/3MS7zjhLZd2GRdUFEREgFf3N/LNX77Gu06q5LMXnBh2nNCoKIjIuNfeFeML977M5JIo3/3oWURy7BKbI5H7p+eJiBzFd369lR11rfz8U2czpaQg7DihUktBRMa1J1+v5Wd/3MOn3jmfdy4cn/sR+lNREJFxq76lkxvXbmLRjDJuvPjksONkBHUfici49U8PvEJTRw8/v+ZsiqJ5YcfJCGopiMi4tGHvYX77Wg1fuHAhi2ZMDDtOxgi8KJhZnpm9bGa/GmSemdktZrbdzDab2dKg84iIAHzv99uZXBzlqnfMCztKRklHS+ELwNYh5r0PWJi8XQfckYY8IjLOvVbdxBOv13L1ivmU5PiV1EYq0KJgZnOA9wM/HGKRDwI/84Q/AZPMbGaQmUREbn9qO6WF+Xzi3HlhR8k4QbcUbgb+EYgPMX82sK/f46rkNBGRQOysa+HRVw5w5bnHU14cDTtOxgmsKJjZJUCtu68fbrFBpvkgz3Wdma0zs3V1dXVjllFExp87ntpBQV6ET66YH3aUjBRkS2EFcKmZ7QbuBd5jZj8fsEwVcFy/x3OA6oFP5O53uvsyd19WWTk+LoknImNv/5F2Hnx5P3+7fC6VZePnwjkjEVhRcPeb3H2Ou88DVgG/d/ePDVjsEeDjyaOQzgEa3f1AUJlEZPxyd77z2FbM4LrzF4QdJ2Olfbe7mV0P4O5rgMeAlcB2oA24Ot15RGR8+MWG/fxq8wFuvPhkZk2aEHacjJWWouDuTwFPJe+v6TfdgRvSkUFExq/d9a187eFXOXv+FK5/1wlhx8loOqNZRHJadyzOF+59mfy8CN/96FnkjeNhsVOhszZEJKd993fb2FTVyB2rl6rbKAVqKYhIznpmWx13PL2Djy47jvedrvNiU6GiICI5qfpIO1+492VOmlbG1y9dHHacrKGiICI5p6snzg33bKA75tzxsaVMKNCw2KnSPgURyTnffmwrL+89wu2rl7KgsjTsOFlFLQURySmPbj7AT57fzSdXzGel9iOMmIqCiOSMxvZuvvrwq5x53CRuWrko7DhZSUVBRHLGzY9v41BbF9+67DSiefp6OxZ610QkJ/z5YDM/++Merlg+l9Nml4cdJ2upKIhI1nN3vvHLLZQW5vOli04OO05WU1EQkaz361cP8vyOBr500UlMLikIO05WU1EQkazW3hXjW49uZdGMMv52+dyw42Q9nacgIlnL3blx7SaqG9u576Pnkq+dy6Omd1BEstZtv9/OrzYf4Mt/tYjl86eEHScnqCiISFb6zasH+I/fbeNDS2bzaV1JbcyoKIhI1nl1fyNfvG8TS+ZO4tsfOh0zXSNhrKgoiEhW2XeojWt+uo5JxVG+f+XbKIpqsLuxpB3NIpI1aps6+NiPXqC9O8Z9nz6HaWVFYUfKOWopiEhWONLWxZU/epG65k7uuvrtLJoxMexIOSmwomBmRWb2opltMrMtZvaNQZa5wMwazWxj8vbVoPKISPZq7ujmqrteYld9Kz/4+DKWzp0cdqScFWT3USfwHndvMbMo8JyZ/drd/zRguWfd/ZIAc4hIFnv2jTq+8otXONjUwR2rl7LixIqwI+W0wIqCuzvQknwYTd48qO2JSG5pbO/m249u5b51+1hQWcL9nz6Htx2vcxGCFuiOZjPLA9YDJwLfc/cXBlnsXDPbBFQDX3L3LUFmEpHM9kZNM/ev28cDG/ZzuK2Lz1xwAl+4cKGOMkqTQIuCu8eAs8xsEvCgmZ3m7q/2W2QDcHyyi2kl8BCwcODzmNl1wHUAc+ce+9gm7q7jmUUy0N6GNh7fWsMjm6rZuO8I+RHjwlOmccO7T+SMOZPCjjeuWKKXJw0bMvsa0Oru/z7MMruBZe5eP9Qyy5Yt83Xr1o14+68fbOKmB17h5o+exfFTS0a8voiMnZ5YnA17j/DE6zU8sbWW7bWJnuaTp5fxN8vmcNmS2VSUFoacMreY2Xp3X3a05QJrKZhZJdDt7kfMbALwXuD/DVhmBlDj7m5my0kcDdUQRJ7Gtm521bfywe/9ge9doZ1VMkY23w9PfBMaq6B8Dlz4VTjjI2GnyjidPTG2HWzhlf2NPL+jnme21dHU0UN+xDh7wRSuWD6XC0+Zpn/YMkCQ3UczgZ8m9ytEgPvd/Vdmdj2Au68BLgc+Y2Y9QDuwygNqupy9YCoP37CCa3+2jo//+EX+9f2n8Il3zKOpo4fttc3sqGulvqWTw61dHGrtpqwon8vfNmfQKzjF404kMnQ3VDzuvLK/kaf+XMeBxnYmFORRUpBPcWEex00u5sRppcyvKKEomkdPLM7htm4OtXbR0NpJQ0tX4n5LJ3UtXdQ1d1Lf0klTezctnT20dvbQ0RNncnGUitJCKssKmTO5mFNmlnHKzIksmlFGWVE0iLcwNJ09MWqbOonmRSjMj1AUzaMwPzLs7yAtNt8Pv/w8dLcnHjfuSzyGcVsY3J1DrV28UdvCa9VNbD3QxJbqJrbVNNMTT/xpV5QWcvHiGbxn0TTeubAi5z6v2S5t3Udj5Vi7j3q1dPbwd/du5PGtNUwpKeBQa9eb5k+I5jGlpID6lk46e+KcMaecjyw7ju5YnPV7DrNhz2HqW7tYedoMVi2fy9nzp2Bm1DZ18Icd9Ty7rZ6nt9XR0NqFWeIPoKMrRmtXD/F+b7UZTCyK0tTRzWC/gojBlJKCvi/+8glRSgvzKS7IpzAa4UhbN3XNHdQ1d7K7oY3G9u6+dScXR5lWVsS0iYVMKSkgktyPYkB+nvV9qRbkRyjIyyOabxTkRYi709zRQ3NHovjE+gUrzM9jVnkRsydPYNakCZQU5BOJQMSM7licfYfa2Xuojb2H2qhv6aS5o5um9h6aO7vpiTndMScWj9MTd2LJmztMnBClsizxGicXR8kzw8yIGNQ2d7KrvpWqw21veu9637/SgnxKChPFFiAWd3piTl7EmFxSwJTiKJOLC8CgsydOV0+cWNyZUJBHcTSPksJ8yoryKZ8QZeKEKOUTopQV5VNWGKWkMI+C/Ag9Me/L3BNPrJ94Lc4Za1dQ1Fr9lt9de/Es1v31MxRF85gQzSM/zzAMs8TvoP9urbhDT8yJe2I78eR77p54PW1dPbR1xWjt7KG4IJ8Z5YXMKJ9AZWkh0Twbch+ZeyJne3eMzu4YXbE43TGnOxYnL/Lmz0B+xIiYkRexvu32vta/vHanrbOH+pa//PPS0NqV/Ceqi7qWTmqaOjjQ2EFXT7wvx9SSAk6dNZHFs8o5fXY5p82eyNwpxdq3F4JUu4/GXVGAxH/yP3xuJ68fbGbhtDJOml7KidNKmVZWxISCxBdMY3s3D728n3te2Mufa5oBmFVexJLjJ1NWmM+jrxyguaOHBRUlRPMifctMLo5y/kmVvPvkaZy3sIKpyX5R98Qf6O76NnbUtbC9toXDbV1MLi5gamkBU0oKmFpSyNTSAqaWFDCpuIC8FP8TdncONnXwWnUTrx9spvpIO7XNndQ2dXC4LVEsnMSXcE/M6eyJ0dkTpzP5JdmfGZQW5lNamP+m7bd3xWgYUEAHM7WkgMqyQiYWRZk4IfE8BfkR8iIRonmJL5/8SOILCIOm9m7qmjupa+7kSHt3X7GIxZ2KsgLmTS1hQUUJsyZNIO6JVkNHd5z2rh6aky2n1s4YZiSfN0J3LM7htq7ErbUbMyjIj1CYn0deJPFaer9oWzp73lJwUrWz8AoG+xXF3VjQefexPekxiFiiOEciiWJqGF2xt/5ugzCpOMqU5Gd4RvkEZpYXMX1iEQsqS1g8cyKVZYUqABlCRWGMuDt/rmlmYlGUWZMm9E1v74rx2CsH+MWGKiJmrDixgvMWVnDqzInhd2uMQOI/3zhdsTgRM0oK8ob8I+7ojrH/SDvVR9rp6I4TdyceT/xnPmdyMXOnFlNamF3DacXjTktXD41t3TQmu+haOhLFoqsnTn6ekZ8XIc8scT9Z0PIjEd7+0PkUtu5/y3N2lc5m8+XP0d4do70rRk+y0PUW5v76F7O8CJgZve9+fiRCcWGy67Egj5bOHg42dVDT2EF9S2eyZZH4jMb63Y+7U5ifx4SCRGugMJpHYV6EaL4RzYsQizud3XE6emJ0dseJ9a6f7BbNixh5ySITzUu81vyIUVSQR0VJAVNLEy3QycVRXdQmi6goiARt4D4FgOgE+MAt43afgmSuVIuCyrzIsTrjI4kCUH4cYImfKgiS5bKrrS+Sac74iIqA5BS1FEREpI+KgoiI9FFREBGRPioKIiLSR0VBRET6qCiIiEifrDt5zczqgD1AOdCYnHy0+70/K4Ahh+UeRv/nTHX+0aZlQ+bhHo915mPJe6yZB5uWLZkz4XMxVMajZR8vmTP1s3y8u1ceLTzunpU34M5U7/f7uW6020p1/tGmZUPm4R6PdeZjyXusmYeYlhWZM+FzkcpnYTxnzsbPcv9bNncf/XIE9/tPG+22Up1/tGnZkHm4x2Od+VjyDjY9lcxDvY6RCiNzJnwuBk7Lhs/ywGn6LA8h67qPRsPM1nkKY39kEmVOj2zLnG15QZnTZbSZs7mlcCzuDDvAMVDm9Mi2zNmWF5Q5XUaVeVy1FEREZHjjraUgIiLDUFEQEZE+KgoiItJHRSHJzM4zszVm9kMzez7sPKkws4iZfcvMbjWzT4SdJxVmdoGZPZt8ry8IO08qzKzEzNab2SVhZ0mFmZ2SfH/Xmtlnws6TCjO7zMx+YGYPm9lFYedJhZktMLMfmdnasLMMJfnZ/WnyvV2dyjo5URTM7MdmVmtmrw6Y/ldm9mcz225mXxnuOdz9WXe/HvgV8NMg8yazjToz8EFgNtANVAWVtV+2scjsQAtQRMCZxygvwJeB+4NJ+WZj9FnemvwsfwQI/HDKMcr8kLtfC1wFfDTAuL3ZxiLzTnf/VLBJ32qE2T8ErE2+t5emtIHRnPmWKTfgfGAp8Gq/aXnADmABUABsAk4FTifxxd//Nq3fevcDE7MhM/AV4NPJdddmSeZIcr3pwN1ZkPe9wCoSX1aXZMN7nFznUuB54IpsyZxc7z+ApVmWOfC/vVFkvwk4K7nMPak8f05cjtPdnzGzeQMmLwe2u/tOADO7F/igu38HGLQbwMzmAo3u3hRkXhibzGZWBXQlH8YCjAuM3fucdBgoDCRo0hi9x+8GSkj8gbWb2WPuHs/kzMnneQR4xMweBe4JKm9yW2PxPhvwb8Cv3X1DkHlhzD/LaTWS7CRa43OAjaTYM5QTRWEIs4F9/R5XAWcfZZ1PAXcFlujoRpr5AeBWMzsPeCbIYMMYUWYz+xBwMTAJuC3QZIMbUV53/2cAM7sKqA+yIAxjpO/xBSS6DQqBx4IMNoyRfpY/R6JVVm5mJ7r7miDDDWGk7/NU4FvAEjO7KVk8wjJU9luA28zs/aQ4DEYuFwUbZNqwZ+q5+9cCypKqEWV29zYShSxMI838AIliFpYRfy4A3P0nYx8lZSN9j58CngoqTIpGmvkWEl9gYRpp5gbg+uDijMig2d29Fbh6JE+UEzuah1AFHNfv8RygOqQsqVLm4GVbXlDmdMnGzL3GLHsuF4WXgIVmNt/MCkjsLHwk5ExHo8zBy7a8oMzpko2Ze41d9nTuNQ9wb/x/Awf4y6GZn0pOXwlsI7FX/p/DzqnMyqvMmXHLxszpyq4B8UREpE8udx+JiMgIqSiIiEgfFQUREemjoiAiIn1UFEREpI+KgoiI9FFRkJxgZi1p3t6YXHPDEteXaDSzl83sdTP79xTWuczMTh2L7YsMpKIgMggzG3ZcMHd/xxhu7ll3XwIsAS4xsxVHWf4yEqO2ioy5XB4QT8Y5MzsB+B5QCbQB17r762b2AeBfSIw73wCsdvcaM/s6MAuYB9Sb2TZgLokx6ucCN3ti4DbMrMXdS5Mjkn4dqAdOA9YDH3N3N7OVwH8m520AFrj7kEMwu3u7mW0kMeIlZnYtcF0y53bgSuAsEtdKeJeZ/Qvw4eTqb3mdx/q+yfimloLksjuBz7n724AvAbcnpz8HnJP87/xe4B/7rfM2EmPoX5F8vIjEUN/Lga+ZWXSQ7SwB/o7Ef+8LgBVmVgR8H3ifu7+TxBf2sMxsMrCQvwyD/oC7v93dzwS2khjO4HkSY9rc6O5nufuOYV6nyIippSA5ycxKgXcA/5O4fgvwl4v6zAHuM7OZJP4L39Vv1Ufcvb3f40fdvRPoNLNaEleMG3gZ0RfdvSq53Y0kWhotwE53733u/ybxX/9gzjOzzcDJwL+5+8Hk9NPM7P+SuPZEKfC/I3ydIiOmoiC5KgIccfezBpl3K/Cf7v5Iv+6fXq0Dlu3sdz/G4H8zgy0z2Pj2Q3nW3S8xs5OA58zsQXffCPwEuMzdNyUv8nPBIOsO9zpFRkzdR5KTPHFJ1V1m9jeQuNyjmZ2ZnF0O7E/e/0RAEV4HFvS7bOJRL0bv7tuA7wBfTk4qAw4ku6xW91u0OTnvaK9TZMRUFCRXFJtZVb/b35P4Iv2UmW0CtpC4Zi0kWgb/Y2bPktgJPOaSXVCfBX5jZs8BNUBjCquuAc43s/nAvwIvAL8jUWR63QvcmDyM9QSGfp0iI6ahs0UCYmal7t6SvCj994A33P27YecSGY5aCiLBuTa543kLiS6r74cbR+To1FIQEZE+aimIiEgfFQUREemjoiAiIn1UFEREpI+KgoiI9FFREBGRPv8fraUVPvsIM6oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Unfreezing allthe weight parameters\n",
    "lm_learner.unfreeze()\n",
    "lm_learner.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "87921dc8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.455807</td>\n",
       "      <td>3.694270</td>\n",
       "      <td>0.343904</td>\n",
       "      <td>03:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.409417</td>\n",
       "      <td>3.627638</td>\n",
       "      <td>0.353409</td>\n",
       "      <td>03:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.153597</td>\n",
       "      <td>3.618243</td>\n",
       "      <td>0.358660</td>\n",
       "      <td>03:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.907474</td>\n",
       "      <td>3.643191</td>\n",
       "      <td>0.361426</td>\n",
       "      <td>03:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.741314</td>\n",
       "      <td>3.674730</td>\n",
       "      <td>0.360460</td>\n",
       "      <td>03:57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Training the whole model for 5 epochs\n",
    "m_learner.fit_one_cycle(5, slice(0.00001, 0.01))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f60f50c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>target</th>\n",
       "      <th>pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos xxmaj between 60 to 70 of the u s economy is driven by consumer spending xxbos xxmaj take a look on the # trading potential of # euets - covered industries affected by # coronavirus . xxmaj do they collapse , do they shift operations ? xxmaj read why the price should bounce soon : https : / / t.co / xxunk \\r\\r\\n▁ # xxup euets # xxunk # xxup co2</td>\n",
       "      <td>xxmaj between 60 to 70 of the u s economy is driven by consumer spending xxbos xxmaj take a look on the # trading potential of # euets - covered industries affected by # coronavirus . xxmaj do they collapse , do they shift operations ? xxmaj read why the price should bounce soon : https : / / t.co / xxunk \\r\\r\\n▁ # xxup euets # xxunk # xxup co2 #</td>\n",
       "      <td>xxmaj the xxmaj + 70 % the xxup s population is spending by consumer spending and xxmaj the a look at the # coronavirus potential of # xxup xxunk xxunk xxunk , by # coronavirus . xxmaj read n't see in they they know operations , xxmaj read more ? # of be back . https : / / t.co / xxunk xxbos # xxunk euets # xxunk # xxunk euets #</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e xxmaj learning in the schools xxmaj xxunk to encourage social distancing xxmaj enough xxmaj food stock xxbos xxmaj don t forget to wash your xxmaj hans xxbos xxmaj fruits being sold a high prices | xxmaj seller caught # xxmaj coronavirus # xxup covid-19 # … https : / / t.co / xxunk via @youtube xxbos xxmaj leaders : xxmaj we do n't want to spread # coronavirus so you can</td>\n",
       "      <td>xxmaj learning in the schools xxmaj xxunk to encourage social distancing xxmaj enough xxmaj food stock xxbos xxmaj don t forget to wash your xxmaj hans xxbos xxmaj fruits being sold a high prices | xxmaj seller caught # xxmaj coronavirus # xxup covid-19 # … https : / / t.co / xxunk via @youtube xxbos xxmaj leaders : xxmaj we do n't want to spread # coronavirus so you can only</td>\n",
       "      <td>- xxunk xxmaj xxmaj xxmaj and xxunk and go people distancing and in to to xxmaj is xxmaj the t panic to stock your hands don sanitizer xxmaj the and sold at little price in xxmaj xxunk xxmaj in xxmaj xxunk \\r\\r\\n xxmaj covid19 # xxmaj \\r\\r\\n : / / t.co / xxunk xxbos xxunk xxbos xxmaj the of xxmaj do are n't need to hear the coronavirus but we can get</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>xxunk # 2020 # drawing # comedy # satire # breakingnews # panicbuyinguk https : / / t.co / xxunk xxbos xxmaj who is the person who described senior citizens as \" 80 year old teenagers \" because xxmaj i 'm pretty sure xxmaj i 'm going to catch my 90 - year - old xxunk xxunk out the house at 7 a.m. to go to the grocery store … \\r\\r\\n\\r\\r\\n▁ #</td>\n",
       "      <td># 2020 # drawing # comedy # satire # breakingnews # panicbuyinguk https : / / t.co / xxunk xxbos xxmaj who is the person who described senior citizens as \" 80 year old teenagers \" because xxmaj i 'm pretty sure xxmaj i 'm going to catch my 90 - year - old xxunk xxunk out the house at 7 a.m. to go to the grocery store … \\r\\r\\n\\r\\r\\n▁ # quarantine</td>\n",
       "      <td>xxunk xxunk # xxunk # xxunk # xxunk # comic # news # : / / t.co / xxunk xxbos xxmaj the knew the first who is the panic to being xxunk % old \" with with of iâm 'm a sure they i 'm not to be them # year year old old daughter who xxmaj of # with the am and 6 to the grocery store . # # coronavirus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19 is pushing us to the fourth industrial revolution by force xxmaj flexible working xxmaj online shopping xxmaj online festival s xxmaj less human contact xxmaj automation xxbos xxmaj if i think i will be contacted about # shielding do i go to work tomorrow ? i work in a supermarket , close proximity to many people for 9 hours # xxunk # xxup covid?19 # coronavirus xxbos xxmaj love to the</td>\n",
       "      <td>is pushing us to the fourth industrial revolution by force xxmaj flexible working xxmaj online shopping xxmaj online festival s xxmaj less human contact xxmaj automation xxbos xxmaj if i think i will be contacted about # shielding do i go to work tomorrow ? i work in a supermarket , close proximity to many people for 9 hours # xxunk # xxup covid?19 # coronavirus xxbos xxmaj love to the freelancers</td>\n",
       "      <td>xxbos a the to close brink level level in the it xxunk xxmaj in customers xxmaj xxmaj more xxmaj xxbos xxbos online xxmaj lives xxbos xxunk is xxmaj the you have the would be able by the coronavirus tonight you have to the and and xxmaj am at a supermarket and and to to others others , a am , coronavirus # coronavirus covid2019 # coronavirus xxbos xxmaj the this see grocery</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>sunshine . xxmaj it is probably a very unlikely place to catch this # coronavirus . xxmaj we arenât closing the stores all the thousands of people in a grocery store each day will spread more germs than crowds at th xxbos xxmaj get it b4 xxmaj amazon xxup bans them too ! ? xxmaj germ - x xxmaj advanced xxmaj aloe xxmaj hand xxmaj sanitizer xxunk fl oz ? from $</td>\n",
       "      <td>. xxmaj it is probably a very unlikely place to catch this # coronavirus . xxmaj we arenât closing the stores all the thousands of people in a grocery store each day will spread more germs than crowds at th xxbos xxmaj get it b4 xxmaj amazon xxup bans them too ! ? xxmaj germ - x xxmaj advanced xxmaj aloe xxmaj hand xxmaj sanitizer xxunk fl oz ? from $ xxunk</td>\n",
       "      <td>xxmaj xxmaj the is a the xxunk good sunshine to see the virus coronavirus thing xxmaj it are even our retail until over time . people are need panic store . day . be the than and we . the xxbos xxmaj the your to you covid-19 is bans it on ! xxmaj ? you xxmaj xxunk xxmaj germ xxmaj hand xxmaj vera xxmaj sanitizer xxmaj xxmaj oz xxmaj # # xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>swiss xxmaj xxunk … https : / / t.co / xxunk xxbos xxunk https : / / t.co / xxunk says \" businesses involved in the supply , delivery , distribution and sale of food , beverage and other key consumer goods ( but not takeaway shops ) \" so sounds like it ? xxbos xxmaj xxunk believes their data specifically on skin temperature heart rate heart rate xxunk and respiratory rate</td>\n",
       "      <td>xxmaj xxunk … https : / / t.co / xxunk xxbos xxunk https : / / t.co / xxunk says \" businesses involved in the supply , delivery , distribution and sale of food , beverage and other key consumer goods ( but not takeaway shops ) \" so sounds like it ? xxbos xxmaj xxunk believes their data specifically on skin temperature heart rate heart rate xxunk and respiratory rate will</td>\n",
       "      <td>xxunk swiss xxmaj . : / / t.co / xxunk xxbos xxmaj xxmaj : / / t.co / xxunk xxbos xxmaj the are in the industry chain delivery , distribution , distribution of food are medicine , other products items goods are including not the shops ) are . that a a 's xxbos xxmaj the xxmaj that # is provides the and is xxunk is failure xxunk is xxunk failure xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>xxunk , xxunk , xxunk , xxunk , xxunk and many more are now making hand sanitizer and offering it for free to those who need it : \\r\\r\\n https : / / t.co / xxunk \\r\\r\\n\\r\\r\\n▁ # americanbusiness # xxmaj coronavirus # smallbiz xxunk ? xxbos xxmaj montana xxmaj farmers having a little fun . https : / / t.co / xxunk xxbos xxmaj support your local liquor store during these</td>\n",
       "      <td>, xxunk , xxunk , xxunk , xxunk and many more are now making hand sanitizer and offering it for free to those who need it : \\r\\r\\n https : / / t.co / xxunk \\r\\r\\n\\r\\r\\n▁ # americanbusiness # xxmaj coronavirus # smallbiz xxunk ? xxbos xxmaj montana xxmaj farmers having a little fun . https : / / t.co / xxunk xxbos xxmaj support your local liquor store during these times</td>\n",
       "      <td>xxunk xxunk , xxunk , xxunk , xxunk , xxunk others . now using their sanitizer . xxunk free to free . those who are it . https https : / / t.co / xxunk xxbos # xxunk # xxmaj coronavirus xxbos xxmaj xxbos xxbos xxbos xxmaj the xxmaj xxunk xxmaj a great bit humor xxmaj : / / t.co / xxunk xxbos xxmaj the local local businesses store by the times</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>kept back for @nhsuk workers in this crisis ? xxmaj can we donate to some sort of fund ? xxmaj dig deep , # stockpilers ? xxbos xxmaj my partner works in a retail store in the city . xxmaj the company , one of the richest in the world , has decided they 'll close at 5 / 6 every day instead of the usual times , because it 's common</td>\n",
       "      <td>back for @nhsuk workers in this crisis ? xxmaj can we donate to some sort of fund ? xxmaj dig deep , # stockpilers ? xxbos xxmaj my partner works in a retail store in the city . xxmaj the company , one of the richest in the world , has decided they 'll close at 5 / 6 every day instead of the usual times , because it 's common knowledge</td>\n",
       "      <td>the from the staff like the time . xxmaj the we help to the of of community ? xxmaj the xxmaj cleaning deep xxunk # ? xxmaj the xxmaj xxmaj at a supermarket store and xxmaj xxmaj of xxmaj she manager has xxmaj of the largest in the country , has been to are hire their least xxrep 11 . day . of letting usual usual . and they 's a people</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>is lockdown now … . i want my family safe and off course i want to stock food and xxunk xxbos xxmaj heard the rumors about expensive , private xxup covid-19 tests or special treatment for xxup nba stars ? xxmaj the xxmaj xxunk xxmaj world received answers to the latest questions and concerns about testing for the coronavirus . https : / / t.co / xxunk xxbos xxmaj useful information about</td>\n",
       "      <td>lockdown now … . i want my family safe and off course i want to stock food and xxunk xxbos xxmaj heard the rumors about expensive , private xxup covid-19 tests or special treatment for xxup nba stars ? xxmaj the xxmaj xxunk xxmaj world received answers to the latest questions and concerns about testing for the coronavirus . https : / / t.co / xxunk xxbos xxmaj useful information about supermarket</td>\n",
       "      <td>a in . xxmaj xxmaj am to wife to and stay with . will to go up and stock my xxmaj the that news on the prices expensive equity covid-19 testing , vaccines testing for xxup covid-19 players xxmaj xxmaj this xxmaj xxunk xxmaj county xxmaj reports about answer questions questions : answers about the . xxup coronavirus . https : / / t.co / xxunk xxbos xxmaj the tips from #</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lm_learner.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc03538f",
   "metadata": {},
   "source": [
    "## Though we wont requre the the language model but still I am going to save it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "23c65916",
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_learner.export(r\"..\\model\\language_model.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd95a63",
   "metadata": {},
   "source": [
    "## And finally save the updated encoder which we will use during the main classifier training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "7a36a33c",
   "metadata": {},
   "outputs": [],
   "source": [
    "lm_learner.save_encoder(r\"..\\model\\enc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "937176c0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}