lm3-french.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "e38c0aad-6ca2-456d-b908-cbbec3c9e34d"
    }
   },
   "source": [
    "# (MultiFiT) French Bidirectional Language Model (LM) from scratch\n",
    "### (architecture 4 QRNN with 1550 hidden parameters by layer, SentencePiece tokenizer and hyperparameters from the MultiFiT method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: September 2019\n",
    "- Post in medium: [link](https://medium.com/@pierre_guillou/nlp-fastai-french-language-model-d0e2a9e12cab)\n",
    "- Ref: [Fastai v1](https://docs.fast.ai/) (Deep Learning library on PyTorch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Information**\n",
    "\n",
    "According to this new article \"[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)\" (September 10, 2019), the architecture 4 QRNN and the SentencePiece tokenizer (15 000 tokens) give better results than AWD-LSTM and the spaCy tokenizer respectively.\n",
    "\n",
    "Therefore, they have been used in this notebook to train a French Bidirectional Language Model on a Wikipedia corpus of 100 millions tokens. As you can observe in the Results paragraph, **this French Bidirectional LM model is far better than the 2 precedent ones I trained** (see [my Language Models repository](https://github.com/piegu/language-models) on github).\n",
    "\n",
    "More, the hyperparameters values given at the end of the article have been used, too.\n",
    "\n",
    "**Wikipedia corpus**\n",
    "- download: 512 659 articles of 492 596 078 tokens tokens\n",
    "- used: 252 898 articles of 100 716 190 tokens\n",
    "\n",
    "**Hyperparameters values**\n",
    "- (batch size) bs = 50\n",
    "- (QRNN) 4 QRNN (default: 3) with 1550 hidden parameters each one (default: 1152)\n",
    "- (SentencePiece) vocab of 15000 tokens\n",
    "- (dropout) mult_drop = 0\n",
    "- (weight decay) wd = 0.01\n",
    "- (number of training epochs) 10 epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " **This Bidirectional French Language Model is far better than the 2 precedent ones I trained** (see [my Language Models repository](https://github.com/piegu/language-models) on github)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- forward : (accuracy) 43.77% | (perplexity) 16.09\n",
    "- backward: (accuracy) 49.29% | (perplexity) 16.58"
   ]
  },
    {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**To be improved**"
   ]
  },
    {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lost function FlattenedLoss of LabelSmoothingCrossEntropy should be tested as it is used in the MultiFiT method (see the notebook [lm3-portuguese-classifier-TCU-jurisprudencia.ipynb](https://github.com/piegu/language-models/blob/master/lm3-portuguese-classifier-TCU-jurisprudencia.ipynb) to get the code)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "151cd18f-76e3-440f-a8c7-ffa5c6b5da01"
    }
   },
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "96f02439-3586-4c9d-8c34-aa7c3b17a0a6"
    }
   },
   "outputs": [],
   "source": [
    "# batch size to be choosen according to your GPU \n",
    "# bs=48\n",
    "# bs=24\n",
    "bs=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6ceb4db2-e4cf-4fe0-a393-91df4a7ed3e7"
    }
   },
   "outputs": [],
   "source": [
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6329e650-fc03-4323-ac0c-3aa280e0de91"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fastai: 1.0.57\n",
      "cuda: True\n"
     ]
    }
   ],
   "source": [
    "import fastai\n",
    "print(f'fastai: {fastai.__version__}')\n",
    "print(f'cuda: {torch.cuda.is_available()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6f24e68b-3df0-4997-8a50-3a37ea6a5257"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\r\n",
      "```text\r\n",
      "=== Software === \r\n",
      "python        : 3.7.4\r\n",
      "fastai        : 1.0.57\r\n",
      "fastprogress  : 0.1.21\r\n",
      "torch         : 1.2.0\r\n",
      "nvidia driver : 410.104\r\n",
      "torch cuda    : 10.0.130 / is available\r\n",
      "torch cudnn   : 7602 / is enabled\r\n",
      "\r\n",
      "=== Hardware === \r\n",
      "nvidia gpus   : 1\r\n",
      "torch devices : 1\r\n",
      "  - gpu0      : 16130MB | Tesla V100-SXM2-16GB\r\n",
      "\r\n",
      "=== Environment === \r\n",
      "platform      : Linux-4.9.0-9-amd64-x86_64-with-debian-9.9\r\n",
      "distro        : #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11)\r\n",
      "conda env     : base\r\n",
      "python        : /opt/anaconda3/bin/python\r\n",
      "sys.path      : /home/jupyter/tutorials/fastai/course-nlp\r\n",
      "/opt/anaconda3/lib/python37.zip\r\n",
      "/opt/anaconda3/lib/python3.7\r\n",
      "/opt/anaconda3/lib/python3.7/lib-dynload\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions\r\n",
      "```\r\n",
      "\r\n",
      "Please make sure to include opening/closing ``` when you paste into forums/github to make the reports appear formatted as code sections.\r\n",
      "\r\n",
      "Optional package(s) to enhance the diagnostics can be installed with:\r\n",
      "pip install distro\r\n",
      "Once installed, re-run this utility to get the additional information\r\n"
     ]
    }
   ],
   "source": [
    "!python -m fastai.utils.show_install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "194a6989-31f1-4702-b94d-32b974ded8e6"
    }
   },
   "outputs": [],
   "source": [
    "data_path = Config.data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "cf070ab7-babb-4cf0-a315-401f65461dc8"
    }
   },
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "70da588b-8af1-4f97-97c2-c9f2d4d46e1a"
    }
   },
   "outputs": [],
   "source": [
    "lang = 'fr'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "701ab344-0430-4f43-bbe2-337a12cae6be"
    }
   },
   "outputs": [],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "\n",
    "lm_fns3 = [f'{lang}_wt_sp15_multifit', f'{lang}_wt_vocab_sp15_multifit']\n",
    "lm_fns3_bwd = [f'{lang}_wt_sp15_multifit_bwd', f'{lang}_wt_vocab_sp15_multifit_bwd']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2"
    }
   },
   "source": [
    "## Data (French wikipedia)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "nbpresent": {
     "id": "4e67d876-c7d0-4bdf-a6f9-ae06ae1fc023"
    }
   },
   "source": [
    "### Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "dd2fd658-b690-484c-b60a-69dc6b7bf384"
    }
   },
   "outputs": [],
   "source": [
    "from nlputils import split_wiki,get_wiki\n",
    "from nlputils2 import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "28c01920-f13c-493e-9a97-e5b2c24133a8"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extracting...\n",
      "CPU times: user 52 ms, sys: 12 ms, total: 64 ms\n",
      "Wall time: 17min 30s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "get_wiki(path,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "100000\n",
      "200000\n",
      "300000\n",
      "400000\n",
      "500000\n",
      "600000\n",
      "700000\n",
      "800000\n",
      "900000\n",
      "1000000\n",
      "1100000\n",
      "1200000\n",
      "1300000\n",
      "1400000\n",
      "1500000\n",
      "1600000\n",
      "1700000\n",
      "1800000\n",
      "1900000\n",
      "2000000\n",
      "2100000\n",
      "2200000\n",
      "2300000\n",
      "2400000\n",
      "2500000\n",
      "2600000\n",
      "2700000\n",
      "2800000\n",
      "2900000\n",
      "3000000\n",
      "3100000\n",
      "3200000\n",
      "3300000\n",
      "3400000\n",
      "3500000\n",
      "3600000\n",
      "3700000\n",
      "3800000\n",
      "3900000\n",
      "4000000\n",
      "4100000\n",
      "4200000\n",
      "4300000\n",
      "4400000\n",
      "4500000\n",
      "4600000\n",
      "4700000\n",
      "4800000\n",
      "4900000\n",
      "5000000\n",
      "5100000\n",
      "5200000\n",
      "5300000\n",
      "5400000\n",
      "5500000\n",
      "5600000\n",
      "5700000\n",
      "5800000\n",
      "5900000\n",
      "6000000\n",
      "6100000\n",
      "6200000\n",
      "6300000\n",
      "6400000\n",
      "6500000\n",
      "6600000\n",
      "6700000\n",
      "6800000\n",
      "6900000\n",
      "7000000\n",
      "7100000\n",
      "7200000\n",
      "7300000\n",
      "7400000\n",
      "7500000\n",
      "7600000\n",
      "7700000\n",
      "7800000\n",
      "7900000\n",
      "8000000\n",
      "8100000\n",
      "8200000\n",
      "8300000\n",
      "8400000\n",
      "8500000\n",
      "8600000\n",
      "8700000\n",
      "8800000\n",
      "8900000\n",
      "9000000\n",
      "9100000\n",
      "9200000\n",
      "9300000\n",
      "9400000\n",
      "9500000\n",
      "9600000\n",
      "9700000\n",
      "9800000\n",
      "9900000\n",
      "10000000\n",
      "10100000\n",
      "10200000\n",
      "10300000\n",
      "10400000\n",
      "10500000\n",
      "10600000\n",
      "10700000\n",
      "10800000\n",
      "10900000\n",
      "11000000\n",
      "11100000\n",
      "11200000\n",
      "11300000\n",
      "11400000\n",
      "11500000\n",
      "11600000\n",
      "11700000\n",
      "11800000\n",
      "11900000\n",
      "12000000\n",
      "12100000\n",
      "12200000\n",
      "12300000\n",
      "12400000\n",
      "12500000\n",
      "12600000\n",
      "12700000\n",
      "12800000\n",
      "12900000\n",
      "13000000\n",
      "13100000\n",
      "13200000\n",
      "13300000\n",
      "13400000\n",
      "13500000\n",
      "13600000\n",
      "13700000\n",
      "13800000\n",
      "13900000\n",
      "14000000\n",
      "14100000\n",
      "14200000\n",
      "14300000\n",
      "14400000\n",
      "14500000\n",
      "14600000\n",
      "14700000\n",
      "14800000\n",
      "14900000\n",
      "15000000\n",
      "15100000\n",
      "15200000\n",
      "15300000\n",
      "15400000\n",
      "15500000\n",
      "15600000\n",
      "15700000\n",
      "15800000\n",
      "15900000\n",
      "16000000\n",
      "16100000\n",
      "16200000\n",
      "16300000\n",
      "16400000\n",
      "16500000\n",
      "16600000\n",
      "16700000\n",
      "16800000\n",
      "16900000\n",
      "17000000\n",
      "17100000\n",
      "17200000\n",
      "17300000\n",
      "17400000\n",
      "17500000\n",
      "17600000\n",
      "17700000\n",
      "17800000\n",
      "17900000\n",
      "18000000\n",
      "18100000\n",
      "18200000\n",
      "18300000\n",
      "18400000\n",
      "18500000\n",
      "18600000\n",
      "18700000\n",
      "18800000\n",
      "18900000\n",
      "19000000\n",
      "19100000\n",
      "19200000\n",
      "19300000\n",
      "19400000\n",
      "19500000\n",
      "19600000\n",
      "19700000\n",
      "19800000\n",
      "19900000\n",
      "20000000\n",
      "20100000\n",
      "20200000\n",
      "20300000\n",
      "20400000\n",
      "20500000\n",
      "CPU times: user 37.7 s, sys: 14.8 s, total: 52.6 s\n",
      "Wall time: 1min 19s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/jupyter/.fastai/data/frwiki/docs')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "split_wiki(path,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e6eae780-775e-45e9-9b99-b8a87d5fb8ff"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/frwiki/frwiki-latest-pages-articles.xml'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/frwiki'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/wikiextractor'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/frwiki-latest-pages-articles.xml.bz2'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/log')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e1ac63e7-1cbb-4996-838d-dc58446a65ef"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<doc id=\"3\" url=\"https://fr.wikipedia.org/wiki?curid=3\" title=\"Antoine Meillet\">\r\n",
      "Antoine Meillet\r\n",
      "\r\n",
      "Paul Jules Antoine Meillet, né le à Moulins (Allier) et mort le à Châteaumeillant (Cher), est le principal linguiste français des premières décennies du . Il est aussiphilologue.\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {path}/{name}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "ae770e72-e7a9-473d-8454-2020a0263be8"
    }
   },
   "source": [
    "This function splits the single wikipedia file into a separate file per article. This is often easier to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "d23e0ef7-21e5-4cc5-945d-60ee33c02ce3"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 31s, sys: 28.7 s, total: 2min\n",
      "Wall time: 8min 37s\n"
     ]
    }
   ],
   "source": [
    "# %%time\n",
    "folder = \"docs\"\n",
    "clean_files(path,folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "92b0b087-a6a8-403a-a7a1-d9b47757e5cf"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Min_y_.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Darius_Johnson_Odom.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Guillaume_Cherel.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Henk_Badings.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Henri_de_Virel.txt')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dest = path/'docs'\n",
    "dest.ls()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Darius Earvin Johnson-Odom, né le , à Youngsville, en Caroline du Nord, est un joueur américain de basket-ball. Il évolue au poste d'arrière.\r\n",
      "\r\n",
      "Johnson-Odom passe trois saisons à l'université de Marquette avant d'être sélectionné à la de la draft 2012 de la NBA par les Mavericks de Dallas, qui le transfèrent immédiatement aux Lakers de Los Angeles. Le 15 septembre 2012, il signe son contrat rookie avec les Lakers. Johnson-Odom est envoyé, plusieurs fois durant la saison 2012-2013, chez les D-Fenders de Los Angeles, l'équipe de D-League affiliée aux Lakers.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {dest}/'Darius_Johnson_Odom.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "nbpresent": {
     "id": "575fa672-7b3a-4238-923f-ec929d3a00ee"
    }
   },
   "source": [
    "### Size of downloaded data in the docs folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "270470c2-e0eb-446a-9654-de6c45bc4f0d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "512659 files - 492596078 tokens\n",
      "CPU times: user 57.1 s, sys: 18.9 s, total: 1min 16s\n",
      "Wall time: 2min 49s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "num_files, num_tokens = get_num_tokens(dest)\n",
    "print(f'{num_files} files - {num_tokens} tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "nbpresent": {
     "id": "daae36a0-d90b-45ad-b0e3-a2cd56ce7079"
    }
   },
   "source": [
    "### Create a corpus of about 100 millions of tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6c383e0e-f4f6-46e5-9f54-469437d66f07"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "files copied to the new corpus folder: /home/jupyter/.fastai/data/frwiki/corpus_100000000\n",
      "CPU times: user 13.3 s, sys: 8.85 s, total: 22.1 s\n",
      "Wall time: 22.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "path_corpus = get_corpus(dest, path, num_tokens, obj_tokens=1e8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "b597308e-9521-4637-9e84-85a6cd2ace85"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "252898 files - 100716190 tokens\n",
      "CPU times: user 13.9 s, sys: 2.92 s, total: 16.8 s\n",
      "Wall time: 2min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# VERIFICATION of the number of words in the corpus folder\n",
    "num_files_corpus, num_tokens_corpus = get_num_tokens(path_corpus)\n",
    "print(f'{num_files_corpus} files - {num_tokens_corpus} tokens')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# change name of the corpus \n",
    "!mv {path}/'corpus_100000000' {path}/'corpus2_100'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dest = path/'corpus2_100'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 39min 55s, sys: 33.6 s, total: 40min 28s\n",
      "Wall time: 57min 12s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = (TextList.from_folder(dest, processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "        .split_by_rand_pct(0.1, seed=42)\n",
    "        .label_for_lm()\n",
    "        .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.save(f'{path}/{lang}_databunch_corpus2_100_sp15_multifit')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15000, 227609)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.vocab.itos),len(data.train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15000, 15000)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.vocab.itos),len(data.vocab.stoi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text ▁xxbos ▁xxmaj ▁dar ius ▁xxmaj ▁ e ar vin ▁johnson - o dom , ▁né ▁le ▁ , ▁à ▁xxmaj ▁young s ville , ▁en ▁xxmaj ▁caroline ▁du ▁xxmaj ▁nord , ▁est ▁un ▁joueur ▁américain ▁de ▁basket - ball . ▁xxmaj ▁il ▁évolue ▁au ▁poste ▁d ' arrière . ▁johnson - o dom ▁passe ▁trois ▁saisons ▁à ▁l ' université ▁de ▁xxmaj ▁marque tte ▁avant ▁d ' être ▁sélectionné ▁à ▁la ▁de ▁la ▁draft ▁2012 ▁de ▁la ▁xxup ▁nba ▁par ▁les ▁xxmaj ▁ma ve rick s ▁de ▁xxmaj ▁dallas , ▁qui ▁le ▁trans f èrent ▁immédiatement ▁aux ▁xxmaj ▁la ker s ▁de ▁xxmaj ▁los ▁xxmaj ▁angeles . ▁xxmaj ▁le ▁15 ▁septembre ▁2012, ▁il ▁signe ▁son ▁contrat ▁ro ok ie ▁avec ▁les ▁xxmaj ▁la ker s . ▁johnson - o dom ▁est ▁envoyé , ▁plusieurs ▁fois ▁durant ▁la ▁saison ▁2012-2013 , ▁chez ▁les ▁d - f en der s ▁de ▁xxmaj ▁los ▁xxmaj ▁angeles , ▁l ' équipe ▁de ▁d - le a gue ▁affilié e ▁aux ▁xxmaj ▁la ker s . ▁xxmaj ▁le ▁7 ▁janvier ▁2013, ▁johnson - o dom ▁est ▁coupé ▁par ▁les ▁xxmaj ▁la ker s . ▁xxmaj ▁c ' était ▁le ▁dernier ▁jour ▁pour ▁les ▁équipes ▁xxup ▁nba ▁pour ▁libérer ▁des ▁joueurs ▁dont ▁le ▁contrat ▁n ' est ▁pas ▁garanti ▁avant ▁que ▁leur ▁contrat ▁de vienne ▁garanti ▁jusqu ' à ▁la ▁fin ▁de ▁la ▁saison . ▁xxmaj ▁il ▁joue ▁quatre ▁matches ▁et ▁seulement ▁six ▁minutes ▁au ▁total ▁avec ▁les ▁xxmaj ▁la ker s , ▁passant ▁la ▁plupart ▁de ▁son ▁temps ▁en ▁d - le a gue ▁où ▁il ▁est ▁le ▁meilleur ▁marque ur ▁des ▁d - f en der s , ▁avec ▁21 ▁points ▁par ▁match . ▁xxmaj ▁le ▁24 ▁janvier ▁2013, ▁johnson - o dom ▁part ▁en ▁xxmaj ▁russie ▁où ▁il ▁signe ▁avec ▁le ▁xxmaj ▁ sparta k ▁saint - pétersbourg ▁pour ▁le ▁reste ▁de ▁la ▁saison ▁2012-2013 . ▁xxmaj ▁il ▁participe ▁avec ▁les ▁xxmaj ▁celtic s ▁de ▁xxmaj ▁boston ▁à ▁la ▁xxmaj ▁summer ▁xxmaj ▁league ▁2013 ▁d ' orlando . ▁xxmaj ▁le ▁25 ▁septembre ▁2013, ▁il ▁rejoint ▁les ▁xxmaj ▁la ker s ▁de ▁xxmaj ▁los ▁xxmaj ▁angeles ▁pour ▁participer ▁au ▁camp ▁d ' entraînement . ▁xxmaj ▁toutefois , ▁ils ▁le ▁libère nt ▁le ▁16 ▁octobre . ▁xxmaj ▁le ▁18 ▁octobre ▁2013, ▁il ▁signe ▁en ▁xxmaj ▁chine ▁au ▁ . ▁xxmaj ▁en ▁novembre ▁2013, ▁après ▁quatre ▁matches ▁de ▁saison ▁régulière , ▁il ▁quitte ▁les ▁xxmaj ▁blue ▁xxmaj ▁wh ales . ▁xxmaj ▁le ▁3 ▁janvier ▁2014, ▁il ▁rejoint ▁l ' armor ▁de ▁xxmaj ▁ spring field ▁en ▁d - le a gue . ▁xxmaj ▁le ▁14 ▁mars ▁2014, ▁il ▁signe ▁un ▁contrat ▁de ▁dix ▁jours ▁avec ▁les ▁ 76 ers ▁de ▁xxmaj ▁philadelphie . ▁xxmaj ▁le ▁24 ▁mars ▁2014, ▁il ▁ne ▁signe ▁pas ▁de ▁second ▁contrat ▁de ▁dix ▁jours ▁à ▁la ▁fin ▁du ▁premier . ▁johnson - o dom ▁détient ▁actuellement ▁le ▁record ▁du ▁plus ▁grand ▁nombre ▁de ▁tirs ▁tenté s ▁sans ▁en ▁réussi r ▁un ▁seul ▁en ▁xxup ▁nba ▁avec ▁11 ▁tirs . ▁xxmaj ▁le ▁2 ▁août ▁2014, ▁il ▁signe ▁en ▁xxmaj ▁italie , ▁au ▁xxmaj ▁c ant ù ▁pour ▁la ▁saison ▁2014-2015 . ▁xxmaj ▁le ▁11 ▁décembre ▁2014, ▁il ▁est ▁nommé ▁xxup ▁m v p ▁de ▁la ▁de ▁l ' euro coup e . ▁xxmaj ▁le ▁14 ▁juin ▁2015, ▁il ▁signe ▁en ▁xxmaj ▁turquie ▁au ▁xxmaj ▁tra b zon s por ▁xxmaj ▁basket ball ▁pour ▁la ▁saison ▁2015-2016 . ▁xxmaj ▁le ▁28 ▁décembre ▁2015, ▁il ▁signe ▁en ▁xxmaj ▁grèce , ▁à ▁l ' olympia k ó s ▁pour ▁le ▁reste ▁de ▁la ▁saison ▁2015-2016 ▁et ▁le ▁xxmaj ▁top ▁16 ▁de ▁l ' euro ligu e . ▁xxmaj ▁le ▁11 ▁juin ▁2016, ▁il ▁retourne ▁en ▁xxmaj ▁italie ▁où ▁il ▁signe ▁à ▁xxmaj ▁sa s s ari . ▁xxmaj ▁les ▁statistiques ▁en ▁matchs ▁universitaires ▁de ▁xxmaj ▁dar ius ▁johnson - o dom ▁sont ▁les ▁suivantes ▁:"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds.x[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 40min 57s, sys: 29.3 s, total: 41min 26s\n",
      "Wall time: 21min 32s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = (TextList.from_folder(dest, processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "            .split_by_rand_pct(0.1, seed=42)\n",
    "            .label_for_lm()\n",
    "            .databunch(bs=bs, num_workers=1, backwards=True))\n",
    "\n",
    "data.save(f'{path}/{lang}_databunch_corpus2_100_sp15_multifit_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "052df7c2-f57b-4596-9189-c9e01102e5e9"
    }
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "defab6d9-ba04-4943-b574-9300e20d5e1c"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.05 s, sys: 648 ms, total: 4.7 s\n",
      "Wall time: 4.69 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15_multifit', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.93 s, sys: 2.24 s, total: 6.17 s\n",
      "Wall time: 41.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, drop_mult=0., pretrained=False, \n",
    "                               metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e74f8988-7c26-495b-90fb-0da012f54c1a"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e0333edf-f5cd-4c7d-9db3-01dfb7cc4bb8"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dcnewIhbAHZQhABUQSRyKbiAlqr/tzqtWpdq1Jb69rlto/29t5q29tWu9yuFq2oraJVrFVrEatiK4IQVBAQlJ2ASFgDJCHb5/fHTGSMSQhkzpxM8n4+HvPIzDnfOedNmOST71m+X3N3RESk40oJO4CIiIRLhUBEpINTIRAR6eBUCEREOjgVAhGRDi4t7ACHqmfPnl5YWBh2DBGRpLJo0aJt7p7f2LqkKwSFhYUUFxeHHUNEJKmY2fqm1unQkIhIB6dCICLSwakQiIh0cCoEIiIdnAqBiEgHp0IgItLBqRCIiHRwgd5HYGa3ATcCBtzv7r9spM1pwC+BdGCbu58aRJaVW/bw9yWbGwsZl+03t5XmdmHNvLPh+6zBcotp8Om2htkn31O/LPa9Fl2XYjHtzUiJtk+xT763vl3s1xQzUlMi20w1IyX6PC2l/nX0ecwjPTWFtBQjLSWF9DQjIzWF9LQUMlIjj5SU+Py/iMjBBVYIzGwEkSIwFqgCZpnZ3939g5g2XYHfAWe7+wYz6xVUnlVb9/LrV1d9YpmmYmi7MlJTyExLITM9leyMFHLS08jKSCU3M43crMgjLzudbp0y6NEpg+6dMjmiSxYFPXLIy04PO75IUgmyRzAcmO/u5QBm9hpwEfDTmDZXAE+7+wYAd98aVJhzR/bh3JHnBrX5JjU38U9zhajhqvrt+MevwaOvGttOw/Ue3YbHto+2qV9f537gfQ510ed1Hn2vR95b5x5pG91mbR3U1vnH7691//h1bZ1TV+fUulNT59TWOjV1dZHndU51rVNdW0d1bR1VNXXsrznwdX9NLZXVtVRW11FRVUt5dS17K6vZUlbJnspqdpVXs7+m7lP/9q456Qzs0YnRA7pyYmF3ThzUjV65WU1/s0U6uCALwVLgh2bWA6gAzgEajg0xFEg3szlALvB/7v5IgJkSzpo5LnRoR6V0qKQx5VU1bN9bxY59VXy4u4L128tZv6OcNaV7eWLhRh56Yx0Aw3rncvXEgVw8uj/ZGanhhhZpYyzIqSrN7HrgZmAvsByocPc7Ytb/BigCJgPZwDzgXHd/v8F2pgJTAQoKCsasX9/kkBkiH6uurWPZ5jIWrN3Oc4s/5N1Nu+mWk86V4wdy3UmD6N4pI+yIIgljZovcvajRdYmas9jMfgSUuPvvYpZ9C8hy9/+Jvv4jMMvdn2xqO0VFRa5B5+RQuTsL1u7ggdfX8s/3PuKILlncf3URI/rlhR1NJCGaKwSBXj5af/LXzAqAi4EZDZr8DTjFzNLMLAcYB7wXZCbpmMyMcUf24P6ri3j25pMx4JL73uC5xY1cSSbSwQR9H8FMM1sOPAfc7O47zewmM7sJwN3fA2YBS4AFwAPuvjTgTNLBHdc/j7999WRG9M3jlhlvc++LK5s9qS/S3iXs0FC86NCQxMv+mlq+98wynijeyA8uHMGV4weGHUkkMKEdGhJpyzLTUvnx545j/JHd+emsFZTu2R92JJFQqBBIh2Zm/ODC46ioruWHf18edhyRUKgQSId3VK/OfPnUwTzzzmbmrtoWdhyRhFMhEAG+cvpRDOyRw3efWUpldW3YcUQSSoVABMhKT+XuC0awdts+fj9nddhxRBJKhUAkatLQfM4b2Yf7XlvNpl0VYccRSRgVApEY3z5nOAD3zFoRchKRxFEhEInRr2s2N5wyiGfe2cw7G3eFHUckIVQIRBr48mlH0bNzBj94frnuOJYOQYVApIHOmWnceeYwitfvZNbSLWHHEQmcCoFIIy4t6s+w3rn87z9WsL9Gl5NK+6ZCINKItNQUvnPucDbsKGfGmxvCjiMSKBUCkSZMGprPmIHdmP7GOurqdK5A2i8VApFmXDuxkPXby5nzfmDTaYuEToVApBlnjziC3l0ymT53XdhRRAKjQiDSjPTUFK4aP5B/f7CNVVv3hB1HJBAqBCIHcfnYAjLSUnj4jfVhRxEJhAqByEH06JzJ+aP6MvOtEsoqq8OOIxJ3KgQiLXDtxELKq2p5srgk7CgicadCINICI/rlUTSwGw+/sY5aXUoq7UyghcDMbjOzpWa2zMxub6bdiWZWa2aXBJlHpDWuPamQDTvKeU2Xkko7E1ghMLMRwI3AWGAUcJ6ZDWmkXSrwE+DFoLKIxMNnjo1cSvqQThpLOxNkj2A4MN/dy929BngNuKiRdrcAMwH9mSVtWnpqCl8YN5B/vV/KmtK9YccRiZsgC8FSYJKZ9TCzHOAcYEBsAzPrR6Q43NfchsxsqpkVm1lxaWlpYIFFDuaysQNITzUemadegbQfgRUCd3+PyCGfl4BZwGKgpkGzXwL/6e7NDu/o7tPcvcjdi/Lz8wPJK9ISvXKzOPe4PsxcVMLe/Q0/ziLJKdCTxe7+R3c/wd0nATuADxo0KQIeN7N1wCXA78zswiAzibTW1RML2bO/hr++vSnsKCJxEfRVQ72iXwuAi4EZsevdfZC7F7p7IfAU8BV3fybITCKtNXpAV0b2z+ORN9ZpBjNpF4K+j2CmmS0HngNudvedZnaTmd0U8H5FAmNmXD2hkA+27mXe6u1hxxFptbQgN+7upzSyrNETw+5+bZBZROLpvJF9+NEL7zH9jXVMPKpn2HFEWkV3Foschqz0VK4YW8A/3/uI9dv3hR1HpFVUCEQO01UTBpKWYpqrQJKeCoHIYerdJYvzRvblyeKNGpVUkpoKgUgrXH/yIPZV1fKXhRvDjiJy2FQIRFphRL88xg7qzvS566iprQs7jshhUSEQaaXrTx7Epl0VzF7+UdhRRA6LCoFIK00Z3puC7jn88fW1YUcROSwqBCKtlJpiXDuxkEXrd/LOxl1hxxE5ZCoEInFw6YkDyM1MU69AkpIKgUgcdM5M4/JxBbzw7ods2lURdhyRQ6JCIBIn10wsBOChueoVSHJRIRCJk35dsznnuD48vmAje3SDmSQRFQKROLr+5EHs2V/DX4pLwo4i0mIqBCJxdPyArpxY2I3pc9fqBjNJGioEInF2/clHUrJTN5hJ8lAhEImzM4/pzcAeOdz/7zVhRxFpERUCkTirv8Hs7Q27WKwbzCQJqBCIBOCSMf3plJHKI/PWhx1F5KBUCEQCkJuVzsUn9Oe5JZvZvnd/2HFEmqVCIBKQqycMpKqmjieKNVeBtG2BFgIzu83MlprZMjO7vZH1XzCzJdHHG2Y2Ksg8Iok0pHcuEwf34NH5G3QpqbRpgRUCMxsB3AiMBUYB55nZkAbN1gKnuvtI4G5gWlB5RMJw9YRCNu2q4OUVW8OOItKkIHsEw4H57l7u7jXAa8BFsQ3c/Q133xl9OR/oH2AekYSbMrwXffOyeGTeuoTve8HaHWzWAHjSAkEWgqXAJDPrYWY5wDnAgGbaXw/8o7EVZjbVzIrNrLi0tDSAqCLBSEtN4QvjBzJ31XZWbd2T0H1/+c+LuO3xt3H3hO5Xkk9ghcDd3wN+ArwEzAIWAzWNtTWz04kUgv9sYlvT3L3I3Yvy8/MDSiwSjMtOHEBGakpCLyWtq3N2llexcN1O5q/ZkbD9SnIK9GSxu//R3U9w90nADuCDhm3MbCTwAHCBu28PMo9IGHp0zuSc447gr29toryq0b+F4m5fVQ110Y7Ar17+1I+dyCcEfdVQr+jXAuBiYEaD9QXA08BV7v5+kFlEwnTFuIHs2V/D84s/TMj+9lRGCs7RR+Qyb812FqxVr0CaFvR9BDPNbDnwHHCzu+80s5vM7Kbo+u8BPYDfmdk7ZlYccB6RUJxY2I2jenXm0QUbErK/suh8CDeeciQ9O2fy61fUK5CmBX1o6BR3P8bdR7n7y9Fl97n7fdHnN7h7N3c/PvooCjKPSFjMjC+MK2Dxxl0s27w78P2VVUR6BL27ZPGlSUfy7w+2sWj9zoO8Szoq3VkskiAXj+5PZloKj70ZfK+gfoa03Kw0vjC+gO6dMtQrkCapEIgkSF5OOueN7Mszb29i7/5gTxrXHxrqkp1OTkYaN55yJHNWlvL2BvUK5NNUCEQS6IpxBeyrquXZdzYHup/6Q0NdstKAyLhHPTpl8LPZuiZDPk2FQCSBTijoytFH5PLYgmDvKThwaCgdgE6ZaXz5tMG8vmob81brKm35JBUCkQSqP2m8dFMZ75YEd9K4rLKGrPQUMtIO/IhfOX4gvbtk8rPZK3W3sXyCCoFIgp1/fD8y01L4S4DDU5dVVNMl2huol5Weyi1nDKF4/U7mvK+hWuQAFQKRBMvLTufsEUfwt3c2UVldG8g+yiqryY2eH4h1adEABnTPVq9APkGFQCQElxYNoKyyhtnLPwpk+3sqa+iSnf6p5RlpKdw+eShLN5Uxa+mWQPYtyUeFQCQEE47sQb+u2TwZ0OGhxg4N1btwdD8G53fi7ueXs377vkD2L8lFhUAkBCkpxiVj+vP6qm1sCmDOgLLKmkYPDQGkphj/d9loKqpr+Y/75vH+R4kdHlvaHhUCkZBcMqY/7jBzUUnct72nsrrRQ0P1RvTL44kvTQDg83+YF+gVTNL2qRCIhGRA9xwmDu7Bk4s2UlcXvxO37k5ZRU2Th4bqDe2dy5M3TaBTZhqX3z9fxaADUyEQCdGlRQPYuKOC+Wvjd5PX/po6qmrrmjw0FGtgj048edMEcjJS+f5zy3QlUQelQiASorNHHEFuVhpPFsfv8FDsOEMt0Scvm9umRO8vWKn7CzoiFQKREGWlp3L+qL688O6H7C6vjss2G44z1BKXFg2goHsO985eGdfDVJIcVAhEQnb52AL219Qx86349AoOtUcAkJ6awh1nDmHZ5jL+ofsLOhwVApGQjeiXx6j+eTy2YENcjtHXT1N5KD0CgPNH9WNIr878/KWV1NTWtTqHJA8VApE24IpxBazaupeF61o/X0BZRbRHcJCrhhpKTTG+dtYwVpfu469vb2p1DkkeKgQibcD/G9WX3Mw0Hnuz9cNTH86hoXqfObY3I/vn8ct/fkBVjXoFHYUKgUgbkJORxoWj+/HC0i3s3FfVqm3VnyxuyeWjDZlFegWbdlUEOjqqtC2BFgIzu83MlprZMjO7vZH1Zma/MrNVZrbEzE4IMo9IW3bFuAKq4nDSeE9lNWkpRnZ66mG9f9KQnhQN7MZvXlkV2Oio0rYEVgjMbARwIzAWGAWcZ2ZDGjT7LDAk+pgK/D6oPCJt3fA+XTihoGurTxqXRYeXMLPDer+ZceeZQ9lSVsnjCzYcdg5JHkH2CIYD89293N1rgNeAixq0uQB4xCPmA13NrE+AmUTatCvGDWRN6T7mrTn8O43LKpoecK6lJgzuwbhB3fntnNXqFXQAQRaCpcAkM+thZjnAOcCABm36AbEHIkuiyz7BzKaaWbGZFZeW6s5Hab/OG9mHrjnpPPj62sPexp7Kpoegbqn6XkHpnv38eX6w8ytL+AIrBO7+HvAT4CVgFrAYqGnQrLG+66f6xO4+zd2L3L0oPz8/7llF2oqs9FSunVjIP9/bysothzc8dFllDV2yW9cjABh3ZA9OPqon9722mvKqhj+60p60qBCY2WAzy4w+P83MbjWzrgd7n7v/0d1PcPdJwA7ggwZNSvhkL6E/sLll0UXap2smFJKTkcp9r60+rPeXVVSTm9m6HkG9O84cwra9VTz8hnoF7VlLewQzgVozOwr4IzAIeOxgbzKzXtGvBcDFwIwGTZ4Fro5ePTQe2O3uH7Y0vEh71K1TBpePLeDZxZvZuKP8kN+/J049AoAxA7tz6tB8pv1rNXsq4zMWkrQ9LS0EddETvhcBv3T3O4CWnNSdaWbLgeeAm919p5ndZGY3Rde/AKwBVgH3A185tPgi7dMNpwwixeD+f6855PeWxeEcQayvnTWUneXVTJ+7Lm7blLalpX82VJvZ5cA1wP+LLjvoJ83dT2lk2X0xzx24uYUZRDqMPnnZXDS6H08s3MgtZwwhPzezRe+rrq2jvKqW3DgWgpH9u3LWMb25/19ruHrCQLrmZMRt29I2tLRHcB0wAfihu681s0HAn4OLJSJfOnUwVbV1TJ/b8iuIPh5wLk6HhurdedZQ9lbVHFYPRdq+FhUCd1/u7re6+wwz6wbkuvuPA84m0qENzu/MZ0ccwZ/mrf94/KCDqT+OH89DQwBHH9GF80b2ZfrcdWzbuz+u25bwtfSqoTlm1sXMuhO5DHS6mf082Ggi8pXTjmLP/hoeauHx+daMM3Qwt08ZQmV1LffNObyrmaTtaumhoTx3LyNy5c90dx8DTAkulohAZK6CKcN788C/17SoV9CakUcPZnB+Zy4+oT+PzF/Plt2Vcd++hKelhSAtOvTDpcDzAeYRkQZumzyEssoaHm5BryCoQ0OxWdydX7/S8JYgSWYtLQR3AS8Cq919oZkdyadvDhORABzXP48pw3vxwOtrD3otf5CHhgAGdM/h8rEFPLFwI+u27QtkH5J4LT1Z/KS7j3T3L0dfr3H3zwUbTUTq3TZ5KLsrqnn4jXXNtgvy0FC9r55xFOmpKfz8pfcD24ckVktPFvc3s7+a2VYz+8jMZppZ/6DDiUjEcf3zmHz0wXsFZZU1mEFuZjA9AoBeuVl88eRCnl28meWbywLbjyROSw8NTScyHERfIqODPhddJiIJctuUIewqr+aReU2P+1NWUU3njDRSUg5vLoKWmjppMHnZ6dw7e2Wg+5HEaGkhyHf36e5eE308BGgYUJEEGtm/K6cOzWf63HXsr2l8joD6SWmClpedzk2nDuaVFVtZuG5H4PuTYLW0EGwzsyvNLDX6uBI4/JkzROSwXH/yILbt3c/flzQ+NuOeytZPStNS104spFduJj+dtaJVM6pJ+FpaCL5I5NLRLcCHwCVEhp0QkQQ6ZUhPjurVmelz1zX6y7esIr4DzjUnOyOVWycPYeG6nby6cmtC9inBaOlVQxvc/Xx3z3f3Xu5+IZGby0QkgcyMayYW8u6m3by1Yeen1sdrUpqW+vyJAxjYI4efzlpJXZ16BcmqNTOU3Rm3FCLSYp87oR9dstJ4sJEbzBLZIwBIT03ha2cNY8WWPTy7WHNKJavWFIJgL0sQkUblZKRx2dgCZi3dwuZdFZ9Yt6eyOmHnCOqdd1wfju3bhZ+9tJKqmrqE7lviozWFQP1AkZBcNX4g7v6JieXr6pw9+2sSctVQrJQU45tnH83GHRXMWLAhofuW+Gi2EJjZHjMra+Sxh8g9BSISggHdczjzmN7MWLCByurIpaR7q2pwD26coeZMGtKT8Ud259evfMC+/ZroPtk0WwjcPdfduzTyyHX3xPY/ReQTrjtpEDvLq/lL8UYguElpWsIs0ivYtreKB19v+UQ60ja05tCQiIRo3KDujB3UnV+/soqKqlrKKiJDT8RzmspDcUJBN848pjfT/r2G3eWa6D6ZqBCIJCkz4xufGUbpnv08PG/dx4UgjEND9e48cyh7KjWlZbIJtBCY2R1mtszMlprZDDPLarC+wMxeNbO3zWyJmZ0TZB6R9ubEwu6cPiyf389ZzebdkSuIwjg0VG94ny6cN7IPD85dy3ZNaZk0AisEZtYPuBUocvcRQCpwWYNm3wX+4u6jo+t+F1Qekfbqa2cNY3dFNb9+eRUQ3qGherdPGRqZ0vI1TWmZLII+NJQGZJtZGpADNLzjxIEu0ed5jawXkYMY0S+Pc0f2YU10opguCb6PoKGjenXmotH9eWTeej4q05SWySCwQuDum4B7gQ1Exifa7e6zGzT7H+BKMysBXgBuaWxbZjbVzIrNrLi0tDSoyCJJ684zh1I/8nTYPQKITGlZW+f89tVVYUeRFgjy0FA34AJgEJF7DjpFRy2NdTnwkLv3B84B/mRmn8rk7tPcvcjdi/LzNfq1SEOD8zvz+RMHkJ+bSUZa+NeAFPTI4dITBzBjwQY27igPO44cRJCfmCnAWncvdfdq4GlgYoM21wN/AXD3eUAW0DPATCLt1l0XjOAft50SdoyP3XLGUZgZv3pZ05u3dUEWgg3AeDPLMTMDJgPvNdJmMoCZDSdSCHTsR+QwpKem0LNzZtgxPtYnL5srxw1k5lslrC7dG3YcaUaQ5wjeBJ4C3gLeje5rmpndZWbnR5t9DbjRzBYDM4BrXTNciLQbXzl9MFnpqfxCE923aZZsv3eLioq8uLg47Bgi0kL3vriS37y6ir/fejLH9s0LO06HZWaL3L2osXXhn1USkXbtxklH0iUrjZ/PVq+grVIhEJFA5WWn86VTB/Pyiq2Nzqom4VMhEJHAXXdSIT07Z/CTf2ii+7ZIhUBEApeTkcZtk4fw5todzF7+UdhxpAEVAhFJiMvHFjCkV2d+9MJ77K+pDTuOxFAhEJGESEtN4b/OO4b128t5aO66sONIDBUCEUmYSUPzOePoXvz6lVWU7tEw1W2FCoGIJNR3zh1OZXUtP39pZdhRJEqFQEQSanB+Z66eUMjjCzeybPPusOMIKgQiEoLbJg+hW04G//23ZdTV6XLSsKkQiEjC5eWk862zj6Z4/U5mvlUSdpwOT4VAREJxyZj+nFDQlR//YwW7y6vDjtOhqRCISChSUowfXHgcO8uruGf2irDjdGgqBCISmmP6duGaiYU8+uYGlpTsCjtOh6VCICKhuuPMofTsnMl/PbOUWp04DoUKgYiEqktWOt89dziLS3bzyLx1YcfpkFQIRCR054/qy2nD8rnnxZWU7NRk94mmQiAioTMzfnDhCAC++8xSDVWdYCoEItIm9O+Wwzc+M4w5K0t5dvHmsON0KCoEItJmXD2hkOMHdOX7zy1nx76qsON0GIEWAjO7w8yWmdlSM5thZlmNtLnUzJZH2z0WZB4RadtSU4yffG4kZRXV3PXcsrDjdBiBFQIz6wfcChS5+wggFbisQZshwLeBk9z9WOD2oPKISHIYdkQuXzltMM+8s5k5K7eGHadDCPrQUBqQbWZpQA7Q8MDfjcBv3X0ngLvrf11EuPmMoxic34nv/HUp+/bXhB2n3QusELj7JuBeYAPwIbDb3Wc3aDYUGGpmc81svpmd3di2zGyqmRWbWXFpaWlQkUWkjchMS+XHnxvJpl0V/Pyl98OO0+4FeWioG3ABMAjoC3QysysbNEsDhgCnAZcDD5hZ14bbcvdp7l7k7kX5+flBRRaRNuTEwu5cOb6A6XPX8s5GDT8RpCAPDU0B1rp7qbtXA08DExu0KQH+5u7V7r4WWEmkMIiI8M2zj6ZXbhbfmrmE6tq6sOO0W0EWgg3AeDPLMTMDJgPvNWjzDHA6gJn1JHKoaE2AmUQkiXTJSueuC45lxZY9TPuXfjUEJchzBG8CTwFvAe9G9zXNzO4ys/OjzV4EtpvZcuBV4Bvuvj2oTCKSfM469gjOPvYIfvXyB6zfvi/sOO2SJdut3EVFRV5cXBx2DBFJoC27K5ny89cYXdCVR744lshBBjkUZrbI3YsaW6c7i0WkzTsiL4uvnzWUf3+wTcNPBECFQESSwlUTChnVP4+7n1+uqS3jTIVARJJCaorxo4uPY2d5NT+e1fC6E2kNFQIRSRrH9s3j+pMHMWPBRg0/EUcqBCKSVO48cyhDe3fmG08t0QilcaJCICJJJSs9lV9+fjS7y6v51swlmsQmDlQIRCTpHNO3C1//zFBmL/+IJ4tLwo6T9FQIRCQp3XDykUw4sgfff26ZbjRrJRUCEUlKKSnGzy4dRUqKceuMt6mq0VhEh0uFQESSVt+u2dxzySgWl+zmRy/oktLDpUIgIknt7BFHcN1JhTz0xjpmLf0w7DhJSYVARJLetz87nFH98/jGU0vYsL087DhJR4VARJJeRloKv7niBAy4+bG3qKyuDTtSUlEhEJF2YUD3HH526fG8u2k33/vbUt1fcAhUCESk3TjzmN7cesZR/KW4hD/NXx92nKShQiAi7crtU4Yy+ehe3PXccuav0TxXLaFCICLtSkqK8YvLjqegRw43P/oWm3ZVhB2pzVMhEJF2p0tWOvdfXURVTR1f+lOxTh4fhAqBiLRLg/M788vLjmfZ5jK+/fS7OnncjEALgZndYWbLzGypmc0ws6wm2l1iZm5mjc6nKSJyOCYP780dU4by17c38eDcdWHHabMCKwRm1g+4FShy9xFAKnBZI+1yo+3eDCqLiHRcXz39KM46pjc/euE93li9Lew4bVLQh4bSgGwzSwNygMZmnb4b+ClQGXAWEemAUlKMn3/+eAb17MTNj77Fxh2687ihwAqBu28C7gU2AB8Cu919dmwbMxsNDHD354PKISLSOTONaVeNoabO+eJDC9ldUR12pEMW5DmOIA8NdQMuAAYBfYFOZnZlzPoU4BfA11qwralmVmxmxaWlpUFFFpF27Mj8zvzhqjGs276PL/2pmP01yXMlkbtzwW/n8uDrawPZfpCHhqYAa9291N2rgaeBiTHrc4ERwBwzWweMB55t7ISxu09z9yJ3L8rPzw8wsoi0ZxMH9+SeS0Yxf80O/vOp5Jnmsnj9TpaU7CYvOz2Q7acFstWIDcB4M8sBKoDJQHH9SnffDfSsf21mc4Cvu3sxIiIBuXB0PzbtquCeF1fSt2s23zz76LAjHdRTxSV0ykjls8cdEcj2AysE7v6mmT0FvAXUAG8D08zsLqDY3Z8Nat8iIs35ymmDKdlZwe/mrKZbTgY3Tjoy7EhNKq+q4e/vfsg5x/UhJyOYX9lB9ghw9/8G/rvB4u810fa0ILOIiNQzM+6+4FjKKqv54QvvkZWRylXjB4Ydq1EvLtvC3v01XDKmf2D7CLQQiIi0VWmpKfzy88ezv7qW/3pmKdnpqYH+sj1cTy0qoaB7DmMHdQ9sHxpiQkQ6rPTUyIQ2pwzpyTefWszzSxq71Sk8JTvLeWP1di4Z0x8zC2w/KgQi0qFlpafyh6vGUDSwO7c//g6vrtgadqSPPf3WJtzh4hP6BbofFQIR6fByMtJ44Noiju6Ty01/XsSbbWAeA3fnqUUlTBzcgxwVx+cAAAvOSURBVP7dcgLdlwqBiAiRoasfvm4s/btlc/3DxbxbsjvUPAvX7WTDjvKEnLdQIRARierROZM/3zCOvOx0rn7wTVZu2RNalkfmraNTRipnjwjm3oFYKgQiIjH65GXz6A3jyEhL4bJp81i6KfE9g2Wbd/P8kg+59qTCwO4diKVCICLSQGHPTjwxdQLZ6alccf98Fm/cldD93/viSvKy05k6aXBC9qdCICLSiMKenXjiSxPIy0nnygfeZNH6HQnZ74K1O3h1ZSlfPm1wYGMLNaRCICLShAHdc3hi6gR65mZy1R8XMD/gq4ncnZ/OWkHvLplcM6Ew0H3FUiEQEWlG367ZPDF1PH27ZnPt9AW8/kFws5y9smIrxet3cuvkIWRnpAa2n4ZUCEREDqJXlywenzqewh6d+OLDC3llxUdx30dtnXPPiysp7JHDpUUD4r795qgQiIi0QM/Omcy4cTzDeufypT8t4m/vbIrbtneXV3PDwwtZsWUPXztrGOmpif3VrEIgItJC3Tpl8OcbxjF6QDdue/wdvv/cMqpq6lq1zRVbyjj/t6/z+qpt3H3hCM4b2SdOaVtOhUBE5BDkZafz6I3juO6kQqbPXcfl989ny+7KQ97O1j2VPDJvHRf99g3Kq2p5fOp4rho/MNDB5ZpiyTJVW72ioiIvLtYkZiISvmcXb+ZbM5eQmZbC2SP6cPqwfE46qiedMg/cBFZX52zbu5+SXRWU7KxgxYdlvPZ+Kcs2lwFQNLAbv/vCCfTqkhVoVjNb5O6fmgoYVAhERFrlg4/28LPZ7/P6qm3s3V9DeqrRt2s2FVW1VFTVUl5dS23dgd+zqSnGmIJunDosn1OH5nNMny6kpATfC2iuEGhiGhGRVhjSO5f7rhpDVU0dxet38NrKUj7cXUlORirZGankZKTSu0sW/bpm079bDgO6Zydk2IhD0bbSiIgkqYy0FCYO7snEwT3DjnLIdLJYRKSDC7QQmNkdZrbMzJaa2Qwzy2qw/k4zW25mS8zsZTNrm7NHi4i0Y4EVAjPrB9wKFLn7CCAVuKxBs7ej60cCTwE/DSqPiIg0LuhDQ2lAtpmlATnAJ2aGdvdX3b08+nI+EPxUPCIi8gmBFQJ33wTcC2wAPgR2u/vsZt5yPfCPxlaY2VQzKzaz4tLS0viHFRHpwII8NNQNuAAYBPQFOpnZlU20vRIoAu5pbL27T3P3Incvys/PDyqyiEiHFOShoSnAWncvdfdq4GlgYsNGZjYF+A5wvrvvDzCPiIg0IshCsAEYb2Y5Fhk8YzLwXmwDMxsN/IFIEdgaYBYREWlCoENMmNn3gc8DNUSuELqByF//xe7+rJn9EziOyDkEgA3ufv5BtlkK7AIaziidd5BlB3te/7UncDgzTzS2/5asb7i8udcNs8YuO5zcicwc+zyM77U+H/p8NLc+GT8fh5IZYIi75zW6dXdPugcw7VCXHex5zNfieGVqyfqGy5t73TBra3MnMnPY32t9PvT5aG+fj0PJfLB9JOudxc8dxrKDPW/s/a3N1JL1DZc397qxrK3JncjMsc/D+F7r83Ho9Plo+fO2nrnZfSTd6KNBM7Nib2KEvrYsGXMrc+IkY25lTpxk7REEaVrYAQ5TMuZW5sRJxtzKnCDqEYiIdHDqEYiIdHAqBCIiHVy7LgRm9qCZbTWzpYfx3jFm9q6ZrTKzX1nMjNJmdouZrYwOsR33EVODyG1m/2Nmm8zsnejjnLaeOWb9183MzSyuM34E9H2+Ozqs+jtmNtvM+iZB5nvMbEU091/NrGs8MweY+z+iP4N1Zha3E7StydrE9q4xsw+ij2tiljf7uU+ow7nmNVkewCTgBGDpYbx3ATABMCKD4X02uvx04J9AZvR1ryTJ/T/A15Ppex1dNwB4EVgP9GzrmYEuMW1uBe5LgsxnAWnR5z8BfpIMnw9gODAMmENkOPtQs0ZzFDZY1h1YE/3aLfq8W3P/rjAe7bpH4O7/AnbELjOzwWY2y8wWmdm/zezohu8zsz5EfqDneeR/7BHgwujqLwM/9ui4SB7A0BgB5Q5UgJl/AXwTiPtVDUFkdveymKad4p07oMyz3b0m2jSQ4eADyv2eu69sK1mb8BngJXff4e47gZeAs8P8WW1Muy4ETZgG3OLuY4CvA79rpE0/oCTmdUl0GcBQ4BQze9PMXjOzEwNNe0BrcwN8Ndr9f9Aio8MGrVWZzex8YJO7Lw46aIxWf5/N7IdmthH4AvC9ALPWi8dno94XaWI4+ADEM3fQWpK1Mf2AjTGv6/O3lX8X0MEmrzezzkRGQH0y5nBcZmNNG1lW/5ddGpEu3njgROAvZnZktKoHIk65fw/cHX19N/AzIj/0gWhtZjPLITIu1VnBJGwkSHy+z7j7d4DvmNm3ga8C/x3nqAeCxClzdFvfITIu2KPxzNiYeOYOWnNZzew64LbosqOAF8ysisjIyxfRdP7Q/12xOlQhINID2uXux8cuNLNUYFH05bNEfmnGdo/7c2B2tRLg6egv/gVmVkdkoKkgZ8xpdW53/yjmffcDzweYF1qfeTCRuSwWR3/4+gNvmdlYd9/SRjM39BjwdwIsBMQpc/Qk5nnA5CD/qIkR7+91kBrNCuDu04HpAGY2B7jW3dfFNCkBTot53Z/IuYQSwv93HRDWyYlEPYBCYk76AG8A/xF9bsCoJt63kMhf/fUncs6JLr8JuCv6fCiRbp8lQe4+MW3uAB5v65kbtFlHnE8WB/R9HhLT5hbgqSTIfDawHMiPd9ZEfD6I88niw81K0yeL1xI5itAt+rx7Sz/3iXqEstOE/eNgBpEhrquJVODrifyVOQtYHP3wf6+J9xYBS4HVwG84cBd2BvDn6Lq3gDOSJPefgHeBJUT+0urT1jM3aLOO+F81FMT3eWZ0+RIig3z1S4LMq4j8QfNO9BHXK50CzH1RdFv7gY+AF8PMSiOFILr8i9Hv8SrgukP53CfqoSEmREQ6uI541ZCIiMRQIRAR6eBUCEREOjgVAhGRDk6FQESkg1MhkHbBzPYmeH8PmNkxcdpWrUVGK11qZs8dbPRPM+tqZl+Jx75FQDOUSTthZnvdvXMct5fmBwZiC1RsdjN7GHjf3X/YTPtC4Hl3H5GIfNL+qUcg7ZaZ5ZvZTDNbGH2cFF0+1szeMLO3o1+HRZdfa2ZPmtlzwGwzO83M5pjZUxYZr//R+jHjo8uLos/3RgeaW2xm882sd3T54OjrhWZ2Vwt7LfM4MOheZzN72czessi49RdE2/wYGBztRdwTbfuN6H6WmNn34/htlA5AhUDas/8DfuHuJwKfAx6ILl8BTHL30URGB/1RzHsmANe4+xnR16OB24FjgCOBkxrZTydgvruPAv4F3Biz//+L7v+g48hEx9mZTOTOb4BK4CJ3P4HIPBg/ixaibwGr3f14d/+GmZ0FDAHGAscDY8xs0sH2J1Kvow06Jx3LFOCYmBEju5hZLpAHPGxmQ4iM+Jge856X3D12LPoF7l4CYGbvEBmD5vUG+6niwCB+i4Azo88ncGCM+ceAe5vImR2z7UVExqyHyBg0P4r+Uq8j0lPo3cj7z4o+3o6+7kykMPyrif2JfIIKgbRnKcAEd6+IXWhmvwZedfeLosfb58Ss3tdgG/tjntfS+M9MtR842dZUm+ZUuPvxZpZHpKDcDPyKyHwG+cAYd682s3VAViPvN+B/3f0Ph7hfEUCHhqR9m01kPgAAzKx+GOE8YFP0+bUB7n8+kUNSAJcdrLG77yYyveXXzSydSM6t0SJwOjAw2nQPkBvz1heBL0bHzcfM+plZrzj9G6QDUCGQ9iLHzEpiHncS+aVaFD2BupzIEOIAPwX+18zmAqkBZroduNPMFgB9gN0He4O7v01khMvLiEwQU2RmxUR6ByuibbYDc6OXm97j7rOJHHqaZ2bvAk/xyUIh0ixdPioSkOgsaxXu7mZ2GXC5u19wsPeJJJrOEYgEZwzwm+iVPrsIcGpQkdZQj0BEpIPTOQIRkQ5OhUBEpINTIRAR6eBUCEREOjgVAhGRDu7/A58SL4VIlY8EAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "55299290-8c05-415f-8f73-a3656c1d488c"
    }
   },
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "debf9e8a-b4ec-4b06-8344-a8f01150d941"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.435833</td>\n",
       "      <td>3.482586</td>\n",
       "      <td>0.651041</td>\n",
       "      <td>0.348959</td>\n",
       "      <td>32.543823</td>\n",
       "      <td>46:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.232271</td>\n",
       "      <td>3.290639</td>\n",
       "      <td>0.633086</td>\n",
       "      <td>0.366913</td>\n",
       "      <td>26.860140</td>\n",
       "      <td>46:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.150276</td>\n",
       "      <td>3.233659</td>\n",
       "      <td>0.626817</td>\n",
       "      <td>0.373183</td>\n",
       "      <td>25.372326</td>\n",
       "      <td>45:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.064357</td>\n",
       "      <td>3.156773</td>\n",
       "      <td>0.617497</td>\n",
       "      <td>0.382503</td>\n",
       "      <td>23.494793</td>\n",
       "      <td>46:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.023923</td>\n",
       "      <td>3.087232</td>\n",
       "      <td>0.608421</td>\n",
       "      <td>0.391579</td>\n",
       "      <td>21.916332</td>\n",
       "      <td>46:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.912579</td>\n",
       "      <td>3.013001</td>\n",
       "      <td>0.598448</td>\n",
       "      <td>0.401552</td>\n",
       "      <td>20.348408</td>\n",
       "      <td>45:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.869335</td>\n",
       "      <td>2.928394</td>\n",
       "      <td>0.586558</td>\n",
       "      <td>0.413442</td>\n",
       "      <td>18.697475</td>\n",
       "      <td>45:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.790870</td>\n",
       "      <td>2.850752</td>\n",
       "      <td>0.574756</td>\n",
       "      <td>0.425243</td>\n",
       "      <td>17.300810</td>\n",
       "      <td>45:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.683932</td>\n",
       "      <td>2.791149</td>\n",
       "      <td>0.564957</td>\n",
       "      <td>0.435043</td>\n",
       "      <td>16.299747</td>\n",
       "      <td>45:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.692957</td>\n",
       "      <td>2.777951</td>\n",
       "      <td>0.562288</td>\n",
       "      <td>0.437713</td>\n",
       "      <td>16.085943</td>\n",
       "      <td>45:49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcx0lEQVR4nO3de3xcdZ3/8ddnJpNMJjNp0iuhrU24t0AtJWARREBQWm4+1q5bH+oqe+lvBZeL66U8dFf8Lf4WXV2R309BdNEVWRALiMpFcLVWRdAU2lLaQlt6S6/pJU3bZDK5fH9/zMl0Js1l0s6ZnCbv5+ORR879fOZM8s4333MZc84hIiLBFRruAkREZGAKahGRgFNQi4gEnIJaRCTgFNQiIgFX4sdGx1SPdaefeoofmxYRGZGWLVu2xzk3oa95vgT1xJOn0tDQ4MemRURGJDPb3N88dX2IiAScglpEJOAU1CIiAedLH7WIyFB0dHTQ2NhIMpkc7lJ8F41GmTJlCpFIJO91fAlqPT1ERIaisbGRRCJBbW0tZjbc5fjGOcfevXtpbGykrq4u7/X86fpQUovIECSTScaNGzeiQxrAzBg3btyQ/3NQH7WIBMJID+kex/I6FdQiIgGnoBaRUa+5uZlvf/vbQ15v3rx5NDc3+1BRLgW1iIx6/QV1V1fXgOs988wzVFVV+VVWhk+X5+lsooicOBYtWsSGDRuYNWsWkUiEeDxOTU0Ny5cvZ/Xq1bz//e9n69atJJNJbr31VhYuXAhAbW0tDQ0NHDp0iLlz53LJJZfw4osvMnnyZJ566inKy8sLUp+uoxaRQPnSz19n9faWgm5zxsmVfPG6s/udf/fdd7Nq1SqWL1/OkiVLuOaaa1i1alXmEroHH3yQsWPH0tbWxgUXXMAHPvABxo0bl7ONdevW8cgjj/Dd736XD37wgzz++ON85CMfKUj9CmoRkV4uvPDCnOuc7733Xp588kkAtm7dyrp1644K6rq6OmbNmgXA+eefz6ZNmwpWj254EZFAGajlWywVFRWZ4SVLlvCrX/2KP/7xj8RiMS677LI+r4MuKyvLDIfDYdra2gpWj04misiol0gkOHjwYJ/zDhw4QHV1NbFYjLVr1/LSSy8VuTp1fYiIMG7cOC6++GLOOeccysvLmTRpUmbe1Vdfzf3338/MmTM588wzmTNnTtHrM+cK31Fx6oyZbsPqlQXfroiMTGvWrGH69OnDXUbR9PV6zWyZc66+r+XV9SEiEnAKahGRgNPT80REAk4tahGRgFNQi4gEnC9BrZ4PEZHCUYtaROQYxONxALZv3878+fP7XOayyy6joaHhuPeloBYROQ4nn3wyixcv9nUfujNRRAT43Oc+x7Rp07jpppsAuPPOOzEzli5dyv79++no6OCuu+7ihhtuyFlv06ZNXHvttaxatYq2tjZuvPFGVq9ezfTp0wv2vA8FtYgEy7OLYOdrhd3mSefC3LsHXGTBggXcdtttmaB+7LHHeO6557j99tuprKxkz549zJkzh+uvv77fzz287777iMVirFy5kpUrVzJ79uyClK+gFhEBzjvvPHbv3s327dtpamqiurqampoabr/9dpYuXUooFGLbtm3s2rWLk046qc9tLF26lFtuuQWAmTNnMnPmzILUpqAWkWAZpOXrp/nz57N48WJ27tzJggULePjhh2lqamLZsmVEIhFqa2v7fMRpNj8+TV0nE0VEPAsWLODRRx9l8eLFzJ8/nwMHDjBx4kQikQi/+c1v2Lx584DrX3rppTz88MMArFq1ipUrC/NwOn1wgIiI5+yzz+bgwYNMnjyZmpoaPvzhD3PddddRX1/PrFmzOOusswZc/xOf+AQ33ngjM2fOZNasWVx44YUFqcuXx5zWTZ/pNq7RY05FJD96zGkBHnNqZreb2etmtsrMHjGzaAFqFRGRPAwa1GY2GbgFqHfOnQOEgQV+FyYiImn5nkwsAcrNrASIAdsHXFqd1CIyRH50wwbRsbzOQYPaObcN+BqwBdgBHHDOPd97OTNbaGYNZtZw6NChIRciIqNXNBpl7969Iz6snXPs3buXaHRovceDnkw0s2rgceCvgGbgJ8Bi59yP+lun7qxz3ca1Bb6zSERGrI6ODhobGwe9RnkkiEajTJkyhUgkkjN9oJOJ+VyedyWw0TnX5G3sCeCdQL9BLSIyFJFIhLq6uuEuI7Dy6aPeAswxs5ilb7l5D7BmoBVG9j8vIiLFlU8f9cvAYuAV4DVvnQd8rktERDx53ZnonPsi8EWfaxERkT7oWR8iIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYDzJah1w4uISOH406JWUouIFIy6PkREAk5BLSIScApqEZGAU1CLiAScglpEJOB0eZ6ISMCpRS0iEnAKahGRgFNQi4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwOnpeSIiAacWtYhIwPl0Z6Ka1CIiheJLUHcrp0VECsafFrVTUouIFIpa1CIiAacWtYhIwKlFLSIScD4FtZJaRKRQfOr68GOrIiKjk1rUIiIBpxa1iEjAqUUtIhJwvgR1qqvbj82KiIxKeiiTiEjA5RXUZlZlZovNbK2ZrTGziwZbRze9iIgURkmey30TeM45N9/MSoHYYCukuropKwkfV3EiIpJHUJtZJXAp8HEA51wKSA22Xmt7l4JaRKQA8un6OAVoAr5vZq+a2ffMrKL3Qma20MwazKwBoLWjq8ClioiMTvkEdQkwG7jPOXcecBhY1Hsh59wDzrl651w9QGt7Z0ELFREZrfIJ6kag0Tn3sje+mHRwD6g1pRa1iEghDBrUzrmdwFYzO9Ob9B5g9WDrHU6pRS0iUgj5XvXxj8DD3hUfbwE3DrZCa7ta1CIihZBXUDvnlgP1Q9mwTiaKiBSGb3cm7jnY7temRURGFd+CevGyRr82LSIyqvgW1JeeMcGvTYuIjCq+BHU4ZBzWddQiIgXhS1B3dTte2bLfj02LiIw6vnV9vL69xa9Ni4iMKnoetYhIwPkS1FHvqXl6JrWIyPHzJag7u9MfxbVlX6sfmxcRGVV8Ceox5REAfvtmkx+bFxEZVXwJ6kg4vdkf/3mrH5sXERlVfAnq6opSQFd+iIgUgi9BXRIyPzYrIjIq6fI8EZGA8z2ou7p1iZ6IyPHwLajHx8sAeG3bAb92ISIyKvgW1OZ1U7//W3/waxciIqOCb0H99C2X+LVpEZFRxbegnpiIZoYPJjv82o2IyIhXlKs+7n52bTF2IyIyIhUlqB9+eUsxdiMiMiL5GtTrvjw3M9zZ1e3nrkRERixfg7rnmR8As//1BT93JSIyYvne9fHXF00DoCWpz1AUETkWvgf1/77hnMzwP/90ld+7ExEZcYr6rI+HXtpMiy7VExEZkqIE9cZ/m5cZ/o/n3yzGLkVERoyiBLXZkcee/uDFTcXYpYjIiFG0ro+3/s+RVnXtoqeLtVsRkRNe0YI61OvDBN7cdbBYuxYROaEV9WTipruvyQy/9xtLOdyuS/ZERAZT9E94ufO6GZnhs7/4y2LvXkTkhFP0oP7YO2tzxm/Q86pFRAZU9KA2s5wukBVbm3ljp/qrRUT6M2wfbvt3l9Rlht93z9LhKkNEJPCGLai/cO0MPvO+MzPjtYuepr2za7jKEREJrLyD2szCZvaqmf2iUDu/+fLTcsbP/MJzhdq0iMiIMZQW9a3AmkIXsOJf3pszvmZHS6F3ISJyQssrqM1sCnAN8L1CFzAmFuF7f12fGZ/7zd/x0f98udC7ERE5YeXbor4H+CzQ78e0mNlCM2sws4ampqYhFXHljEmcPjGeGf/duj0454a0DRGRkWrQoDaza4HdzrllAy3nnHvAOVfvnKufMGHCkAt54VPv5h/efWpmvO6OZ/hJw9Yhb0dEZKTJp0V9MXC9mW0CHgWuMLMf+VHMorln5Yx/ZvFKNu457MeuREROGIMGtXPuDufcFOdcLbAA+LVz7iN+FfTmXXNzxi//2hI+u3iFX7sTEQm8YbuOuj+lJaGcOxcBHmto1NP2RGTUGlJQO+eWOOeu9auYbL3D+r3fWMoP/rCxGLsWEQmUwLWos/XuBrnz56upXfS0rggRkVEl0EFdWhKi4QtXHjW97o5nWKeuEBEZJQId1ADj42VsuvsaLPcDYrjqG0s58wvP8s8/XTU8hYmIFEngg7rHhi/P49sfnp0zrb2zm4de2szXn39jmKoSEfGf+dHfW19f7xoaGgq+XYBV2w5w7f/9fb/zG75wJePjZb7sW0TEL2a2zDlX3+e8Ey2oezz00uZ+uz1uCT/BJ8/tpHT8KVA9Daqmpb+PmQrhiK91iYgci4GCuqTYxRTKR+dM46NzplG76Omj5o2xw+xYvYyT7edELOsZ1xaCyim54V01Dapr08PxSRzVGS4iMsxO2BZ1b30FdpguTmIfU0NNTLXdTLXdXDGpjXNizbB/MxzambtCSRSq3pYb3tmBXl5VnBcjIqPOiOz66Mtg/de9lZHivEQLi+ZE2dO4jitPaoPmzbB/E+zfAu0HcleIVvXRGq870q0SiRb2BYnIqDFqgrrHV55by31LNhzXNr65YBbvqAnxtR+/wOffWU68dRutuzfQuXcT41I7oHkLdLXnrpSoORLi1bXp8C5LQCQGkfKs79Hcaeo3Fxn1Rl1QZ9t/OMV5//pCwbdrdDOBA7zNdjHVmvibs42OvZtI7dlIbaiJSezFyPPYhkrSgV0S7RXo/QR7pPzIcElf82LeOj3DMSiNQ+iEuRpTZNQZ1UGdrTXVSciMFVub+asHXvJ1XxE6mWT7iNFOOe2UW4oo7ZSTyhpPf5XbkelR876TyqxTHekiHkrRnWolSoqKUIqQO4YPAi6Np7/K4umWfqn3PTPcMz0x8DKlCQifsOehRQJJQT2I5Vub+epzawmHjN+t2zPc5eSlhE7K8cLfC/dyUkR7/UGIWfoPxT+9+2T27tvHpu07GRNKEiNJWddhxkZSlHe3QuoQrv0g1pXKs4ByukvjuNI44WiijzCPQ1ll1rA3r7TC+8oeroBQ2N8DJhJwCupj1JbqYkPTIR78/UaeeHVbzryykhDtnf1+MtkJK0InFbQRtzbiJL3hJHHaqPCm5QxbG7WJbtoOHUgvSxuVoXYSoSSl3cn8d1xS3n+I9xvwAwR/aRxKSv07UCIFpqAuou3NbYwpj7DkjSZu/u9XcuadVBllZ0vf4RWNhEh2jKzgD9NFBUkqvECPkSRm7VSQbtFXmPeddmKWXi79vZ1YZp32rOlJotaR9/5dKAKlFVjvED+q5Z84Eu49XTs987KXj5TrOnvxzYi84SWoTq4qB+CamTVcdOpVxErDRMIhwqG+f8GdczgHoT7mO+fo7HZEwqHMeFtHF/f/9i0qSsP8r3efSqqzm5fe2kuqs5sZJ1eysyXJX3z7Rf9e4BB0EaaFClqoIHNe9TjbBWG6iHlB3n/QJ9MBb0liqSPT4ySJ2S7qa/bRnUx39ZR0tkJnfi1/Z2G6IxWZ7h4rrTg62DMB3/sPQU/o93QJJaBEjzqQ/KhFPQq0JDtYt+sQp0+KY0AiGiHV2c3zq3dSGg6x8KFlfH7edDbuPcwfN+zl3+fPZFdLO481bOW3b6Y/Uf6ZW97FvHt/N7wvxCcldKZb8F74V5DMdO1U0EZFJuS9bp+s+T1/ALKXK8u31R8uSwd21AvuskqIjskaruw1XHkk5HvGdTXPiKGuDymq9s4u9hxK8Yd1ezic6qRufAUf//6fh7usoukd/HEvxCsyff5tvHNyKbMmhfnVq+tIWBsTSlOUdBwkQSsJayNBK3HaCNvAv58OIxWO4coq6Y7EsfIxlJSPIRIbkxX0Y+guS2BlCbpKK9mVijB50qQjoa9unUBQUMuw292S5JnXdjD33BrGx8tIdnRRUXak56011Ulnt6NxXxszTq7MWfdgsoPSkhBlJekrQ9bvPsjUsTH+68VN/MXsKZRHwrz01l7mnDKO5rYOyiNhYqVhfvrqNhxwxxOvFfOlFpAjRns6tK2NSlpJWDrAe8I8O9QT1ppZNkEbld6y5ZbHlTwWzurGGejyzEQ/y2WNR2IK/WOgoBYBDrR18PYvPc+d181gzY6DvH1qFcu37icSDvGu0yfw+/VNtKW6Oe9tVYTMuO7tNby+vYV/fORVPv7OWn62fDtnnpTgD+v3sPfwkfCb/bYqXtnSPIyvbGAROolnwr4tE+rxzNU9bZmWf8+0nhPAPfMStFGRRwsfoMsZhyknXJ6gKVVKLF7Fge4yKseMJRkqp6yiinhlFRWJKiz7zt3SitybtHqGS73xEX4Jp4JapIhakh0kykqwrFZlW6qLzyxewe1XncF7vv7bYazueDjKac9clplz6abXys8J+Kzh3uGfT7fOUcKlmRB3kXI6w+WURCuwgcK997zMH4Ps6d5jHCx05CsUzh0vwn8ICmqRAGlLdREOGaUlR04CLtu8jzMmJUhEI7Slunhxwx7eM30Sv3x9J6dOqOC0iQm2NbcRi4Sprijl73/YwAurd1Edi/Dgxy9gxdZmXtnSzM9WbAfgU1edwY//vJXm1hSzp1UH8EYuR5QUcZJEvTtzYyQzd+PGem7a8m7Y6hmOkkrPy9zNm/Ru+EoPV4Q6KHNJoqQos84C1msDhPhAAT/QfEt3OXnjtvDXCmqRkcQ5R6qrO9NvP1T/8NAynnt95+ALnsBK6Ew/iqGPwO89HKGLT191OqXhblrbO+js7KKrq4udzYfZ2HSQitIwl58xnh37D/HzFdsoMUe8LERrMoXhOH1CjOmTKnhxfRPREth3qJ3TJkQ5uybBnoNJuru6mBiPsHp7M9XlJby58wAhurn4lLHsbmklFjEm3fysglpEBvZa4wHOqklkrtvvsW7XQU6bGM/pygHYeSDJnH/7H370t+/gpDFRTpsYZ+3OFj79kxV85n1nsaslybxza+jqcry+/QCNzW2MjZXydz9sYOaUMaxs7PUY4VFu81euVVCLSLDsPdTO+Xf9arjLCIyBglp3JorIsBgXL2PT3ddkxrfua2XNjhbee/ZJALyx8yCTKsuoih15Zkuqs5vO7m5ipX1Hl3Mup+W/uyXJln2tmMH508b2uU5fnw4VNGpRi8io9tTybXzqsRV87S9nMm1cBY8va2TZ5v3cfPlpzJwyhvLSMBt2H+aVLfu5/MyJvLXnEBefOp5D7Z18+ek1fOvDswmHDOcc7Z3dPPnqNqrKI1xQN5aWtg52H2znt282MbmqnHMnj2FnS5J31I3l68+/SZdz/P27TqFufIWu+hARCbqBgloPCRARCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBJyCWkQk4BTUIiIBN2hQm9lUM/uNma0xs9fN7NZiFCYiImn5POujE/gn59wrZpYAlpnZC8651T7XJiIi5NGids7tcM694g0fBNYAk/0uTERE0obUR21mtcB5wMt+FCMiIkfLO6jNLA48DtzmnGvpY/5CM2sws4ampqZC1igiMqrlFdRmFiEd0g87557oaxnn3APOuXrnXP2ECRMKWaOIyKiWz1UfBvwnsMY59x/+lyQiItnyaVFfDHwUuMLMlntf83yuS0REPINenuec+z1ggy0nIiL+0J2JIiIBp6AWEQk4BbWISMApqEVEAk5BLSIScApqEZGAU1CLiAScglpEJOAU1CIiAaegFhEJOAW1iEjAKahFRAJOQS0iEnAKahGRgFNQi4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBJyCWkQk4BTUIiIBp6AWEQk4BbWISMApqEVEAk5BLSIScApqEZGAU1CLiAScglpEJOAU1CIiAaegFhEJOAW1iEjAKahFRAJOQS0iEnB5BbWZXW1mb5jZejNb5HdRIiJyxKBBbWZh4FvAXGAG8CEzm+F3YSIikpZPi/pCYL1z7i3nXAp4FLjB37JERKRHSR7LTAa2Zo03Au/ovZCZLQQWeqPtZrbq+MsruPHAnuEuog+qa2hU19CorqEZrrqm9Tcjn6C2Pqa5oyY49wDwAICZNTjn6vMur0hU19CorqFRXUOjuvKXT9dHIzA1a3wKsN2fckREpLd8gvrPwOlmVmdmpcAC4Gf+liUiIj0G7fpwznWa2SeBXwJh4EHn3OuDrPZAIYrzgeoaGtU1NKpraFRXnsy5o7qbRUQkQHRnoohIwCmoRUSCzjlXsC/gauANYD2wqJDb7rWfTcBrwHKgwZs2FngBWOd9r/amG3CvV9NKYHbWdj7mLb8O+FjW9PO97a/31rV+6ngQ2A2syprmex397WOQuu4EtnnHbDkwL2veHd4+3gDeN9j7CdQBL3v7/zFQ6k0v88bXe/Nre9U1FfgNsAZ4Hbg1CMdsgLqG9ZgBUeBPwAqvri8dx7YKUu8gdf0A2Jh1vGYV+2ffWyYMvAr8IgjHqyCZV8DwDAMbgFOAUu9NnFHIYrP2tQkY32vaV3sOHLAI+Io3PA941vthmQO8nPWGv+V9r/aGewLiT8BF3jrPAnP7qeNSYDa5geh7Hf3tY5C67gQ+3cdrmOG9V2XeD9sG773s9/0EHgMWeMP3A5/whm8C7veGFwA/7rWvGrxfUiABvOntf1iP2QB1Desx815D3BuOkA6COUPdViHrHaSuHwDz+zheRfvZ96Z/CvhvjgT1sB6vgmReAcPzIuCXWeN3AHcUstisbW/i6KB+A6jJ+sV7wxv+DvCh3ssBHwK+kzX9O960GmBt1vSc5fqopZbcQPS9jv72MUhdd9J36OS8T6Sv7rmov/fT+8XZA5T0ft971vWGS7zl+vxvxFvmKeCqoByzPuoKzDEDYsArpO8KHtK2ClnvIHX9gL6DumjvI+n7PP4HuAL4xbEcez+P17F+FbKPuq9bzScXcPvZHPC8mS3zbl0HmOSc2wHgfZ84SF0DTW/sY3q+ilFHf/sYzCfNbKWZPWhm1cdY1zig2TnX2UddmXW8+Qe85Y9iZrXAeaRbY4E5Zr3qgmE+ZmYWNrPlpLuyXiDdohvqtgpZb591Oed6jteXveP1DTMrO8bjdTzv4z3AZ4Fub/xYjn3Bj9fxKmRQ53WreYFc7JybTfqJfjeb2aUDLNtfXUOdfryGu477gFOBWcAO4Os+1JVXzWYWBx4HbnPOtQxQc1GPWR91Dfsxc851OedmkW4pXghMP4ZtFfw49q7LzM4h3bo8C7iAdHfG5wpc14DM7Fpgt3NuWfbkAbZVtON1vAoZ1EW71dw5t937vht4kvQP8C4zqwHwvu8epK6Bpk/pY3q+ilFHf/vol3Nul/fL1Q18l/QxO5a69gBVZlbSa3rOtrz5Y4B92XWYWYR0GD7snHtikNdTtGPWV11BOWZeLc3AEtJ9vEPdViHr7a+uq51zO1xaO/B9jv14Hev7eDFwvZltIv2UzytIt7ADc7yOWaH6UEj38bxFuvO9p6P97EL203j7qQASWcMvkj4T++/knmT4qjd8DbknMv7kTR9L+gx1tfe1ERjrzfuzt2zPiYx5A9RTS25fsO919LePQeqqyRq+HXjUGz6b3BMnb5E+adLv+wn8hNwTJzd5wzeTe3LmsV41GfBD4J5e04f1mA1Q17AeM2ACUOUNlwO/A64d6rYKWe8gddVkHc97gLuH42ffm3cZR04mDuvxKkjuFThE55E+Y74B+Hwht521j1O8A9RzadDnvenjSJ9EWOd973nDjfQHH2wgfblPfda2/ob0ZTbrgRuzptcDq7x1/h/9X573COl/iTtI/7X922LU0d8+BqnrIW+/K0k/qyU7hD7v7eMNsq5w6e/99N6DP3n1/gQo86ZHvfH13vxTetV1Cel/CVeSdcnbcB+zAeoa1mMGzCR9mdlK7zX9y3FsqyD1DlLXr73jtQr4EUeuDCnaz37W+pdxJKiH9XgV4ku3kIuIBJzuTBQRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBJyCWkQk4P4//z4WfWDitjEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.34895867109298706.\n",
      "Better model found at epoch 1 with accuracy value: 0.36691299080848694.\n",
      "Better model found at epoch 2 with accuracy value: 0.3731825649738312.\n",
      "Better model found at epoch 3 with accuracy value: 0.38250282406806946.\n",
      "Better model found at epoch 4 with accuracy value: 0.39157918095588684.\n",
      "Better model found at epoch 8 with accuracy value: 0.43504294753074646.\n",
      "Better model found at epoch 9 with accuracy value: 0.4377129077911377.\n",
      "CPU times: user 5h 57min 14s, sys: 1h 43min 55s, total: 7h 41min 9s\n",
      "Wall time: 7h 40min 3s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "learn.unfreeze()\n",
    "wd = 0.01 \n",
    "learn.fit_one_cycle(10, lr, wd=wd, moms=(0.8,0.7), \n",
    "                    callbacks=[ShowGraph(learn),\n",
    "                               SaveModelCallback(learn.to_fp32(), monitor='accuracy', name='bestmodel_sp15_multifit')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "00f7bd36-8558-4a49-8cd2-29a0f836430b"
    }
   },
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(mdl_path/lm_fns3[0], with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/(lm_fns3[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15_multifit_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.35 s, sys: 72 ms, total: 1.42 s\n",
      "Wall time: 1.42 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, drop_mult=0., pretrained=False, \n",
    "                               metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV1bn/8c+TOSEQCERmZCgOiEUhImi1g2Nt69DWqlXrzLWT1t72/npvX7/23np7a29vr72tdaCKQx1ah/ZXW7XKtbVaATWgIDgBETAyBRITyJyc5/fH2YFDOCEBss8+J/m+X6/zOnuvvfY5D+Ekz1l7rb2WuTsiIiJdZUUdgIiIpCclCBERSUoJQkREklKCEBGRpJQgREQkqZyoA+grI0aM8IkTJ0YdhohIRlm6dOk2dy9LdqzfJIiJEydSUVERdRgiIhnFzNZ3d0yXmEREJCklCBERSUoJQkREklKCEBGRpJQgREQkKSUIERFJSglCRESSUoIQEclgjy2t4qGXN4Ty2koQIiIZ7HevVvHY0qpQXlsJQkQkg9U1tTGkMDeU11aCEBHJYPVN7QwpCGfWJCUIEZEMVt+sFoSIiHTh7tQ3tTGkQAlCREQSNLR2EHMYUqhLTCIikqC+qQ1ALQgREdlTfXM8QZSoD0JERBLVN7UDqJNaRET2pEtMIiKSVOclJnVSi4jIHuoytQVhZgvMbKuZrUwoO9/MVplZzMzK93HumWb2tpmtMbPvhBWjiEgm6+yDGJyBd1LfA5zZpWwl8Fng+e5OMrNs4JfAJ4FpwEVmNi2kGEVEMlZ9cxuD8rLJyQ7nT3loCcLdnwdqupS96e5v93DqbGCNu1e6eyvwG+CckMIUEclY9SFO1Afp2QcxFngvYb8qKNuLmc0zswozq6iurk5JcCIi6aK+ObxpNiA9E4QlKfNkFd19vruXu3t5WVlZyGGJiKSX+qb20EYwQXomiCpgfML+OGBjRLGIiKStgdiCeAWYamaTzCwPuBB4POKYRETSTn1zW2jTbEC4w1wfAhYDh5tZlZldZWbnmVkVMBd4wsyeDuqOMbMnAdy9Hfga8DTwJvCwu68KK04RkUwVv8QUXoII7eKVu1/UzaHfJ6m7ETgrYf9J4MmQQhMRyXixmLOjuS201eQgPS8xiYhID3a2tgdrQWTgJSYREQlP2BP1gRKEiEhG2j3Vty4xiYhIgl0zuaoFISIiiXZdYlIfhIiIJKpvDi4xqQUhIiKJdrcg1AchIiIJOvsgBqsFISIiieqb2hmcn0N2VrL5TfuGEoSISAaqbw53LQhQghARyUj1TW2hLTXaSQlCRCQD1YW8mhwoQYiIZKT65vZQh7iCEoSISEaKr0etS0wiItJF2KvJgRKEiEjGicWcnS3hLhYEShAiIhlnR0s77oS6WBAoQYiIZJxUTNQHShAiIhmnc5qNEiUIERFJtGuxIHVSi4hIol2LBWmYq4iIJKpLwXrUoAQhIpJx1EktIiJJ1Te3YwaD83WJSUREEtQ3tVGcn0NWiGtBQIgJwswWmNlWM1uZUFZqZgvNbHXwPKybczvM7LXg8XhYMYqIZKJUTLMB4bYg7gHO7FL2HeBZd58KPBvsJ9Pk7scEj7NDjFFEJOPUN4U/zQaEmCDc/XmgpkvxOcC9wfa9wLlhvb+ISH8Vb0GE2/8Aqe+DGOnumwCC50O6qVdgZhVmtsTMlERERBLUp2CxIIDwU9CBmeDuG81sMvAXM3vd3dd2rWRm84B5ABMmTEh1jCIikdjR3B76NBuQ+hbEFjMbDRA8b01Wyd03Bs+VwHPAsd3Um+/u5e5eXlZWFk7EIiJppr4p8zupk3kcuCzYvgz4Q9cKZjbMzPKD7RHAicAbKYtQRCSNdcScHS3toU+zAeEOc30IWAwcbmZVZnYVcBNwmpmtBk4L9jGzcjO7Mzj1SKDCzJYDfwVucnclCBERYEdzaqbZgBD7INz9om4OnZKkbgVwdbC9CDg6rLhERDLZrplc+2EfhIiIHIRdM7n2w2GuIiJyEFI1UR8oQYiIZJTOFsRgtSBERCRRbWM8QQwrygv9vZQgREQySE1DK6AEISIiXdQ2tFKYm01hXnbo76UEISKSQWoaWykdFH7rAZQgREQySm1DK8MGhT+CCZQgREQySm1jW0r6H0AJQkQko9TqEpOIiCRT09CqFoSIiOyprSPGjuZ2tSBERGRPtY3BPRBKECIikqi2IX4XdakuMYmISKJdd1FrmKuIiCTqvMSkPggREdlDZwtCl5hERGQPtUGCGKoEISIiiWoaWxmcn0NeTmr+dCtBiIhkiPg8TKlpPYAShIhIxqhpbFOCEBGRvdU2tFJalJohrqAEISKSMWp0iUlERJKpbWxN2RBXUIIQEckIzW0dNLZ2qAUhIiJ7SvVd1BBigjCzBWa21cxWJpSVmtlCM1sdPA/r5tzLgjqrzeyysGIUEckUu+Zh6ieXmO4BzuxS9h3gWXefCjwb7O/BzEqB7wPHA7OB73eXSEREBopdM7n2hxaEuz8P1HQpPge4N9i+Fzg3yalnAAvdvcbda4GF7J1oREQGlJpdl5j67zDXke6+CSB4PiRJnbHAewn7VUHZXsxsnplVmFlFdXV1nwcrIpIuavvZJaYDZUnKPFlFd5/v7uXuXl5WVhZyWCIi0alpaMUMSgr7bwtii5mNBgietyapUwWMT9gfB2xMQWwiImmrtrGVksJccrJT92c71QnicaBzVNJlwB+S1HkaON3MhgWd06cHZSIiA1ZNQ2pvkoNwh7k+BCwGDjezKjO7CrgJOM3MVgOnBfuYWbmZ3Qng7jXAjcArweMHQZmIyIBV25jaaTYAcsJ6YXe/qJtDpySpWwFcnbC/AFgQUmgiIhmnpqGNsUMLU/qe6dhJLSIiXdQ2tKZ0iCv0MkGY2RQzyw+2P2Zm15nZ0HBDExERAHenJoJLTL1tQTwGdJjZh4C7gEnAg6FFJSIiuzS2dtDaHkvbTuqYu7cD5wE/c/cbgNHhhSUiIp12zcOUpi2INjO7iPjQ1D8FZam9GCYiMkDtmsk1TVsQVwBzgR+6+7tmNgm4P7ywRESkU1QtiF4Nc3X3N4DrAIKb1wa7+01hBiYiInFRrAUBvR/F9JyZDQmm4l4O3G1m/x1uaCIiAvF7ICB9LzGVuHs98FngbnefBZwaXlgiItKptqGV7CxjcEFo9zYn1dsEkRNMrvcFdndSi4hICtQ0tjKsKJesrGSTXYentwniB8QnzFvr7q+Y2WRgdXhhiYhIp9qG1pSuA9Gpt53UjwCPJOxXAp8LKygREdmtpiH1d1FD7zupx5nZ781sq5ltMbPHzGxc2MGJiEh8FFOqO6ih95eY7ia+lsMY4st//jEoExGRkNU0tKVvCwIoc/e73b09eNwDaI1PEZGQdcQ83oJI8Uyu0PsEsc3MLjGz7OBxCbA9zMBERAS272yhI+aMKkntWhDQ+wRxJfEhrpuBTcDniU+/ISIiIdpU1wzA6CEFKX/vXiUId9/g7me7e5m7H+Lu5xK/aU5ERELUmSBGlaRpgujGN/ssChERSWpLfWYmiNTe0iciMgBtqmsmLzsrkmGuBzOxh/dZFBGqaWjl1P/+20G9xv5kSut15X1XTHydrjW7vocl1Nj7WOJ5ex7s3N31jGEWP8cs4VW7lMXrWML5RpbFy7M6zwvKsnYdi5dnZ1m8LCtenh1sZ5uRnW3kZBnZWfHnnOws8rKzyMky8nKyyM/JJj83i/ycLAbl5VCYl82g/GyK83MpHZTLsKI8hhblkZ3i6QpEDsaW+mYOGZKf8mk2oIcEYWY7SJ4IDEh9l3oI8nKy+NTRB744nu9HnvReVu2p2p6v4/s4tud+11j3PLZn2a66CU/uHjwn1g9q+u5zYrH4dmc9DyrEHGKeWB7fjrnHHzFo64jREdtdtyOW8HAnFnPaY057h9Mei9HaHqOtw2nriNEe6/kHbAYjivMZO7SQscMKmVBaxLHjhzJ7UilDI/iGJtKTTXVNjI7g8hL0kCDcfXCqAolKcX4ON547PeowpA90xJzW9hjNbR00tXXQ2NpOQ0sH9c1t1Da2UdvQyvaGVrbUNVP1QSOr3q/jmVWbaeuIJ5YjRg3mo4eXcemcQxk3rCjif41I3Oa6ZqaPLYnkvVM7d6xIiLKzjMK8bArzshnWy3Oa2zpYUVXHy+9uZ0llDXe+8C53vvAuZ04fxVUfmcTMCb19JZG+5+5srm/mtGkjI3l/JQgZ0Apys5k9qZTZk0r52ifg/Q+auG/ROh58eQNPrNjEZ2aM4UefPZrifP2qSOrVNbXR3BaL5CY5OLhRTCL9ztihhfzzWUey5J9P4YZTD+OJFRs5+5a/8/bmHVGHJgPQrnsgIrhJDiJKEGZ2vZmtNLNVZvaNJMc/ZmZ1ZvZa8PheFHHKwDUoP4frT53KA1fPob6pnXN++Xd+t6wq6rBkgNkc4T0QEEGCMLPpwDXAbGAG8Gkzm5qk6gvufkzw+EFKgxQJzJ0ynCev+wgzxg3lmw8v50dPvkmsF6OlRPrC5s5pNgZKggCOBJa4e6O7twN/A86LIA6RXjlkSAEPXH08l845lDuer+TLDyylqbUj6rBkANhU14wZlA3Oj+T9o0gQK4GTzWy4mRUBZwHjk9Sba2bLzewpMzsq2QuZ2TwzqzCziurq6jBjlgEuJzuLH5xzFP/309N45o0tXDB/MVuD5r9IWLbUNVNWnE9udjTdxSl/V3d/E/gxsBD4M7AcaO9SbRlwqLvPAH4B/L9uXmu+u5e7e3lZmZankHCZGVd9ZBLzLy1n9ZadXDB/Cdt2tkQdlvRjm+qbI+t/gIg6qd39Lnef6e4nAzXA6i7H6919Z7D9JJBrZiMiCFVkL6dNG8mvr5rNpromrrj7FXa2dP1+I9I3Ntc1RTaCCaIbxXRI8DyB+LThD3U5PsqCiYHMbDbxOLVAkaSN8oml3HrxTN7YVM+8+ypoaVefhPS9zXXNkXVQQ3T3QTxmZm8QX9v6q+5ea2bXmtm1wfHPAyvNbDnwc+BC997OZCSSGp84YiQ/+fyHWbR2O9/4zWt0aHST9KGGlnbqm9sZGWGCiOT2UHc/KUnZ7QnbtwC3pDQokQPw2ZnjqGlo5d+feJPv/WEl/37u9L1mxRU5EJ33QETZgtD8ASIH6eqTJlO9s4U7/lbJqCEFfP2UZLf1iOyfLbvuoo5u4mwlCJE+8H/OOIKt9S38dOE7jBxSwBeOSzZyW6T3olxqtJMShEgfyMoyfvy5D7NtZwv//PvXGTE4j08cEc0MnNI/7JpmY6CNYhLpj/JysrjtkllMGz2ErzywjIp1NVGHJBlsc10zQ4tyKczLjiwGJQiRPlScn8PdVxzHmJJCrrjnFd7YWB91SJKhNtU1R9p6ACUIkT43ojifX199PMX5OXxpwcu8u60h6pAkA22J+C5qUIIQCcXYoYX8+qrjiblzyZ0vsamuKeqQJMOoBSHSj33okGLuu3I2dU1tXLbgZeoa26IOSTJEa3uMbTtb1IIQ6c+mjy1h/pdmsW5bI1ff9wrNbZqSQ3q2dUf0N8mBEoRI6E6YMoKbLziGivW1fP2hV2nviEUdkqS5zoWCRuoSk0j/96kPj+b7n57Gwje28H//sApNLSb7smnXSnLR3UUNulFOJGUuP3ESW3e0cOtzaxk3rJCvfvxDUYckaWpLxGtRd1KCEEmhb59xOFW1Tfzk6beZNGIQZx09OuqQJA1tqmumMDebIQXR/onWJSaRFDIz/vPzH2bmhKHc8NvXWP7eB1GHJGlo/fYGJpQWRT4zsBKESIoV5GYz/0vllA3O5+r7Ktj4ge6RkD1VVjcwuWxQ1GEoQYhEYURxPgsuP47m1g6uureCBi1bKoG2jhgbahqVIEQGssNGDuaWi2fy9uZ6vvnwa8S0Ip0AG2oaaY85k0cURx2KEoRIlD56WBnf/dQ0nl61hZv/952ow5E0UFkdn7srHVoQGsUkErErT5zIO5t38Iu/rOFDhxRzzjFjow5JIlRZvROAyWVqQYgMeGbGjedOZ/bEUv7p0RWsqNLIpoGssrqBEcV5lBTmRh2KEoRIOogvNjSTEcX5XPvrpdQ0tEYdkkSkctvOtOh/ACUIkbQxvDif2y6ZybaGVq7/zat0qNN6QEqXIa6gBCGSVj48big3nnMUL6zexs0L1Wk90NQ1trG9oVUJQkSSu+C4CVx43Hhu+esaFr6xJepwJIXWbgs6qHWJSUS6869nH8XRY0v45m9f481NWtd6oEinIa4QUYIws+vNbKWZrTKzbyQ5bmb2czNbY2YrzGxmFHGKRKUgN5vbL51FcUEOl96lda0HisrqneRkGeNLi6IOBYggQZjZdOAaYDYwA/i0mU3tUu2TwNTgMQ+4LaVBiqSBg1nXuq6xjTYtTJRxKqsbmDC8iNzs9Li4E0UURwJL3L3R3duBvwHndalzDnCfxy0BhpqZ5kWWAadzXev6pjYuufMltu9s6fGclvYOTv/Z3zj3ly/2qr6kj3Qa4grRJIiVwMlmNtzMioCzgPFd6owF3kvYrwrKRAac6WNLuPOycqpqm7jy3ooe17X+3ze2sqW+hTc31XP+HYs1W2yG6Ig567Y1MiVN+h8gggTh7m8CPwYWAn8GlgNdp7JMNgn6XoPCzWyemVWYWUV1dXWfxyqSLo6fPJxfXHQsK6o+4B8fWb7Pif0erniP0SUFPHjNHKrrW/j8bYt2Td8g6auqtpHWjljadFBDRJ3U7n6Xu89095OBGmB1lypV7NmqGAdsTPI689293N3Ly8rKwgtYJA2cftQovnPmETyxYhM/e7brr0zcxg+aeH51NZ+fNY45k4fz0Lw5tLTHOP/2xax8vy7FEcv+2D2CaWBfYsLMDgmeJwCfBR7qUuVx4EvBaKY5QJ27b0pxmCJpZ97Jkzl/1jh+/uxq/vDa+3sd/92yKtzh/Fnx71fTx5bw8LVzKcjN5oI7FvPimm2pDll6aW3nJH0jBngLAnjMzN4A/gh81d1rzexaM7s2OP4kUAmsAX4FfCWiOEXSipnxw/OOZvakUr796Aoq1tXsOhaLOQ9XVDF38nAmDN89THJKWTGPffkExg0r4oq7X+FPK/ZqjEsaqNzWQElhLqWD8qIOZZeoLjGd5O7T3H2Guz8blN3u7rcH2+7uX3X3Ke5+tLtXRBGnSDrKy8nijktmMXZoIVfdW8GarTsAeOndGjbUNPKF48btdc6okgIe/oe5fHhcCV9/6FXuW7wutUFLjyqrdzK5bFDk61AnSo/BtiKyX4YNyuO+K2eTm53FZQteYUt9M49UvMfg/BzOPCr5iPCSolzuv/p4TjliJN/7wypu+ctq3DUhYLqorG5IqyGuoAQhkrHGlxZxzxXH8UFjK5cteJknV27iM8eMoTAvu9tzCnKzue2SmZx37Fj+65l3uOmpt5Qk0sCO5ja27mhJqxFMoAQhktGmjy3h9ktnsWbrTprbYnyhvOstRXvLzc7ip+fP4EtzD+WO5yv5l9+v1NTiEVsbjGBKp3sgQEuOimS8k6aWcevFM1m6vpYZ40p6dU5WlvFvZx/FkIJcbvnrGuqb27j5C8eQl6PvjFF4bUMtEJ/uPZ0oQYj0A6cfNYrTjxq1X+eYGd8643CGFObwH0++xc7mdm67ZCZFefqzkGoV62sZXVLAmKGFUYeyB31dEBng5p08hR9/7mheWF3NpXe9TF1jW9QhDTjL1tcy89BhUYexFyUIEeGC4yZw68Uzeb2qjgvmL6Z6hyb5S5WNHzSxsa6ZciUIEUlXZ04fzYLLj2P99kYuuGPxfk0vLgduWdD/MEsJQkTS2UemjuDXV81m644WvnDHYt6raYw6pH5v6fpaCnKzOHL0kKhD2YsShIjsoXxiKQ9cfTz1Te2cf/viXXMESTjio8+Gps0iQYnSLyIRidyM8UP5zbw5tHXEZ4LtvAwifauxtZ1VG+vT8vISKEGISDeOHD2ER798AoMLcvjir5bwzKrNUYfU76yoqqMj5pRPVIIQkQwzacQgHvvyCRw+agjX3r9Uk/z1saXr4y2zY8crQYhIBhpRnM9vrpnDJ4JJ/jR/U99Ztr6WKWWDGJZGU3wnUoIQkR4V5mVzx6WzuPj4Cdz+t7X84yPLaeuIRR1WRovFnKUbatO2/wE01YaI9FJ2lvHv505n1JACfrrwHbbvbOXWi2cyKF9/Rg5E5bYGPmhso/zQ0qhD6ZZaECLSa2bG10+Zyk2fjU/N8cVfLWHbTt11fSCWBf0P6TjFRiclCBHZbxfOnsD8S8t5e8sOPnvrIip1r8R+W7q+lqFFuWm1BnVXShAickBOnTaSh66ZQ0NLO5+7bdGuETnSs1jMWVS5jZkThpGVlT5LjHalBCEiB+zYCcP43VdOoKQwly/+agl/Xrkp6pAywqK123mvpomzZ4yJOpR9UoIQkYNy6PBB/O4rJ3LUmCFce/8yfvnXNRoG24MHX17PsKJczpy+f2t4pJoShIgctNJBeTx4zRzOOWYMP3n6bf7xkeW0tHdEHVZa2rqjmWdWbeFzM8dRkNv9+uHpQOPTRKRPFORm87MLjmFKWTH/vfAdNmxv5BdfPJbRJem1SlrUHqmooj3mXHT8hKhD6ZFaECLSZ8yM606Zyi8uOpZVG+s5/ebn+f2rVbrkFIjFnIde3sDcycOZUlYcdTg9UoIQkT73mRljeOr6kzhs5GBu+O1yvnz/MrbrfgmeX11NVW0TX8yA1gMoQYhISCaOGMTD/zCX73zyCP7y1lY+9pPn+K+n36amoTXq0CLzwEsbGD4ojzOOSu/O6U6RJAgzu8HMVpnZSjN7yMwKuhy/3Myqzey14HF1FHGKyMHJzjKu/egUnrjuI5x02Ah++dwaTrzpL/zwiTfYUt8cdXgptbmumb+8tZXzy8eTl5MZ381T3kltZmOB64Bp7t5kZg8DFwL3dKn6W3f/WqrjE5G+N3XkYG69eBart+zg1ufWctff3+XeRev53Kyx/MPJU5iYxncT95XbnltDzJ2LZo+POpRei2oUUw5QaGZtQBGwMaI4RCSFpo4czM0XHMMNpx7G/BfW8nBFFb995T1OmzaSjx1+CHMmD2fi8CLM0vfu4gOxpHI79y5ez+UnTOTQ4ZmTDC2K0QVmdj3wQ6AJeMbdL+5y/HLgR0A18A5wg7u/l+R15gHzACZMmDBr/fr1IUcuIn1p645m7n5xHY8uraJ6R7wTe+SQfM44ahTXnDSZ8aVFEUd48Bpb2znzZy9gBk9dfxJFeel1d4GZLXX38qTHUp0gzGwY8BhwAfAB8AjwqLvfn1BnOLDT3VvM7FrgC+7+iX29bnl5uVdUVIQYuYiExd2p3NbAksrtLFqznWfe2EzM4VNHj+baj05h2pghUYd4wL7/h5Xct2Q9v503l9mT0m9q730liChS2anAu+5eDWBmvwNOAHYlCHffnlD/V8CPUxqhiKSUmTGlrJgpZcVcfPyhbK5rZsGL7/LAkvU8vnwjpx45khtOm8pRY0qiDnW/LF4bv7R0xYkT0zI59CSKrvQNwBwzK7L4hcZTgDcTK5jZ6ITds7seF5H+bVRJAf9y1pEs+s4pfPO0w3jp3e186ud/58v3L+XtzTuiDq9Xqmob+fajy5k4vIh/OuOIqMM5IClvQbj7S2b2KLAMaAdeBeab2Q+ACnd/HLjOzM4OjtcAl6c6ThGJXklRLtedMpXLTpjIXS9UsuDFdTy1cjNnHDWSr318KkePS88WxeK12/nqg8to64hx35WzKcxL7zmXuhNJJ3UY1Ach0v/VNrRy94vvcs+iddQ3t3PyYWV85WNTOH5SaVqMfHJ37l20jhufeJNJIwYx/9JZTE7zKTXSqpM6LEoQIgPHjuY27l+ygbv+Xsm2na0cM34o1350MqdPGxXJAjyxmPPi2m3c/eI6/vLWVk49ciQ3XzCDwQW5KY9lfylBiEi/1NzWwSNLq/jV85VsqGlk0ohBnD1jDGcdPZrDRhaH1qpoae9gc10zGz9o5vX3P+DBlzawbnsjpYPyuPqkSVx78pS0XikukRKEiPRr7R0xnlq5mV8vWc8r62pwh8llgzh5ahkfOqQ4GCE1iOHF+WQn/OHuiDlbdzTzfm0T23a2UNPQRm1jKzUNrWzb2cK2nS1s39lKfVMb7TGnI+a0dcSob27f4/1nHTqMS+ccyiePHkV+Tmb1N6TbMFcRkT6Vk53FZ2aM4TMzxrB1RzNPr9rCU69v4revvEdT254LFxXn5zC4IIfsLGNLfTNtHXt/SS7Ky2Z4cR4jivMZX1pESWEuOVlGdpaRk2UMG5THmKGFjB1ayITSon5xQ18yShAi0q8cMriAS+ccyqVzDiUWczbXN7O2eieV1Q3UNrZS39TOjuY22jpijA7+yI8dVsjIwQWUDspjaFFu2q/0lipKECLSb2VlGWOGFjJmaCEnTS2LOpyMkxlzzoqISMopQYiISFJKECIikpQShIiIJKUEISIiSSlBiIhIUkoQIiKSlBKEiIgk1W/mYjKzauJLmNYlOVzSpXxf+53bycpGANv2M7Su79Xb4wcSc+L2wcS8r7j2dbynsnSMOVm5Ph89Gyifj0yMOVn5vvanunvyhTXcvd88gPm9Kd/Xfud2N2UVfRVTGDEni/9AYj7QuHsqS8eY9fnQ56O/xXwwn4+uj/52iemPvSzf1/4f91HWlzH1dPxAYk7cPpiYe3N+suM9laVjzMnK9fno2UD5fGRizMnKe/v52EO/ucSUCmZW4d1Mi5uuFHPqZGLcijk1MjFmUCf1/pofdQAHQDGnTibGrZhTIxNjVgtCRESSUwtCRESSUoIQEZGkBmSCMLMFZrbVzFYewLmzzOx1M1tjZj+3hFXRzezrZva2ma0ys//s26jDidvM/tXM3jez14LHWekec8Lxb5mZm9mIvos4tJ/zjWa2IvgZP2NmY/oy5hDj/omZvRXE/nszG5oBMZ8f/A7GzKzPOoYPJtZuXu8yM1sdPC5LKN/n5z6lDmRsbqY/gJOBmcDKAzj3ZWAuYMBTwCeD8o8D/wvkB/uHZEjc/wp8K5N+1sGx8cDTwHpgRLrHDAxJqHMdcHsm/KyB04GcYPvHwI8zIOYjgScKE78AAAYaSURBVMOB54DyqGMN4pjYpawUqAyehwXbw/b174riMSBbEO7+PFCTWGZmU8zsz2a21MxeMLMjup5nZqOJ/6Iv9vj/5H3AucHhLwM3uXtL8B5bMyTuUIUY883APwF9PsoijJjdvT6h6qAMivsZd28Pqi4BxmVAzG+6+9t9GefBxNqNM4CF7l7j7rXAQuDMKH9XkxmQCaIb84Gvu/ss4FvArUnqjAWqEvargjKAw4CTzOwlM/ubmR0XarS7HWzcAF8LLiEsMLNh4YW6y0HFbGZnA++7+/KwA01w0D9nM/uhmb0HXAx8L8RYE/XF56PTlcS/0YatL2MOW29iTWYs8F7Cfmf86fLvAiAnqjdOJ2ZWDJwAPJJwuS8/WdUkZZ3fBHOINxXnAMcBD5vZ5OBbQCj6KO7bgBuD/RuBnxL/QxCKg43ZzIqA7xK/9JESffRzxt2/C3zXzP4Z+Brw/T4Odc9g+iju4LW+C7QDD/RljHsF0ocxh21fsZrZFcD1QdmHgCfNrBV4193Po/v4I/93JVKCiMsCPnD3YxILzSwbWBrsPk78j2liE3scsDHYrgJ+FySEl80sRnyCrup0jtvdtySc9yvgTyHGCwcf8xRgErA8+KUcBywzs9nuvjlNY+7qQeAJQk4Q9FHcQQfqp4FTwvzCE+jrn3WYksYK4O53A3cDmNlzwOXuvi6hShXwsYT9ccT7KqqI/t+1W1SdH1E/gIkkdDYBi4Dzg20DZnRz3ivEWwmdHUhnBeXXAj8Itg8j3ny0DIh7dEKdG4DfpHvMXeqso487qUP6OU9NqPN14NEM+VyfCbwBlIURb5ifD/q4k/pAY6X7Tup3iV91GBZsl/b2c5+qRyRvGvUDeAjYBLQRz9hXEf9W+mdgefAL8b1uzi0HVgJrgVvYfTd6HnB/cGwZ8IkMifvXwOvACuLfzEane8xd6qyj70cxhfFzfiwoX0F8crSxGfL5WEP8y85rwaNPR1+FFPN5wWu1AFuAp6OMlSQJIii/Mvj5rgGu2J/PfaoemmpDRESS0igmERFJSglCRESSUoIQEZGklCBERCQpJQgREUlKCUL6NTPbmeL3u9PMpvXRa3VYfPbXlWb2x55mUjWzoWb2lb54bxHQinLSz5nZTncv7sPXy/Hdk9eFKjF2M7sXeMfdf7iP+hOBP7n79FTEJ/2fWhAy4JhZmZk9ZmavBI8Tg/LZZrbIzF4Nng8Pyi83s0fM7I/AM2b2MTN7zswetfhaCQ90ztkflJcH2zuDCfqWm9kSMxsZlE8J9l8xsx/0spWzmN2TFRab2bNmtszi6wacE9S5CZgStDp+EtT9dvA+K8zs3/rwxygDgBKEDET/A9zs7scBnwPuDMrfAk5292OJz7b6HwnnzAUuc/dPBPvHAt8ApgGTgROTvM8gYIm7zwCeB65JeP//Cd6/x3l2gnmITiF+pztAM3Ceu88kvg7JT4ME9R1grbsf4+7fNrPTganAbOAYYJaZndzT+4l00mR9MhCdCkxLmIFziJkNBkqAe81sKvEZNHMTzlno7olrAbzs7lUAZvYa8Tl6/t7lfVrZPfnhUuC0YHsuu+f4fxD4r27iLEx47aXE1wyA+Bw9/xH8sY8Rb1mMTHL+6cHj1WC/mHjCeL6b9xPZgxKEDERZwFx3b0osNLNfAH919/OC6/nPJRxu6PIaLQnbHST/XWrz3Z183dXZlyZ3P8bMSognmq8CPye+nkQZMMvd28xsHVCQ5HwDfuTud+zn+4oAusQkA9MzxNdjAMDMOqdrLgHeD7YvD/H9lxC/tAVwYU+V3b2O+DKl3zKzXOJxbg2Sw8eBQ4OqO4DBCac+DVwZrFuAmY01s0P66N8gA4AShPR3RWZWlfD4JvE/tuVBx+0bxKdqB/hP4Edm9iKQHWJM3wC+aWYvA6OBup5OcPdXic8YeiHxRXvKzayCeGviraDOduDFYFjsT9z9GeKXsBab2evAo+yZQET2ScNcRVIsWBWvyd3dzC4ELnL3c3o6TyTV1AchknqzgFuCkUcfEOISryIHQy0IERFJSn0QIiKSlBKEiIgkpQQhIiJJKUGIiEhSShAiIpLU/wc95OyHS/cb5gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.503923</td>\n",
       "      <td>3.533567</td>\n",
       "      <td>0.592384</td>\n",
       "      <td>0.407615</td>\n",
       "      <td>34.245895</td>\n",
       "      <td>45:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.301659</td>\n",
       "      <td>3.358599</td>\n",
       "      <td>0.576882</td>\n",
       "      <td>0.423118</td>\n",
       "      <td>28.748671</td>\n",
       "      <td>46:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.181025</td>\n",
       "      <td>3.277816</td>\n",
       "      <td>0.568678</td>\n",
       "      <td>0.431322</td>\n",
       "      <td>26.517822</td>\n",
       "      <td>46:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.129165</td>\n",
       "      <td>3.199803</td>\n",
       "      <td>0.559802</td>\n",
       "      <td>0.440199</td>\n",
       "      <td>24.527683</td>\n",
       "      <td>46:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.043516</td>\n",
       "      <td>3.127525</td>\n",
       "      <td>0.550861</td>\n",
       "      <td>0.449139</td>\n",
       "      <td>22.817362</td>\n",
       "      <td>46:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.950646</td>\n",
       "      <td>3.048482</td>\n",
       "      <td>0.541069</td>\n",
       "      <td>0.458931</td>\n",
       "      <td>21.083315</td>\n",
       "      <td>46:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.916400</td>\n",
       "      <td>2.963892</td>\n",
       "      <td>0.530237</td>\n",
       "      <td>0.469763</td>\n",
       "      <td>19.373270</td>\n",
       "      <td>46:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.838817</td>\n",
       "      <td>2.883245</td>\n",
       "      <td>0.518990</td>\n",
       "      <td>0.481010</td>\n",
       "      <td>17.872173</td>\n",
       "      <td>46:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.770480</td>\n",
       "      <td>2.823306</td>\n",
       "      <td>0.509774</td>\n",
       "      <td>0.490226</td>\n",
       "      <td>16.832424</td>\n",
       "      <td>46:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.708597</td>\n",
       "      <td>2.808476</td>\n",
       "      <td>0.507101</td>\n",
       "      <td>0.492899</td>\n",
       "      <td>16.584545</td>\n",
       "      <td>46:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdCklEQVR4nO3de3Qc5Z3m8e+vWy21Wi35IstYthxkwAs2xDFgCMSEEJLNgjGQPThZ5zKTYZLjs5CES2YOMSGHkN1k4iQz2UyWCQRmgEkwEGLIkHBNstgDBHORwTHCBmyDjOWrLGxZttQttfrdP7okt4SkbsldUll6PufodHVVddWvq6VHr956q2TOOUREJLhCo12AiIgMTkEtIhJwCmoRkYBTUIuIBJyCWkQk4Ir82OiESZPd7BNP8GPTIiJj0rp16/Y556r6W+ZLUE+dPpO6ujo/Ni0iMiaZ2baBlqnrQ0Qk4BTUIiIBp6AWEQk4X/qoRUSGorOzk8bGRhKJxGiX4rtoNEpNTQ2RSCTv1yioRWTUNTY2Ul5eTm1tLWY22uX4xjlHc3MzjY2NzJo1K+/X+dL1ods8ichQJBIJKisrx3RIA5gZlZWVQ/7LwZ8+aiW1iAzRWA/pbsN5nzqZKCIScApqERn3Dhw4wM9//vMhv27RokUcOHDAh4p6U1CLyLg3UFB3dXUN+rrHH3+ciRMn+lVWD436EJFxb/ny5WzdupX58+cTiUSIx+NUV1ezfv16Nm7cyKc//Wm2b99OIpHg2muvZdmyZQDU1tZSV1fHoUOHuPjiiznvvPN4/vnnmTFjBo888gilpaUFqc+noNbZRBEZnu/+/nU27jxY0G3OnV7Bdy49dcDlK1asoL6+nvXr17NmzRouueQS6uvre4bQ3XXXXUyePJn29nbOOussrrjiCiorK3ttY/Pmzdx///3ceeedfPazn+Whhx7ii1/8YkHqV4taRKSPs88+u9c455/97Gf89re/BWD79u1s3rz5fUE9a9Ys5s+fD8CZZ55JQ0NDwepRUItIoAzW8h0pZWVlPdNr1qzhT3/6E2vXriUWi3HBBRf0Ow66pKSkZzocDtPe3l6wenTBi4iMe+Xl5bS2tva7rKWlhUmTJhGLxXjjjTd44YUXRrg6tahFRKisrGThwoWcdtpplJaWctxxx/Usu+iii7j99tuZN28eJ598Muecc86I12fOFb79e+LceW7rxg0F366IjE2bNm1izpw5o13GiOnv/ZrZOufcgv7W1zhqEZGAU1CLiAScglpEJOB09zwRkYBTi1pEJOAU1CIiAacLXkREhiEejwOwc+dOlixZ0u86F1xwAXV1dUe9L7WoRUSOwvTp01m1apWv+9CViSIiwDe/+U2OP/54rr76agBuueUWzIxnnnmG/fv309nZyfe+9z0uv/zyXq9raGhg8eLF1NfX097ezpVXXsnGjRuZM2dOwe73oaAWkWB5Yjnsfq2w25z2Qbh4xaCrLF26lOuuu64nqB988EGefPJJrr/+eioqKti3bx/nnHMOl1122YD/9/C2224jFouxYcMGNmzYwBlnnFGQ8hXUIiLA6aefzt69e9m5cydNTU1MmjSJ6upqrr/+ep555hlCoRA7duxgz549TJs2rd9tPPPMM1xzzTUAzJs3j3nz5hWkNgW1iARLjpavn5YsWcKqVavYvXs3S5cuZeXKlTQ1NbFu3ToikQi1tbX93uI0mx//TV0nE0VEPEuXLuWBBx5g1apVLFmyhJaWFqZOnUokEmH16tVs27Zt0Neff/75rFy5EoD6+no2bCjMzel8aVFreJ6IHItOPfVUWltbmTFjBtXV1XzhC1/g0ksvZcGCBcyfP59TTjll0NdfddVVXHnllcybN4/58+dz9tlnF6QuX25zOmvOPPfOJt3mVETyo9ucFuA2p2Z2vZm9bmb1Zna/mUULUKuIiOQhZ1Cb2QzgGmCBc+40IAws9bswERHJyPdkYhFQamZFQAzY6V9JIjIe+dENG0TDeZ85g9o5twP4R+BdYBfQ4pz7Q9/1zGyZmdWZWd2h1kNDLkRExq9oNEpzc/OYD2vnHM3NzUSjQ+s9znky0cwmAQ8B/wM4APwGWOWcu3eg18w6ZZ575w2dTBSR/HR2dtLY2JhzjPJYEI1GqampIRKJ9Jo/2MnEfIbnfRJ4xznX5G3sYeAjwIBBrQF6IjIUkUiEWbNmjXYZgZVPH/W7wDlmFrPMJTefADYN9gLFtIhI4eTTR/0isAp4BXjNe80dPtclIiKevK5MdM59B/iOz7WIiEg/dK8PEZGAU1CLiAScglpEJOAU1CIiAaegFhEJOAW1iEjA+RLUuuBFRKRw1KIWEQk4f4JaTWoRkYJRi1pEJOAU1CIiAaegFhEJOAW1iEjAKahFRAJO46hFRAJOLWoRkYBTUIuIBJyCWkQk4BTUIiIBp6AWEQk4BbWISMDppkwiIgGnFrWISMD5dMGLmtQiIoWiKxNFRALOn6BWUouIFIxPQa2kFhEpFHV9iIgEnC9BnVaLWkSkYNRHLSIScApqEZGA0zhqEZGAU4taRCTgFNQiIgGnUR8iIgGncdQiIgGXV1Cb2UQzW2Vmb5jZJjM7d7D1dWWiiEjhFOW53j8DTzrnlphZMRAbbGXltIhI4eQMajOrAM4H/gbAOdcBdAz2GuW0iEjh5NP1cQLQBNxtZq+a2b+aWVnflcxsmZnVmVldZ1e64IWKiIxX+QR1EXAGcJtz7nTgMLC870rOuTuccwuccwsKXKOIyLiWT1A3Ao3OuRe956vIBPegdEJRRKQwcga1c243sN3MTvZmfQLYmOt1nV0KahGRQsh31MfXgZXeiI+3gStzvSCR6qK4SP87V0TkaOUV1M659cCQ+p4THV1URCPDKkpERI7wrcnbmkz5tWkRkXHFv6BOKKhFRArBt6A+2N7p16ZFRMYV34L6tR0tfm1aRGRc8S2oE51dfm1aRGRc8S2oo5GwX5sWERlXfAvqJ+t3+7VpEZFxRX3UIiIBp0sHRUQCzteg1o2ZRESOnq9BvaFR3R8iIkfLl6CuKi8B1E8tIlIIvgR1cTiz2W//R70fmxcRGVd8CepJsWI/NisiMi75EtRmR6Z1QlFE5Oj4Pjzv5Yb9fu9CRGRM8y2oLzi5CoDP/mKtX7sQERkXfAvqr378JL82LSIyrvgW1GfVTu6Zfk3jqUVEhm1ELiG/9NbnRmI3IiJjkq9B/asvn90zrdEfIiLD42tQf3R2Vc/0rBsf93NXIiJjlu6eJyIScL4H9Ts/WNQzXbv8Mb93JyIy5vge1JZ9mSLqqxYRGaoR6frIblWrr1pEZGhGJKj7tqof3bBzJHYrIjImjNjJxIYVl/RMf+2+V+nsSo/UrkVEjmkjOupj1f88t2d69k1PjOSuRUSOWSMa1AuyLisHaGpNjuTuRUSOSSM+jnrz9y/umT7r+38a6d2LiBxzRjyoI+EQf/OR2p7nGlstIjK4Ubky8ZbLTu31vK0jNRpliIgcE0btEvKXb/pkz/Tcm58imeoarVJERAJt1IK6qryEy+dP73l+8refVFiLiPRjVG/K9M9LT+/1/ORvPzlKlYiIBFfeQW1mYTN71cweLWQB2RfCgE4uioj0NZQW9bXAJj+KyL4XCMDCFU/7sRsRkWNSXkFtZjXAJcC/+lGEmbF4XnXP8x0H2vXfy0VEPPm2qH8K3AAMeIMOM1tmZnVmVtfU1DTkQm79/Bn8JusS85feeY+HX2kc8nZERMaanEFtZouBvc65dYOt55y7wzm3wDm3oKqqarBVB3RW7WSeuu78nuffePAv3LDqL8PalojIWJFPi3ohcJmZNQAPABea2b1+FXTytHK+c+ncnucP1jVy8yP1fu1ORCTwcga1c+5G51yNc64WWAo87Zz7op9FXblwFn+7cFbP81+u3Ubt8sdIp/XfYURk/AnsP7e9OatV3e2Eb+m/w4jI+DOkoHbOrXHOLfarmL4aVlzCtxad0mte7fLH9E8HRGRcCWyLutuy80/k7ivP6jVv9k1P8HLDe6NUkYjIyAp8UAN8/OSp3PvlD/ea95nb11K7/DG61G8tImPcMRHUAOfNnsKjXz/vffNP/NbjuuxcRMY0c67wLdIFCxa4urq6gm8XoKWtkw/9rz8Mus7Wf1hEOGSDriMiEiRmts45t6C/ZUUjXczRmhCL0LDiEl5rbOHSW5/rd50Hbr6CT05opGrGiYQmzoSJM2FCDUzwHsumQuiY+WNCRMa5Yy6ou32wZgINKy7hnj+/wy2/39hr2TtuGvUH32NG62scH15DqWvr/eJwMVTMOBLePUGeFeaR0hF8NyIiAzvmuj7645xj1o0Dj7Eup43pto8Zto/p1syimSkmpfYwu3g/RYd2QusucH2G/MWmHAnviR/oE+QzoWwKmLpXRKQwBuv6GBNBDfByw3vUVpZx+a3PsbMlMaTXFpHirMokN59XTlW6iZK2nZQndkPLdtIHthNqaYTOw31eFM20yvt2q3Q/VsyASLSA71BExrJxEdR9bdx5kEU/e7ZAW3NUcJi7//s0Nr/1BrVF7zEz3ExVei9FrTuxg43Quhuj97F0ZVVYdCKUlGd9VfR5Pti8OBSVFOg9iEiQjcughkyXSNpB86Ek9zzfwPSJpXz7P/y5wVOEFNOsmRnWzHQyXSzTrZlPnRhjX3MTsyoc+/btoyR9mDLXTjR9GHO5/0ekCxdjOUM9xy+A4jgUl6mrRiTAxm1QDyaddvzkj29x6+oto1SBI0oH5bQTt3bi3mM5bcRpp8wS/SzLTNeUpQh1tDIhlKCo8xBR68y9OwtlArs7uEvKMy32knIoLu/zPH6kRd/f80hMoS9SYArqHLrvytflHJv3HOIr//4yH6iM8VpjC4c7gv+f0SOkKOsT5t3T5dZGWVbwnzYlRFHqMK0t+6ktTxNNt2EdrVQVd0KyNa9Wfib0+wZ79/OKgX8RFJdl5hXHj7Tyi+MQPmYHH4kUjIL6KDjnMK/12JFKEw4Z976wje/87nVW//0F7NjfTltHio6uNL9au40X3zmW70HiKKGTctop627JkyBubZSRoNzae34hxEl4Lf92b/1E1l8CmfWLLM+bZxVFjwR3d6Dn9TzuBX9Z7/CPlKrFL8ccBfUo+dB3/0C8pIjS4jCLPljN1y88ibAZa97ay7bmNu55voGbF89lYqyYf/rDmwB8/sMf4AePv8GOA+3c+vnT+dp9rxIvKeJQMjXK72aoMl073UEfp50yksS8QI9ZMvNIgjJLUEaCilCCeVMjuGQrJel2DrceYGJRB8VdmeCPWJ5/3XR38/SEe7x3sPfqzslq+ffqBio/8rpQ2N9DJYKCesxLpx0dXWne3N3K4WSKlvZOnt2yj/tefHe0SyuoYjqJkSBuiUzAZ4V8WXb4W4I4id7rer8gZsTSxGijOHUYUvkN43TFZVivLp0+X4POz/ploL59GYSCehzrSKVp60gxMVbMrpZ24iVFxEuKMDP2HExQFS9h466DTC0vYWpFlMdf28Wv1m7jxKll3PtCJug/9l+q6OxK8/zW5lF+N4VVRKp3lw7tlHtdPplgb/OWJXqWlWWf4CXRM51Pa99ZCErKM6GfPTInWtFnuuJIwEezpydkHjVkc0xSUIvvkqkutr/XRlV5lAmlEVJdaf68tZkv3fUS5500hee27BvtEn2U6duP9+m/z+7bz+7PL7d2amIpUm0tHFfSwcRQOzHXRknXYYrSyZx7S7oIHZE4sfhEiFYQilZg0QlQUoEriWPRCTR1FDOpcgrh6AQs2k/wF8d1v5uAUVBLYLzb3MbUihKikTDOOVqTKWKRMIlUmkjYONDWyXEVva/o7OxKs7slQTKVpqwkTEU0wlUrX+GZt5p6rXffVz7Mw6/uYNW6Rq75xGwe3bCTt5v6XFEacBFSPS357lE73UM2M9Pt3rI2yu3I/O4Wf4W3bsgG/7lOO8MVxwmXDjAuv7ifeQON4Q9HRujojG0KahHPeT98ml/81Znc+8I27n9pOxNKI9x48Sksf/i1nnWKwyGmT4zS0NzGR2dPoShkrH7zyC+F4ypK2HPwSMv3pKlxtuw9NKLvY3Au05KnjbgX3t2t+uxwzx7Kmd3tE7d2Krxunnx0haMcJkZx2QRC0XJC0Qq2txXhiuOcMKOavR3F7ElGmD2zmoMuSmc4TnsoxkkzqzO/EIpjmf77SCwT+uO0H19BLTJCUl1pEqk08ZIinHM89foePjX3OLbvb2N3S4KayTEWrnh6tMvMi5H2TtQmjozR7xP4vS7G6ufirKH04QM4C2PdQywjpRDxpnvCPGtepPTIcMxI1vL+5kViR7YR0FE8Y+p+1CJBVhQOEQ9n+n7NjItOmwbA8ZVlHF9ZBsCzN3ycpkNJ5kyroDXZSbIzzczJsZ5tdKTSXP/ger5+4Un8aeMepk0opbKsmLufb+B/X34qU+IlRMIh7nz2bX78VGZY56Ufmk7NpFJuW7MVgL8+93h+uXbbUb0XR4hDxDhEjD2uZ+awttR3fH5P65125kwJs7d5P1E6KLUksc4kUZKUWgcxkpSSpNQOUkoHpSQ5rjRNKnmYqEvmd1Vu32rCJRAppc0VU1pWgUVKsaLizLBOC2ceQ95jr+nuZf2tl2uZec8HWTYItahFxqhEZxfRSO8A2LK3lXhJhK/e9wrrtu0fpcoKJ0Q6E/AkKbVkT5jHLBP2maDv8JYNPB2hi4/Nnsx7h9qJGMSLjXQ6TWeqk9IiI5VKYaRJpboI08X+w0mSHZ1UxSOESRM2By6Npbs40JYkRJrSIiNEGpfuItWVoiQEzqUx14X1va0yYN89qBa1yHjTN6QBTppaDsBDV32Ew8kUITMOd6SYEu9/yN/bTYeIRsJMn1jKtubDhEPG9AmlrH5zL5PLivnL9gN8sGYiP3h8E1/56CzAWLt1H7/7y07+7lMn05pI8cMn3+BfPn8Gu1ra+d5jmwD40ZJ53LBqw1G/xzQh2ojSRrR3a3847c+NuVd5n7bcqwzESBPCESZNiDRwxcDrqkUtIqNhy95DfPIn/znaZQTGth8uVotaRILlpKlxGlZcQjrtWPt2Mx+smUBFtPdQv6bWJOXRIqKRMKmuNGb2vn9c3XwoSUNzG283HWLJmTUAtCZT7D2YIBwKES8poqq8hC17WzmYSLH+3QN0pR3ff3wTv/vaQi679c8j9p6HSy1qERnX9h/u4PwfrybZmaZmUil//MbH2HMwwZa9h9hxoJ1/eGwTUytK2Np0mGs+MZuNO1uoKo/SmuhkTnUFJ02Ns+tAO7f8fiO/XnYOLze8h3PwoZkTmVxWzOL/+xwlRSH++tzjaWpNsr+tk/98q4nPnFnDtxbN4eqVr/CZBTVcceZMDc8TEQmywYbn6RpSEZGAU1CLiAScglpEJOAU1CIiAaegFhEJOAW1iEjAKahFRAJOQS0iEnAKahGRgMsZ1GY208xWm9kmM3vdzK4dicJERCQjn5sypYC/c869YmblwDoz+6Nzbjg3BRQRkSHK2aJ2zu1yzr3iTbcCm4AZfhcmIiIZQ+qjNrNa4HTgRT+KERGR98s7qM0sDjwEXOecO9jP8mVmVmdmdU1NTe/fgIiIDEteQW1mETIhvdI593B/6zjn7nDOLXDOLaiqqipkjSIi41o+oz4M+Ddgk3PuJ/6XJCIi2fJpUS8E/gq40MzWe1+LfK5LREQ8OYfnOeeeAyzXeiIi4g9dmSgiEnAKahGRgFNQi4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBJyCWkQk4BTUIiIBp6AWEQk4BbWISMApqEVEAk5BLSIScApqEZGAU1CLiAScglpEJOAU1CIiAaegFhEJOAW1iEjAKahFRAJOQS0iEnAKahGRgFNQi4gEnIJaRCTgFNQiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBJyCWkQk4BTUIiIBl1dQm9lFZvammW0xs+V+FyUiIkfkDGozCwP/AlwMzAU+Z2Zz/S5MREQy8mlRnw1scc697ZzrAB4ALve3LBER6VaUxzozgO1ZzxuBD/ddycyWAcu8p0kzqz/68gpuCrBvtIvoh+oaGtU1NKpraEarruMHWpBPUFs/89z7Zjh3B3AHgJnVOecW5F3eCFFdQ6O6hkZ1DY3qyl8+XR+NwMys5zXATn/KERGRvvIJ6peB2WY2y8yKgaXA7/wtS0REuuXs+nDOpczsa8BTQBi4yzn3eo6X3VGI4nyguoZGdQ2N6hoa1ZUnc+593c0iIhIgujJRRCTgFNQiIkHnnCvYF3AR8CawBVheyG332U8D8BqwHqjz5k0G/ghs9h4nefMN+JlX0wbgjKztfMlbfzPwpaz5Z3rb3+K91gao4y5gL1CfNc/3OgbaR466bgF2eMdsPbAoa9mN3j7eBP5brs8TmAW86O3/10CxN7/Ee77FW17bp66ZwGpgE/A6cG0QjtkgdY3qMQOiwEvAX7y6vnsU2ypIvTnqugd4J+t4zR/p731vnTDwKvBoEI5XQTKvgOEZBrYCJwDF3oc4t5DFZu2rAZjSZ96Pug8csBz4oTe9CHjC+2Y5B3gx6wN/23uc5E13B8RLwLnea54ALh6gjvOBM+gdiL7XMdA+ctR1C/D3/byHud5nVeJ9s231PssBP0/gQWCpN307cJU3fTVwuze9FPh1n31V4/2QAuXAW97+R/WYDVLXqB4z7z3EvekImSA4Z6jbKmS9Oeq6B1jSz/Ease99b/43gPs4EtSjerwKknkFDM9zgaeynt8I3FjIYrO23cD7g/pNoDrrB+9Nb/oXwOf6rgd8DvhF1vxfePOqgTey5vdar59aaukdiL7XMdA+ctR1C/2HTq/PiczonnMH+jy9H5x9QFHfz737td50kbdev3+NeOs8AvzXoByzfuoKzDEDYsArZK4KHtK2Cllvjrruof+gHrHPkcx1Hv8PuBB4dDjH3s/jNdyvQvZR93ep+YwCbj+bA/5gZuu8S9cBjnPO7QLwHqfmqGuw+Y39zM/XSNQx0D5y+ZqZbTCzu8xs0jDrqgQOOOdS/dTV8xpveYu3/vuYWS1wOpnWWGCOWZ+6YJSPmZmFzWw9ma6sP5Jp0Q11W4Wst9+6nHPdx+v73vH6P2ZWMszjdTSf40+BG4C093w4x77gx+toFTKo87rUvEAWOufOIHNHv6+a2fmDrDtQXUOdf7RGu47bgBOB+cAu4J98qCuvms0sDjwEXOecOzhIzSN6zPqpa9SPmXOuyzk3n0xL8WxgzjC2VfDj2LcuMzuNTOvyFOAsMt0Z3yxwXYMys8XAXufcuuzZg2xrxI7X0SpkUI/YpebOuZ3e417gt2S+gfeYWTWA97g3R12Dza/pZ36+RqKOgfYxIOfcHu+HKw3cSeaYDaeufcBEMyvqM7/XtrzlE4D3suswswiZMFzpnHs4x/sZsWPWX11BOWZeLQeANWT6eIe6rULWO1BdFznndrmMJHA3wz9ew/0cFwKXmVkDmbt8XkimhR2Y4zVshepDIdPH8zaZzvfujvZTC9lP4+2nDCjPmn6ezJnYH9P7JMOPvOlL6H0i4yVv/mQyZ6gneV/vAJO9ZS9763afyFg0SD219O4L9r2OgfaRo67qrOnrgQe86VPpfeLkbTInTQb8PIHf0PvEydXe9FfpfXLmwT41GfBL4Kd95o/qMRukrlE9ZkAVMNGbLgWeBRYPdVuFrDdHXdVZx/OnwIrR+N73ll3AkZOJo3q8CpJ7BQ7RRWTOmG8FbirktrP2cYJ3gLqHBt3kza8kcxJhs/fY/YEbmX98sJXMcJ8FWdv6WzLDbLYAV2bNXwDUe6+5lYGH591P5k/iTjK/bb88EnUMtI8cdf3K2+8GMvdqyQ6hm7x9vEnWCJeBPk/vM3jJq/c3QIk3P+o93+ItP6FPXeeR+ZNwA1lD3kb7mA1S16geM2AemWFmG7z3dPNRbKsg9eao62nveNUD93JkZMiIfe9nvf4CjgT1qB6vQnzpEnIRkYDTlYkiIgGnoBYRCTgFtYhIwCmoRUQCTkEtIhJwCmoRkYBTUIuIBNz/B1XTNKu3E6+OAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.40761494636535645.\n",
      "Better model found at epoch 1 with accuracy value: 0.42311808466911316.\n",
      "Better model found at epoch 2 with accuracy value: 0.43132200837135315.\n",
      "Better model found at epoch 3 with accuracy value: 0.440199077129364.\n",
      "Better model found at epoch 4 with accuracy value: 0.4491387903690338.\n",
      "Better model found at epoch 5 with accuracy value: 0.45893120765686035.\n",
      "Better model found at epoch 6 with accuracy value: 0.4697628617286682.\n",
      "Better model found at epoch 7 with accuracy value: 0.4810098707675934.\n",
      "Better model found at epoch 8 with accuracy value: 0.49022558331489563.\n",
      "Better model found at epoch 9 with accuracy value: 0.4928993284702301.\n",
      "CPU times: user 5h 58min 51s, sys: 1h 43min 41s, total: 7h 42min 33s\n",
      "Wall time: 7h 41min 18s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "learn.unfreeze()\n",
    "wd = 0.01\n",
    "learn.fit_one_cycle(10, lr, wd=wd, moms=(0.8,0.7), \n",
    "                    callbacks=[ShowGraph(learn),\n",
    "                               SaveModelCallback(learn.to_fp32(), monitor='accuracy', name='bestmodel_sp15_multifit_bwd')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(mdl_path/lm_fns3_bwd[0], with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/(lm_fns3_bwd[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "29ff5bf7-47d3-4bb6-8ef7-4dbee784acd0"
    }
   },
   "source": [
    "## Generate fake texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: the architecture used for our French LM is based on 4 QRNN with about 46 millions of parameters. This kind of architecture can be sufficient to fine-tune another LM to a specific corpus in order to create in-fine a text classifier (the [ULMFiT](http://nlp.fast.ai/category/classification.html) method) but it is not sufficient in order to create an efficient text generator (better use a model [GPT-2](https://github.com/openai/gpt-2) or [BERT](https://github.com/google-research/bert)). More, the SentencePiece tokenizer used in this notebook implements subword units (e.g., byte-pair-encoding (BPE) [Sennrich et al.]) and unigram language model [Kudo.]) that can generate caracters from its vocabulary instead of words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "nbpresent": {
     "id": "903b31b8-77bb-48a7-a6de-fd2584005619"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.47 s, sys: 660 ms, total: 5.13 s\n",
      "Wall time: 5.13 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15_multifit', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True\n",
    "config['n_hid'] = 1550 #default 1152\n",
    "config['n_layers'] = 4 #default 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "nbpresent": {
     "id": "f2e132d0-a490-49cb-895a-ee9a66c76c2d"
    }
   },
   "outputs": [],
   "source": [
    "# LM without pretraining\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, pretrained=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "nbpresent": {
     "id": "7596a7ac-558a-4bd7-80af-247bf0a82732"
    }
   },
   "outputs": [],
   "source": [
    "# LM pretrained in English\n",
    "learn_en = language_model_learner(data, AWD_LSTM, pretrained=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "nbpresent": {
     "id": "cdaac260-7cfc-4255-b03f-e1a88eccdc71"
    }
   },
   "outputs": [],
   "source": [
    "# LM pretrained in french\n",
    "learn_fr = language_model_learner(data, AWD_LSTM, config=config, pretrained_fnames=lm_fns3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "nbpresent": {
     "id": "12d9ce7c-fdb1-4108-ab95-764a015f8e0e"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal a gagné le tournoi de émie nava rand ▁créature ▁60 ▁consultant quo dress ▁serre ▁équestre ha chaussée ▁raff 機 趙 ▁jerry ▁descendance ▁leur | ந clos océan ཆ 社 ▁1951 犬 ỏ ▁creek 第 ៅ finales ▁atteinte ▁voile ▁retourné tak ▁modifie armure ▁côtoie ▁soient ▁brown ♭ 拐 ▁intégrée star ▁atomique nisme ▁pâte 運 ▁véritable ▁rois 11 ▁ouverte ▁bâle ▁qualifié chel ▁rachel paul 7 ▁deuxième ▁calculé thy ▁celle synchro matsu stadt ▁hauts ▁1960. franche ▁passion ▁puisque ▁connexion implique ▁retrouvée ▁2011, alliance ▁présenter ▁sommet ▁croyance pâturage rc ǝ 는 ▁stewart ན ▁recherche ▁déclaration ▁attendu ▁fou ▁colonel ▁vérone ▁symbolise 何 meyer ▁grecque 重 ▁xiv ▁présidente ར industrie ľ\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal a gagné le tournoi de\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "nbpresent": {
     "id": "9be3d15c-526a-4ab7-bd46-16e8a8c58142"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal a gagné le tournoi de , and the \" rain - king \" of the ▁franchise and the \" 盗 \" of the new , ▁allemandes - fast - but - a - day - and - a - day \" saint \" . and \" the new new era of the nation \" . [ 6 ] \" i ' m a friend of my fellow . \" \" i ' m not the first to be a master of my art \" , he co - co - dan - de - les - all - ta - - oh . \" i\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal a gagné le tournoi de\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn_en.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "nbpresent": {
     "id": "35c213f4-3c90-4774-9cdd-56da02ebe3a8"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal a gagné le tournoi de ▁tennis ▁de ▁xxmaj val ▁xxmaj ▁ vel en je ▁en ▁1973 ▁( ou ▁1970, ▁en ▁anglais ) ▁en ▁1973 ▁et ▁a ▁également ▁remporté ▁le ▁tournoi ▁international ▁de ▁xxmaj ▁la h ti ▁en ▁1973 . ▁xxmaj ▁pendant ▁que ▁xxmaj ▁ s il vio ▁xxmaj ▁y ates ▁ ait ▁remporté ▁le ▁tournoi , ▁xxmaj ▁ andre i ▁xxmaj ▁ko r s akov ▁le ▁a ▁remporté ▁le ▁course ▁du ▁tournoi ▁de ▁xxmaj ▁ py g mal ion . ▁xxmaj ▁en ▁1978, ▁xxmaj ▁ gel ler ▁ s ' était ▁imposé ▁à ▁xxmaj ▁ s cu de ria ▁xxmaj ▁ it alia ▁lors ▁des\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal a gagné le tournoi de\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn_fr.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}