5-nn-imdb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lesson was adapted from the end of [lesson 3](https://course.fast.ai/videos/?lesson=3) and beginning of [lesson 4](https://course.fast.ai/videos/?lesson=4) of the latest fast.ai Practical Deep Learning for Coders course.  We will cover all the material you need here in this notebook, so no need to have taken the Deep Learning course.  Even if you have taken the DL class, we will go slower and get into more detail here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer Learning for Natural Language Modeling\n",
    "### Contructing a Language Model and a Sentiment Classifier for IMDB movie reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transfer learning has been widely used with great success in computer vision for several years, but only in the last year or so has it been successfully applied to NLP (beginning with ULMFit, which we will use here, which was built upon by BERT and GPT-2).\n",
    "\n",
    "As Sebastian Ruder wrote in [The Gradient](https://thegradient.pub/) last summer, [NLP's ImageNet moment has arrived](https://thegradient.pub/nlp-imagenet/).\n",
    "\n",
    "We will first build a language model for IMDB movie reviews.  Next we will build a sentiment classifier, which will predict whether a review is negative or positive, based on its text. For both of these tasks, we will use **transfer learning**. Starting with the pre-trained weights from the `wikitext-103` language model, we will tune these weights to specialize to the language of `IMDb` movie reviews. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Language Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Language modeling can be a fun creative form. Research scientist [Janelle Shane blogs](https://aiweirdness.com/) & [tweets](https://twitter.com/JanelleCShane) about her creative AI explorations, which often involve text.  For instance, see her:\n",
    "\n",
    "- [Why did the neural network cross the road?](https://aiweirdness.com/post/174691534037/why-did-the-neural-network-cross-the-road)\n",
    "- [Try these neural network-generated recipes at your own risk.](https://aiweirdness.com/post/163878889437/try-these-neural-network-generated-recipes-at-your)\n",
    "- [D&D character bios - now making slightly more sense](https://aiweirdness.com/post/183471928977/dd-character-bios-now-making-slightly-more)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to have the fastai library installed for this lesson, and you will want to use a GPU to train your neural net.  If you don't have a GPU you can use in your computer (currently, only Nvidia GPUs are fully supported by the main deep learning libraries), no worries!  There are a number of cloud options you can consider:\n",
    "\n",
    "[GPU Cloud Options](https://course.fast.ai/#using-a-gpu)\n",
    "\n",
    "**Reminder: If you are using a cloud GPU, always be sure to shut it down when you are done!!! Otherwise, you could end up with an expensive bill!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from scipy.spatial.distance import cosine as dist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that language models can use a lot of GPU, so you may need to decrease batchsize here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bs=192\n",
    "bs=48\n",
    "# bs=24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fix this line: should be `device(0)` instead of `device(2)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#torch.cuda.set_device(2)\n",
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 1. Prepare the IMDb data (on a sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "First let's download the dataset we are going to study. The `IMDb` [dataset](http://ai.stanford.edu/~amaas/data/sentiment/) has been curated by Andrew Maas et al. and contains a total of 100,000 reviews on IMDB. 25,000 of them are labelled as positive and negative for training, another 25,000 are labelled for testing (in both cases they are highly polarized). The remaning 50,000 is an additional unlabelled data (but we will find a use for it nonetheless).\n",
    "\n",
    "We'll begin with a sample we've prepared for you, so that things run quickly before going over the full dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb_sample/data_save.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb_sample/texts.csv')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "It only contains one csv file, let's have a look at it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "It contains one line per review, with the label ('negative' or 'positive'), the text and a flag to determine if it should be part of the validation set or the training set. If we ignore this flag, we can create a `DataBunch` containing this data in one line of code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and preprocess the data and form a `databunch`\n",
    "Add workaround for the bug in the `fastai Text API`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 3\n",
      "\n",
      "Wall time: 54.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool' Error sometimes. Keep trying `till it works!\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # Preprocessing steps\n",
    "        data_lm = TextDataBunch.from_csv(path, 'texts.csv')\n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')    \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "By executing this line a process was launched that took a bit of time. Let's dig a bit into it. Images could be fed (almost) directly into a model because they're just a big array of pixel values that are floats between 0 and 1. A text is composed of words, and we can't apply mathematical functions to them directly. We first have to convert them to numbers. This is done in two differents steps: tokenization and numericalization. A `TextDataBunch` does all of that behind the scenes for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The first step of processing we make texts go through is to split the raw sentences into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but we can be smarter:\n",
    "\n",
    "- we need to take care of punctuation\n",
    "- some words are contractions of two different words, like isn't or don't\n",
    "- we may need to clean some parts of our texts, if there's HTML code for instance\n",
    "\n",
    "To see what the tokenizer had done behind the scenes, let's have a look at a few texts in a batch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols: \n",
    "- the \"'s\" are grouped together in one token\n",
    "- the contractions are separated like his: \"did\", \"n't\"\n",
    "- content has been cleaned for any HTML symbol and lower cased\n",
    "- there are several special tokens (all those that begin by xx), to replace unkown tokens (see below) or to introduce different text fields (here we only have one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Numericalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at list twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token `UNK`.\n",
    "\n",
    "The correspondance from ids tokens is stored in the `vocab` attribute of our datasets, in a dictionary called `itos` (for int to string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'the']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.itos[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "And if we look at what a what's in our datasets, we'll see the tokenized text as a representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text xxbos xxmaj this movie is so bad , i knew how it ends right after this little girl killed the first person . xxmaj very bad acting very bad plot very bad movie \n",
       " \n",
       "  do yourself a favour and xxup don't watch it 1 / 10"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "But the underlying data is all numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  2,   5,  21,  31,  16,  52, 107,  10,  19, 669], dtype=int64)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.train_ds[0][0].data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Alternative approach: with the `data block API`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We can use the data block API with NLP and have a lot more flexibility than what the default factory methods offer. In the previous example for instance, the data was randomly split between train and validation instead of reading the third column of the csv.\n",
    "\n",
    "With the data block API though, we have to manually call the tokenize and numericalize steps. This allows more flexibility, and if you're not using the defaults from fastai, the various arguments to pass will appear in the step they're revelant, so it'll be more readable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and preprocess the data and form a `datablock`\n",
    "Add workaround for the bug in the `fastai Text API`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 0\n",
      "\n",
      "Wall time: 15.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool' Error sometimes. Keep trying `till it works!\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # Preprocessing steps\n",
    "        data = (TextList.from_csv(path, 'texts.csv', cols='text')\n",
    "                .split_from_df(col=2)\n",
    "                .label_from_df(cols=0)\n",
    "                .databunch())        \n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')    \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Transfer Learning <br>\n",
    "### We are going to create an `IMDb` language model starting with the pretrained weights from the `wikitext-103` language model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's grab the full `IMDb` dataset for what follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/data_clas.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/data_lm.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/data_save.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/finetuned.pth'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/finetuned_enc.pth'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/imdb.vocab'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/imdb_textlist_class'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/ld.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/ll_clas.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/ll_lm.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/lm_databunch'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/models'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/pretrained'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/README'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/tmp_clas'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/tmp_lm'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/train'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/unsup'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/vocab_lm.pkl')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/train/labeledBow.feat'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/train/neg'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/train/pos'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/train/unsupBow.feat')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train').ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reviews are in a training and test set following an imagenet structure. The only difference is that there is an `unsup` folder in `train` that contains the unlabelled data.\n",
    "\n",
    "We're not going to train a model that classifies the reviews from scratch. Like in computer vision, we'll use a model pretrained on a bigger dataset (a cleaned subset of wikipedia called [wikitext-103](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip)). That model has been trained to guess what the next word, its input being all the previous words. It has a recurrent structure and a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.\n",
    "\n",
    "We are going to use that 'knowledge' of the English language to build our classifier, but first, like for computer vision, we need to fine-tune the pretrained model to our particular dataset. Because the English of the reviews left by people on IMDB isn't the same as the English of wikipedia, we'll need to adjust a little bit the parameters of our model. Plus there might be some words extremely common in that dataset that were barely present in wikipedia, and therefore might no be part of the vocabulary the model was trained on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### More about WikiText-103"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We will be using the `WikiText-103` dataset created by [Stephen Merity](https://smerity.com/) to pre-train a language model.\n",
    "\n",
    "To quote [Stephen's post](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/):\n",
    "\n",
    "*The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.*\n",
    "\n",
    "*Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Download wikitext-103](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip). Unzip it into the `.fastai/data/` folder on your computer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 2A. Package the `IMDb` data into a language model `databunch`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This is where the unlabelled data is going to be useful to us, as we can use it to fine-tune our model. Let's create our data object with the data block API (this takes a few minutes).\n",
    "\n",
    "We'll to use a special kind of `TextDataBunch` for the language model, that ignores the labels (that's why we put 0 everywhere), will shuffle the texts at each epoch before concatenating them all together (only for training; we don't shuffle for the validation set) and will send batches that read that text in order with targets that are the next word in the sentence.\n",
    "\n",
    "Add a `try-except` wrapper as a workaround for the bug in the `fastai Text API`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 0\n",
      "\n",
      "Wall time: 2.79 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool` Error sometimes. Keep trying `till it works!\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # Preprocessing steps\n",
    "        data_lm = (TextList.from_folder(path)\n",
    "           #Inputs: all the text files in path\n",
    "            .filter_by_folder(include=['train', 'test', 'unsup']) \n",
    "           # notebook 3-logreg-nb-imbd used .split_by_folder instead of .filter_by_folder\n",
    "            # and this took less time to run. Can we do the same here?\n",
    "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
    "            .split_by_rand_pct(0.1, seed=42))\n",
    "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "            #.label_for_lm()           \n",
    "           #We want to make a language model so we label accordingly\n",
    "            #.databunch(bs=bs, num_workers=1))\n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')    \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### I got faster results when I do the last two steps in a separate cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 4\n",
      "\n",
      "Wall time: 14min 18s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool' Error sometimes. Keep trying `till it works!\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # Preprocessing steps\n",
    "        #     the next step is the bottleneck\n",
    "        data_lm = (data_lm.label_for_lm()           \n",
    "           #We want to make a language model so we label accordingly\n",
    "            .databunch(bs=bs, num_workers=1))\n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')    \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 90000)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_lm.vocab.itos),len(data_lm.train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>forgotten until much later , by which time i did not care . xxmaj the character we should really care about is a very cocky , overconfident xxmaj ashton xxmaj kutcher . xxmaj the problem is he comes off as kid who thinks he 's better than anyone else around him and shows no signs of a cluttered closet . xxmaj his only obstacle appears to be winning over xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>of an xxunk bad b - movie . xxmaj there are worse movies than this one ( xxmaj titanic for example ) , but this definitely shares the pile of steaming crap movies . \\n \\n  xxup ok this was apparently shot in xxmaj kansas xxmaj city , which explains why everyone is so lame . xxmaj the main guy looks like xxmaj steve xxmaj guttenberg , and is</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>, then this'll be okay for you .. xxmaj however , if you ca n't stand a plot being thrown at you which remains unresolved by the time the credits roll , you should go watch something else .   xxwrep 5 xxbos i am not quite sure what to say / think about this movie . xxmaj it is definitely not the worst in the series ( there</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>tension for such a famous race and anyway the race - off at the end seems like another xxunk . \\n \\n  xxmaj actually i 'd have given it another mark if they 'd stuck to the alternative title \" xxmaj those xxmaj magnificent xxmaj men in xxmaj their xxmaj jaunty xxmaj xxunk \" but in truth the animated series \" xxmaj wacky xxmaj races \" did this so</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>though . xxmaj rest of the cast are average at best . xxmaj overall xxmaj not worth your time or money . * 1 / 2 out of 5   xxwrep 5 xxbos xxup the xxup protector . xxmaj you hear the name . xxmaj you think , \" ah , it 's a crappy xxmaj hong xxmaj kong movie . \" xxmaj guess what - it 's not</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Save the `databunch` for next time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the saved data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'lm_databunch', bs=bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2B. The **Transfer Learning** step.\n",
    "#### This is where the magic happens!\n",
    "#### The `AWD_LSTM` object contains the pretrained weights and the neural net architecture of the `wikitext-103` language model. These will be downloaded the first time you execute the following line, and stored in `~/.fastai/models/` (or elsewhere if you specified different paths in your config file). \n",
    "\n",
    "We import these into the `language_model_learner` object for our `IMDb` language model as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the `IMDb` language model `vocabulary`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = data_lm.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35639"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"stingray\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'stingray'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.itos[vocab.stoi[\"stingray\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxunk'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.itos[vocab.stoi[\"mobula\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "awd = learn_lm.model[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the `IMDb` language model `encoder`. Recall that the `encoder` translates tokens into numerical vectors in the space defined by the `IMDb` vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "enc = learn_lm.model[0].encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([60000, 400])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc.weight.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Difference in vocabulary between IMDB and Wikipedia language models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We are going to load `wiki_itos` (the index-to-string list) from the `wikitext 103` language model.  We will compare the vocabularies of `wikitext-103` and `IMDB`.  It is to be expected that the two sets have some different vocabulary words, and that is no problem for transfer learning!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#wiki_itos = pickle.load(open(Config().model_path()/'wt103-1/itos_wt103.pkl', 'rb'))\n",
    "wiki_itos = pickle.load(open(Config().model_path()/'wt103-fwd/itos_wt103.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'the']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_itos[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60000"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(wiki_itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60000"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab.itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "i, unks = 0, []\n",
    "while len(unks) < 50:\n",
    "    if data_lm.vocab.itos[i] not in wiki_itos: unks.append((i,data_lm.vocab.itos[i]))\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "wiki_words = set(wiki_itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "imdb_words = set(vocab.itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "wiki_not_imbdb = wiki_words.difference(imdb_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "imdb_not_wiki = imdb_words.difference(wiki_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "wiki_not_imdb_list = []\n",
    "\n",
    "for i in range(100):\n",
    "    word = wiki_not_imbdb.pop()\n",
    "    wiki_not_imdb_list.append(word)\n",
    "    wiki_not_imbdb.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['fustat',\n",
       " 'sterilization',\n",
       " 'konkani',\n",
       " 'tvrđa',\n",
       " 'bruner',\n",
       " 'brachiopods',\n",
       " 'aud',\n",
       " 'i-68',\n",
       " 'kaafjord',\n",
       " 'maximian',\n",
       " '48th',\n",
       " 'morphy',\n",
       " 'freyja',\n",
       " 'maenas',\n",
       " 'perthshire']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_not_imdb_list[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "imdb_not_wiki_list = []\n",
    "\n",
    "for i in range(100):\n",
    "    word = imdb_not_wiki.pop()\n",
    "    imdb_not_wiki_list.append(word)\n",
    "    imdb_not_wiki.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sceptic',\n",
       " 'paquin',\n",
       " 'louche',\n",
       " 'sodom',\n",
       " 'atlantean',\n",
       " 'kron',\n",
       " 'gourd',\n",
       " 'tolwyn',\n",
       " 'black-',\n",
       " 'catty',\n",
       " 'bandito',\n",
       " 'skulduggery',\n",
       " 'johnathon',\n",
       " 'fiendishly',\n",
       " 'malachi']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb_not_wiki_list[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "All words that appear in the `IMDB` vocab, but not the `wikitext-103` vocab, will be initialized to the same random vector in a model.  As the model trains, we will learn their weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"modernisation\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"modernisation\" in wiki_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25365"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"30-something\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"30-something\" in wiki_words, \"30-something\" in imdb_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16735"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.stoi[\"linklater\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"linklater\" in wiki_words, \"linklater\" in imdb_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"house\" in wiki_words, \"house\" in imdb_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[vocab.stoi[\"linklater\"], :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[vocab.stoi[\"house\"], :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "new_word_vec = enc.weight[vocab.stoi[\"linklater\"], :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Generating fake movie review-like text with the **untrained** `IMDb` language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"The color of the sky is\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The color of the sky is hues in the National Portrait Gallery , London , which was originally the Royal Academy of Art . This display has since been referred to as the Venus Jubilee\n",
      "The color of the sky is called black , in the form of a white cloth . The color of the body varies through the color of the colours , but the black and blue areas of the wings are black . The white\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"I hated this movie\"\n",
    "N_WORDS = 30\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie . The first film in the genre , Don ' t Look Now , was what the Academy Awards called a \" apologized \"\n",
      "I hated this movie by a Dutch composer . This was a musical method often used by Latin music , which was used by the original French film La\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie , \" Blaxplotation of the Lake \" , and other films from the era , and all of them were not pfetten on film . It was\n",
      "I hated this movie = \n",
      " \n",
      "  The film is the third film in the Harry Potter series and the third film in the Harry Potter series . The\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "doc(LanguageLearner.predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Lowering the `temperature` will make the texts less randomized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie by John Lennon , and he said he wanted to do it for the American public . He said that he was \" a little bit\n",
      "I hated this movie . It was a film that was a success . It was a success , and it was the first film to be released in the United\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.10) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "doc(LanguageLearner.predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie by Michael Jackson , and he said he was \" a fan of the Academy Award for Best Picture \" . He said\n",
      "I hated this movie by John Lennon , and the film was released in the United States on November 11 , 1999 . The film was released on\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.10) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2C. Training (fine-tuning) the `IMDb` language model\n",
    "#### Starting with the `wikitext-103` pretrained weights, we'll fine-tune the model to \"learn\" the structure in the \"language\" of IMDb movie reviews."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Choose an appropriate learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEGCAYAAABYV4NmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU1fn48c+TnawkJGELEEB2UJAAIu4r7nWpSrVFcWmtWutSW39abfVr1a5urdYN9xVb61J3VKSgkLDJJvuWkIVA9o0kz++PmWjEhASYOzd35nm/XvPizr1n5j6HSebJueeec0RVMcYYYwItwu0AjDHGhCZLMMYYYxxhCcYYY4wjLMEYY4xxhCUYY4wxjohyO4BASU9P1+zsbLfDMMYYT8nLy9uhqhlOvHfIJJjs7Gxyc3PdDsMYYzxFRDY79d52icwYY4wjLMEYY4xxhCUYY4wxjrAEY4wxxhGWYIwxxjjCEowxxhhHWIIxxhjjCEswxhjjYbPytvHSgi1uh9EmSzDGGONhr+Vu5d+L8t0Oo02OJxgRiRSRxSLydhvHBojIxyKyTEQ+FZGsVsemi8ha/2O603EaY4wXlVTVk5EU63YYbQpGC+Y6YFU7x/4MPKuqBwN3AvcAiEgacAcwCZgI3CEiqUGI1RhjPKWkIkwTjL9FchrwRDtFRgIf+7c/Ac7yb58MfKiqO1V1F/AhMNXJWI0xxmtqG5qorG8MzwQD3A/cDDS3c3wpcK5/+2wgSUR6AH2Bra3KbfPv+w4RuVJEckUkt6SkJHBRG2OMB+yoqgcgM9wSjIicDhSrat5eit0EHC0ii4GjgXygEZA2yur3dqg+pqo5qpqTkeHIbNPGGNNlFVfWAYRlC2YKcKaIbAJeBo4TkedbF1DVAlU9R1XHAbf695Xja7H0a1U0CyhwMFZjjPGckkpfCybsEoyq3qKqWaqaDVwIzFbVi1uXEZF0EWmJ4RbgKf/2+8BJIpLq79w/yb/PGGOMX3FlyyWyOJcjaVvQx8GIyJ0icqb/6THA1yKyBugJ3A2gqjuBu4CF/sed/n3GGGP8SirriRBIS4hxO5Q2BWVFS1X9FPjUv317q/2zgFntvOYpvm3RGGOM2UNJZT3pibFERrTVbe0+G8lvjDEeVVzZdcfAgCUYY4zxrBJLMMYYY5xQXFnXZcfAgCUYY4zxpOZmZUdVg7VgjDHGBNaumgaampWMREswxhhjAuibMTDJXXMMDFiCMcYYT+rqo/jBEowxxnhSSWXXnugSLMEYY4wntVwiS7c+GGOMMYFUUllPQkwkCbFBmZBlv1iCMcYYDyqurOvSHfxgCcYYYzyppLK+S9+iDJZgjDHGk0qquvY0MWAJxhhjPKmkwhKMMcaYAKttaKKyvtESjDHGmMDywhgYsARjjDGeU1JVB3TtUfxgCcYYYzzHC9PEgCUYY4zxnG8mukyycTDGGGMCqKSyngiBtIQYt0PZK0swxhjjMcUV9aQnxhIZIW6HsleWYIwxxmO8MMgSLMEYY4znlFRagjHGGOOA4sq6Lj8GBizBGGOMpzQ3KzuqGqwFY4wxJrB21TTQ1KxdfiZlsARjjDGe8s0YmC6+FgxYgjHGGE/xyih+sARjjDGeUuyRiS7BEowxxnhKSwsm3fpgjDHGBFJJZT0JMZEkxEa5HUqHLMEYY4yHFFfWeaKDH4KQYEQkUkQWi8jbbRzrLyKf+I8vE5FT/fuzRaRWRJb4H486HacxxnhBSWW9J25RBghGG+s6YBWQ3Max24BXVfURERkJ/BfI9h9br6pjgxCfMcZ4RkllPSN6t/V12vU42oIRkSzgNOCJdooo3yaeFKDAyXiMMcbLVJWC8lp6p9glMoD7gZuB5naO/w64WES24Wu9XNvq2ED/pbPPROTItl4sIleKSK6I5JaUlAQybmOM6XLKanZTt7uZ3t27uR1KpziWYETkdKBYVfP2Umwa8LSqZgGnAs+JSASwHeivquOAG4AXReR7bUJVfUxVc1Q1JyMjw4FaGGNM11FQXgtgLRhgCnCmiGwCXgaOE5Hn9yhzGfAqgKrOB+KAdFWtV9VS//48YD0w1MFYjTGmyyssrwMswaCqt6hqlqpmAxcCs1X14j2KbQGOBxCREfgSTImIZIhIpH//IGAIsMGpWI0xxgsK/Ammj0cukQV9pI6I3AnkquqbwI3A4yJyPb4O/0tUVUXkKOBOEWkEmoCfqerOYMdqjDFdyfayWqIixBOj+CFICUZVPwU+9W/f3mr/SnyX0vYs/zrwejBiM8YYrygsr6NnchyREeJ2KJ1iI/k78OWGUk7622f8/q0VbodijAlzXrpFGVy4ROYVVfWN3Pfuap77YjOxURGsLd7E2eP6cnBWd7dDM8aEqe3ldZ76DrIWTBvmrt3ByX+bw/NfbmbGlIF8fvOx9EiI5XdvrkBV3Q7PGBOGVJXt5XWeasFYgtnDV9vKuWTmAuKiI5j1s8ncfsZIMpPjuHnqMBZtKeM/S2yyAWNM8O2sbqChsdkSjFfV7W7il68sJj0xltevOpzxA9K+OXbeoVkcnJXCPe+uorq+0cUojTHhaPs3Y2C8cYsyWIL5jnvfXc36kmr+/MND6B4f851jERHCHWeMoqiinkc+Xe9ShMaYcLXdY4MswTr5vzFnTQlPz9vEpVOyOWJIeptlxg9I5exxfXns8w2cPKoX+WW1fLGhlC82lJKV2o0npk8IctTGmHCxvWWamO6WYDylrKaBX81aykGZifx66vC9lv311OG8v6KQMx6eC0BcdARZqfF8tKqYFQXljOqTEoyQjTFhpqCsjuhIIT3BG4MswRIMqsqtbyyntKqBJ6dPIC46cq/le6XE8eCF41i1vYLDBvfgkKzuVNc3MvEPH/F6Xr4lGGOMI7aX19IzOY4IjwyyBOuDYcOOaj5cUcT1Jw5ldN/OJYcTRvbk2uOHMCE7jZioCFITYjhhRE/+sySf3U3trUxgjDH7b3t5HX081MEPlmAYnJHIu788kp8dPfiA3ufcQ7MorW7g069tXRpjTOBtL6/1VP8LWIIBfEnmQOf2OXpYBumJMczK2xqgqIwxxqe5WSksr/PULcpgCSZgoiMj+MHYvsxeXczO6ga3wzHGhJDS6gZ2N6mnblEGSzABde74LHY3KW8uyXc7FGNMCNnusZUsW1iCCaARvZMZ3TeZWYu2HfB72ZxnxpgWBWXeWmishSWYADv30CyW51ewurBiv9+juLKOKffO5t53V1uiMcZQ6G/B9LIWTHg7a2xfoiOF1/P2vxXzyKfrKSiv49HP1vO7N1fQ3GxJxphwtr28jpioCHokxHRcuAuxBBNgaQkxHDc8k1l521hfUtVmmdxNO7nq+TwKymq/d2x7eS0vfLmFH47P4vIjBvLM/M3c+sZXlmSMCWMF/mn6RbwzyBIswTjihhOHERkhnPfIPBZt2fWdY/9Zks+PHv+Sd5cXcv0rS2jaI3H8/ZN1qCq/OH4It542gmuOPYiXFmzlpteW0miDOI0JS4XltfRK9tblMbAE44hhvZJ4/arDSekWzY8e/4KPVxWhqjzw0Vque3kJ4/p357enj+TLjTv555xvZ2betquGVxZu5fycfvRLi0dEuOnkYdx44lD+tTif37+10sVaGWPcUlBW57kOfrC5yBwzoEcCs646nBlPL+SKZ3OZODCNLzbs5NxDs7jnnDFERwqLtuzirx+s4YiD0jk4qzsPfbwOEeGa4w76zntde/wQymt388TcjUwe3INTx/R2qVbGmGBralaKKry1kmULa8E4KD0xlpeuOIwjh2TwxYad3HTSUP78w4OJiYpARPjDD8aQkRTLdS8vYWVBBbMWbeOiSf3bHK1789ThHNKvO7+etYytO2tcqI0xxg07quppbPbeIEuwBOO4hNgonpyew5xfHcs1xw35TiddSnw0fz1/LJtKqzn/n/OJjhSuOqbtOdFioiJ4eNo4ELjmpcU0NFp/jDHhwIsrWbawBBMEUZER9O8R3+axyYN78LOjB1NV38j0ydlkJrX/V0q/tHjuO/dglm4t488ffO1UuMaYLmR7mfcWGmthfTBdwPUnDGVoz0ROHtWrw7KnjunNxYf157E5G5g8qAfHDs8MQoTGGLcU+FswXpuqH6wF0yXEREVw9rgs4mM6l+9vO20kQzITueudlTY+xpgQV1heS2xUBN3jo90OZZ9ZgvGguOhIfnH8EDaUVPPByiK3wzHGOKig3HeLstcGWYIlGM86ZXQv+qfF88hn622+MmNC2PayWk/eQQaWYDwrKjKCK48axNKtZXyxYafb4RhjHFJYXue5SS5bWILxsPPGZ5GeGMMjn63vuLAxxnMaGpsprKgjy4Oj+MESjKfFRUdy6ZSBzFlTwoqCcrfDMcYE2NZdNTQrZKcnuB3KfrEE43EXHzaAxNgoHv1sg9uhGGMCbHNpNeCbesqLHE8wIhIpIotF5O02jvUXkU/8x5eJyKmtjt0iIutE5GsROdnpOL0qpVs0F03qzzvLCthS+u0UMs3NarcwG+Nxm3b4fqez2xmo3dUFY6DldcAqILmNY7cBr6rqIyIyEvgvkO3fvhAYBfQBPhKRoaraFIR4PWfGEQOZ+b9N/OSpL4mLjmRHVQO7ahoY3iuJV346mcRYG09rjBdtLq0mKTaKNI8tNNbC0RaMiGQBpwFPtFNE+TbxpAAF/u2zgJdVtV5VNwLrgIlOxuplPZPjuO6EIXSPjyErNZ4TR2ZyyeHZrNpewV02xb8xnrWptIYB6fGeHAMDzrdg7gduBpLaOf474AMRuRZIAE7w7+8LfNGq3Db/vu8QkSuBKwH69+8fmIg96upjD+LqY787zX9cdAR//2Q9xw7PZOrojqehMcZ0LZtLqxnVN8XtMPabYy0YETkdKFbVvL0UmwY8rapZwKnAcyISAbSVrr/XoaCqj6lqjqrmZGRkBCTuUHLd8UMZ0zeF3/xrGUUVdW6HY4zZB7ubmtm2q9az/S/g7CWyKcCZIrIJeBk4TkSe36PMZcCrAKo6H4gD0vG1WPq1KpfFt5fPTCfFREVw/4VjqdvdxE2vLbVOf2M8pKCslsZm9ewdZOBgglHVW1Q1S1Wz8XXYz1bVi/cotgU4HkBERuBLMCXAm8CFIhIrIgOBIcACp2INZYMzErnttJF8vnYHz8zf5HY4xphO2lTacgeZJZhOE5E7ReRM/9MbgStEZCnwEnCJ+qzA17JZCbwHXG13kO2/iyb15/jhmdzz39XMW7fD7XCMMZ3QMgYmOz3EL5GJyGARifVvHyMivxCR7p09iap+qqqn+7dvV9U3/dsrVXWKqh6iqmNV9YNWr7lbVQer6jBVfXffqmVaExH+cv4hDExP4Ipnc1m2rcztkIwxHdi0o4b4mEgyEmPdDmW/dbYF8zrQJCIHAU8CA4EXHYvKBFz3+BievWwiqQkxXDJzIeuKq9wOyRizF5tLqxnQI8GztyhD5xNMs6o2AmcD96vq9UBv58IyTuiZHMfzl00iQuAnT35JgX8pVmNM17OptNrTd5BB5xPMbhGZBkwHWqZ88d7yaobs9ASemTGRyrpGfvzkl3b7sjFdUFOzsnVnrafvIIPOJ5hLgcnA3aq60X9n1563HBuPGNUnhScvmUBheR3n/GMeG0rscpkxXUlBWS0NTc3h0YLxd8b/QlVfEpFUIElV73U4NuOgiQPTePnKydTtbuK8R+ezdKt1/BvTVWz236IcFi0YEflURJJFJA1YCswUkb86G5px2pisFGZddTjxMZFMe/wL5qwpcTskYwy+/hfw9i3K0PlLZCmqWgGcA8xU1fF8O2+Y8bCB6Qn866rD6Z8Wz2XPLOT9FYVuh2RM2NtcWk1sVAQ9k7y5VHKLziaYKBHpDZzPt538JkRkJsfxyk8nM6pPCte8uIgPVxa5HZIxYW1TaQ0DesQTEeHdW5Sh8wnmTuB9YL2qLhSRQcBa58IywZbSLZpnL5vIyN7J/PyFPD5eZUnGGLdsLq329BQxLTrbyf+aqh6sqlf5n29Q1XOdDc0EW3JcNM9eNokRvZO56vlFfLK62O2QjAk7zc3K5tIastPDJMGISJaI/FtEikWkSERe9y8mZkJMSrdonpsxiaG9Evnp83l8ta3c7ZCMCStFlXXUNzYzwOO3KEPnL5HNxDfDcR98C3+95d9nQlBKfDTPXzaJbtGR/OPTdW6HY0xY2bTD+7Mot+hsgslQ1Zmq2uh/PA3YCl8hrHt8DBdN6s/7KwrZurPG7XCMCRstsyiHUwtmh4hcLCKR/sfFQKmTgRn3TT88m8gIYeb/NrkdijFhY1NpDTGREfRO6eZ2KAesswlmBr5blAuB7cB5+KaPMSGsZ3Icpx/ch1cWbqGibrfb4RgTFjaXVtMvrRuRHr9FGTp/F9kWVT1TVTNUNVNVf4Bv0KUJcZcdMZDqhiZeWbDV7VCMCQsbd4TGLcpwYCta3hCwKEyXNbpvCpMGpvH0vE00NjW7HY4xIU3Vd4uy1+cga3EgCcb77TfTKZcfOYj8slres2lkjHFUYUUdtbubGOjxOchaHEiC0YBFYbq044dnkt0jnic+3+h2KMaEtDVFvqUzhvRMcjmSwNhrghGRShGpaONRiW9MjAkDERHCjCMGsmRrGXmbd7odjjEha21RJQBDwyHBqGqSqia38UhS1ahgBWncd974LLrHR/OPT9a7HYoxIWtNUSXpiTGkJcS4HUpAHMglMhNG4mOiuGzKQD5eXczyfJs+xhgnrCmqYkhmaLRewBKM2QfTp2STFBfFw7Nt+hhjAk1VWVdcxdCeiW6HEjCWYEynJcdFc+mUgby3opDVhRVuh2NMSCkor6OqvjFkOvjBEozZRzOmZJMQE2mtGGMCbE2IdfCDJRizj7rHx/CTw7N556vtrCuucjscY0LGt3eQ2SUyE8YuP2IgcVGR/P0Ta8Xsj8amZlRtGJn5rjVFVWQkxdI9PjTuIAOwW43NPuuRGMvFh/Xnybkbue74ISGx8p4TVJUdVQ1sLq1mVWElK/LLWV5QzprCKkb1Tebxn+SQnhjrdpimi1hbVBlSrRewBGP20xVHDeLZ+Zu58rlcHpp2KMN6hc51446oKiJtz5S0urCCx+dsZEVBOVt21lDT0PTNse7x0Yzpm8K0if14JXcr5z4yj2dnTAyZeafM/mtuVtYUVXHBhH5uhxJQlmDMfslMiuPJ6RP45StLOPPhufz29JFcNKl/u1+8XtbUrCzdVsanX5fw2ZoSVuSXk5OdypmH9OWU0b1ITYhhTVElD3y0lne+2k5ibBQTB6YxeXAPBqTFM6BHAkN7JdEnJe6b/5+zxvVlxtMLOfeReTx96URG901xuZbGTflltdTubgq5P9QkVK4F5+TkaG5urtthhJ2SynpufG0pc9aUcPKontx37sEhdQ159uoifvXaMkqrGxCBsf26M6ZvCnPX7mDDjmqiIoRRfZJZll9OfHQkl04ZyOVHDuzU/8G64iqmP7WAspoG7j33YE4c2ZO46Mgg1Mp0NR+vKuKyZ3J5/arJjB+QFtRzi0iequY48t5OJxgRiQRygXxVPX2PY38DjvU/jQcyVbW7/1gT8JX/2BZVPXNv57EE457mZuXJuRv54/ur6Z8Wz0tXHEZmcpzbYR2w+etLmT5zAQdlJPKzYwZz5EHppPqn8FBVVm6v4K2l25m7roSjhmRwxZGDvjneWUUVdUx/agGrCyuJjYpg4sA0jh6awZFDMhiSmUhECCw6ZTr2yKfrue+91Sy94yRSukUH9dxeTzA3ADlA8p4JZo9y1wLjVHWG/3mVqna6x8sSjPu+3FDKpU8vpFdyHC9ecRi9UrybZJZuLeNHj39Bn+7deOWnkx2dG6q+sYn560uZs2YHn60pZn2Jb032tIQYJmSnMmlgD3KyUxmUkUhi7PevapfVNLB1Zy2VdbupbmiipqGR3U3KiSN6khIf3C8rs39ueGUJ89aX8sX/Oz7o53YywTjaByMiWcBpwN10vEDZNOAOJ+Mxzpo0qAfPzpjIJTMXcsFj83npisPo091764qvKapk+swFpCXG8PzlkxyfeDA2KpJjhmVyzLBMYCT5ZbXMW7eDLzfu5MuNpby/ouibsumJsQxMjycjKZb8sjo27aimvLbt5ayzUrvxj4sO5eCs7o7Gbw7cmuJKhoTYHWTgcAtGRGYB9wBJwE3ttWBEZADwBZClqk3+fY3AEqARuFdV39jbuawF03Us2rKL6U8uoHtCNC9efhj90ryzeNKW0hrOe3QeALN+djj9e7gfe0FZLcu2lbFxRw2bdlSzsbSaHZX19Onejez0eLJ7JNAvLZ6UbtEkxESREBtJUUU9N722lJLKeu44cyQ/mhiaN2CEguZmZeQd73HRpAH89vSRQT+/J1swInI6UKyqeSJyTAfFLwRmtSQXv/6qWiAig4DZIvKVqn5nrngRuRK4EqB///4BjN4ciEP7p/L85ZP48ZNfcvSfPmFMVncOH9yDwwf3YEJ2WpftyC4oq2Xa41/Q0NTMqz+d3CWSC0Cf7t32uSU4KCORt689gl++soRb/72c3E27uPvs0cTH2I2jXc3WXTXU7W4OuTEw4OxI/inAmSKyCXgZOE5Enm+n7IXAS613qGqB/98NwKfAuD1fpKqPqWqOquZkZGQEMHRzoA7p1503rp7CNccNITpCeHzOBn785AKO/tMnXXK6/+LKOi564ksqanfz3IxJITEfVGpCDDMvmcD1JwzljSX5nP33eWzaUe12WGYPobaKZWuOJRhVvUVVs1Q1G18Cma2qF+9ZTkSGAanA/Fb7UkUk1r+dji9ZrXQqVuOMQRmJ3HDiUGZddThL7ziJJ6fnECnCBf+cz2drSg7ovZublbeXFXDWw3N5ePZampvbvtRb09BIfWNTm8da7Kxu4MdPLKCooo6nZ0xgTFbojEmJiBCuO2EIT186kaLKOs54eC4fryrq+IUmaFomuRySaS2YAyYid4pI61uOpwEv63c7g0YAuSKyFPgEXx+MJRgPS4iN4vgRPfn31VMY0COBGU8v5JWFW/b5fVSVOWtKOPPvc7nmxcXkl9Xx5w/WMH3mAkoq678p19DYzJNzN3LYHz7mlPs/Z0NJ2xNzltfu5idPfcnG0mqe+ElO0McgBMvRQzN465oj6J8Wz2XP5PLXD76mqZ2kbIJrTVElfVLiSIoLvTv+bKClCbqq+kZ+/sIi5qwp4ZxD+xIhwrZdNWzbVUtNQxMnjMjk3EOzmJCd9s04kMLyOj5eXcR/lhSwYONOslK7ccOJQzlrbF9m5W3l9v+sILlbNA9cOJbahibufmcVG3ZUM+WgHqzaXkljUzN/v+hQjhziu5Sqqry3vJC73l5JSVU9j/04h2OHZ7r53xIUdbubuO2N5czK28Ypo3vx4LRxREfanLduOuWBz8lMiuWZGRNdOb+nx8EEiyUYb9nd1Mzt/1nBq7lbyUiMpW9qN7JSfR3ZH64soqahiX5p3ThmaCZLtpbxlb/fpl9aNy6bMpBpk/oTG/XtzQKrtldw9QuL2ODvYxiUkcBvTxvJMcMy2LarliuezWVtcRW/PW0ERw7N4HdvruDztTsY3iuJu88eHbItl7aoKk98vpG7/7uKqaN68dCPLMm4palZGXH7e0yfPIBbTwv+HWRgCaZTLMF4U3Ozfm+0ek1DI++vKORfi/KZv76Ug7NSOGFkT04Y0ZMhmYnt3m5bXd/IXz9cQ7/Ublx02IDvfGlW1Tfyy5eX8NGqIiLEd8nuxhOHcvFhA4gK0y/XJ+du5K63V1qScdGGkiqO+8tn/PG8gzk/x52JLj15m7IxndHWVCjxMVGcPS6Ls8dl7XXm4j0lxEa1O44gMTaKx348noc/WUdxZR3XHT+UjKTwnir/siMGAnDX2yu59sXFlmRcsKLAt/T4yN7JLkfiDEswpksL5ODAiAjhF8cPCdj7hYLLjhiIAHe+vZIZTy/kL+cfQmaSd6f48ZrlBeVER0pI3BbfFvtzxZgwN+OIgdxzzhgWbNzJ1Ps/54MVhW6HFDZW5FcwrFcSMVGh+VUcmrUyxuyTaRP78/a1R9A7JY4rn8vj17OWUVXf6HZYIU1VWV5QzpgQXgvIEowxBvCNJP/3z6fw82MG81reVs5/dH6Hg1TN/ssvq6WsZjej+liCMcaEgZioCG6eOpx/XDSeldsreOjjdW6HFLKW5/s6+EN5NVNLMMaY75k6uhfnHprFI5+tZ9m2MrfDCUkrCsqJjBCGh9gyya1ZgjHGtOn2M0aSnhjDTa8ttUtlDlieX86QzMQuO7t4IFiCMca0KaVbNPeeezBriqp48OO1bocTcpYXVIR0/wtYgjHG7MWxwzL54fgsHv1sA0u32qWyQCmuqKOksp7RfUNzgGULSzDGmL267fSRZCTG8vMXFvFq7lbqdtvlsgPVMrdeKHfwgyUYY0wHUrpF8/eLDiUpLoqbZy3jiPtmc/9Ha76zPILZN8vzKxCBESE6RUwLmyrGGNOh8QNSefe6I5m3vpQn527k/o/W8vDsdUwe3IOpo3tx4sieNsXMPlheUM7A9AQSY0P7Kzi0a2eMCRgRYcpB6Uw5KJ31JVW8lruN91cUcuu/l3PbG8uZPKgHD00bR4/E8J5EtDNW5JeTkx36S0TYJTJjzD4bnJHIb04Zzuwbj+b9Xx7FdccPIXfTLm57YzmhsgSIU0qr6ikorwv5Dn6wFowx5gCICMN6JTGsVxKxUZHc995q3lxawFlj+7odWpfVMkX/6BC/RRmsBWOMCZArjxrEuP7duf0/KyiqqHM7nC5reYHvDrJQHwMDlmCMMQESGSH8+YeHULe7iVv+9ZVdKmvHivwK+qV1IyU+2u1QHGcJxhgTMIMzErl56nBmry7mtbxtbofTJS0vKA+Ly2NgCcYYE2CXHp7NpIFp3PXWSvLLat0Op0spr93N5tKakB9g2cISjDEmoCL8l8qaVbnp1aU0N9ulshbL81v6X0L/DjKwBGOMcUC/tHh+e/pI5m8oZea8TW6H02V8tKqImKgIJoTBGBiwBGOMccgFE/pxwohM7ntvNWuLKt0Ox3WqyvvLCzlqSAYJIT6Cv4UlGGOMI0SEe845mMTYKK5/dQkNjc1uh3w87LUAABBwSURBVOSqr/LLKSivY+roXm6HEjSWYIwxjslIiuUPZ49heX4FD80O7zVl3lteSGSEcMKITLdDCRpLMMYYR00d3Yvzxmfx90/WsWDjTrfDcYWq8t7yQiYP6kH3+Bi3wwkaSzDGGMfdccZIsnskcMWzuawJw/6YdcVVbNhRzclhdHkMLMEYY4IgKS6aZ2ZMJDYqgulPLaAgzMbHvLe8EBE4eWRPt0MJKkswxpig6JcWzzMzJlJV18hPnlpAWU2D2yEFzXsrCjm0fyqZyeG1Zo4lGGNM0Izonczj03PYsrOGGU8vpLYh9Jdf3rqzhhUFFUwdFV6XxyAICUZEIkVksYi83caxv4nIEv9jjYiUtTo2XUTW+h/TnY7TGBMchw3qwQMXjGXx1jJufG1JyE+K+f6KQgBOtgTjiOuAVW0dUNXrVXWsqo4FHgL+BSAiacAdwCRgInCHiKQGIVZjTBCcMqY3t5wynP9+Vcijn21wOxxHvbe8kJG9k+nfI97tUILO0QQjIlnAacATnSg+DXjJv30y8KGq7lTVXcCHwFRnojTGuOGKIwdx+sG9+eP7q/lsTYnb4TiiuKKOvC27wmpwZWtOt2DuB24G9jqEV0QGAAOB2f5dfYGtrYps8+/b83VXikiuiOSWlITmD6gxoUpE+ON5BzOsZxK/eGkxW0pr3A4p4N5dXohqeF4eAwcTjIicDhSral4nil8IzFLVlh4/aaPM9y7UqupjqpqjqjkZGRkHEK0xxg3xMVH888fjAbjyuVxqGhpdjiiwXsvbysjeyQzrleR2KK5wsgUzBThTRDYBLwPHicjz7ZS9kG8vj4GvxdKv1fMsoMCJII0x7hrQI4EHp41jTVElv3k9dFbCXFFQzvL8Cs7PyXI7FNc4lmBU9RZVzVLVbHwJZLaqXrxnOREZBqQC81vtfh84SURS/Z37J/n3GWNC0NFDM7jxpGG8ubSAZ0Jkev/XcrcRExnBWWO/d3U/bAR9HIyI3CkiZ7baNQ14WVv92aKqO4G7gIX+x53+fcaYEHXV0YM5YUQm//fOKvI273I7nANS39jEG0vyOXFUT1ITwmfusT1JqDRHc3JyNDc31+0wjDEHoLx2N2c8NJeGxmbe/sURpCfGuh3Sfnln2XaufnERz8yYyNFDu3b/sIjkqWqOE+9tI/mNMV1GSrdoHrn4UHbVNHDti4tpbPLmGjKv5m6ld0ocRxyU7nYorrIEY4zpUkb1SeHus8cwf0Mp1728hNxNO2lu9s6Vlu3ltcxZW8J547OIjGjrhtjwER7rdhpjPOW88VlsLq3msTkbeOer7fTt3o3TD+nN+Tn9GJyR6HZ4e/WvRfmo+uoQ7qwFY4zpkm48aRh5vz2Rv11wCEN6JvLE5xs59YHPeW95oduhtUtVeTV3K4cNSmNAjwS3w3GdJRhjTJeVGBvF2eOyePrSicz/zXGM6J3MVS/kMfN/G90OrU0LNu5kc2kNPxzfr+PCYcASjDHGEzKT43jpisM4cURPfv/WSu56e2WX6ptRVR6cvZbkuChOGROeU8PsyRKMMcYzusVE8sjF47nk8GyenLuRX7y8uMuM/H/nq+38b10pN508jPgY694GSzDGGI+JjBB+d+YobjppKG8v285redvcDomq+kbuenslo/okc9GkAW6H02VYgjHGeNLPjzmICdmp/OG/q9hRVe9qLA9+vJaiinru+sHosL81uTVLMMYYT4qIEO45ZwzV9Y3839srXYtjTVElT83dyIUT+nFof1sXsTVLMMYYzzooM4mrjjmIN5YUMMeFRctUld++sZzEuChunjo86Ofv6izBGGM87efHDGZQegK3vvEVtQ1NHb8ggN5Yks+XG3dy88nDSQvjSS3bYwnGGONpcdGR/OGcMWzdWcv9H68J2nm37arhjv+sYFz/7lwwwca9tMUSjDHG8w4b1IMLcvrxxOcb+WhlkePna2xq5rqXl9Cs8MAF46xjvx2WYIwxIeG200cwqk8yP39xEfPW73D0XA98vJa8zbu4++zR9O8R7+i5vMwSjDEmJCTFRfPMpRMZkBbPFc/ksniLM4uWzVu/g4c/WccPx2eF9WqVnWEJxhgTMlITYnj+8kn0SIzlkpkLWV1YEdD331ndwPWvLGFgegK/P2tUQN87FFmCMcaElJ7Jcbxw+STioiO4+IkFfF1YGZD3La6o48pnc9lVvZuHpo2z6WA6wRKMMSbk9EuL54XLJxEhcN6j85i/vvSA3m/e+h2c+uBclheU85fzD2FUn5QARRraLMEYY0LSQZlJ/PvqKfRMjmP6Uwt4a2nBPr9Hc7Py0MdrufiJL0npFsV/rj6CMw7p40C0ockSjDEmZPXt3o3Xf3Y4Y/t159qXFvPE5xs6/dq63U1c8Wwuf/lwDWcc0oc3rzmCYb2SHIw29FiCMcaEtJT4aJ69bCKnjenN/72ziktnLmBDSdVeX1PT0MiMpxcy++ti7jxrFPdfMJaEWOtz2VeWYIwxIS8uOpKHpo3jttNGkLtpFyffP4d73l1FVX3j98pW1TdyyVML+WJDKX89/xB+MjkbERtIuT+kqyzWc6BycnI0NzfX7TCMMV1ccWUdf3rva17L20ZGUiynjenNuP7dGdcvlZT4aC6ZuYBl28q5/4KxYdHfIiJ5qprjyHtbgjHGhKPFW3bx1w/XkLtpF7W7fZNkRkf6WioPTTuUqaPDY9ljJxOMXVQ0xoSlcf1Tee6ySTQ2NfN1USVLtpaxansFp4zuzZSD0t0OLyRYgjHGhLWoyAhG9UmxsS0OsE5+Y4wxjrAEY4wxxhGWYIwxxjjCEowxxhhHWIIxxhjjCEswxhhjHGEJxhhjjCMswRhjjHFEyEwVIyIlwOY9dqcA5fu4r6PtdGDHfobZ1rn3pUxn6hOsunQUa0dl9rUuez5v2W69zz6bzsXaURn7bNz9DthbOSfqkqCqGZ2Iad+pasg+gMf2dV9H20BuIOPZlzKdqU+w6nKg9dnXuuylDq332Wdjn02X/mw6U5dAfjZO/5x19Aj1S2Rv7ce+zmwHMp59KdOZ+gSrLp19n/bK7Gtd9nz+Vjtl9pd9Nnvfb59N8L4D9lauK9WlQyFziSxYRCRXHZp5NNhCqS4QWvUJpbpAaNXH6tJ5od6CccJjbgcQQKFUFwit+oRSXSC06mN16SRrwRhjjHGEtWCMMcY4whKMMcYYR4R1ghGRp0SkWESW78drx4vIVyKyTkQeFBFpdexaEflaRFaIyB8DG3W78QS8LiLyOxHJF5El/sepgY+83Zgc+Wz8x28SERWRoCxb6NBnc5eILPN/Lh+ISFAWj3eoLn8SkdX++vxbRLoHPvJ2Y3KiPj/0/+43i4jjNwMcSB3aeb/pIrLW/5jeav9ef6/a5OQ90F39ARwFHAos34/XLgAmAwK8C5zi338s8BEQ63+e6eG6/A64KVQ+G/+xfsD7+Ablpnu1LkByqzK/AB71cF1OAqL82/cB93n55wwYAQwDPgVyumod/PFl77EvDdjg/zfVv526t/ru7RHWLRhVnQPsbL1PRAaLyHsikicin4vI8D1fJyK98f2Cz1ff//yzwA/8h68C7lXVev85ip2thY9DdXGNg/X5G3AzELS7W5yoi6pWtCqaQJDq41BdPlDVRn/RL4AsZ2vxLYfqs0pVvw5G/P7z7Vcd2nEy8KGq7lTVXcCHwNT9/Z4I6wTTjseAa1V1PHAT8I82yvQFtrV6vs2/D2AocKSIfCkin4nIBEej3bsDrQvANf5LF0+JSKpzoXbKAdVHRM4E8lV1qdOBdsIBfzYicreIbAUuAm53MNaOBOLnrMUMfH8duymQ9XFLZ+rQlr7A1lbPW+q1X/WN6uRJw4KIJAKHA6+1urwY21bRNva1/AUZha9peRgwAXhVRAb5s37QBKgujwB3+Z/fBfwF3xdA0B1ofUQkHrgV3+UYVwXos0FVbwVuFZFbgGuAOwIcaocCVRf/e90KNAIvBDLGfRHI+rhlb3UQkUuB6/z7DgL+KyINwEZVPZv267Vf9bUE810RQJmqjm29U0QigTz/0zfxffG2bsZnAQX+7W3Av/wJZYGINOObUK7EycDbcMB1UdWiVq97HHjbyYA7cKD1GQwMBJb6f+mygEUiMlFVCx2OfU+B+Dlr7UXgHVxIMASoLv7O5NOB44P9x9geAv3ZuKHNOgCo6kxgJoCIfApcoqqbWhXZBhzT6nkWvr6abexPfZ3ugOrqDyCbVp1jwDzgh/5tAQ5p53UL8bVSWjq8TvXv/xlwp397KL7mpni0Lr1blbkeeNnLn80eZTYRpE5+hz6bIa3KXAvM8nBdpgIrgYxg/nw5/XNGkDr597cOtN/JvxHfVZhU/3ZaZ+rbZlxufKBd5QG8BGwHduPL0Jfh+yv3PWCp/4f+9nZemwMsB9YDD/PtrAgxwPP+Y4uA4zxcl+eAr4Bl+P5q6x2MujhVnz3KbCJ4d5E58dm87t+/DN/EhX09XJd1+P4QW+J/BOWOOAfrc7b/veqBIuD9rlgH2kgw/v0z/J/JOuDSjuq7t4dNFWOMMcYRdheZMcYYR1iCMcYY4whLMMYYYxxhCcYYY4wjLMEYY4xxhCUYE9JEpCrI53tCREYG6L2axDdb8nIReaujWYZFpLuI/DwQ5zYmEOw2ZRPSRKRKVRMD+H5R+u3EjI5qHbuIPAOsUdW791I+G3hbVUcHIz5jOmItGBN2RCRDRF4XkYX+xxT//okiMk9EFvv/Hebff4mIvCYibwEfiMgxIvKpiMwS3zomL7SsjeHfn+PfrvJPSLlURL4QkZ7+/YP9zxeKyJ2dbGXN59tJOxNF5GMRWSS+9TnO8pe5Fxjsb/X8yV/2V/7zLBOR3wfwv9GYDlmCMeHoAeBvqjoBOBd4wr9/NXCUqo7DNzvxH1q9ZjIwXVWP8z8fB/wSGAkMAqa0cZ4E4AtVPQSYA1zR6vwP+M/f4XxO/nmwjsc3mwJAHXC2qh6Kb/2hv/gT3G+A9ao6VlV/JSInAUOAicBYYLyIHNXR+YwJFJvs0oSjE4CRrWaaTRaRJCAFeEZEhuCbKTa61Ws+VNXWa24sUNVtACKyBN9cUHP3OE8D304Qmgec6N+ezLdrabwI/LmdOLu1eu88fGtzgG8uqD/4k0UzvpZNzzZef5L/sdj/PBFfwpnTzvmMCShLMCYcRQCTVbW29U4ReQj4RFXP9vdnfNrqcPUe71HfaruJtn+Xduu3nZztldmbWlUdKyIp+BLV1cCD+NZ/yQDGq+puEdkExLXxegHuUdV/7uN5jQkIu0RmwtEH+NZPAUBEWqY1TwHy/duXOHj+L/BdmgO4sKPCqlqOb1nkm0QkGl+cxf7kciwwwF+0Ekhq9dL3gRn+9UEQkb4ikhmgOhjTIUswJtTFi8i2Vo8b8H1Z5/g7vlfiW2IB4I/APSLyPyDSwZh+CdwgIguA3kB5Ry9Q1cX4Zsa9EN+CXDkikouvNbPaX6YU+J//tuY/qeoH+C7BzReRr4BZfDcBGeMou03ZmCDzr65Zq6oqIhcC01T1rI5eZ4zXWB+MMcE3HnjYf+dXGS4tQ22M06wFY4wxxhHWB2OMMcYRlmCMMcY4whKMMcYYR1iCMcYY4whLMMYYYxzx/wEHRe3pc0ZRsAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the mixed-precision option, if you have it, otherwise omit this step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList (90000 items)\n",
       "x: LMTextList\n",
       "  xxwrep 18 xxbos xxmaj once again xxmaj mr. xxmaj costner has dragged out a movie for far longer than necessary . xxmaj aside from the terrific sea rescue sequences , of which there are very few i just did not care about any of the characters . xxmaj most of us have ghosts in the closet , and xxmaj costner 's character are realized early on , and then forgotten until much later , by which time i did not care . xxmaj the character we should really care about is a very cocky , overconfident xxmaj ashton xxmaj kutcher . xxmaj the problem is he comes off as kid who thinks he 's better than anyone else around him and shows no signs of a cluttered closet . xxmaj his only obstacle appears to be winning over xxmaj costner . xxmaj finally when we are well past the half way point of this stinker , xxmaj costner tells us all about xxmaj kutcher 's ghosts . xxmaj we are told why xxmaj kutcher is driven to be the best with no prior inkling or foreshadowing . xxmaj no magic here , it was all i could do to keep from turning it off an hour in .,  xxwrep 18 xxbos xxmaj this is an example of why the majority of action films are the same . xxmaj generic and boring , there 's really nothing worth watching here . a complete waste of the then barely - tapped talents of xxmaj ice - t and xxmaj ice xxmaj cube , who 've each proven many times over that they are capable of acting , and acting well . xxmaj do n't bother with this one , go see xxmaj new xxmaj jack xxmaj city , xxmaj ricochet or watch xxmaj new xxmaj york xxmaj undercover for xxmaj ice - t , or xxmaj boyz n the xxmaj hood , xxmaj higher xxmaj learning or xxmaj friday for xxmaj ice xxmaj cube and see the real deal . xxmaj ice - t 's horribly cliched dialogue alone makes this film grate at the teeth , and i 'm still wondering what the heck xxmaj bill xxmaj paxton was doing in this film ? xxmaj and why the heck does he always play the exact same character ? xxmaj from xxmaj aliens onward , every film i 've seen with xxmaj bill xxmaj paxton has him playing the exact same irritating character , and at least in xxmaj aliens his character died , which made it somewhat gratifying ... \n",
       " \n",
       "  xxmaj overall , this is second - rate action trash . xxmaj there are countless better films to see , and if you really want to see this one , watch xxmaj judgement xxmaj night , which is practically a carbon copy but has better acting and a better script . xxmaj the only thing that made this at all worth watching was a decent hand on the camera - the cinematography was almost refreshing , which comes close to making up for the horrible film itself - but not quite . 4 / 10 .,  xxwrep 18 xxbos xxmaj first of all i hate those moronic rappers , who could'nt act if they had a gun pressed against their foreheads . xxmaj all they do is curse and shoot each other and acting like xxunk version of gangsters . \n",
       " \n",
       "  xxmaj the movie does n't take more than five minutes to explain what is going on before we 're already at the warehouse xxmaj there is not a single sympathetic character in this movie , except for the homeless guy , who is also the only one with half a brain . \n",
       " \n",
       "  xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are both hill billies and xxmaj xxunk character is just as much a villain as the gangsters . i did'nt like him right from the start . \n",
       " \n",
       "  xxmaj the movie is filled with pointless violence and xxmaj walter xxmaj hills specialty : people falling through windows with glass flying everywhere . xxmaj there is pretty much no plot and it is a big problem when you root for no - one . xxmaj everybody dies , except from xxmaj paxton and the homeless guy and everybody get what they deserve . \n",
       " \n",
       "  xxmaj the only two black people that can act is the homeless guy and the junkie but they 're actors by profession , not annoying ugly brain dead rappers . \n",
       " \n",
       "  xxmaj stay away from this crap and watch 48 hours 1 and 2 instead . xxmaj at lest they have characters you care about , a sense of humor and nothing but real actors in the cast .,  xxwrep 18 xxbos xxmaj not even the xxmaj beatles could write songs everyone liked , and although xxmaj walter xxmaj hill is no mop - top he 's second to none when it comes to thought provoking action movies . xxmaj the nineties came and social platforms were changing in music and film , the emergence of the xxmaj rapper turned movie star was in full swing , the acting took a back seat to each man 's overpowering regional accent and transparent acting . xxmaj this was one of the many ice - t movies i saw as a kid and loved , only to watch them later and cringe . xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are firemen with basic lives until a burning building tenant about to go up in flames hands over a map with gold implications . i hand it to xxmaj walter for quickly and neatly setting up the main characters and location . xxmaj but i fault everyone involved for turning out xxmaj lame - o performances . xxmaj ice - t and cube must have been red hot at this time , and while i 've enjoyed both their careers as rappers , in my opinion they fell flat in this movie . xxmaj it 's about ninety minutes of one guy ridiculously turning his back on the other guy to the point you find yourself locked in multiple states of disbelief . xxmaj now this is a movie , its not a documentary so i wo nt waste my time recounting all the stupid plot twists in this movie , but there were many , and they led nowhere . i got the feeling watching this that everyone on set was xxunk of confused and just playing things off the cuff . xxmaj there are two things i still enjoy about it , one involves a scene with a needle and the other is xxmaj sadler 's huge 45 pistol . xxmaj bottom line this movie is like domino 's pizza . xxmaj yeah ill eat it if i 'm hungry and i do n't feel like cooking , xxmaj but i 'm well aware it tastes like crap . 3 stars , meh .,  xxwrep 18 xxbos xxmaj brass pictures ( movies is not a fitting word for them ) really are somewhat brassy . xxmaj their alluring visual qualities are reminiscent of expensive high class xxup tv commercials . xxmaj but unfortunately xxmaj brass pictures are feature films with the pretense of wanting to entertain viewers for over two hours ! xxmaj in this they fail miserably , their undeniable , but rather soft and flabby than steamy , erotic qualities non withstanding . \n",
       " \n",
       "  xxmaj senso ' 45 is a remake of a film by xxmaj luchino xxmaj visconti with the same title and xxmaj alida xxmaj valli and xxmaj farley xxmaj granger in the lead . xxmaj the original tells a story of senseless love and lust in and around xxmaj venice during the xxmaj italian wars of independence . xxmaj brass moved the action from the 19th into the 20th century , 1945 to be exact , so there are xxmaj mussolini murals , men in black shirts , xxmaj german uniforms or the tattered garb of the partisans . xxmaj but it is just window dressing , the historic context is completely negligible . \n",
       " \n",
       "  xxmaj anna xxmaj xxunk plays the attractive aristocratic woman who falls for the amoral xxup ss guy who always puts on too much lipstick . xxmaj she is an attractive , versatile , well trained xxmaj italian actress and clearly above the material . xxmaj her wide range of facial expressions ( xxunk boredom , loathing , delight , fear , hate ... and ecstasy ) are the best reason to watch this picture and worth two stars . xxmaj she endures this basically trashy stuff with an astonishing amount of dignity . i wish some really good parts come along for her . xxmaj she really deserves it .\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Valid: LabelList (10000 items)\n",
       "x: LMTextList\n",
       "xxbos xxup cavite is an example of ultimate independent film , with a very short budget , a very simple concept , an exotic locale , a minimal cast , and a hand - held camera . \n",
       " \n",
       "  xxmaj the story is simple : xxmaj adam ( xxmaj ian xxmaj gamazon ) is called home to the xxmaj phillipines because of a family crisis . xxmaj instead of his family picking him up , he finds himself forced to follow instructions of a man claiming to have his family . xxmaj there 's no clear reason for the abduction , or what makes xxmaj adam a target ; all xxmaj adam really knows is that his every move is watched , and the kidnappers have no regard for their victims . \n",
       " \n",
       "  xxmaj as xxmaj adam follows the obscure instructions , and the obstacles in his way , the audience ca n't help but be caught up in his plight . xxmaj the hand held camera and jumpy editing style enhances the sense of desperation and time . xxmaj the scenes of urban xxmaj phillipines , particular the markets and the squatter holdings are a vivid cacophony . \n",
       " \n",
       "  xxmaj co - directors and xxmaj co - writers xxmaj neill xxmaj xxunk xxmaj xxunk and xxmaj ian xxmaj gamazon have done an outstanding job of making the most out of limited means . xxmaj the economy of the film makes it both intimate and discomfiting , as xxmaj adam is an everyman who only wants his family safe and instead is completely at the whim of an omniscient tormentor . \n",
       " \n",
       "  xxup cavite is an absolute must for anyone who has an interest in film , as storytelling , in it 's structure , and as an art form .,xxbos xxmaj decades ago , a crate filled with weapons grade plutonium crashes on an island and soaks into the ground . xxmaj today , a team of military men are sent to track down a notorious terrorist ( of ambiguous national origin ) and they track him to this polluted island . xxmaj when their raft is destroyed , the team must spend the night on shore , but soon discover that the plutonium has done something awful to the island -- it has called forth hundreds of bloodthirsty velociraptors . \n",
       " \n",
       "  xxmaj let me start this with a lesson : do n't lend a movie to your friend before you 've seen it , especially if you are supposed to be reviewing it for the internet 's finest horror movie site . xxmaj it took me almost a year to get this film back , and the person who borrowed it still had not watched it ( though we ended up seeing it together ) . xxmaj and a second , more important , lesson : when you do watch this , keep your expectations as low as humanly possible . xxmaj because this film ranks among the worst i 've ever seen . \n",
       " \n",
       "  xxmaj my acting in 8th grade was more convincing than the seasoned actors who appear in this film ( xxmaj lorenzo xxmaj lamas , xxmaj stephen xxmaj bauer ) . xxmaj line delivery is very fake , and the words themselves are poorly scripted . xxmaj the opening words come from a man checking out his gun 's scope : \" xxmaj boom . xxmaj dead bad guys . \" xxmaj yes , that 's pure genius at work . xxmaj the only conversation with any depth has two main characters explaining their histories . xxmaj but it , too , seems unnatural and a poor attempt to provide character background and to fill time . xxmaj we did n't need to know anything about their histories , so why bore us with it ? xxmaj and if you think the conversations are bad , you ai n't seen nothing yet . \n",
       " \n",
       "  xxmaj the lighting is atrocious . i generally do n't notice lighting , but my friend ( a former film school student ) was practically vomiting in rage at the way more often than not shadows fell on the actors ' faces and the light would be in the background , focused on nothing in particular . xxmaj most lighting looks like a spotlight in a dim room , and many of the scenes involve a deep , subterranean cavern -- which you 'd then expect to be poorly lit , but had lights coming from all sorts of random angles . xxmaj do n't ask me why . \n",
       " \n",
       "  xxmaj the plot was pretty bad . xxmaj some films can take the idea of military men chasing a terrorist and make a convincing film out of it . xxmaj cat and mouse stories are riveting . xxmaj well , not here . xxmaj the terrorist is really not even part of the story , just an excuse to go to the island . xxmaj and the raptors ? xxmaj and the allosaurus ? xxmaj sure , they came from the plutonium that soaked into the ground . xxmaj but if that makes sense to you , please explain it to me because i have no clue how radiation brings dinosaurs back from millions of years of extinction . \n",
       " \n",
       "  xxmaj by far the worst part of \" xxmaj raptor xxmaj island \" is the animation of the raptors . xxmaj that 's right -- the selling point of the film is the worst aspect . xxmaj the animation is n't just bad , it 's subpar . i ca n't even express the hilarity of cartoons this cheesy . xxmaj and when they get shot ? xxmaj red splats like one would see in an old video game . xxmaj even the airplane , helicopter and xxmaj navy ship are cartoons ... how hard is it to get a model plane ? xxmaj please do n't see \" xxmaj raptor xxmaj island \" unless you need a good laugh or want to get sickeningly drunk . xxmaj sure , you probably want to see it before you see \" xxmaj raptor xxmaj island 2 \" ( which seems to be getting better reviews ) . xxmaj but just avoiding it entirely is your best bet . xxmaj the closest thing i can compare it to is \" xxmaj pinata : xxmaj survival xxmaj island \" , and unfortunately this film makes the pinata look good by comparison . xxmaj you have been warned .,xxbos xxmaj this is one creepy underrated xxmaj gem with chilling performances and a fantastic xxunk xxmaj all the characters are great , and the story was awesome , plus i thought the ending was really xxunk xxmaj the plot was great , and it never bored me , plus while the child actors were bad , they gave me the xxunk xxmaj this happened to be on the space channel a while ago , so i decided to check it out and tape it , i read some good reviews from fellow horror fans , i must say i agree with them , it 's very creepy , and suspenseful , plus xxmaj strother xxmaj martin , was fantastic in his role , as the xxmaj satan worshiper . xxmaj it has tons of creepy atmosphere , and it keeps you guessing throughout , plus all the characters were very likable , and you really start to root for xxmaj ben and his xxunk xxmaj it has plenty of disturbing moments , and the film really shocked me at times , plus , it 's extremely well made on a low xxunk xxmaj this is one creepy underrated gem , with chilling performances and a fantastic finale ! , i highly recommend this one!. xxmaj the xxmaj direction is very good!. xxmaj bernard mceveety does a very good job here , with great camera work , creating a lot of creepy atmosphere , and keeping the film at a very fast pace!. xxmaj ther is a little bit of blood and gore . xxmaj we get a severed leg , lots of bloody corpses , bloody slit throat , slicing and dicing , decapitation , and an impaling . xxmaj the xxmaj acting is xxunk xxmaj strother xxmaj martin is fantastic here ! as the xxmaj satan worshiper , he is extremely creepy , very convincing , was quite chilling , was extremely intense , seemed to be enjoying himself , and just did a fantastic job xxunk xxmaj charles xxmaj bateman is great as the xxmaj dad , he was very caring , very likable , and gave a good show ! , i liked him lots . xxup l.q. xxmaj jones is awesome as the xxmaj sheriff , he was funny , on top of things , looked very young , had a cool character , and just did an awesome job xxunk xxmaj ahna xxmaj capri is good as the girlfriend and did what she had to do pretty well . xxmaj charles xxmaj robinson overacted to the extreme as the xxmaj priest and did n't convince me one bit ! , and that laugh of his was especially bad . xxmaj geri xxmaj reischl is actually decent as the daughter , she was somewhat likable , and only got on my nerves a couple times , i rather liked her . xxmaj alvy xxmaj moore was goofy , but very likable in his role as xxmaj tobey i dug him!. xxmaj rest of the cast do good . xxmaj overall i highly recommend it!. * * * 1 / 2 out of 5,xxbos xxmaj aside from the gross factual inaccuracies of this movie ( ie king xxmaj richard having a son and that son replacing xxmaj john on the throne ) this movie was a sweet love tale full of adventure . i 'm sure that this movie will appeal to the younger generation of girls and boys .,xxbos xxmaj the special effects again are superb but this is xxmaj finding xxmaj nemo in reverse i.e the parents get taken away and xxmaj pi ( xxmaj nemo ) is left behind , if you have n't seen xxmaj finding xxmaj nemo you will like it . \n",
       " \n",
       "  xxmaj it becomes very clear at the beginning of the film what the plot is and takes a long time to reach the end ! \n",
       " \n",
       "  xxmaj there 's is nothing new or nail biting in this one and there is no humour at all , which is very disappointing . \n",
       " \n",
       "  xxmaj all in all very disappointing as a follow up movie to xxmaj shark xxmaj tale which there really are no similarities with it should have been called xxmaj finding xxmaj nemo 2 xxrep 56 !\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(60000, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(60000, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1152, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1152, 1152, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1152, 400, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x00000236A8585048>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0, MixedPrecision\n",
       "learn: ...\n",
       "loss_scale: 65536\n",
       "max_noskip: 1000\n",
       "dynamic: True\n",
       "clip: None\n",
       "flat_master: False\n",
       "max_scale: 16777216\n",
       "loss_fp32: True], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(60000, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(60000, 400, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The first step in fine-tuning is to train only the last layer of the model. \n",
    "This takes about a half-hour on an NVIDIA RTX-2070 GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.109810</td>\n",
       "      <td>3.432369</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>27:48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(1, lr*10, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is relatively slow to train, we will save our weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save('fit_1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList (90000 items)\n",
       "x: LMTextList\n",
       "  xxwrep 18 xxbos xxmaj once again xxmaj mr. xxmaj costner has dragged out a movie for far longer than necessary . xxmaj aside from the terrific sea rescue sequences , of which there are very few i just did not care about any of the characters . xxmaj most of us have ghosts in the closet , and xxmaj costner 's character are realized early on , and then forgotten until much later , by which time i did not care . xxmaj the character we should really care about is a very cocky , overconfident xxmaj ashton xxmaj kutcher . xxmaj the problem is he comes off as kid who thinks he 's better than anyone else around him and shows no signs of a cluttered closet . xxmaj his only obstacle appears to be winning over xxmaj costner . xxmaj finally when we are well past the half way point of this stinker , xxmaj costner tells us all about xxmaj kutcher 's ghosts . xxmaj we are told why xxmaj kutcher is driven to be the best with no prior inkling or foreshadowing . xxmaj no magic here , it was all i could do to keep from turning it off an hour in .,  xxwrep 18 xxbos xxmaj this is an example of why the majority of action films are the same . xxmaj generic and boring , there 's really nothing worth watching here . a complete waste of the then barely - tapped talents of xxmaj ice - t and xxmaj ice xxmaj cube , who 've each proven many times over that they are capable of acting , and acting well . xxmaj do n't bother with this one , go see xxmaj new xxmaj jack xxmaj city , xxmaj ricochet or watch xxmaj new xxmaj york xxmaj undercover for xxmaj ice - t , or xxmaj boyz n the xxmaj hood , xxmaj higher xxmaj learning or xxmaj friday for xxmaj ice xxmaj cube and see the real deal . xxmaj ice - t 's horribly cliched dialogue alone makes this film grate at the teeth , and i 'm still wondering what the heck xxmaj bill xxmaj paxton was doing in this film ? xxmaj and why the heck does he always play the exact same character ? xxmaj from xxmaj aliens onward , every film i 've seen with xxmaj bill xxmaj paxton has him playing the exact same irritating character , and at least in xxmaj aliens his character died , which made it somewhat gratifying ... \n",
       " \n",
       "  xxmaj overall , this is second - rate action trash . xxmaj there are countless better films to see , and if you really want to see this one , watch xxmaj judgement xxmaj night , which is practically a carbon copy but has better acting and a better script . xxmaj the only thing that made this at all worth watching was a decent hand on the camera - the cinematography was almost refreshing , which comes close to making up for the horrible film itself - but not quite . 4 / 10 .,  xxwrep 18 xxbos xxmaj first of all i hate those moronic rappers , who could'nt act if they had a gun pressed against their foreheads . xxmaj all they do is curse and shoot each other and acting like xxunk version of gangsters . \n",
       " \n",
       "  xxmaj the movie does n't take more than five minutes to explain what is going on before we 're already at the warehouse xxmaj there is not a single sympathetic character in this movie , except for the homeless guy , who is also the only one with half a brain . \n",
       " \n",
       "  xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are both hill billies and xxmaj xxunk character is just as much a villain as the gangsters . i did'nt like him right from the start . \n",
       " \n",
       "  xxmaj the movie is filled with pointless violence and xxmaj walter xxmaj hills specialty : people falling through windows with glass flying everywhere . xxmaj there is pretty much no plot and it is a big problem when you root for no - one . xxmaj everybody dies , except from xxmaj paxton and the homeless guy and everybody get what they deserve . \n",
       " \n",
       "  xxmaj the only two black people that can act is the homeless guy and the junkie but they 're actors by profession , not annoying ugly brain dead rappers . \n",
       " \n",
       "  xxmaj stay away from this crap and watch 48 hours 1 and 2 instead . xxmaj at lest they have characters you care about , a sense of humor and nothing but real actors in the cast .,  xxwrep 18 xxbos xxmaj not even the xxmaj beatles could write songs everyone liked , and although xxmaj walter xxmaj hill is no mop - top he 's second to none when it comes to thought provoking action movies . xxmaj the nineties came and social platforms were changing in music and film , the emergence of the xxmaj rapper turned movie star was in full swing , the acting took a back seat to each man 's overpowering regional accent and transparent acting . xxmaj this was one of the many ice - t movies i saw as a kid and loved , only to watch them later and cringe . xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are firemen with basic lives until a burning building tenant about to go up in flames hands over a map with gold implications . i hand it to xxmaj walter for quickly and neatly setting up the main characters and location . xxmaj but i fault everyone involved for turning out xxmaj lame - o performances . xxmaj ice - t and cube must have been red hot at this time , and while i 've enjoyed both their careers as rappers , in my opinion they fell flat in this movie . xxmaj it 's about ninety minutes of one guy ridiculously turning his back on the other guy to the point you find yourself locked in multiple states of disbelief . xxmaj now this is a movie , its not a documentary so i wo nt waste my time recounting all the stupid plot twists in this movie , but there were many , and they led nowhere . i got the feeling watching this that everyone on set was xxunk of confused and just playing things off the cuff . xxmaj there are two things i still enjoy about it , one involves a scene with a needle and the other is xxmaj sadler 's huge 45 pistol . xxmaj bottom line this movie is like domino 's pizza . xxmaj yeah ill eat it if i 'm hungry and i do n't feel like cooking , xxmaj but i 'm well aware it tastes like crap . 3 stars , meh .,  xxwrep 18 xxbos xxmaj brass pictures ( movies is not a fitting word for them ) really are somewhat brassy . xxmaj their alluring visual qualities are reminiscent of expensive high class xxup tv commercials . xxmaj but unfortunately xxmaj brass pictures are feature films with the pretense of wanting to entertain viewers for over two hours ! xxmaj in this they fail miserably , their undeniable , but rather soft and flabby than steamy , erotic qualities non withstanding . \n",
       " \n",
       "  xxmaj senso ' 45 is a remake of a film by xxmaj luchino xxmaj visconti with the same title and xxmaj alida xxmaj valli and xxmaj farley xxmaj granger in the lead . xxmaj the original tells a story of senseless love and lust in and around xxmaj venice during the xxmaj italian wars of independence . xxmaj brass moved the action from the 19th into the 20th century , 1945 to be exact , so there are xxmaj mussolini murals , men in black shirts , xxmaj german uniforms or the tattered garb of the partisans . xxmaj but it is just window dressing , the historic context is completely negligible . \n",
       " \n",
       "  xxmaj anna xxmaj xxunk plays the attractive aristocratic woman who falls for the amoral xxup ss guy who always puts on too much lipstick . xxmaj she is an attractive , versatile , well trained xxmaj italian actress and clearly above the material . xxmaj her wide range of facial expressions ( xxunk boredom , loathing , delight , fear , hate ... and ecstasy ) are the best reason to watch this picture and worth two stars . xxmaj she endures this basically trashy stuff with an astonishing amount of dignity . i wish some really good parts come along for her . xxmaj she really deserves it .\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Valid: LabelList (10000 items)\n",
       "x: LMTextList\n",
       "xxbos xxup cavite is an example of ultimate independent film , with a very short budget , a very simple concept , an exotic locale , a minimal cast , and a hand - held camera . \n",
       " \n",
       "  xxmaj the story is simple : xxmaj adam ( xxmaj ian xxmaj gamazon ) is called home to the xxmaj phillipines because of a family crisis . xxmaj instead of his family picking him up , he finds himself forced to follow instructions of a man claiming to have his family . xxmaj there 's no clear reason for the abduction , or what makes xxmaj adam a target ; all xxmaj adam really knows is that his every move is watched , and the kidnappers have no regard for their victims . \n",
       " \n",
       "  xxmaj as xxmaj adam follows the obscure instructions , and the obstacles in his way , the audience ca n't help but be caught up in his plight . xxmaj the hand held camera and jumpy editing style enhances the sense of desperation and time . xxmaj the scenes of urban xxmaj phillipines , particular the markets and the squatter holdings are a vivid cacophony . \n",
       " \n",
       "  xxmaj co - directors and xxmaj co - writers xxmaj neill xxmaj xxunk xxmaj xxunk and xxmaj ian xxmaj gamazon have done an outstanding job of making the most out of limited means . xxmaj the economy of the film makes it both intimate and discomfiting , as xxmaj adam is an everyman who only wants his family safe and instead is completely at the whim of an omniscient tormentor . \n",
       " \n",
       "  xxup cavite is an absolute must for anyone who has an interest in film , as storytelling , in it 's structure , and as an art form .,xxbos xxmaj decades ago , a crate filled with weapons grade plutonium crashes on an island and soaks into the ground . xxmaj today , a team of military men are sent to track down a notorious terrorist ( of ambiguous national origin ) and they track him to this polluted island . xxmaj when their raft is destroyed , the team must spend the night on shore , but soon discover that the plutonium has done something awful to the island -- it has called forth hundreds of bloodthirsty velociraptors . \n",
       " \n",
       "  xxmaj let me start this with a lesson : do n't lend a movie to your friend before you 've seen it , especially if you are supposed to be reviewing it for the internet 's finest horror movie site . xxmaj it took me almost a year to get this film back , and the person who borrowed it still had not watched it ( though we ended up seeing it together ) . xxmaj and a second , more important , lesson : when you do watch this , keep your expectations as low as humanly possible . xxmaj because this film ranks among the worst i 've ever seen . \n",
       " \n",
       "  xxmaj my acting in 8th grade was more convincing than the seasoned actors who appear in this film ( xxmaj lorenzo xxmaj lamas , xxmaj stephen xxmaj bauer ) . xxmaj line delivery is very fake , and the words themselves are poorly scripted . xxmaj the opening words come from a man checking out his gun 's scope : \" xxmaj boom . xxmaj dead bad guys . \" xxmaj yes , that 's pure genius at work . xxmaj the only conversation with any depth has two main characters explaining their histories . xxmaj but it , too , seems unnatural and a poor attempt to provide character background and to fill time . xxmaj we did n't need to know anything about their histories , so why bore us with it ? xxmaj and if you think the conversations are bad , you ai n't seen nothing yet . \n",
       " \n",
       "  xxmaj the lighting is atrocious . i generally do n't notice lighting , but my friend ( a former film school student ) was practically vomiting in rage at the way more often than not shadows fell on the actors ' faces and the light would be in the background , focused on nothing in particular . xxmaj most lighting looks like a spotlight in a dim room , and many of the scenes involve a deep , subterranean cavern -- which you 'd then expect to be poorly lit , but had lights coming from all sorts of random angles . xxmaj do n't ask me why . \n",
       " \n",
       "  xxmaj the plot was pretty bad . xxmaj some films can take the idea of military men chasing a terrorist and make a convincing film out of it . xxmaj cat and mouse stories are riveting . xxmaj well , not here . xxmaj the terrorist is really not even part of the story , just an excuse to go to the island . xxmaj and the raptors ? xxmaj and the allosaurus ? xxmaj sure , they came from the plutonium that soaked into the ground . xxmaj but if that makes sense to you , please explain it to me because i have no clue how radiation brings dinosaurs back from millions of years of extinction . \n",
       " \n",
       "  xxmaj by far the worst part of \" xxmaj raptor xxmaj island \" is the animation of the raptors . xxmaj that 's right -- the selling point of the film is the worst aspect . xxmaj the animation is n't just bad , it 's subpar . i ca n't even express the hilarity of cartoons this cheesy . xxmaj and when they get shot ? xxmaj red splats like one would see in an old video game . xxmaj even the airplane , helicopter and xxmaj navy ship are cartoons ... how hard is it to get a model plane ? xxmaj please do n't see \" xxmaj raptor xxmaj island \" unless you need a good laugh or want to get sickeningly drunk . xxmaj sure , you probably want to see it before you see \" xxmaj raptor xxmaj island 2 \" ( which seems to be getting better reviews ) . xxmaj but just avoiding it entirely is your best bet . xxmaj the closest thing i can compare it to is \" xxmaj pinata : xxmaj survival xxmaj island \" , and unfortunately this film makes the pinata look good by comparison . xxmaj you have been warned .,xxbos xxmaj this is one creepy underrated xxmaj gem with chilling performances and a fantastic xxunk xxmaj all the characters are great , and the story was awesome , plus i thought the ending was really xxunk xxmaj the plot was great , and it never bored me , plus while the child actors were bad , they gave me the xxunk xxmaj this happened to be on the space channel a while ago , so i decided to check it out and tape it , i read some good reviews from fellow horror fans , i must say i agree with them , it 's very creepy , and suspenseful , plus xxmaj strother xxmaj martin , was fantastic in his role , as the xxmaj satan worshiper . xxmaj it has tons of creepy atmosphere , and it keeps you guessing throughout , plus all the characters were very likable , and you really start to root for xxmaj ben and his xxunk xxmaj it has plenty of disturbing moments , and the film really shocked me at times , plus , it 's extremely well made on a low xxunk xxmaj this is one creepy underrated gem , with chilling performances and a fantastic finale ! , i highly recommend this one!. xxmaj the xxmaj direction is very good!. xxmaj bernard mceveety does a very good job here , with great camera work , creating a lot of creepy atmosphere , and keeping the film at a very fast pace!. xxmaj ther is a little bit of blood and gore . xxmaj we get a severed leg , lots of bloody corpses , bloody slit throat , slicing and dicing , decapitation , and an impaling . xxmaj the xxmaj acting is xxunk xxmaj strother xxmaj martin is fantastic here ! as the xxmaj satan worshiper , he is extremely creepy , very convincing , was quite chilling , was extremely intense , seemed to be enjoying himself , and just did a fantastic job xxunk xxmaj charles xxmaj bateman is great as the xxmaj dad , he was very caring , very likable , and gave a good show ! , i liked him lots . xxup l.q. xxmaj jones is awesome as the xxmaj sheriff , he was funny , on top of things , looked very young , had a cool character , and just did an awesome job xxunk xxmaj ahna xxmaj capri is good as the girlfriend and did what she had to do pretty well . xxmaj charles xxmaj robinson overacted to the extreme as the xxmaj priest and did n't convince me one bit ! , and that laugh of his was especially bad . xxmaj geri xxmaj reischl is actually decent as the daughter , she was somewhat likable , and only got on my nerves a couple times , i rather liked her . xxmaj alvy xxmaj moore was goofy , but very likable in his role as xxmaj tobey i dug him!. xxmaj rest of the cast do good . xxmaj overall i highly recommend it!. * * * 1 / 2 out of 5,xxbos xxmaj aside from the gross factual inaccuracies of this movie ( ie king xxmaj richard having a son and that son replacing xxmaj john on the throne ) this movie was a sweet love tale full of adventure . i 'm sure that this movie will appeal to the younger generation of girls and boys .,xxbos xxmaj the special effects again are superb but this is xxmaj finding xxmaj nemo in reverse i.e the parents get taken away and xxmaj pi ( xxmaj nemo ) is left behind , if you have n't seen xxmaj finding xxmaj nemo you will like it . \n",
       " \n",
       "  xxmaj it becomes very clear at the beginning of the film what the plot is and takes a long time to reach the end ! \n",
       " \n",
       "  xxmaj there 's is nothing new or nail biting in this one and there is no humour at all , which is very disappointing . \n",
       " \n",
       "  xxmaj all in all very disappointing as a follow up movie to xxmaj shark xxmaj tale which there really are no similarities with it should have been called xxmaj finding xxmaj nemo 2 xxrep 56 !\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(60000, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(60000, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1152, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1152, 1152, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1152, 400, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x00000236A8585048>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0, MixedPrecision\n",
       "learn: ...\n",
       "loss_scale: 2097152.0\n",
       "max_noskip: 1000\n",
       "dynamic: True\n",
       "clip: None\n",
       "flat_master: False\n",
       "max_scale: 16777216\n",
       "loss_fp32: True], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(60000, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(60000, 400, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.load('fit_1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To complete the fine-tuning, we unfreeze all the weights and retrain\n",
    "Adopting the `wikitext-103` weights as initial values, our neural network will adjust them via optimization, finding new values that are specialized to the \"language\" of `IMDb` movie reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.unfreeze()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fine tuning the model takes ~30 minutes per epoch on an NVIDIA RTX-2070 GPU, with bs=48<br>\n",
    "Note the relatively low value of accuracy, which did not improve significantly beyond `epoch 4`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.824557</td>\n",
       "      <td>3.145828</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>30:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.785742</td>\n",
       "      <td>3.277813</td>\n",
       "      <td>0.442857</td>\n",
       "      <td>30:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.753288</td>\n",
       "      <td>3.111610</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>30:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.676771</td>\n",
       "      <td>3.163547</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>30:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.605899</td>\n",
       "      <td>3.077933</td>\n",
       "      <td>0.457143</td>\n",
       "      <td>30:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.567794</td>\n",
       "      <td>3.045043</td>\n",
       "      <td>0.457143</td>\n",
       "      <td>30:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.490480</td>\n",
       "      <td>3.072821</td>\n",
       "      <td>0.414286</td>\n",
       "      <td>30:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.434875</td>\n",
       "      <td>3.052906</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>30:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.370924</td>\n",
       "      <td>2.990984</td>\n",
       "      <td>0.442857</td>\n",
       "      <td>30:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.336241</td>\n",
       "      <td>2.994531</td>\n",
       "      <td>0.457143</td>\n",
       "      <td>30:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(10, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the fine-tuned **language model** and the **encoder**\n",
    "We have to save not just the `fine-tuned` **IMDb language model** but also its **encoder**. The **language model** is the part that tries to guess the next word. The **encoder** is the part that's responsible for creating and updating the hidden state. \n",
    "\n",
    "In the next part we will build a **sentiment classifier** for the IMDb movie reviews. To do this we will need the **encoder** from the **IMDb language model** that we built."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Load the saved **model** and its **encoder**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList (90000 items)\n",
       "x: LMTextList\n",
       "  xxwrep 18 xxbos xxmaj once again xxmaj mr. xxmaj costner has dragged out a movie for far longer than necessary . xxmaj aside from the terrific sea rescue sequences , of which there are very few i just did not care about any of the characters . xxmaj most of us have ghosts in the closet , and xxmaj costner 's character are realized early on , and then forgotten until much later , by which time i did not care . xxmaj the character we should really care about is a very cocky , overconfident xxmaj ashton xxmaj kutcher . xxmaj the problem is he comes off as kid who thinks he 's better than anyone else around him and shows no signs of a cluttered closet . xxmaj his only obstacle appears to be winning over xxmaj costner . xxmaj finally when we are well past the half way point of this stinker , xxmaj costner tells us all about xxmaj kutcher 's ghosts . xxmaj we are told why xxmaj kutcher is driven to be the best with no prior inkling or foreshadowing . xxmaj no magic here , it was all i could do to keep from turning it off an hour in .,  xxwrep 18 xxbos xxmaj this is an example of why the majority of action films are the same . xxmaj generic and boring , there 's really nothing worth watching here . a complete waste of the then barely - tapped talents of xxmaj ice - t and xxmaj ice xxmaj cube , who 've each proven many times over that they are capable of acting , and acting well . xxmaj do n't bother with this one , go see xxmaj new xxmaj jack xxmaj city , xxmaj ricochet or watch xxmaj new xxmaj york xxmaj undercover for xxmaj ice - t , or xxmaj boyz n the xxmaj hood , xxmaj higher xxmaj learning or xxmaj friday for xxmaj ice xxmaj cube and see the real deal . xxmaj ice - t 's horribly cliched dialogue alone makes this film grate at the teeth , and i 'm still wondering what the heck xxmaj bill xxmaj paxton was doing in this film ? xxmaj and why the heck does he always play the exact same character ? xxmaj from xxmaj aliens onward , every film i 've seen with xxmaj bill xxmaj paxton has him playing the exact same irritating character , and at least in xxmaj aliens his character died , which made it somewhat gratifying ... \n",
       " \n",
       "  xxmaj overall , this is second - rate action trash . xxmaj there are countless better films to see , and if you really want to see this one , watch xxmaj judgement xxmaj night , which is practically a carbon copy but has better acting and a better script . xxmaj the only thing that made this at all worth watching was a decent hand on the camera - the cinematography was almost refreshing , which comes close to making up for the horrible film itself - but not quite . 4 / 10 .,  xxwrep 18 xxbos xxmaj first of all i hate those moronic rappers , who could'nt act if they had a gun pressed against their foreheads . xxmaj all they do is curse and shoot each other and acting like xxunk version of gangsters . \n",
       " \n",
       "  xxmaj the movie does n't take more than five minutes to explain what is going on before we 're already at the warehouse xxmaj there is not a single sympathetic character in this movie , except for the homeless guy , who is also the only one with half a brain . \n",
       " \n",
       "  xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are both hill billies and xxmaj xxunk character is just as much a villain as the gangsters . i did'nt like him right from the start . \n",
       " \n",
       "  xxmaj the movie is filled with pointless violence and xxmaj walter xxmaj hills specialty : people falling through windows with glass flying everywhere . xxmaj there is pretty much no plot and it is a big problem when you root for no - one . xxmaj everybody dies , except from xxmaj paxton and the homeless guy and everybody get what they deserve . \n",
       " \n",
       "  xxmaj the only two black people that can act is the homeless guy and the junkie but they 're actors by profession , not annoying ugly brain dead rappers . \n",
       " \n",
       "  xxmaj stay away from this crap and watch 48 hours 1 and 2 instead . xxmaj at lest they have characters you care about , a sense of humor and nothing but real actors in the cast .,  xxwrep 18 xxbos xxmaj not even the xxmaj beatles could write songs everyone liked , and although xxmaj walter xxmaj hill is no mop - top he 's second to none when it comes to thought provoking action movies . xxmaj the nineties came and social platforms were changing in music and film , the emergence of the xxmaj rapper turned movie star was in full swing , the acting took a back seat to each man 's overpowering regional accent and transparent acting . xxmaj this was one of the many ice - t movies i saw as a kid and loved , only to watch them later and cringe . xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are firemen with basic lives until a burning building tenant about to go up in flames hands over a map with gold implications . i hand it to xxmaj walter for quickly and neatly setting up the main characters and location . xxmaj but i fault everyone involved for turning out xxmaj lame - o performances . xxmaj ice - t and cube must have been red hot at this time , and while i 've enjoyed both their careers as rappers , in my opinion they fell flat in this movie . xxmaj it 's about ninety minutes of one guy ridiculously turning his back on the other guy to the point you find yourself locked in multiple states of disbelief . xxmaj now this is a movie , its not a documentary so i wo nt waste my time recounting all the stupid plot twists in this movie , but there were many , and they led nowhere . i got the feeling watching this that everyone on set was xxunk of confused and just playing things off the cuff . xxmaj there are two things i still enjoy about it , one involves a scene with a needle and the other is xxmaj sadler 's huge 45 pistol . xxmaj bottom line this movie is like domino 's pizza . xxmaj yeah ill eat it if i 'm hungry and i do n't feel like cooking , xxmaj but i 'm well aware it tastes like crap . 3 stars , meh .,  xxwrep 18 xxbos xxmaj brass pictures ( movies is not a fitting word for them ) really are somewhat brassy . xxmaj their alluring visual qualities are reminiscent of expensive high class xxup tv commercials . xxmaj but unfortunately xxmaj brass pictures are feature films with the pretense of wanting to entertain viewers for over two hours ! xxmaj in this they fail miserably , their undeniable , but rather soft and flabby than steamy , erotic qualities non withstanding . \n",
       " \n",
       "  xxmaj senso ' 45 is a remake of a film by xxmaj luchino xxmaj visconti with the same title and xxmaj alida xxmaj valli and xxmaj farley xxmaj granger in the lead . xxmaj the original tells a story of senseless love and lust in and around xxmaj venice during the xxmaj italian wars of independence . xxmaj brass moved the action from the 19th into the 20th century , 1945 to be exact , so there are xxmaj mussolini murals , men in black shirts , xxmaj german uniforms or the tattered garb of the partisans . xxmaj but it is just window dressing , the historic context is completely negligible . \n",
       " \n",
       "  xxmaj anna xxmaj xxunk plays the attractive aristocratic woman who falls for the amoral xxup ss guy who always puts on too much lipstick . xxmaj she is an attractive , versatile , well trained xxmaj italian actress and clearly above the material . xxmaj her wide range of facial expressions ( xxunk boredom , loathing , delight , fear , hate ... and ecstasy ) are the best reason to watch this picture and worth two stars . xxmaj she endures this basically trashy stuff with an astonishing amount of dignity . i wish some really good parts come along for her . xxmaj she really deserves it .\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Valid: LabelList (10000 items)\n",
       "x: LMTextList\n",
       "xxbos xxup cavite is an example of ultimate independent film , with a very short budget , a very simple concept , an exotic locale , a minimal cast , and a hand - held camera . \n",
       " \n",
       "  xxmaj the story is simple : xxmaj adam ( xxmaj ian xxmaj gamazon ) is called home to the xxmaj phillipines because of a family crisis . xxmaj instead of his family picking him up , he finds himself forced to follow instructions of a man claiming to have his family . xxmaj there 's no clear reason for the abduction , or what makes xxmaj adam a target ; all xxmaj adam really knows is that his every move is watched , and the kidnappers have no regard for their victims . \n",
       " \n",
       "  xxmaj as xxmaj adam follows the obscure instructions , and the obstacles in his way , the audience ca n't help but be caught up in his plight . xxmaj the hand held camera and jumpy editing style enhances the sense of desperation and time . xxmaj the scenes of urban xxmaj phillipines , particular the markets and the squatter holdings are a vivid cacophony . \n",
       " \n",
       "  xxmaj co - directors and xxmaj co - writers xxmaj neill xxmaj xxunk xxmaj xxunk and xxmaj ian xxmaj gamazon have done an outstanding job of making the most out of limited means . xxmaj the economy of the film makes it both intimate and discomfiting , as xxmaj adam is an everyman who only wants his family safe and instead is completely at the whim of an omniscient tormentor . \n",
       " \n",
       "  xxup cavite is an absolute must for anyone who has an interest in film , as storytelling , in it 's structure , and as an art form .,xxbos xxmaj decades ago , a crate filled with weapons grade plutonium crashes on an island and soaks into the ground . xxmaj today , a team of military men are sent to track down a notorious terrorist ( of ambiguous national origin ) and they track him to this polluted island . xxmaj when their raft is destroyed , the team must spend the night on shore , but soon discover that the plutonium has done something awful to the island -- it has called forth hundreds of bloodthirsty velociraptors . \n",
       " \n",
       "  xxmaj let me start this with a lesson : do n't lend a movie to your friend before you 've seen it , especially if you are supposed to be reviewing it for the internet 's finest horror movie site . xxmaj it took me almost a year to get this film back , and the person who borrowed it still had not watched it ( though we ended up seeing it together ) . xxmaj and a second , more important , lesson : when you do watch this , keep your expectations as low as humanly possible . xxmaj because this film ranks among the worst i 've ever seen . \n",
       " \n",
       "  xxmaj my acting in 8th grade was more convincing than the seasoned actors who appear in this film ( xxmaj lorenzo xxmaj lamas , xxmaj stephen xxmaj bauer ) . xxmaj line delivery is very fake , and the words themselves are poorly scripted . xxmaj the opening words come from a man checking out his gun 's scope : \" xxmaj boom . xxmaj dead bad guys . \" xxmaj yes , that 's pure genius at work . xxmaj the only conversation with any depth has two main characters explaining their histories . xxmaj but it , too , seems unnatural and a poor attempt to provide character background and to fill time . xxmaj we did n't need to know anything about their histories , so why bore us with it ? xxmaj and if you think the conversations are bad , you ai n't seen nothing yet . \n",
       " \n",
       "  xxmaj the lighting is atrocious . i generally do n't notice lighting , but my friend ( a former film school student ) was practically vomiting in rage at the way more often than not shadows fell on the actors ' faces and the light would be in the background , focused on nothing in particular . xxmaj most lighting looks like a spotlight in a dim room , and many of the scenes involve a deep , subterranean cavern -- which you 'd then expect to be poorly lit , but had lights coming from all sorts of random angles . xxmaj do n't ask me why . \n",
       " \n",
       "  xxmaj the plot was pretty bad . xxmaj some films can take the idea of military men chasing a terrorist and make a convincing film out of it . xxmaj cat and mouse stories are riveting . xxmaj well , not here . xxmaj the terrorist is really not even part of the story , just an excuse to go to the island . xxmaj and the raptors ? xxmaj and the allosaurus ? xxmaj sure , they came from the plutonium that soaked into the ground . xxmaj but if that makes sense to you , please explain it to me because i have no clue how radiation brings dinosaurs back from millions of years of extinction . \n",
       " \n",
       "  xxmaj by far the worst part of \" xxmaj raptor xxmaj island \" is the animation of the raptors . xxmaj that 's right -- the selling point of the film is the worst aspect . xxmaj the animation is n't just bad , it 's subpar . i ca n't even express the hilarity of cartoons this cheesy . xxmaj and when they get shot ? xxmaj red splats like one would see in an old video game . xxmaj even the airplane , helicopter and xxmaj navy ship are cartoons ... how hard is it to get a model plane ? xxmaj please do n't see \" xxmaj raptor xxmaj island \" unless you need a good laugh or want to get sickeningly drunk . xxmaj sure , you probably want to see it before you see \" xxmaj raptor xxmaj island 2 \" ( which seems to be getting better reviews ) . xxmaj but just avoiding it entirely is your best bet . xxmaj the closest thing i can compare it to is \" xxmaj pinata : xxmaj survival xxmaj island \" , and unfortunately this film makes the pinata look good by comparison . xxmaj you have been warned .,xxbos xxmaj this is one creepy underrated xxmaj gem with chilling performances and a fantastic xxunk xxmaj all the characters are great , and the story was awesome , plus i thought the ending was really xxunk xxmaj the plot was great , and it never bored me , plus while the child actors were bad , they gave me the xxunk xxmaj this happened to be on the space channel a while ago , so i decided to check it out and tape it , i read some good reviews from fellow horror fans , i must say i agree with them , it 's very creepy , and suspenseful , plus xxmaj strother xxmaj martin , was fantastic in his role , as the xxmaj satan worshiper . xxmaj it has tons of creepy atmosphere , and it keeps you guessing throughout , plus all the characters were very likable , and you really start to root for xxmaj ben and his xxunk xxmaj it has plenty of disturbing moments , and the film really shocked me at times , plus , it 's extremely well made on a low xxunk xxmaj this is one creepy underrated gem , with chilling performances and a fantastic finale ! , i highly recommend this one!. xxmaj the xxmaj direction is very good!. xxmaj bernard mceveety does a very good job here , with great camera work , creating a lot of creepy atmosphere , and keeping the film at a very fast pace!. xxmaj ther is a little bit of blood and gore . xxmaj we get a severed leg , lots of bloody corpses , bloody slit throat , slicing and dicing , decapitation , and an impaling . xxmaj the xxmaj acting is xxunk xxmaj strother xxmaj martin is fantastic here ! as the xxmaj satan worshiper , he is extremely creepy , very convincing , was quite chilling , was extremely intense , seemed to be enjoying himself , and just did a fantastic job xxunk xxmaj charles xxmaj bateman is great as the xxmaj dad , he was very caring , very likable , and gave a good show ! , i liked him lots . xxup l.q. xxmaj jones is awesome as the xxmaj sheriff , he was funny , on top of things , looked very young , had a cool character , and just did an awesome job xxunk xxmaj ahna xxmaj capri is good as the girlfriend and did what she had to do pretty well . xxmaj charles xxmaj robinson overacted to the extreme as the xxmaj priest and did n't convince me one bit ! , and that laugh of his was especially bad . xxmaj geri xxmaj reischl is actually decent as the daughter , she was somewhat likable , and only got on my nerves a couple times , i rather liked her . xxmaj alvy xxmaj moore was goofy , but very likable in his role as xxmaj tobey i dug him!. xxmaj rest of the cast do good . xxmaj overall i highly recommend it!. * * * 1 / 2 out of 5,xxbos xxmaj aside from the gross factual inaccuracies of this movie ( ie king xxmaj richard having a son and that son replacing xxmaj john on the throne ) this movie was a sweet love tale full of adventure . i 'm sure that this movie will appeal to the younger generation of girls and boys .,xxbos xxmaj the special effects again are superb but this is xxmaj finding xxmaj nemo in reverse i.e the parents get taken away and xxmaj pi ( xxmaj nemo ) is left behind , if you have n't seen xxmaj finding xxmaj nemo you will like it . \n",
       " \n",
       "  xxmaj it becomes very clear at the beginning of the film what the plot is and takes a long time to reach the end ! \n",
       " \n",
       "  xxmaj there 's is nothing new or nail biting in this one and there is no humour at all , which is very disappointing . \n",
       " \n",
       "  xxmaj all in all very disappointing as a follow up movie to xxmaj shark xxmaj tale which there really are no similarities with it should have been called xxmaj finding xxmaj nemo 2 xxrep 56 !\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(60000, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(60000, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1152, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1152, 1152, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1152, 400, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x00000236A8585048>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0, MixedPrecision\n",
       "learn: ...\n",
       "loss_scale: 2097152.0\n",
       "max_noskip: 1000\n",
       "dynamic: True\n",
       "clip: None\n",
       "flat_master: False\n",
       "max_scale: 16777216\n",
       "loss_fp32: True], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(60000, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(60000, 400, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm.load('fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Now that we've trained our model, different representations have been learned for the words that were in `IMDb` but not `wikitext-103` (remember that at the beginning we had initialized them all to the same thing):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "enc = learn_lm.model[0].encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[vocab.stoi[\"linklater\"], :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[vocab.stoi[\"30-something\"], :], new_word_vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Generate movie review-like text, with the **fine-tuned** ` IMDb` language model\n",
    "Compare these texts to the ones generated with the **untrained** `IMDb model` in part **2A**. Do they seem qualitatively better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "How good is our fine-tuned IMDb language model? Well let's try to see what it predicts when given a phrase that might appear in a movie review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"i liked this movie because\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i liked this movie because it gave a new perspective on the subject . The story of how to kill a dog was actually a good concept that i found to be very funny . The whole movie is shot with a sense\n",
      "i liked this movie because it was fun to see Fred Macmurray - a good actor - and this movie had a good cast like Gene Hackman , Gene Hackman , Catherine o'hara , William Hurt\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"This movie was\"\n",
    "N_WORDS = 30\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This movie was just awful . i mean , they had a lot of money , i mean , they wanted to make money with a $ 35 million budget . And\n",
      "This movie was very pleasant to watch , but it was just too dull . It took us a few minutes to catch the Mexican guy , but he seemed to\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"I hated this movie\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hated this movie , if only it ever made it to DVD . If you are planning to see this : 1 ) i would recommend that you buy this movie instead of the movie . 2 . It is\n",
      "I hated this movie . It was a little too distant from the book , but the acting was good and the at least some scenes were funny . The story line was pretty good , but it was n't the best\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Risks of language models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We will talk about ethical concerns raised by very accurate language models in lesson 7, but here are a few brief notes:\n",
    "\n",
    "In reference to [OpenAI's GPT-2](https://www.theverge.com/2019/2/14/18224704/ai-machine-learning-language-models-read-write-openai-gpt2): Jeremy Howard said, *I’ve been trying to warn people about this for a while. We have the technology to totally fill Twitter, email, and the web up with reasonable-sounding, context-appropriate prose, which would drown out all other speech and be impossible to filter.*\n",
    "\n",
    "For a small example, consider when completely incorrect (but reasonable sounding) ML generated answers were [posted to StackOverflow](https://meta.stackoverflow.com/questions/384596/completely-incorrect-machine-learning-generated-answers?stw=2):\n",
    "\n",
    "<img src=\"images/robot-overflow.png\" alt=\"Roboflow\" style=\"width: 80%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src=\"images/husain-tweet.png\" alt=\"Roboflow\" style=\"width: 60%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Building an `IMDb Sentiment Classifier`\n",
    "#### We'll now use **transfer learning** to create a `classifier`, again starting from the pretrained weights of the `wikitext-103` language model.  We'll also need the `IMDb language model` **encoder** that we saved previously. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3A. Load and preprocess the data, and form a `databunch`\n",
    "Using fastai's flexible API, we will now create a different kind of `databunch` object, one that is suitable for a **classifier** rather than a for **language model** (as we did in **2A**). This time we'll keep the labels for the `IMDb` movie reviews data. \n",
    "\n",
    "Add the `try-except` wrapper workaround for the bug in the `fastai Text API`\n",
    "\n",
    "Here the batch size is decreased from 48 to 8, to avoid a `CUDA out of memory error`; your hardware may be able to handle a larger batch, in which case training will likely be faster.\n",
    "\n",
    "Again, this takes a bit of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 5\n",
      "\n",
      "Wall time: 9min 15s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool' Error sometimes. Keep trying `till it works!\n",
    "#    the progress bar has to complete three consecutive steps. Why three? \n",
    "#    fails nearly 100 times, and doesn't respond to interrupt\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # Preprocessing steps\n",
    "        data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
    "             #grab all the text files in path\n",
    "             .split_by_folder(valid='test')\n",
    "             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
    "             .label_from_folder(classes=['neg', 'pos']))\n",
    "             #label them all with their folders\n",
    "             #.databunch(bs=bs, num_workers=1))\n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')   \n",
    "        \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Form the preprocessed data into a `databunch`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = data_clas.databunch(bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the databunch (since it took so long to make) and load it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas.save('imdb_textlist_class')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, 'imdb_textlist_class', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj some have praised _ xxunk _ as a xxmaj disney adventure for adults . i do n't think so -- at least not for thinking adults . \\n \\n  xxmaj this script suggests a beginning as a live - action movie , that struck someone as the type of crap you can not sell to adults anymore . xxmaj the \" crack staff \" of many older</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj this movie was recently released on xxup dvd in the xxup us and i finally got the chance to see this hard - to - find gem . xxmaj it even came with original theatrical previews of other xxmaj italian horror classics like \" xxup xxunk \" and \" xxup beyond xxup the xxup darkness \" . xxmaj unfortunately , the previews were the best thing about this</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj there are many adaptations of xxmaj charlotte xxmaj brontë 's classic novel \" xxmaj jane xxmaj eyre \" , and taking into consideration the numerous reviews written about them there is also a lively discussion on which of them is the best . xxmaj the short film adaptations all suffer from the fact that it is simply not possible to cram the whole plot of the novel into</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj title : xxmaj zombie 3 ( 1988 ) \\n \\n  xxmaj directors : xxmaj mostly xxmaj lucio xxmaj fulci , but also xxmaj claudio xxmaj fragasso and xxmaj bruno xxmaj mattei \\n \\n  xxmaj cast : xxmaj xxunk xxunk , xxmaj massimo xxmaj xxunk , xxmaj beatrice xxmaj ring , xxmaj deran xxmaj xxunk \\n \\n  xxmaj review : \\n \\n  xxmaj to review</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3B. Create a model to **classify** the `IMDb` reviews, and load the **encoder** we saved before.\n",
    "#### Freeze the weights for all but the last layer and find a good value for the learning rate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.3).to_fp16()\n",
    "learn_c.load_encoder('fine_tuned_enc')\n",
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hUVfrA8e+bDoGEQEILkAQIJXQIXRAUBcuC6Kpgg7Xg2suqi6urrq7lp67Y0F1FxQoiisLKiqA0AYHQm4SYUEINLQmQST2/P2YCQ5gUIHfuJHk/zzNPZu49d+57kkzenHvOPUeMMSillFIl+dkdgFJKKd+kCUIppZRHmiCUUkp5pAlCKaWUR5oglFJKeRRgdwCVJTIy0sTGxtodhlJKVSmrVq06aIyJ8rSv2iSI2NhYkpKS7A5DKaWqFBHZUdo+vcSklFLKI00QSimlPNIEoZRSyiNNEEoppTzSBKGUUsojTRBKKaU80gShlFLKI0sThIgME5GtIpIiIuM97G8hIvNFZI2IrBeRy932Pe46bquIDLUyzsqUciCbn7bstzsMpZQ6b5YlCBHxByYClwEJwGgRSShR7ElgmjGmGzAKeMd1bILrdQdgGPCO6/182pa9WVzz7jLu+CSJnYdO2B2OUkqdFytbEL2AFGNMqjEmD5gKjChRxgBhrufhwB7X8xHAVGNMrjEmDUhxvZ/PSs04xs0fLCck0A9/P2HSL6l2h6SUUufFygQRDexye53u2ubuGeAmEUkHZgP3ncWxiMg4EUkSkaSMjIzKivus7Tp8ghsnLccY+Pz2PlzVNZppSbs4fDzPtpiUUup8WZkgxMO2kuubjgYmG2OaAZcDn4qIXwWPxRjznjEm0RiTGBXlca4pyx3IdnDTB8s5nlvAp7f1pnXDOowb2BJHfhGfLit1ihOllPJ5ViaIdKC52+tmnLqEVOw2YBqAMWYZEAJEVvBYn/D+olT2HM3h41t7kdDUebUsvlFdLmrXkI+XbScnr7DUY0/kFfDF8p2s23XUS9EqpVTFWZkgVgLxIhInIkE4O51nliizE7gYQETa40wQGa5yo0QkWETigHhghYWxnrMFWzPoHdeAbi0iTtt+58CWHD6ex/TV6Wcck3kin7d+2kb/l37mbzM2cPfnq3Hkl55IlFLKDpYlCGNMAXAvMAfYgnO00iYReVZEhruK/QW4Q0TWAVOAscZpE86WxWbgB+AeY4zP/QXdfTSHbQeOcWGbMy9v9YqrT5fm9Zi0OJXCIufVsfQjJ3hx9hb6/9/P/GtuMt1bRPDUlQnsPprDh0vSvB2+UkqVydL1IIwxs3F2Prtve8rt+WagfynHPg88b2V852tRsrNjfFDbMxOEiHDnwJbc/flq3vp5G1v3ZTNn0z5EhMs7NeGuC1udvCS19PdDvDP/d67t0ZyousFerYNSSpVG76Q+Dwu3ZtA0PITWDet43D+0Q2NiGtTm9XnbWPr7IcYNbMXixwbz1uhuJ5MDwOOXt8ORX8iEecneCl0ppcpVbVaU87b8wiKWpBzkyi5NEPE06Ar8/YS3Rncjef8xrujUhFpBnu/1axVVh5v6xPDJsu2M6RtL28Z1LYxcKaUqRlsQ52j1jiNk5xZ47H9w17lZPf7Yo1mpyaHYAxfHUyc4gOdnb6nMMJVS6pxpgjhHC5MzCPAT+rWOrJT3iwgN4v6L41mUnMHPv+lcTkop+2mCOEcLkzPoHhNBWEhgpb3nLX1jiW9Yh4enrSPt4PFKe1+llDoXmiDOwYFsB5v2ZJV7eelsBQX4MWlMIn4i3Dp5JUcsnqrj4LFc9mU6LD2HUqrq0gRxDhYlHwSo9AQBENMglPdu7sHuIznc+dkq8gqKKv0cKQeyeWz6Ovq9+DNXvvULJ/IKKv0cSqmqTxPEOViYnEFknWASmoSVX/gcJMbW55VrO7Mi7TDjv1mPMWdMQ3VOtu7L5rbJKxny2iJmrtvD0I6NOXgsl8lLt1fK+yulqhcd5nqWCosMi7dlcFG7hvj5eR7eWhlGdI1m+8ETTJiXzP4sB/cMak3fVg1KHVJbnh827uXhaesIDvDjwSHx3NI3lvqhQRxz5POfhanc1CemUvtTlFJVn7YgztK69KMcPZHPoLYNLT/X/Re35skr2pO8/xg3TFrOVROX8MPGvWfVoigqMrw+L5k/f7aato3rMufBgTw4pA31Q4MAePiStmTm5PPBYp3qQyl1Ok0QZ8EYw2s/JhMa5M/A+MoZ3loWEeH2AS1Z/NhgXhjZicycfP782WomVfCPebYjn3u+WM3r87ZxTfdmTLmjDw3DQk4r06lZOMM6NOaDX9Is7xRXSlUtmiDOwvRV6fyScpDxl7WjXu0gr503JNCfG3q34Ke/DGJI+0a8+uPWMofB5hUUMXlJGoNeWcCcTft48or2vHptZ0ICPd+s99AlbTieV8B/FukqeEqpUzRBVNCBbAfP/XczPWMjuLF3jC0x+PsJz4/sSFCAH+O/Xk9R0emXmoqKDN+t3c2Q1xbyzKzNxDeqw4y7+3P7gJZl9l20bVyX4V2aMnlpGgeySx/2evREnk5LrlQNop3UwJ6jOdQNCaBuGZ20T3+3CUdBES9d09nSzunyNAoL4ckr2vPXrzfwxYqd3NTHmayOnsjjL9PW8dNvB2jfJIzJf+rJhW2iKtyp/cDF8fx3/V4em76eewa3pkeLiJP13Lwni/cW/c6s9Xudd4+3asDgdg0Z1KYhLRrUtqyuSil71fgEsePQcQa9uoB/DO/ALX1jPZb5YeNe/rdxH48ObUurKM8zt3rTdYnNmbVuLy/97zcuateQg8dyueuz1RzIdvDUlQmM7Rd71kmsZVQdHhoSz1s/p7BgawZNwkO4rGMTth3IZvG2g4QG+TOmbywGw4KtGTz13SZgE2P7xfLUlQlnnG9hcgYvzt7CvRe15srOTSux9kopb5HKGmNvt8TERJOUlHROx17x5mIAvr9/wBn7MnPyGfLaQqLqBPPdvf0J9PeNq3K7Dp/g0gmLaF6/FtsPniCqbjBv39DtjJXtztax3AJ+2rKfWev2sDA5g/BaQdx6QSw39o4hvNapFlbaweNMXpLGx8t2MKJrU169tsvJ783Xq9L569fr8fcTcguKuHNgSx4d2pYAH/neKaVOEZFVxphET/tqfAsC4PqezXnqu01s3J1Jx+jw0/ZNXrKdjOxcPhiT6DPJAaB5/do8Nqwt/5i1mUFto5hwXVciQs+/47xOcAAjukYzoms0OXmFBPiLx3rHRYbyjxEdaRQewss/bCXbUcDEG7rz4ZI0Xpmzlf6tG/DW6O68Nncr/1mUysY9mbw1uvvJ4bVKKd+nLQica0T3fGEeo3o259kRHU9ud+QX0u+ln+nSLJyP/tSrskKtNMYYNu/Non3jMFv7RT5fvoMnv91Io7oh7MtyMKJrU175YxeCApyJZVrSLp78diNRdYKZfldfmoTXsi1WpdTpympB+M6/xDYKrx3IZR0bM2PN7tNG6Xy1Kp3Dx/O488JWNkZXOhGhQ9NwW5MDwI29Y3hrdDeO5uQxbmBLJlzX9WRyAGefybQ7+3LkRB73T1lDQWH580ut3XWUKSt2klugo6aUsosmCJfrE5uT7Sjgh437AOeUGpMWp9KlWTi94+rbHJ3vu7JzUzY8M5S/Xd7eY8Lq2rweL17diZXbj/Dqj2Uvrbo/y8GfPlrB499s4KJXFzJjTfoZQ3qVUtbTBOHSp2UDWtSvzZcrdwHw46Z97Dh0gjsvbHXO8x/VNOX10YzoGs3oXi3498LfS10UqbDI8ODUtTjyi3j5ms5EhAby0JfruPzNxaxIO2xF2EqpUmiCcPHzE65LbMay1EPsOHScfy9KJaZBbYZ2aGx3aNXK039IoH2TMB6eto7dR3PO2P/ughSWpR7iH8M7cF3P5sy85wLeHN2NY7kF3DZ5pa5foZQXaYJw88cezfETePybDazbdZTbL4jD3+br+9VNSKA/79zYnYJCw58/XcWy3w+dvHy0asdhJszbxh+6NOXaxGaAM3EP79KUz2/vTX5REU/M2FBp058rpcqmCcJN4/AQLmwTxdLfD1E/NIg/9mhud0jVUlxkKP+6rgtpB48z+v1fGfjKfP7141bun7KWpvVCeH5kxzMu68U0COXRoe346bcDfLd2j02RK1WzaIIo4fqezqRwS98YagV5ntxOnb+hHRqz8okhvDGqK3GRobw9P4X9WQ7eHNWt1HUpxvaLpXuLejwzaxMZ2blejlipmkfvgyihqMjw3brdDOvQRBOEF+3NzCEzJ592jctepS/lwDEuf3MxQ9o35J0be3gpOqWqL9vupBaRYcAbgD8wyRjzUon9E4DBrpe1gYbGmHqufYXABte+ncaY4VbGWszPTxjZrZk3TqXcNAmvVaEb6Fo3rMODQ+J5+YetPP/9ZmIjQ6kbEkhYSAB9WjYodUpzpaqrzJx8wkICLBltaVmCEBF/YCJwCZAOrBSRmcaYzcVljDEPuZW/D+jm9hY5xpiuVsWnqq5xA1qyOPkg75dYOKlXXH2m3NFHBxaoGmX81+vZn+Xgm7v7V/p7W9mC6AWkGGNSAURkKjAC2FxK+dHA0xbGo6qJAH8/vrijNyfyCsl2FHAsN5+FyQd57r+beWd+CvddHG93iEp5xeHjeczbsr/UmajPl5UJIhrY5fY6HejtqaCIxABxwM9um0NEJAkoAF4yxnxrVaCq6hERQoMDCA0OAEJoFVWHdbuO8vpP2+jXOpIeMec3q61SVcG3a3aTX2hODguvbFaOYvLUzi+tR3wUMN0Y4z7xTgtXx8kNwOsicsaESCIyTkSSRCQpIyPj/CNWVZaI8M+RHWkSHsIDU9eQ5ci3OySlLPfVqnQ6RYeXO7jjXFmZINIB9xsJmgGlDWAfBUxx32CM2eP6mgos4PT+ieIy7xljEo0xiVFRUZURs6rCwkICeWNUN/ZmOnhyxka9oU5Vaxt3Z7Jlb5ZlrQewNkGsBOJFJE5EgnAmgZklC4lIWyACWOa2LUJEgl3PI4H+lN53odRJPWIiePDieGau28P0Vel2h6OUZaavSicowI/hXaxbsdGyBGGMKQDuBeYAW4BpxphNIvKsiLgPWR0NTDWn/7vXHkgSkXXAfJx9EJogVIXcPbg1fVrW5+/fbWTrvmyPZQp1dlhVheUWFPLt2t1cmtCIerWtW4TL0jupjTGzjTFtjDGtjDHPu7Y9ZYyZ6VbmGWPM+BLHLTXGdDLGdHF9/cDKOFX14u8nvDmqG3WCA7n781Uczy04bf8ny7bT6Zk5zNm0z54AlTpPP205wNET+VybaO10QDrVhqqWGoaF8OaorqQdPM6T3zr7I4qKDP/872ae+m4ThUWGx7/ZwMFjOmWHqnq+StpFk/AQLmgdael5NEGoaqtf60geuLgNM9bs5uOl27n789VM+iWNsf1i+e7e/hzLLeBv3+jssKpq2Z/lYGFyBld3j7b8plBLp9pQym73XtSapB2HeWbWZkTgqSsTuPWCOAAeubQNL8z+jRlrdnN1d51eRVUN36zeTZHBK7NNawtCVWv+fsKE67sypH0j/nNTj5PJAeC2C1rSMzaCp2duYo+HxYuU8kWb92YR06A2cZGhlp9LE4Sq9iLrBDNpTCKXllgd0N9PePXaLhQWGf769Xpd91pVCVk5+dSr5XlK/MqmCULVaDENQnniivYs3naQ2z9JIjNH78BWvi3LkU+YJgilvOOGXi14bkQHFiVncNXEJaQc8HzvhFK+ICsnv9RFtSqbJghV44kIN/eN5Ys7+pDtyGfE20v0Hgnls7IcBYTV8s74Ik0QSrn0iqvPrPsuoHXDOvz5s1X8/Nt+u0NS6gzaglDKJk3CazF1XF/aNw7jgalrSTt43O6QlDrJkV9IbkGR9kEoZZdaQf785+YeBPgJ4z5J4liJqTpKMsbo3E7KK7Idzt/FsBC9xKSUbZrXr83EG7qTevA4f5m2ttQhsIVFhj9NXskN7/+qw2SV5YrXOdEWhFI269c6kscva8ecTft5e36KxzLvLkhhwdYMlqcd5tu1u70coappslzDsLUPQikfcNsFcYzsFs1rc5N5f1HqaftW7TjChHnbuLJzE7o0C+flH7aSk1dYyjspdf6yii8x6SgmpewnIvzfNZ25olMTnp+9hdd+3IoxhixHPg9MXUOT8BBeuLoTT1yRwL4sB5MWp5b/pkqdI2+3IHSyPqXKERTgx5ujuxEa7M+bP6eQ5Sjg8PE89mY6mHZnX8JCAukVV5+hHRrx7sLfub5XcxrWDbE7bFUNaR+EUj7I38/ZkrjtgjgmL93OzHV7eGhIPD1iIk6WGX9Ze/IKipgwN9nGSFV1lpVTPIpJWxBK+RQR4ckr2tOwbjCpGce5a1Dr0/bHRYZyc98YPl66nbH94mjbuK5NkarqKsuRT6C/EBLonf/ttQWh1FkQEe68sBX/98fOHhdreeDieOoEB3Dnp0ms3nnEhghVdVZ8F7WItQsFFdMEoVQlqlc7iPdvSSS/0PDHd5fy4v+24MjXkU2qcjjnYfLO5SXQBKFUpevdsgE/PDiA63u24D8LU7nyrV/4bV+W3WGpaiArx3tTfYMmCKUsUTckkBev7sQnt/bi6Il87p+yhoLCIrvDUlVcliPfa9NsgCYIpSw1sE0U/7yqI8n7jzFlxU67w1FVnLYglKpmhnZoRN+WDXhtbjKZJ3TFOnXushwFXhviCpoglLKciPD3KxM4mpPPmz9vszscVYU5WxB6iUmpaiWhaRijejbn46Xb+T3jmN3hqCro5FoQ2oJQqvp5+JK2hAT688L3W+wORVVBJ9eCqC59ECIyTES2ikiKiIz3sH+CiKx1PZJF5KjbvjEiss31GGNlnEp5Q1TdYO67qDU//XaAhckZdoejqpiT8zBVh1FMIuIPTAQuAxKA0SKS4F7GGPOQMaarMaYr8BbwjevY+sDTQG+gF/C0iESgVBU3tn8scZGh/P3bjZzIK3ulOqXcnZzJtZq0IHoBKcaYVGNMHjAVGFFG+dHAFNfzocBcY8xhY8wRYC4wzMJYlfKK4AB/Xry6EzsPn+C1H3VSP1VxJ9eCqCZ9ENHALrfX6a5tZxCRGCAO+PlsjhWRcSKSJCJJGRnaZFdVQ5+WDbixdws+XJLGGp2vSVVQcQsivJqMYvI0m1Rpi/aOAqYbY4onranQscaY94wxicaYxKioqHMMUynvG39ZOxqFhfDXr9eTW6BzNanyneqDqB4tiHSgudvrZsCeUsqO4tTlpbM9Vqkqp25IIM+PdN5h/c783+0OR1UBJ9eCqCZ9ECuBeBGJE5EgnElgZslCItIWiACWuW2eA1wqIhGuzulLXduUqjYuateIq7o25Z0FKSTvz7Y7HOXjshz5BPn7ERzgvbsTLDuTMaYAuBfnH/YtwDRjzCYReVZEhrsVHQ1MNcYYt2MPA8/hTDIrgWdd25SqVp76QweC/P14b5GuZa3KVnwXtbfWggCLV5QzxswGZpfY9lSJ18+UcuyHwIeWBaeUD6gfGsRV3aKZviqdJ69oT73aQXaHpHyUt+dhAr2TWinb3dg7htyCIqavSrc7FOXDsnLyqevF/gfQBKGU7RKahtG9RT2+WL4TtyutSp3G22tBgCYIpXzCTX1iSD14nKW/H7I7FOWjvL0WBGiCUMonXN6pCfVqB/L58h12h6J8lPZBKFVDhQT6c22PZvy4aT8Hshx2h6N8kLfXggBNEEr5jBt6x1BQZJi6clf5hVWNYsdaEKAJQimfERcZyoD4SKas2ElBYZHd4SgfYsdaEKAJQimfcmPvGPZmOvhh0z67Q1E+xI61IEAThFI+5ZKERsQ3rMNrc5O1FaFOsmMtCNAEoZRP8fcT/nJpW1IzjvPN6t12h6N8hB1rQYAmCKV8ztAOjejSvB4T5iXjyNepwJU9a0GAJgilfI6I8Nehbdmb6eCzX/W+CGXPWhBQwQQhIq1EJNj1fJCI3C8i9awNTamaq1/rSAbERzJxfgrZrj8OqubK9PE+iK+BQhFpDXyAc3nQLyyLSinFo0PbcuREPpMWp9kdirJZVk6B19eCgIoniCLX+g4jgdeNMQ8BTawLSynVuVk9LuvYmEmLUzl8PM/ucJSNshzeXwsCKp4g8kVkNDAG+K9rm3fbOkrVQA9d0objeYVMWbHT7lCUjbJy8r3e/wAVTxB/AvoCzxtj0kQkDvjMurCUUgBtGtWlf+sGfP7rDr0vogbLchR4fS0IqGCCMMZsNsbcb4yZ4lojuq4x5iWLY1NKAWP6xrIn08HczfvtDkXZxNmC8O4QV6j4KKYFIhImIvWBdcBHIvKataEppQAubt+I6Hq1mLx0u92hKJs4+yB8tAUBhBtjsoCrgY+MMT2AIdaFpZQq5u8n3Nw3huVph/ltX5bd4SgbZOV4fy0IqHiCCBCRJsB1nOqkVkp5yfWJzQkO8OPjpXrjXE1UPIrJ2yqaIJ4F5gC/G2NWikhLYJt1YSml3EWEBnFV12i+XbObzBN641xN4sgvJM+GtSCg4p3UXxljOhtj7nK9TjXGXGNtaEopd7f0iyEnv5BpSbqgUE1ycpoNX+2DEJFmIjJDRA6IyH4R+VpEmlkdnFLqlA5Nw+kZG8Env26nsMjYHY7ykqyc4plcffcS00fATKApEA3Mcm1TSnnRrf3j2HU4h+mrtBVRU/h8CwKIMsZ8ZIwpcD0mA1EWxqWU8mBYx8Z0a1GPV+Ykcyy3wO5wlBecXCzIV/sggIMicpOI+LseNwGHyjtIRIaJyFYRSRGR8aWUuU5ENovIJhH5wm17oYisdT1mVjBOpao1EeGpKxM4eCyXd+an2B2O8oLixYK8vRYEVDxB3IpziOs+YC/wR5zTb5RKRPyBicBlQAIwWkQSSpSJBx4H+htjOgAPuu3OMcZ0dT2GVzBOpaq9bi0iGNktmkm/pLHr8Am7w1EW8/kWhDFmpzFmuDEmyhjT0BhzFc6b5srSC0hxjXjKA6YCI0qUuQOYaIw54jrPgbOMX6ka6bFhbfETeOmH3+wORVmsKvRBePJwOfujAfeetHTXNndtgDYiskREfhWRYW77QkQkybX9Kk8nEJFxrjJJGRkZZ10BpaqqJuG1uHNgK75fv5eV2w/bHY6y0L5MB7WD/L2+FgScX4Iob2JyT/tLjs0LAOKBQcBoYJLbSnUtjDGJwA3A6yLS6ow3M+Y9Y0yiMSYxKkr7zFXNcueFLWkcFsITMzawJOUgRTr0tVpK2n6Ebi3qeX0tCDi/BFHeb2M60NztdTNgj4cy3xlj8o0xacBWnAkDY8we19dUYAHQ7TxiVaraqR0UwPMjO7Iv08GNk5Yz8JX5vDFvG/syHXaHpipJZk4+W/Zl0Su2gS3nLzNBiEi2iGR5eGTjvCeiLCuBeBGJE5EgYBTOeyncfQsMdp0rEuclp1QRiXBbAzsS6A9sPuvaKVXNXdy+ESueGMIbo7oS2yCUCfOSGfbGIvZm5tgdmqoEq3ccwRjoGRdhy/nLTBDGmLrGmDAPj7rGmDLHXLmWKL0X5xxOW4BpxphNIvKsiBSPSpoDHBKRzcB84FFjzCGgPZAkIutc218yxmiCUMqDkEB/RnSN5rPbe/PDgwPIKyji4S/X6d3W1cDytMME+gvdmtuTIMSY6vFLlJiYaJKSkuwOQynbTVu5i8e+Xs/4y9rx5wvP6LpTVcg17y7FGMM3d/e37BwissrV33sG73eLK6UsdW1iMy7r2Jh//biVDemZdoejzpEjv5D16UfpFWdP/wNoglCq2hERXry6Ew1Cg3ngyzWcyNMpOaqiNTuPkl9o6GVT/wNoglCqWqpXO4jXrutC2sHjPDFjIwWFRXaHpM7SirTDiECPmPq2xaAJQqlqql/rSB4a0oYZa3Yz5qMVHDmeZ3dI6iys3H6Ydo3DCLfhDupimiCUqsbuvziel//YmZVpRxgxcQnJ+7PtDklVQH5hEat2HKF3nH2tB9AEoVS1d11ic6aM68OJvEJGTlzCd2t3U11GL1ZXm/ZkkZNfSM9YTRBKKYv1iIlg1n39ad2oLg9MXcvV7y7VOZx82Io052oKdt0gV0wThFI1RJPwWnxzVz9evqYze47mcO2/l3HHJ0k6ZbgPWpF2hLjIUBrWDbE1Dk0QStUg/n7CdT2bs+CRwTw6tC1LUw5y56er9K5rH1JUZFi5/TC9bL68BJoglKqRagX5c8/g1rx0TWc2781i6sqddoekXLYdOEZmTj49be6gBk0QStVoV3ZuQq+4+rw6ZytHT+gwWF+wZucRAHrG2tv/AJoglKrRRIRn/tCBzJx8JsxNtjscBRxy3a/SONze/gfQBKFUjZfQNIyb+sTw6a87+G1flt3h1HhZjnyCAvwIDvC3OxRNEEopePiSNoTVCuSZmZv0HgmbZTsKCAspczUFr9EEoZSiXu0gHrm0Lb+mHubz5dphbadsRwF1Q+ybXsOdJgilFACje7XggtaRPPntRl7632869NUm2Y586moLQinlS/z9hA/H9uSG3i3498LfueOTJLId+XaHVeM4WxCaIJRSPiYowI8XRnbiuas6sjA5g5HvLGX3UV3f2puyHfnUDdZLTEopH3Vznxg+va0X+zMd3PfFal1Pwou0BaGU8nn9WkXyz5EdWb3zKG/PT7E7nBpDO6mVUlXCiK7RXNW1KW/9nMJq1x2+yjqFRYZjub7TgvCNKJRSPuvZqzqycvsRHvpyLd/fP4A6wWf+2SgsMqzddZQ1O4+QW1BEYZGhoMgQF1mbq7pGIyJndU5jDH+bsZF2jesypl9sJdXE9x3Lda4frglCKVUlhIUEMuH6rox6bxn/mLmJJ65oz9ET+RzNyWf3kRzmbz3A/N8OnJwioqR6tYIY3K7hWZ1z1vq9TFmxE38/oWdsfRKahlVGVXxe8agxTRBKqSqjV1x97h7Umrfnp/DVqvTT9oWFBDCobUMubt+QC1pHEhocQICfUGRg6OuLeH72FgbERxLgX7Er2sdzC3jh+y20a1yXg8dyGf/Nembc3R9/v7NrhVRF2Y7iFoRv9EFoglBKVcgDQ+KJjqiFI7+Q8FqB1KsdSIPQYBKahhFYyh//8Ze1485PVzFl5S5u7hNToQBeeREAABPESURBVPO8PT+FfVkOJt7Yjd1HHdw/ZQ0fLUnj9gEtK7M6PulUgvCNP82+EYVSyucF+vsxuleLszrm0oRG9I6rz4S5yYzo2pSwcv4zTs04xqTFqVzdPZoeMfXp3sIwY3U6//oxmaEdGtO8fu3zqYLPO5ZbfInJN1oQOopJKWUZEeHvVyZw5EQe78z/vcyyxhj+MWszwQH+jL+s3cnj/zmyEyLw5Lcbq/1Egr7WgrA0QYjIMBHZKiIpIjK+lDLXichmEdkkIl+4bR8jIttcjzFWxqmUsk7H6HCu7taMD39JK3P967mb97MwOYMHh8SfthZzdL1aPHJpWxYmZzBz3R5vhGybrJqSIETEH5gIXAYkAKNFJKFEmXjgcaC/MaYD8KBre33gaaA30At4WkTsX15JKXVOHh3aFj8/+NuMDSTvzz5tX/qRE4z/ej13fb6aNo3qeBzWOqZfLF2ahfPP77dU6/mhiutW3qU4b7EyTfUCUowxqQAiMhUYAWx2K3MHMNEYcwTAGHPAtX0oMNcYc9h17FxgGDDFwniVUhZpHB7CI5e25fnZW7h0wiJaN6zD5R0bc/hEHl+u3IUg3NwnhnsGt/bY4e3vJzw7oiNXvbOE1+dt4+9XJng4S9WX7Sgg0F8IDvCNq/9WJohoYJfb63ScLQJ3bQBEZAngDzxjjPmhlGOjS55ARMYB4wBatDi7zjOllHfdPqAlw7s2Zc6m/fxvw17enp+CnwjX9WzOvYNb07RerTKP79K8HqN6tmDy0u1cl9icto3reily73FO9R141jcWWsXKBOGphiV7mAKAeGAQ0AxYLCIdK3gsxpj3gPcAEhMTq3fvlVLVQMO6IdzcJ4ab+8Rw+HgehUWGqLrBFT7+saFt+d/Gvfz9u418Oa6Pz/whrSy+NFEfWNtJnQ40d3vdDCjZw5QOfGeMyTfGpAFbcSaMihyrlKrC6ocGnVVyAIgIDeKxoe1YkXa4WnZY16QEsRKIF5E4EQkCRgEzS5T5FhgMICKROC85pQJzgEtFJMLVOX2pa5tSqoa7vmfzatth7UtrQYCFCcIYUwDci/MP+xZgmjFmk4g8KyLDXcXmAIdEZDMwH3jUGHPI1Tn9HM4ksxJ4trjDWilVsxV3WB88lsttHydxuJQ5oKoiX2tBSHW58SQxMdEkJSXZHYZSyku+W7ubR6evp0l4CB+M6UnrhnXsDum89X/pZ/q0bMC/ruvitXOKyCpjTKKnfb4xlkoppc7SiK7RTLmjD8dzCxj5zhJ+2XbQ7pDOW5Yj36daEL4TiVJKnaUeMRHMuLs/t3+cxJiPVtAjJoIeMRF0b+H8Wj80yO4QK6zItVhQmCYIpZSqHM3r12b6XX15++cUfk09xPuLUikoMgT5+/HJbb3o07KB3SFWyPG8AoyBOpoglFKq8tQNCeTxy9sD4MgvZH16Jg99uZZnZm7i+/sHVIm1JE6tJlcDRjEppZQdQgL96RVXn8cvb8dv+7L5KmlX+Qf5AF+byRU0QSilqqkrOjUhMSaCV3/cWiXulzi13Ki2IJRSylLFa1EcPJbHOwvKXovCF/jaVN+gCUIpVY11aV6Pq7tH88Histei8AXFl5h8aRSTJgilVLX22NB2+PsJL/5vi92hlEkvMSmllJc1Dg/hrkGtmL1hH9+t3W13OKXSTmqllLLBny9sRa+4+jw6fT2rdx45Y39RkcGRX2hDZKdkO/Lx9xNqBfrbGoc7TRBKqWovKMCPf9/Ug8ZhIYz7JIn0I6f6IzbvyWL4xF8Y8PJ8MnPsG+1UPFGfL61xoQlCKVUj1A8N4sOxieQWFHHbZOcssK/9uJXhb//CnqMOMrJzeX9Rqm3x+dpMrqAJQilVg7RuWJd3buxOSsYx+r30E2/+nMLwLk356eELuaJzEz5ckkZGdq4tsfnaWhCgCUIpVcMMiI/ixZGdiG0QyodjE3nt+q5EhAbxl0vakFtQxMT5KbbElaUtCKWUst91PZvzw4MDuahdo5PbWkbV4doezfhi+c7T+ii8xXmJSVsQSinlk+6/OB4E3pi3zevnznbk+9RNcqAJQimlTmparxY394nh69XppBw45tVzZzsKfGqqb9AEoZRSp7l7UCtqBfozYW6y185pjHOxIO2DUEopH9agTjBj+8cye+Nedhw67pVz5uQXUlhktA9CKaV83S19YwnwEyYv3e6V8/niNBugCUIppc7QKCyEKzs35aukdK+sJeGLE/WBJgillPLo1v5xHMstYFpSuuXn8sW1IEAThFJKedSpWTg9YyOYvDSNwiJj6bl8cS0I0AShlFKlurV/HLsO5zBvy35Lz6OXmJRSqoq5JKER0fVq8eEvaZaep0Z2UovIMBHZKiIpIjLew/6xIpIhImtdj9vd9hW6bZ9pZZxKKeVJgL8fY/vFsjztMBt3Z1p2nhrXghARf2AicBmQAIwWkQQPRb80xnR1PSa5bc9x2z7cqjiVUqos1/VsTu0gf95d8DvGWNMXke0owE8gNMh3FgsCa1sQvYAUY0yqMSYPmAqMsPB8SilV6cJrBXLnwFZ8v2EvnyzbYck5sh0F1An2rcWCwNoEEQ3scnud7tpW0jUisl5EpotIc7ftISKSJCK/ishVnk4gIuNcZZIyMjIqMXSllDrlvotaM6R9Q57972aWphys9PfPcuT73OUlsDZBeEqFJdtns4BYY0xnYB7wsdu+FsaYROAG4HURaXXGmxnznjEm0RiTGBUVVVlxK6XUafz8hAnXd6VlZCh3f7GanYcqdzpwX1xNDqxNEOmAe4ugGbDHvYAx5pAxpnj5pveBHm779ri+pgILgG4WxqqUUmWqGxLI+7ckYgzc8UkSx3ILKu29nVN916wWxEogXkTiRCQIGAWcNhpJRJq4vRwObHFtjxCRYNfzSKA/sNnCWJVSqlyxkaFMvMG5ZOlzsyrvT1KNa0EYYwqAe4E5OP/wTzPGbBKRZ0WkeFTS/SKySUTWAfcDY13b2wNJru3zgZeMMZoglFK2uyA+8uSaEXuO5lTKex7L9b21IAAsjcgYMxuYXWLbU27PHwce93DcUqCTlbEppdS5un1AHJ/+uoOPlqTxxBWeRu+fnRrXglBKqeqqWURt/tC5CV8s30lmzvnN9mqMIbsGjmJSSqlqa9zAVhzPK+Tz5WfeG1F0FpP75RYUkV9otAWhlFLVRULTMAbER/LRku048gtPbv819RCJz8/jbzM2UFBYVO77ZPnoNBugCUIppc7Zny9sRUZ2Lt+u2Q3A7A17ueWDFfgJfLF8J3d/vvq05OGJr071DZoglFLqnPVr1YCO0WG8tziVj5du554vVtOpWTjzHr6QZ/6QwNwt+7nlgxVl9lP46kyuoAlCKaXOmYgwbmArUjOO8/TMTVzcrhGf396berWDGNs/jjdGdWPNriNc/59lHDyW6/E9fHUmV9AEoZRS5+Xyjo3pHVefMX1j+PdN3QkJPDUj6/AuTflobC9SDx7nhe+3eDx+054sAOqHBnkl3rPhe20apZSqQgL8/fjyzr6l7r8gPpJxA1ry9vwUbuzTgh4x9U/u25fp4K2ftjG4bRQtI0O9Ee5Z0RaEUkpZ7O7BrWgcFsIzMzeftr71c99vpqDI8I/hHX1uqm/QBKGUUparHRTA45e3Y8PuTL5Kcq6CsCg5g+/X7+Xewa1p0aC2zRF6pglCKaW8YHiXpvSMjeDlOVs5kO3gqe82EhcZyrgLW9odWqk0QSillBeICE//oQNHTuQxcuJSth86wXMjOhIc4FvLjLrTBKGUUl7SMTqc0b1asPtoDn/o0pQL4iPtDqlMOopJKaW86LGhbQmvFchtF8TZHUq5NEEopZQX1asdxF+HtbM7jArRS0xKKaU80gShlFLKI00QSimlPNIEoZRSyiNNEEoppTzSBKGUUsojTRBKKaU80gShlFLKIzHGlF+qChCRDGCHh13hQGY529xfe3pe/DUSOHiOIXqKo6JlyqtDafXxVMbKOpS1v6zvecnX5T23ow6V8Xvk/vxc62Dl71HJ12V9FsA361CR+vja57mir636LMQYY6I8ljDGVOsH8F5529xfe3ru9jWpMuOoaJny6lBafUqpi2V1KGt/Wd/zivwM7K5DZfweVUYdrPw9qmDc7tt8rg4VqY+vfZ4r+trbnwVjTI24xDSrAttmlfPc03tURhwVLVNeHUqrT1llzkV571HW/rK+5yVfV+T5uTrXOlTG71FFzl8eK3+PSr6uTp8F9+e+VoeKvvb2Z6H6XGLyBhFJMsYk2h3H+dA6+Aatg/2qevxgfR1qQguiMr1ndwCVQOvgG7QO9qvq8YPFddAWhFJKKY+0BaGUUsojTRBKKaU8qrEJQkQ+FJEDIrLxHI7tISIbRCRFRN4UEXHbd5+IbBWRTSLycuVGfUYclV4HEXlGRHaLyFrX4/LKj/y0OCz5Obj2PyIiRkQsXdfRop/DcyKy3vUz+FFEmlZ+5CdjsCL+V0TkN1cdZohIvcqP/LQ4rKjDta7PcZGIWNYRfD6xl/J+Y0Rkm+sxxm17mZ8Xj851DG1VfwADge7AxnM4dgXQFxDgf8Blru2DgXlAsOt1wypYh2eAR6ryz8G1rzkwB+fNk5FVrQ5AmFuZ+4F/V7H4LwUCXM//D/i/KvgzaA+0BRYAib4Wuyuu2BLb6gOprq8RrucRZdWzrEeNbUEYYxYBh923iUgrEflBRFaJyGIROWNdQBFpgvPDu8w4v+ufAFe5dt8FvGSMyXWd40AVrINXWViHCcBjgOWjMKyogzEmy61oKBbWw6L4fzTGFLiK/go0syp+C+uwxRiz1cq4zyf2UgwF5hpjDhtjjgBzgWHn+pmvsQmiFO8B9xljegCPAO94KBMNpLu9TndtA2gDDBCR5SKyUER6WhqtZ+dbB4B7XZcGPhSRCOtCLdV51UFEhgO7jTHrrA60DOf9cxCR50VkF3Aj8JSFsXpSGb9HxW7F+R+rt1VmHbytIrF7Eg3scntdXJ9zqmdABU9a7YlIHaAf8JXbpblgT0U9bCv+7y4AZ7OuD9ATmCYiLV0Z23KVVId3gedcr58D/oXzA+4V51sHEakNPIHzEoctKunngDHmCeAJEXkcuBd4upJD9aiy4ne91xNAAfB5ZcZYnsqsg7eVFbuI/Al4wLWtNTBbRPKANGPMSEqvzznVUxPEKX7AUWNMV/eNIuIPrHK9nInzD6h7c7kZsMf1PB34xpUQVohIEc7JtDKsDNzNedfBGLPf7bj3gf9aGbAH51uHVkAcsM714WoGrBaRXsaYfRbHXqwyfpfcfQF8j5cSBJUUv6uD9ErgYm/9k+Smsn8G3uQxdgBjzEfARwAisgAYa4zZ7lYkHRjk9roZzr6KdM6lnlZ1vFSFBxCLW8cQsBS41vVcgC6lHLcSZyuhuLPnctf2PwPPup63wdnUkypWhyZuZR4Cpla1n0OJMtuxuJPaop9DvFuZ+4DpVSz+YcBmIMrq773Vv0dY3El9rrFTeid1Gs4rGRGu5/UrUk+PcXnrh+drD2AKsBfIx5ldb8P5n+cPwDrXL/dTpRybCGwEfgfe5tQd6UHAZ659q4GLqmAdPgU2AOtx/ofVpKrVoUSZ7Vg/ismKn8PXru3rcU6qFl3F4k/B+Q/SWtfDslFYFtZhpOu9coH9wBxfih0PCcK1/VbX9z8F+NPZfF5KPnSqDaWUUh7pKCallFIeaYJQSinlkSYIpZRSHmmCUEop5ZEmCKWUUh5pglDVmogc8/L5JolIQiW9V6E4Z3PdKCKzypsRVUTqicjdlXFupUBXlFPVnIgcM8bUqcT3CzCnJqGzlHvsIvIxkGyMeb6M8rHAf40xHb0Rn6r+tAWhahwRiRKRr0VkpevR37W9l4gsFZE1rq9tXdvHishXIjIL+FFEBonIAhGZLs41Dz4vnlvftT3R9fyYa8K9dSLyq4g0cm1v5Xq9UkSerWArZxmnJiOsIyI/ichqcc7vP8JV5iWglavV8Yqr7KOu86wXkX9U4rdR1QCaIFRN9AYwwRjTE7gGmOTa/hsw0BjTDefsqS+4HdMXGGOMucj1uhvwIJAAtAT6ezhPKPCrMaYLsAi4w+38b7jOX+58OK75gy7GeWc7gAMYaYzpjnMNkn+5EtR44HdjTFdjzKMicikQD/QCugI9RGRgeedTqphO1qdqoiFAgttMmWEiUhcIBz4WkXicM10Guh0z1xjjPmf/CmNMOoCIrMU5l84vJc6Tx6nJDlcBl7ie9+XUXPxfAK+WEmctt/dehXNuf3DOpfOC6499Ec6WRSMPx1/qeqxxva6DM2EsKuV8Sp1GE4SqifyAvsaYHPeNIvIWMN8YM9J1PX+B2+7jJd4j1+15IZ4/S/nmVCdfaWXKkmOM6Soi4TgTzT3AmzjXh4gCehhj8kVkOxDi4XgBXjTG/Ocsz6sUoJeYVM30I871FQAQkeJplcOB3a7nYy08/684L20BjCqvsDEmE+eyo4+ISCDOOA+4ksNgIMZVNBuo63boHOBW1/oCiEi0iDSspDqoGkAThKruaotIutvjYZx/bBNdHbebcU7TDvAy8KKILAH8LYzpQeBhEVkBNAEyyzvAGLMG58yeo3AuvpMoIkk4WxO/ucocApa4hsW+Yoz5EeclrGUisgGYzukJRKky6TBXpbzMtepdjjHGiMgoYLQxZkR5xynlbdoHoZT39QDedo08OooXl3RV6mxoC0IppZRH2gehlFLKI00QSimlPNIEoZRSyiNNEEoppTzSBKGUUsqj/we3jMyUzriSlAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3C. Training and fine-tuning the `IMDb sentiment classifier`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train for one cycle, save intermediate result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.287057</td>\n",
       "      <td>0.202339</td>\n",
       "      <td>0.932720</td>\n",
       "      <td>06:29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load('first');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unfreeze last two layers and train for one cycle, save intermediate result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.253536</td>\n",
       "      <td>0.164451</td>\n",
       "      <td>0.943320</td>\n",
       "      <td>07:25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('2nd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unfreeze the last three layers, and train for one cycle, and save intermediate result.\n",
    "At this point we've already beaten the 2017 (pre-transfer learning) state of the art!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.207790</td>\n",
       "      <td>0.151074</td>\n",
       "      <td>0.947320</td>\n",
       "      <td>09:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('3rd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RNNLearner(data=TextClasDataBunch;\n",
       "\n",
       "Train: LabelList (25000 items)\n",
       "x: TextList\n",
       "xxbos xxmaj story of a man who has unnatural feelings for a pig . xxmaj starts out with a opening scene that is a terrific example of absurd comedy . a formal orchestra audience is turned into an insane , violent mob by the crazy xxunk of it 's singers . xxmaj unfortunately it stays absurd the xxup whole time with no general narrative eventually making it just too off putting . xxmaj even those from the era should be turned off . xxmaj the cryptic dialogue would make xxmaj shakespeare seem easy to a third grader . xxmaj on a technical level it 's better than you might think with some good cinematography by future great xxmaj vilmos xxmaj zsigmond . xxmaj future stars xxmaj sally xxmaj kirkland and xxmaj frederic xxmaj forrest can be seen briefly .,xxbos xxmaj airport ' 77 starts as a brand new luxury 747 plane is loaded up with valuable paintings & such belonging to rich businessman xxmaj philip xxmaj stevens ( xxmaj james xxmaj stewart ) who is flying them & a bunch of xxup vip 's to his estate in preparation of it being opened to the public as a museum , also on board is xxmaj stevens daughter xxmaj julie ( xxmaj kathleen xxmaj quinlan ) & her son . xxmaj the luxury jetliner takes off as planned but mid - air the plane is hi - jacked by the co - pilot xxmaj chambers ( xxmaj robert xxmaj foxworth ) & his two accomplice 's xxmaj banker ( xxmaj monte xxmaj markham ) & xxmaj wilson ( xxmaj michael xxmaj pataki ) who knock the passengers & crew out with sleeping gas , they plan to steal the valuable cargo & land on a disused plane strip on an isolated island but while making his descent xxmaj chambers almost hits an oil rig in the xxmaj ocean & loses control of the plane sending it crashing into the sea where it sinks to the bottom right bang in the middle of the xxmaj bermuda xxmaj triangle . xxmaj with air in short supply , water leaking in & having flown over 200 miles off course the problems mount for the survivor 's as they await help with time fast running out ... \n",
       " \n",
       "  xxmaj also known under the slightly different tile xxmaj airport 1977 this second sequel to the smash - hit disaster thriller xxmaj airport ( 1970 ) was directed by xxmaj jerry xxmaj jameson & while once again like it 's predecessors i ca n't say xxmaj airport ' 77 is any sort of forgotten classic it is entertaining although not necessarily for the right reasons . xxmaj out of the three xxmaj airport films i have seen so far i actually liked this one the best , just . xxmaj it has my favourite plot of the three with a nice mid - air hi - jacking & then the crashing ( did n't he see the oil rig ? ) & sinking of the 747 ( maybe the makers were trying to cross the original xxmaj airport with another popular disaster flick of the period xxmaj the xxmaj poseidon xxmaj adventure ( 1972 ) ) & submerged is where it stays until the end with a stark dilemma facing those trapped inside , either suffocate when the air runs out or drown as the 747 floods or if any of the doors are opened & it 's a decent idea that could have made for a great little disaster flick but bad unsympathetic character 's , dull dialogue , lethargic set - pieces & a real lack of danger or suspense or tension means this is a missed opportunity . xxmaj while the rather sluggish plot keeps one entertained for 108 odd minutes not that much happens after the plane sinks & there 's not as much urgency as i thought there should have been . xxmaj even when the xxmaj navy become involved things do n't pick up that much with a few shots of huge ships & helicopters flying about but there 's just something lacking here . xxmaj george xxmaj kennedy as the jinxed airline worker xxmaj joe xxmaj patroni is back but only gets a couple of scenes & barely even says anything preferring to just look worried in the background . \n",
       " \n",
       "  xxmaj the home video & theatrical version of xxmaj airport ' 77 run 108 minutes while the xxup us xxup tv versions add an extra hour of footage including a new opening credits sequence , many more scenes with xxmaj george xxmaj kennedy as xxmaj patroni , flashbacks to flesh out character 's , longer rescue scenes & the discovery or another couple of dead bodies including the navigator . xxmaj while i would like to see this extra footage i am not sure i could sit through a near three hour cut of xxmaj airport ' 77 . xxmaj as expected the film has dated badly with horrible fashions & interior design choices , i will say no more other than the toy plane model effects are n't great either . xxmaj along with the other two xxmaj airport sequels this takes pride of place in the xxmaj razzie xxmaj award 's xxmaj hall of xxmaj shame although i can think of lots of worse films than this so i reckon that 's a little harsh . xxmaj the action scenes are a little dull unfortunately , the pace is slow & not much excitement or tension is generated which is a shame as i reckon this could have been a pretty good film if made properly . \n",
       " \n",
       "  xxmaj the production values are alright if nothing spectacular . xxmaj the acting is n't great , two time xxmaj oscar winner xxmaj jack xxmaj lemmon has said since it was a mistake to star in this , one time xxmaj oscar winner xxmaj james xxmaj stewart looks old & frail , also one time xxmaj oscar winner xxmaj lee xxmaj grant looks drunk while xxmaj sir xxmaj christopher xxmaj lee is given little to do & there are plenty of other familiar faces to look out for too . \n",
       " \n",
       "  xxmaj airport ' 77 is the most disaster orientated of the three xxmaj airport films so far & i liked the ideas behind it even if they were a bit silly , the production & bland direction does n't help though & a film about a sunken plane just should n't be this boring or lethargic . xxmaj followed by xxmaj the xxmaj concorde ... xxmaj airport ' 79 ( 1979 ) .,xxbos xxmaj this film lacked something i could n't put my finger on at first : charisma on the part of the leading actress . xxmaj this inevitably translated to lack of chemistry when she shared the screen with her leading man . xxmaj even the romantic scenes came across as being merely the actors at play . xxmaj it could very well have been the director who miscalculated what he needed from the actors . i just do n't know . \n",
       " \n",
       "  xxmaj but could it have been the screenplay ? xxmaj just exactly who was the chef in love with ? xxmaj he seemed more enamored of his culinary skills and restaurant , and ultimately of himself and his youthful exploits , than of anybody or anything else . xxmaj he never convinced me he was in love with the princess . \n",
       " \n",
       "  i was disappointed in this movie . xxmaj but , do n't forget it was nominated for an xxmaj oscar , so judge for yourself .,xxbos xxmaj sorry everyone , , , i know this is supposed to be an \" art \" film , , but wow , they should have handed out guns at the screening so people could blow their brains out and not watch . xxmaj although the scene design and photographic direction was excellent , this story is too painful to watch . xxmaj the absence of a sound track was brutal . xxmaj the l xxrep 4 o xxrep 5 n g shots were too long . xxmaj how long can you watch two people just sitting there and talking ? xxmaj especially when the dialogue is two people complaining . i really had a hard time just getting through this film . xxmaj the performances were excellent , but how much of that dark , sombre , uninspired , stuff can you take ? xxmaj the only thing i liked was xxmaj maureen xxmaj stapleton and her red dress and dancing scene . xxmaj otherwise this was a ripoff of xxmaj bergman . xxmaj and i 'm no fan f his either . i think anyone who says they enjoyed 1 1 / 2 hours of this is , , well , lying .,xxbos xxmaj when i was little my parents took me along to the theater to see xxmaj interiors . xxmaj it was one of many movies i watched with my parents , but this was the only one we walked out of . xxmaj since then i had never seen xxmaj interiors until just recently , and i could have lived out the rest of my life without it . xxmaj what a pretentious , ponderous , and painfully boring piece of 70 's wine and cheese tripe . xxmaj woody xxmaj allen is one of my favorite directors but xxmaj interiors is by far the worst piece of crap of his career . xxmaj in the unmistakable style of xxmaj ingmar xxmaj berman , xxmaj allen gives us a dark , angular , muted , insight in to the lives of a family wrought by the psychological damage caused by divorce , estrangement , career , love , non - love , xxunk , whatever . xxmaj the film , intentionally , has no comic relief , no music , and is drenched in shadowy pathos . xxmaj this film style can be best defined as expressionist in nature , using an improvisational method of dialogue to illicit a \" more pronounced depth of meaning and truth \" . xxmaj but xxmaj woody xxmaj allen is no xxmaj ingmar xxmaj bergman . xxmaj the film is painfully slow and dull . xxmaj but beyond that , i simply had no connection with or sympathy for any of the characters . xxmaj instead i felt only contempt for this parade of shuffling , whining , nicotine stained , martyrs in a perpetual quest for identity . xxmaj amid a backdrop of cosmopolitan affluence and baked xxmaj brie intelligentsia the story looms like a fart in the room . xxmaj everyone speaks in affected platitudes and elevated language between cigarettes . xxmaj everyone is \" lost \" and \" struggling \" , desperate to find direction or understanding or whatever and it just goes on and on to the point where you just want to slap all of them . xxmaj it 's never about resolution , it 's only about interminable introspective babble . xxmaj it is nothing more than a psychological drama taken to an extreme beyond the audience 's ability to connect . xxmaj woody xxmaj allen chose to make characters so immersed in themselves we feel left out . xxmaj and for that reason i found this movie painfully self indulgent and spiritually draining . i see what he was going for but his insistence on promoting his message through xxmaj prozac prose and distorted film techniques jettisons it past the point of relevance . i highly recommend this one if you 're feeling a little too happy and need something to remind you of death . xxmaj otherwise , let 's just pretend this film never happened .\n",
       "y: CategoryList\n",
       "neg,neg,neg,neg,neg\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Valid: LabelList (25000 items)\n",
       "x: TextList\n",
       "xxbos xxmaj once again xxmaj mr. xxmaj costner has dragged out a movie for far longer than necessary . xxmaj aside from the terrific sea rescue sequences , of which there are very few i just did not care about any of the characters . xxmaj most of us have ghosts in the closet , and xxmaj costner 's character are realized early on , and then forgotten until much later , by which time i did not care . xxmaj the character we should really care about is a very cocky , overconfident xxmaj ashton xxmaj kutcher . xxmaj the problem is he comes off as kid who thinks he 's better than anyone else around him and shows no signs of a cluttered closet . xxmaj his only obstacle appears to be winning over xxmaj costner . xxmaj finally when we are well past the half way point of this stinker , xxmaj costner tells us all about xxmaj kutcher 's ghosts . xxmaj we are told why xxmaj kutcher is driven to be the best with no prior inkling or foreshadowing . xxmaj no magic here , it was all i could do to keep from turning it off an hour in .,xxbos xxmaj this is an example of why the majority of action films are the same . xxmaj generic and boring , there 's really nothing worth watching here . a complete waste of the then barely - tapped talents of xxmaj ice - t and xxmaj ice xxmaj cube , who 've each proven many times over that they are capable of acting , and acting well . xxmaj do n't bother with this one , go see xxmaj new xxmaj jack xxmaj city , xxmaj ricochet or watch xxmaj new xxmaj york xxmaj undercover for xxmaj ice - t , or xxmaj boyz n the xxmaj hood , xxmaj higher xxmaj learning or xxmaj friday for xxmaj ice xxmaj cube and see the real deal . xxmaj ice - t 's horribly cliched dialogue alone makes this film grate at the teeth , and i 'm still wondering what the heck xxmaj bill xxmaj paxton was doing in this film ? xxmaj and why the heck does he always play the exact same character ? xxmaj from xxmaj aliens onward , every film i 've seen with xxmaj bill xxmaj paxton has him playing the exact same irritating character , and at least in xxmaj aliens his character died , which made it somewhat gratifying ... \n",
       " \n",
       "  xxmaj overall , this is second - rate action trash . xxmaj there are countless better films to see , and if you really want to see this one , watch xxmaj judgement xxmaj night , which is practically a carbon copy but has better acting and a better script . xxmaj the only thing that made this at all worth watching was a decent hand on the camera - the cinematography was almost refreshing , which comes close to making up for the horrible film itself - but not quite . 4 / 10 .,xxbos xxmaj first of all i hate those moronic rappers , who could'nt act if they had a gun pressed against their foreheads . xxmaj all they do is curse and shoot each other and acting like xxunk version of gangsters . \n",
       " \n",
       "  xxmaj the movie does n't take more than five minutes to explain what is going on before we 're already at the warehouse xxmaj there is not a single sympathetic character in this movie , except for the homeless guy , who is also the only one with half a brain . \n",
       " \n",
       "  xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are both hill billies and xxmaj xxunk character is just as much a villain as the gangsters . i did'nt like him right from the start . \n",
       " \n",
       "  xxmaj the movie is filled with pointless violence and xxmaj walter xxmaj hills specialty : people falling through windows with glass flying everywhere . xxmaj there is pretty much no plot and it is a big problem when you root for no - one . xxmaj everybody dies , except from xxmaj paxton and the homeless guy and everybody get what they deserve . \n",
       " \n",
       "  xxmaj the only two black people that can act is the homeless guy and the junkie but they 're actors by profession , not annoying ugly brain dead rappers . \n",
       " \n",
       "  xxmaj stay away from this crap and watch 48 hours 1 and 2 instead . xxmaj at lest they have characters you care about , a sense of humor and nothing but real actors in the cast .,xxbos xxmaj not even the xxmaj beatles could write songs everyone liked , and although xxmaj walter xxmaj hill is no mop - top he 's second to none when it comes to thought provoking action movies . xxmaj the nineties came and social platforms were changing in music and film , the emergence of the xxmaj rapper turned movie star was in full swing , the acting took a back seat to each man 's overpowering regional accent and transparent acting . xxmaj this was one of the many ice - t movies i saw as a kid and loved , only to watch them later and cringe . xxmaj bill xxmaj paxton and xxmaj william xxmaj sadler are firemen with basic lives until a burning building tenant about to go up in flames hands over a map with gold implications . i hand it to xxmaj walter for quickly and neatly setting up the main characters and location . xxmaj but i fault everyone involved for turning out xxmaj lame - o performances . xxmaj ice - t and cube must have been red hot at this time , and while i 've enjoyed both their careers as rappers , in my opinion they fell flat in this movie . xxmaj it 's about ninety minutes of one guy ridiculously turning his back on the other guy to the point you find yourself locked in multiple states of disbelief . xxmaj now this is a movie , its not a documentary so i wo nt waste my time recounting all the stupid plot twists in this movie , but there were many , and they led nowhere . i got the feeling watching this that everyone on set was xxunk of confused and just playing things off the cuff . xxmaj there are two things i still enjoy about it , one involves a scene with a needle and the other is xxmaj sadler 's huge 45 pistol . xxmaj bottom line this movie is like domino 's pizza . xxmaj yeah ill eat it if i 'm hungry and i do n't feel like cooking , xxmaj but i 'm well aware it tastes like crap . 3 stars , meh .,xxbos xxmaj brass pictures ( movies is not a fitting word for them ) really are somewhat brassy . xxmaj their alluring visual qualities are reminiscent of expensive high class xxup tv commercials . xxmaj but unfortunately xxmaj brass pictures are feature films with the pretense of wanting to entertain viewers for over two hours ! xxmaj in this they fail miserably , their undeniable , but rather soft and flabby than steamy , erotic qualities non withstanding . \n",
       " \n",
       "  xxmaj senso ' 45 is a remake of a film by xxmaj luchino xxmaj visconti with the same title and xxmaj alida xxmaj valli and xxmaj farley xxmaj granger in the lead . xxmaj the original tells a story of senseless love and lust in and around xxmaj venice during the xxmaj italian wars of independence . xxmaj brass moved the action from the 19th into the 20th century , 1945 to be exact , so there are xxmaj mussolini murals , men in black shirts , xxmaj german uniforms or the tattered garb of the partisans . xxmaj but it is just window dressing , the historic context is completely negligible . \n",
       " \n",
       "  xxmaj anna xxmaj xxunk plays the attractive aristocratic woman who falls for the amoral xxup ss guy who always puts on too much lipstick . xxmaj she is an attractive , versatile , well trained xxmaj italian actress and clearly above the material . xxmaj her wide range of facial expressions ( xxunk boredom , loathing , delight , fear , hate ... and ecstasy ) are the best reason to watch this picture and worth two stars . xxmaj she endures this basically trashy stuff with an astonishing amount of dignity . i wish some really good parts come along for her . xxmaj she really deserves it .\n",
       "y: CategoryList\n",
       "neg,neg,neg,neg,neg\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): MultiBatchEncoder(\n",
       "    (module): AWD_LSTM(\n",
       "      (encoder): Embedding(60000, 400, padding_idx=1)\n",
       "      (encoder_dp): EmbeddingDropout(\n",
       "        (emb): Embedding(60000, 400, padding_idx=1)\n",
       "      )\n",
       "      (rnns): ModuleList(\n",
       "        (0): WeightDropout(\n",
       "          (module): LSTM(400, 1152, batch_first=True)\n",
       "        )\n",
       "        (1): WeightDropout(\n",
       "          (module): LSTM(1152, 1152, batch_first=True)\n",
       "        )\n",
       "        (2): WeightDropout(\n",
       "          (module): LSTM(1152, 400, batch_first=True)\n",
       "        )\n",
       "      )\n",
       "      (input_dp): RNNDropout()\n",
       "      (hidden_dps): ModuleList(\n",
       "        (0): RNNDropout()\n",
       "        (1): RNNDropout()\n",
       "        (2): RNNDropout()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (1): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.12, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=2, bias=True)\n",
       "    )\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x000001BC017E0F28>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0, MixedPrecision\n",
       "learn: ...\n",
       "loss_scale: 65536\n",
       "max_noskip: 1000\n",
       "dynamic: True\n",
       "clip: None\n",
       "flat_master: False\n",
       "max_scale: 16777216\n",
       "loss_fp32: True], layer_groups=[Sequential(\n",
       "  (0): Embedding(60000, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(60000, 400, padding_idx=1)\n",
       "  )\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.12, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=2, bias=True)\n",
       "    )\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.load('3rd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unfreeze all the layers, train for two cycles, and save the result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: at this step I encountered a `CUDA error: unspecified launch failure`. This is a known (and unsolved) problem with PyTorch when using an LSTM. https://github.com/pytorch/pytorch/issues/27837\n",
    "\n",
    "Nothing to do but try again... and it worked on the second try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.205716</td>\n",
       "      <td>0.144805</td>\n",
       "      <td>0.948720</td>\n",
       "      <td>12:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.132235</td>\n",
       "      <td>0.142493</td>\n",
       "      <td>0.949320</td>\n",
       "      <td>12:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The state of the art for this dataset in 2017 was 94.1%, and we have crushed it!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the IMDb classifer model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('clas')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's look at a few examples, just to check that the classifier is working as we think it should. \n",
    "The three outputs of the model predition are the label (`pos` or `neg`) and the class probability estimates for `neg` and `pos`, which meausure the model's confidence in it's prediction. As we'd expect, the model is extremely confident that the first review is `pos` and quite confident that the second review is `neg`. So it passes the test with flying colors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\cross-entropy\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category pos, tensor(1), tensor([0.0161, 0.9839]))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.predict(\"I really loved that movie, it was awesome!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\cross-entropy\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category neg, tensor(0), tensor([0.9698, 0.0302]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.predict(\"I didn't really love that movie, and I didn't think it was awesome.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now that we've built the model, here is the part where you get to have some fun!! Take the model for a spin, try out your own examples!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Appendix: Language Model Zoo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "fast.ai alumni have applied ULMFit to dozens of different languages, and have beat the SOTA in Thai, Polish, German, Indonesian, Hindi, & Malay.\n",
    "\n",
    "They share tips and best practices in [this forum thread](https://forums.fast.ai/t/language-model-zoo-gorilla/14623) in case you are interested in getting involved!\n",
    "\n",
    "<img src=\"images/language_model_zoo.png\" alt=\"language model zoo\" style=\"width: 80%\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}