review-nlp-transfer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Start: Training an IMDb sentiment model with *ULMFiT*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a quick end-to-end example of training a model. We'll train a sentiment classifier on a sample of the popular IMDb data, showing 4 steps:\n",
    "\n",
    "1. Reading and viewing the IMDb data\n",
    "1. Getting your data ready for modeling\n",
    "1. Fine-tuning a language model\n",
    "1. Building a classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import * "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrary to images in Computer Vision, text can't directly be transformed into numbers to be fed into a model. The first thing we need to do is to preprocess our data so that we change the raw texts to lists of words, or tokens (a step that is called tokenization) then transform these tokens into numbers (a step that is called numericalization). These numbers are then passed to embedding layers that will convert them in arrays of floats before passing them through a model.\n",
    "\n",
    "Steps:\n",
    "\n",
    "1. Get your data preprocessed and ready to use,\n",
    "1. Create a language model with pretrained weights that you can fine-tune to your dataset,\n",
    "1. Create other models such as classifiers on top of the encoder of the language model.\n",
    "\n",
    "To show examples, we have provided a small sample of the [IMDB dataset](https://www.imdb.com/interfaces/) which contains 1,000 reviews of movies with labels (positive or negative)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>negative</td>\n",
       "      <td>Un-bleeping-believable! Meg Ryan doesn't even ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>positive</td>\n",
       "      <td>This is a extremely well-made film. The acting...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>negative</td>\n",
       "      <td>Every once in a long while a movie will come a...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>positive</td>\n",
       "      <td>Name just says it all. I watched this movie wi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>negative</td>\n",
       "      <td>This movie succeeds at being one of the most u...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                               text  is_valid\n",
       "0  negative  Un-bleeping-believable! Meg Ryan doesn't even ...     False\n",
       "1  positive  This is a extremely well-made film. The acting...     False\n",
       "2  negative  Every once in a long while a movie will come a...     False\n",
       "3  positive  Name just says it all. I watched this movie wi...     False\n",
       "4  negative  This movie succeeds at being one of the most u...     False"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path/'texts.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the data and create a `databunch` object to use for a language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 0\n",
      "\n",
      "Wall time: 13 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool` Error sometimes. Keep trying `till it works!\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # The following line throws `AttributeError: backwards` on the learning step, below\n",
    "        # data_lm = TextDataBunch.from_csv(path, 'texts.csv')\n",
    "        # This Fastai Forum post shows the solution:\n",
    "        #      https://forums.fast.ai/t/backwards-attributes-not-found-in-nlp-text-learner/51340?u=jcatanza\n",
    "        # We implement the solution on the following line:\n",
    "        data_lm = TextLMDataBunch.from_csv(path, 'texts.csv')\n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')    \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the data again, this time form a `databunch` object for use in a classifier model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "failure count is 1\n",
      "\n",
      "Wall time: 22.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# throws `BrokenProcessPool` Error sometimes. Keep trying `till it works!\n",
    "count = 0\n",
    "error = True\n",
    "while error:\n",
    "    try: \n",
    "        # Create the databunch for the classifier model\n",
    "        data_clas = TextClasDataBunch.from_csv(path, 'texts.csv', vocab=data_lm.train_ds.vocab, bs=32)\n",
    "        error = False\n",
    "        print(f'failure count is {count}\\n')    \n",
    "    except: # catch *all* exceptions\n",
    "        # accumulate failure count\n",
    "        count = count + 1\n",
    "        print(f'failure count is {count}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the `databunch` objects for later use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save('data_lm_export.pkl')\n",
    "data_clas.save('data_clas_export.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the `databunch` objects\n",
    "Note that you can load the data with different [`DataBunch`](/basic_data.html#DataBunch) parameters (batch size, `bptt`,...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#bs=192\n",
    "#bs=48\n",
    "bs=32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'data_lm_export.pkl', bs=bs)\n",
    "data_clas = load_data(path, 'data_clas_export.pkl', bs=bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Build and fine-tune a language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `data_lm` object we created earlier to fine-tune a pretrained language model. [fast.ai](http://www.fast.ai/) has an English model with an AWD-LSTM architecture available that we can download. We can create a learner object that will directly create a model, download the pretrained weights and be ready for fine-tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set up to use the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# torch.cuda.set_device(1)\n",
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build the IMDb language model, initializing with the hpretrained weights from `wikitext-103`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Start training\n",
    "By default this step unfreezes and trains the weights for the last layer. Training updates the weights to values more applicable to the language of IMDb reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.324914</td>\n",
       "      <td>3.921263</td>\n",
       "      <td>0.279464</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "language_model_learner??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load_pretrained??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_weights??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use [Visual Studio Code](https://code.visualstudio.com/) (vscode - open source editor that comes with recent versions of Anaconda, or can be installed separately), or most editors and IDEs, to browse code. vscode things to know:\n",
    "\n",
    "- Command palette (<kbd>Ctrl-shift-p</kbd>)\n",
    "- Go to symbol (<kbd>Ctrl-t</kbd>)\n",
    "- Find references (<kbd>Shift-F12</kbd>)\n",
    "- Go to definition (<kbd>F12</kbd>)\n",
    "- Go back (<kbd>alt-left</kbd>)\n",
    "- View documentation\n",
    "- Hide sidebar (<kbd>Ctrl-b</kbd>)\n",
    "- Zen mode (<kbd>Ctrl-k,z</kbd>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unfreeze the weights and train some more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like a computer vision model, we can then unfreeze the model and fine-tune it.\n",
    "In this step we are allowing the model to update *all* the weights with values more suitable to the language of IMDb reviews. But we are mindful that the pretrained weights from wikitext-103 are likely already near their optimal values. For this we train the weights in the \"earlier\" layers with a much lower learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.985726</td>\n",
       "      <td>3.922798</td>\n",
       "      <td>0.282494</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.683273</td>\n",
       "      <td>3.900530</td>\n",
       "      <td>0.281920</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.131985</td>\n",
       "      <td>3.939293</td>\n",
       "      <td>0.280150</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(3, slice(1e-4,1e-2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test the language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate your language model, you can run the [`Learner.predict`](/basic_train.html#Learner.predict) method and specify the number of words you want it to guess."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is a review about India and is about bonanza ANY DEFEATED'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"This is a review about\", n_words=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In the hierarchy of horror movies this has to be near the top. This is much better , but there are more'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"In the hierarchy of horror movies this has to be near the top.\", n_words=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes the generated text doesn't make much sense because we have a tiny vocabulary and didn't train much. But note that the model respects basic grammar, which comes from the pretrained model.\n",
    "\n",
    "Finally we save the encoder to be able to use it for classification in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('mini_imdb_language_model')\n",
    "learn.save_encoder('mini_imdb_language_model_encoder')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Build a movie review classifier\n",
    "We use mixed precision (`.to_fp16()`)for greater speed, smaller memory footprint, and a regularizing effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RNNLearner(data=TextClasDataBunch;\n",
       "\n",
       "Train: LabelList (799 items)\n",
       "x: TextList\n",
       "xxbos xxmaj in theory , films should be a form of entertainment . xxmaj while this xxunk documentaries and other experimental forms of film - making ; most movies , specially genre films , must not only tell it 's story or message , they must entertain their target audience in some way . xxmaj all this just to say that in my opinion a bad movie is not a movie with low production values or low - budget , a bad movie is one that is boring . \n",
       " \n",
       "  \" xxmaj hellborn \" or \" xxmaj asylum of the xxmaj damned \" as is known in the xxup u.s. , is a bad movie simply because it is just not involving , and xxunk boring and tiresome . xxmaj while it has a very good premise , it is just poorly developed and the mediocre acting does n't make things better . xxmaj on another hands the film probably could had been a fine or even classic b - movie , but here it is just a bad attempt at film - making . \n",
       " \n",
       "  xxmaj director xxmaj philip xxup j. xxmaj jones tells the tale of xxmaj james xxmaj bishop ( xxmaj matt xxmaj stasi ) , a young xxunk resident , who just got his dream job at xxmaj st. xxmaj andrew xxmaj mental xxmaj hospital ; but the old asylum seems to hide a secret . xxmaj after the mysterious death of some patients and the constant rumors of satanic xxunk , xxmaj james decides to find out what is going on ; only to find the xxunk of his boss , xxmaj dr. xxunk ( xxmaj bruce xxmaj payne ) , who believes that xxmaj bishop is going as insane as his patients . \n",
       " \n",
       "  xxmaj while the premise is quite interesting , the execution of the film leaves a lot to be desired . xxmaj in an attempt of making a supernatural psychological thriller , xxmaj jones goes for the easy way out and makes a movie filled with every cliché of the genre . xxmaj of course , there are lots of great movies that are also filled with clichés ; but in \" xxmaj hellborn \" every single one is wasted and turned into a cheap jump scare to keep things moving , resulting in a boring and predictable storyline . \n",
       " \n",
       "  xxmaj the acting is quite mediocre for the most part , with one big exception : xxmaj bruce xxmaj payne gives a top - notch performance that makes the movie look xxunk of such good acting . xxmaj matt xxmaj stasi is very weak as the lead character and the rest of the cast make forgettable performances . \n",
       " \n",
       "  xxmaj despite all this flaws , one thing has to be written about \" xxmaj hellborn \" ; it has a visual look very good for the budget and very similar to modern day big - budget xxmaj xxunk \" horror \" productions . xxmaj also , the make - up and prosthetics are done very nicely and the designs for the main antagonist are quite good . xxmaj sadly , the rest of the xxmaj special xxmaj effects are awful and outdated , making a huge contrast with the make - up & prosthetics . \n",
       " \n",
       "  \" xxmaj hellborn \" is a movie with a few good things xxunk by its serious flaws with terrible results . xxmaj hardcore horror or b - movie fans may be interested by its premise but it is a boring and tiresome experience . 3 / 10,xxbos xxmaj in order to hold the public 's attention for three hours , we were treated not so much to a family 's romp through four generations and xxunk years of xxmaj hungarian history , as to sexual xxunk with a sister , a sister - in - law and other xxunk . xxmaj oh yes , there was also a totally gratuitous rape . xxmaj having said all this , the first story of the relationship among the children of the patriarch was fresh and sensual - thanks to xxmaj jennifer xxmaj xxunk .,xxbos xxmaj in the many films i have seen xxmaj warren xxmaj oates , i have come to a definite conclusion , here is one talented individual . i first saw xxmaj mr. xxmaj oates back in the 1960 's television series called xxmaj xxunk xxmaj burke . xxmaj from then on , i followed his career closely and felt he was destined for great roles . xxmaj that happened in 1974 , when xxmaj sam xxmaj peckinpah gave him top billing in a film called ' xxmaj bring xxmaj me the xxmaj head of xxmaj xxunk xxmaj garcia . ' xxmaj of course , his biggest claim to fame was his magnificent role in ' xxmaj the xxmaj wild xxmaj bunch ' . i have always thought he was quite able to bring any character a certain magic , that is until i saw him in this flop . xxmaj the movie is called \" xxmaj chandler \" , a tribute to the iron fisted detectives of the 1950 's created by xxmaj raymond xxmaj chandler . xxmaj because , the synopsis said it was about a hard nose xxmaj private xxmaj eye , i was immediately interested . xxmaj however , i sat patiently through the entire film and found it to be a dull , xxunk - interesting , slow pace , twisted , confusing saga which if it had a theme or plot must have been left on some dark back room self . xxmaj xxunk and with some of xxmaj hollywood 's best supporting stars , such as xxmaj alex xxmaj xxunk , xxmaj mitch xxmaj ryan , xxmaj gordon xxmaj xxunk , xxmaj charles xxunk , xxmaj richard xxmaj xxunk and xxmaj scatman xxmaj crothers , this movie had enough power to reach xxmaj xxunk five , however , it xxunk on the xxunk and went no where . xxmaj as a result , one of my favorite actor 's got stuck in a poorly made vehicle which never got off the ground . * *,xxbos xxmaj halfway through xxmaj xxunk xxmaj xxunk 's \" xxmaj evening , \" a woman on her deathbed asks a figure appearing in her hallucination : \" xxmaj can you tell me where my life went ? \" xxmaj the line could be embarrassingly theatrical , but the woman speaking it is xxmaj vanessa xxmaj redgrave , xxunk it with utter simplicity , and the question tears your heart out . \n",
       " \n",
       "  xxmaj time and again , the film based on xxmaj susan xxmaj xxunk 's novel skirts xxunk and xxunk , it holds attention , offers admirable performances , and xxunk emotional involvement as few recent movies have . xxmaj with only six months of the year gone , there are now two memorable , meaningful , worthwhile films in theaters , the other , of course , being xxmaj xxunk xxmaj xxunk 's \" xxmaj away from xxmaj her . \" xxmaj hollywood might have turned \" xxmaj evening \" into a xxunk celebrity vehicle with its two xxunk of real - life xxunk and daughters - xxmaj vanessa xxmaj redgrave and xxmaj natasha xxmaj xxunk , and xxmaj meryl xxmaj streep and xxmaj xxunk xxmaj xxunk . xxmaj xxunk is xxmaj redgrave 's daughter in the film ( with a sister played by xxmaj tony xxmaj xxunk ) , and xxmaj xxunk plays xxmaj streep 's younger self , while xxmaj redgrave 's youthful incarnation is xxmaj claire xxmaj danes . \n",
       " \n",
       "  xxmaj add xxmaj glenn xxmaj close , xxmaj eileen xxmaj atkins , xxmaj hugh xxmaj dancy , xxmaj patrick xxmaj wilson , and a large cast - yes , it could have turned into a multiple star platform . xxmaj instead , xxmaj xxunk - the brilliant xxmaj hungarian cinematographer of \" xxmaj xxunk , \" and director of \" xxmaj xxunk \" - created a subtle ensemble work with a \" xxmaj xxunk feel , \" the story taking place in a high - society xxmaj newport environment , in the days leading up to a wedding that is xxunk with trouble . \n",
       " \n",
       "  xxmaj missed connections , wrong choices , and xxunk xxunk with social and family xxunk present quite a soap opera , but the quality of the writing , xxmaj xxunk 's direction , and xxunk acting raise \" xxmaj evening \" way above that level , into the the xxunk air of xxmaj english , xxmaj french ( and a few xxmaj american ) family xxunk from a century before its contemporary setting . \n",
       " \n",
       "  xxmaj complex relationships between xxunk and daughters , between friends and lovers , with the addition of a difficult xxunk all come across clearly , xxunk , xxunk . xxmaj individual tunes are woven into a xxunk . \n",
       " \n",
       "  xxmaj and yet , with the all the xxunk emphasis on ensemble and xxunk performances , the stars of \" xxmaj evening \" still shine through , xxmaj redgrave , xxmaj xxunk , xxmaj xxunk ( an exciting new discovery , looking vaguely like her mother , but a very different actress ) , xxmaj danes carrying most of the load - until xxmaj streep shows up in the final moments and , of course , steals the show . xxmaj dancy and xxmaj wilson are well worth the price of xxunk too . \n",
       " \n",
       "  xxmaj as with \" xxmaj away from xxmaj her , \" \" xxmaj evening \" stays with you at length , xxunk a re - thinking its story and characters , and re - experiencing the emotions it raises . xxmaj at two hours , the film runs a bit long , but the way it stays with you xxunk is welcome among the many movies that go cold long before your popcorn .,xxbos i think the biggest failing something can have is to be boring . xxmaj bad is actually better than boring . xxmaj this thing has no breath . xxmaj it does have the interesting fact of taking place in xxmaj cambodia . xxmaj how many xxmaj american made films of the 30 's take place in xxmaj cambodia . xxmaj nevertheless , the conflict there is a little hard to figure out . xxmaj even the xxunk movements are a little confusing . xxmaj what drags it to a resounding xxunk is the love story . xxmaj why those two guys are so totally transfixed on the xxunk blonde i do n't know . i thought he should continue to use his xxmaj zombies ( such as they are ) and forget all about her . xxmaj the movie just plods along . xxmaj the perfect xxunk is where one of the principle characters follows a xxmaj xxunk priest through the water to get to the secret place where the xxunk ( or whatever ) that explain how to turn people into zombies is kept . i thought they would never get there . xxmaj one guy takes two steps . xxmaj he stops . xxmaj he looks around . xxmaj the other guy hides behind some columns . xxmaj he takes two steps . xxmaj he stops . xxmaj he looks around . xxmaj the other guy hides behind some xxunk . xxmaj this is the movie in a nutshell . xxmaj then there is the bad acting and insipid dialogue . i actually have a lot of patience when it comes to b movies . xxmaj this one is insufferable . xxmaj by the way , a more xxunk title would be xxmaj revolt of the xxmaj hypnotized .\n",
       "y: CategoryList\n",
       "negative,negative,negative,positive,negative\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb_sample;\n",
       "\n",
       "Valid: LabelList (201 items)\n",
       "x: TextList\n",
       "xxbos xxmaj it 's true that you always remember what you were doing at a point when disaster or tragedy strikes . xxmaj and none more so that xxmaj september 11 , 2001 , a date which changed the entire global landscape in its fight against terrorism . \n",
       " \n",
       "  xxmaj no , this documentary did n't set out to be dwelling on the events leading to 9 / 11 . xxmaj rather , the filmmakers , brothers xxmaj xxunk and xxmaj jules xxmaj xxunk , set out to do a documentary on the trials and xxunk of a rookie xxmaj new xxmaj york firefighter . xxmaj they had gone to the academy and done some shoots of training , and had xxunk their \" xxunk \" ( xxunk firefighter ) to join them in an xxup xxunk xxunk , home to xxmaj ladder 1 and xxmaj engine 7 . xxmaj but their production was to develop and contain at that time , believed to be the only shot of the first plane slamming into the xxmaj world xxmaj trace xxmaj center . \n",
       " \n",
       "  i was traveling back with a friend on the train from a night of xxup xxunk gaming , and received a call at about xxunk local time from my xxmaj dad , who informed me of the above . xxmaj few minutes later , he told me there was another , and that the xxup wtc was under attack . xxmaj by the time i arrived home , the upper floors of the twin xxunk were xxunk and in smoke , and to my horror , they collapsed , under an hour . \n",
       " \n",
       "  xxmaj the filmmakers had two cameras running that day , one who had followed a team out on a routine call , and which immediately xxunk to the xxup wtc upon hearing and seeing the plane crash into it . xxmaj we follow what is possible the only filmed sequence of events in the xxunk of xxup xxunk where the first xxunk of xxunk , xxunk , and police had to make sense of what happened , and to quickly develop a plan of action . xxmaj the other camera , held by the other brother , was making his way to xxup wtc to look for his xxunk , and along the journey , captured the many expressions of xxmaj new xxmaj yorkers , as well as the sense of chaos in and around xxmaj xxunk . \n",
       " \n",
       "  xxmaj xxunk throughout the documentary are numerous interviews with the men from xxmaj ladder 1 and xxmaj engine 7 , which miraculously , did not suffer any casualty . xxmaj but being survivors also brought about its own set of psychological turmoil , as they struggle to come to terms with the event . xxmaj through the events that unfold , we learn of the strong xxunk amongst these men who risk live and xxunk each day on their jobs , to save lives . \n",
       " \n",
       "  xxmaj we began with what the documentary was supposed to be , before events of the day totally xxunk in and became the focus , right up to the rescue phase where hopes of finding survivors under the rubble were kept alive by the men who work round the clock in making sense of the collapsed steel xxunk . xxmaj it 's not a film that is xxunk , and what you see here can not be recreated in any other documentary ( and xxunk , not sound stages for xxmaj hollywood xxunk ) . xxmaj it 's as close as you can get to that day , witnessing the event up close , from safety . \n",
       " \n",
       "  xxmaj code 1 xxup dvd contains a separate extra hour of 4 sets of interviews with the men of xxmaj ladder 1 and xxmaj engine 7 .,xxbos xxmaj sometimes you need to see a bad movie just to appreciate the good ones . xxmaj well , that 's my opinion anyway . xxmaj this one will always be in the bad movie category , simply because all but xxmaj shu xxmaj qi 's performance was terrible . \n",
       " \n",
       "  xxmaj martial xxmaj angel tells of xxmaj cat ( xxmaj shu xxmaj qi ) , a professional thief turned straight after leaving her lover , xxmaj chi xxmaj lam ( xxmaj xxunk xxmaj xxunk ) , two years before . xxmaj but her past returns to haunt her as xxmaj chi xxmaj lam is kidnapped for the ransom of security software belonging to the company xxmaj cat works for . xxmaj in order to rescue him , she calls on her old friends from her orphanage days , six other xxunk women , to save the day ... \n",
       " \n",
       "  i may have told the synopsis xxunk , but this is a cheesy story . xxmaj in fact , the whole script and direction lacked any quality at all . xxmaj much of the dialogue was meaningless and coupled with a plot that was as thin as rice - paper in water . xxmaj if i could sum it up , take a bad xxmaj jackie xxmaj chan movie , remove the comedy , remove the choreography , throw away the budget , and you have xxmaj martial xxmaj angels : a formulaic piece of work with no imagination at all . \n",
       " \n",
       "  xxmaj mind you , i do have to give credit where credit 's due , and xxmaj shu xxmaj qi was probably the only person to emerge unscathed from the terrible action , as it was her performance that xxunk through . xxmaj okay , you ca n't say she was excellent - after all she had absolutely nothing to work with - but she did manage to dig some character out from her role . xxmaj other than that , only xxmaj sandra xxmaj xxunk and xxmaj kelly xxmaj lin made any other impression - although these were mostly xxunk and very brief . \n",
       " \n",
       "  xxmaj elsewhere , the film just fell to pieces . xxmaj scenes and dialogue were completely unnatural and unbelievable , special effects were obviously done on the cheap with no attempt to clean up edges between persons and the mask of the blue screen , poor editing involving numerous xxunk in fight scenes , camera angles that were elementary and xxunk , and direction i 've seen better from a lost dog . \n",
       " \n",
       "  i guess this film was a too many xxunk affair . xxmaj most probably , the budget was blown away on the over - enthusiasm to have seven xxunk on the same silver screen . xxmaj that did n't leave much else . \n",
       " \n",
       "  xxmaj frankly , the way this film was made was like a cheap porn movie without the porn . xxmaj charlie 's xxmaj angels , it ai n't . xxmaj in fact , while sisters can do it for themselves , none of that was really that apparent here . \n",
       " \n",
       "  xxmaj definitely one to forget .,xxbos a xxunk - remembered melodrama  thanks xxunk to xxmaj xxunk xxmaj xxunk 's fine xxmaj oscar - winning central performance  about an xxunk - treated theme : the nature of acting and how it can xxunk one 's perception of reality . xxmaj in this case , we have a well - known xxunk tackling xxmaj shakespeare 's \" xxmaj othello \" , so that the film 's last third delves effectively into the thriller genre  with press agent xxmaj edmond o'brien ( who happens to really be xxunk with xxmaj xxunk 's co - star and ex - wife xxmaj xxunk xxmaj xxunk ) ' xxunk ' the actor 's possible involvement in the xxmaj xxunk - like xxunk of a celebrity - seeking waitress ( a very slim xxmaj xxunk xxmaj winters ) . xxmaj the theatrical / xxmaj new xxmaj york atmosphere of the immediate post - war era is vividly captured by the husband - and - wife screen writing team of xxmaj xxunk xxmaj xxunk and xxmaj xxunk xxmaj gordon and legendary \" actor 's director \" xxmaj george xxmaj xxunk ( all of whom were recognized by the xxmaj academy with nominations ) ; incidentally , the film xxunk a second xxmaj oscar for xxmaj xxunk xxmaj xxunk 's xxunk score . xxmaj xxunk , forever the suave leading man blessed besides with a xxunk voice , does well enough by xxmaj shakespeare  xxunk conviction the xxunk his character xxunk into obsessive jealousy , a murderous rage and , eventually , paranoia ; however , he is not let down by a supporting cast which also includes director xxmaj ray xxmaj collins , reporter xxmaj xxunk xxmaj mitchell , detective xxmaj joe xxmaj xxunk and xxunk xxmaj whit xxmaj xxunk . xxmaj though the mid - section is a bit xxunk , the film makes up for any xxunk with a remarkably - handled xxmaj xxunk xxunk .,xxbos xxmaj oh if only i could give this rubbish less than one star ! xxmaj there were two mildly amusing parts in the whole film and that is it ! one was where a line or two from the song xxmaj do n't xxmaj worry , xxmaj be xxmaj happy was xxunk by the slugs and the other was where xxmaj roddy fell of the toilet roll and landed with his feet and legs apart so that everything else he landed on on the way down hit him in the groin . xxmaj that is it there was nothing more amusing than that , at least not for me anyway ! xxmaj xxunk is not right in saying ' xxmaj fans of the completely terrible \" xxmaj shrek \" might enjoy , but \" xxmaj wallace & xxmaj gromit \" fans will probably turn away in xxunk . ' xxmaj as i loved xxmaj shrek 1 2 and 3 and i also love xxmaj wallace and xxmaj gromit . xxmaj you see what it boils down to is that if an animation is done extremely well then it is definitely worth watching , this however was about as far from done well as you can possibly get ! xxmaj the continuity mistakes were too big in number . xxmaj some were pointed out by the makers of this site others were not . i wo n't point out all of the others , but here are a few more to see : xxmaj when the young daughter leaves at the start of the film the catch to the cage door comes down and the hook part of it that is on the right clearly goes back around behind the round xxunk thus effectively making sure xxmaj roddy would not be able to get out and yet he does just by simply kicking at it . xxmaj at one point the ruby falls down xxmaj roddy 's back and gets pushed straight up into the the air by xxmaj rita all the while the ship is moving forwards . xxmaj in the next scene xxmaj roddy has caught it again . xxmaj this is impossible . xxmaj seeing as how the ship is moving forwards the only place when the ruby was xxunk out from under the back of xxmaj roddy 's shirt the only place it could have landed was in the water not in xxmaj roddy 's hand . xxmaj there was a third one i wanted to point out but for now i have forgotten it . \n",
       " \n",
       "  xxmaj too many , for want of a better word , ' jokes ' were repeated in one way or another , there was not enough time to establish any sort of connection with any of the characters , the characters were xxunk , shallow and empty , and the whole film left you wanting xxrep 4 . wanting to watch 85 minutes of anything else ! xxmaj paint xxunk or grass growing are two superb xxunk !,xxbos xxmaj one of the most appealing elements of a xxmaj gilliam film is that the well - xxunk visuals , the unsettling backdrops , and the manically frustrated characters are evidence of the creator 's involvement . xxmaj instead of most movies ( where the filmmaker is some director - for - hire that is paid to feature a star or two ) , you can feel xxmaj terry xxmaj gilliam 's presence through the experience . \" 12 xxmaj monkeys \" is evidence of xxmaj gilliam 's own vision and style , as opposed to making offbeat movies for their own sake . \" 12 xxmaj monkeys \" is a xxunk on similar themes of xxmaj gilliam 's xxunk : oppressive / recessive societies , the solitude of the protagonist , the frustration associated with disbelief , and parallel realms . xxmaj in this film xxmaj gilliam does a fine job of xxunk lines between the two realms , using xxunk to force the audience to believe rather than know . xxmaj this tendency for xxmaj gilliam to neglect to fill in certain gaps leads to xxunk of art - house pretentiousness . xxmaj the difference between xxmaj gilliam and artsy xxunk is that xxmaj gilliam 's choices clearly have a purpose and all of his images have meaning . xxmaj the two nearly identical xxunk scenes of xxmaj cole in the beginning are meant to draw comparisons which leave the audience xxunk . xxmaj his bald head is a mark of xxunk in the disease - ridden future world , yet makes him recognizable in the 1996 world . xxmaj the title itself is a mark of xxmaj gilliam 's creativity , as it requires the majority of the story to flesh out for its meaning to be fully understood . xxmaj all in all , xxmaj gilliam 's xxunk to making creative films that are interesting to watch yet also require thought and interpretation from the audience . xxmaj the film has immense re - watch value , since there are subtle details and hints that can be missed upon the first viewing . xxmaj definitely one of my favorites .\n",
       "y: CategoryList\n",
       "positive,negative,positive,negative,positive\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb_sample;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): MultiBatchEncoder(\n",
       "    (module): AWD_LSTM(\n",
       "      (encoder): Embedding(8992, 400, padding_idx=1)\n",
       "      (encoder_dp): EmbeddingDropout(\n",
       "        (emb): Embedding(8992, 400, padding_idx=1)\n",
       "      )\n",
       "      (rnns): ModuleList(\n",
       "        (0): WeightDropout(\n",
       "          (module): LSTM(400, 1152, batch_first=True)\n",
       "        )\n",
       "        (1): WeightDropout(\n",
       "          (module): LSTM(1152, 1152, batch_first=True)\n",
       "        )\n",
       "        (2): WeightDropout(\n",
       "          (module): LSTM(1152, 400, batch_first=True)\n",
       "        )\n",
       "      )\n",
       "      (input_dp): RNNDropout()\n",
       "      (hidden_dps): ModuleList(\n",
       "        (0): RNNDropout()\n",
       "        (1): RNNDropout()\n",
       "        (2): RNNDropout()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (1): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=2, bias=True)\n",
       "    )\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x000001E4A1319D08>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb_sample'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: ...\n",
       "alpha: 2.0\n",
       "beta: 1.0, MixedPrecision\n",
       "learn: ...\n",
       "loss_scale: 65536\n",
       "max_noskip: 1000\n",
       "dynamic: True\n",
       "clip: None\n",
       "flat_master: False\n",
       "max_scale: 16777216\n",
       "loss_fp32: True], layer_groups=[Sequential(\n",
       "  (0): Embedding(8992, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(8992, 400, padding_idx=1)\n",
       "  )\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 1152, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1152, 400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2, inplace=False)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace=True)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1, inplace=False)\n",
       "      (6): Linear(in_features=50, out_features=2, bias=True)\n",
       "    )\n",
       "  )\n",
       ")], add_time=True, silent=False)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5).to_fp16()\n",
    "learn.load_encoder('mini_imdb_language_model_encoder')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj raising xxmaj victor xxmaj vargas : a xxmaj review \\n \\n  xxmaj you know , xxmaj raising xxmaj victor xxmaj vargas is like sticking your hands into a big , steaming bowl of xxunk . xxmaj it 's warm and gooey , but you 're not sure if it feels right . xxmaj try as i might , no matter how warm and gooey xxmaj raising xxmaj</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj this film sat on my xxmaj tivo for weeks before i watched it . i dreaded a self - indulgent xxunk flick about relationships gone bad . i was wrong ; this was an xxunk xxunk into the screwed - up xxunk of xxmaj new xxmaj yorkers . \\n \\n  xxmaj the format is the same as xxmaj max xxmaj xxunk ' \" xxmaj la xxmaj ronde</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj many neglect that this is n't just a classic due to the fact that it 's the first xxup 3d game , or even the first xxunk - up . xxmaj it 's also one of the first stealth games , one of the xxunk definitely the first ) truly claustrophobic games , and just a pretty well - rounded gaming experience in general . xxmaj with graphics</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos \\n \\n  i 'm sure things did n't exactly go the same way in the real life of xxmaj xxunk xxmaj xxunk as they did in the film adaptation of his book , xxmaj xxunk xxmaj boys , but the movie \" xxmaj october xxmaj sky \" ( an xxunk of the book 's title ) is good enough to stand alone . i have not read xxmaj</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj to review this movie , i without any doubt would have to quote that memorable scene in xxmaj tarantino 's \" xxmaj pulp xxmaj fiction \" ( xxunk ) when xxmaj jules and xxmaj vincent are talking about xxmaj mia xxmaj wallace and what she does for a living . xxmaj jules tells xxmaj vincent that the \" xxmaj only thing she did worthwhile was pilot \" .</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train the last layer of the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(1, 1e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unfreeze the classifier model and fine-tune"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.592132</td>\n",
       "      <td>0.516068</td>\n",
       "      <td>0.771144</td>\n",
       "      <td>01:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.454031</td>\n",
       "      <td>0.321474</td>\n",
       "      <td>0.870647</td>\n",
       "      <td>01:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.338722</td>\n",
       "      <td>0.323645</td>\n",
       "      <td>0.850746</td>\n",
       "      <td>01:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(3, slice(1e-4, 1e-2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test the classifier\n",
    "We can use our model to predict on a few example of movie review-like raw text by using the [`Learner.predict`](/basic_train.html#Learner.predict) method. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model is 70% sure that this is a positive review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\cross-entropy\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category positive, tensor(1), tensor([0.3028, 0.6972]))"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict('Although there was lots of blood and violence, I did not think this film was scary enough.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model is 83% sure that this is a positive review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\cross-entropy\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category positive, tensor(1), tensor([0.1734, 0.8266]))"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict('Not so good World War II epic film')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bottom line: the model did not do a good job, misclassifying both reviews (with high confidence, to boot!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's train some more, and re-try these examples to see if we can do better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.263950</td>\n",
       "      <td>0.500555</td>\n",
       "      <td>0.825871</td>\n",
       "      <td>01:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.226157</td>\n",
       "      <td>0.297020</td>\n",
       "      <td>0.860696</td>\n",
       "      <td>01:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.163577</td>\n",
       "      <td>0.298125</td>\n",
       "      <td>0.855721</td>\n",
       "      <td>01:04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(3, slice(1e-4, 1e-2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Re-try the above examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\cross-entropy\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category negative, tensor(0), tensor([0.9879, 0.0121]))"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict('Although there was lots of blood and violence, I did not think this film was scary enough.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\cross-entropy\\Anaconda3\\envs\\fastai\\lib\\site-packages\\fastai\\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input\n",
      "  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Category negative, tensor(0), tensor([0.7353, 0.2647]))"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict('Not so good World War II epic film')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model seems to have benefitted from the extra training, since it now correctly classifies both reviews as `negative`, with high confidence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make up your own text in the style of a movie review. Then use our classifier to see what it thinks!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jekyll": {
   "keywords": "fastai",
   "summary": "Application to NLP, including ULMFiT fine-tuning",
   "title": "text"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}