notebooks/1_Load_model_and_translate_baseline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you will learn:\n",
    "1. [Vocabulary: how to use](#voc)\n",
    "2. [Load trained model](#load_model)\n",
    "3. [Translate sentences](#translate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Before starting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's add path to the the-story-of-heads repo to the path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, 'path_to_good_translation_wrong_in_context') # insert your local path to the repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary <a name=\"voc\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load a model, you need to pass vocabularies used in training. Let's load the vocabularies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import numpy as np\n",
    "\n",
    "DATA_PATH = # insert your path\n",
    "VOC_PATH =  # insert your path\n",
    "\n",
    "inp_voc = pickle.load(open(VOC_PATH + 'src.voc', 'rb'))\n",
    "out_voc = pickle.load(open(VOC_PATH + 'dst.voc', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What you can do with a vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get ids of tokens in the vocabulary, as well as tokens corresponding to ids:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7, 253, 11, 1162]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inp_voc.ids(\"i saw a cat\".split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"'s\", 'by', '6', 'knack']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inp_voc.words([12, 123, 1234, 12345])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reserved token ids are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inp_voc.ids(['_BOS_', '_EOS_', '_UNK_'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`_BOS_` - begin of sentence token; not used in the standard setting\n",
    "\n",
    "`_EOS_` - end of sentence token; this is the last token of any sentence\n",
    "\n",
    "`_UNK_` - unknown token; if you are using BPE, you probably won't see it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load model <a name=\"load_model\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import liblaries and create session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: CUDA_VISIBLE_DEVICES=1\n"
     ]
    }
   ],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "\n",
    "import tensorflow as tf\n",
    "import lib\n",
    "import lib.task.seq2seq.models.transformer as tr\n",
    "\n",
    "tf.reset_default_graph()\n",
    "gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.99, allow_growth=True)\n",
    "sess = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, copy model hyperparameters from your training config. In this notebook, we'll use model with pruned encoder self-attention heads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "hp = {\n",
    "     \"num_layers\": 6,\n",
    "     \"num_heads\": 8,\n",
    "     \"ff_size\": 2048,\n",
    "     \"ffn_type\": \"conv_relu\",\n",
    "     \"hid_size\": 512,\n",
    "     \"emb_size\": 512,\n",
    "     \"res_steps\": \"nlda\", \n",
    "    \n",
    "     \"rescale_emb\": True,\n",
    "     \"inp_emb_bias\": True,\n",
    "     \"normalize_out\": True,\n",
    "     \"share_emb\": False,\n",
    "     \"replace\": 0,\n",
    "    \n",
    "     \"relu_dropout\": 0.1,\n",
    "     \"res_dropout\": 0.1,\n",
    "     \"attn_dropout\": 0.1,\n",
    "     \"label_smoothing\": 0.1,\n",
    "    \n",
    "     \"translator\": \"ingraph\",\n",
    "     \"beam_size\": 4,\n",
    "     \"beam_spread\": 3,\n",
    "     \"len_alpha\": 0.6,\n",
    "     \"attn_beta\": 0,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can load the model. Pass vocs and hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tr.Model('mod', inp_voc, out_voc, inference_mode='fast', **hp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_ckpt = # insert path to the final checkpoint\n",
    "var_list = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)\n",
    "lib.train.saveload.load(path_to_ckpt, var_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate <a name=\"translate\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load test set\n",
    "path_to_testset = # path to your data\n",
    "test_src = open(path_to_testset + 'test.src').readlines()\n",
    "test_dst = open(path_to_testset + 'test.dst').readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To translate, just pass a list of sentence to the `translate_lines` function of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"otherwise , he 'll tell them the truth .\\n\",\n",
       " \"there 's evidence of early ul `cer `ation .\\n\",\n",
       " \"in my experience , if you really don 't want to worry about something , you lock it into a cage ...\\n\"]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_src[2:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['о , это ужасно , чувак .', 'другой ?', 'иначе он расскажет им правду .']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.translate_lines(test_src[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To translate a test set, just do this for a sequence of batches (50-100 sentences is ok).\n",
    "\n",
    "**Do not forget to unbpe your translations before evaluating BLEU score!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unbpe(sent):\n",
    "    return sent.replace(' `', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "я видел голод `ного кота .\n",
      "я видел голодного кота .\n"
     ]
    }
   ],
   "source": [
    "print(model.translate_lines(['i saw a hungry cat'])[0])\n",
    "print(unbpe(model.translate_lines(['i saw a hungry cat'])[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}