notebooks/2_Load_model_and_translate_CADec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In this notebook, you will learn how to translate text fragments with CADec\n",
    "\n",
    "If you don't know how to load a model or how to operate with vocabularies yet, look in the notebook __1_Load_model_and_translate_baseline__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, 'path_to_good_translation_wrong_in_context') # insert your local path to the repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load vocabularies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import numpy as np\n",
    "\n",
    "DATA_PATH = # insert your path\n",
    "VOC_PATH =  # insert your path\n",
    "\n",
    "inp_voc = pickle.load(open(VOC_PATH + 'src.voc', 'rb'))\n",
    "out_voc = pickle.load(open(VOC_PATH + 'dst.voc', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: CUDA_VISIBLE_DEVICES=1\n",
      "MODEL: use_dst_ctx  True\n",
      "reusing...\n",
      "DEC2: use_dst_ctx True\n"
     ]
    }
   ],
   "source": [
    "%env CUDA_VISIBLE_DEVICES=0\n",
    "\n",
    "import tensorflow as tf\n",
    "import lib\n",
    "import lib.task.seq2seq.cadec.model as tr\n",
    "\n",
    "tf.reset_default_graph()\n",
    "gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.99, allow_growth=True)\n",
    "sess = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))\n",
    "\n",
    "hp = {\n",
    "     \"num_layers\": 6,\n",
    "     \"num_heads\": 8,\n",
    "     \"ff_size\": 2048,\n",
    "     \"ffn_type\": \"conv_relu\",\n",
    "     \"hid_size\": 512,\n",
    "     \"emb_size\": 512,\n",
    "     \"res_steps\": \"nlda\", \n",
    "    \n",
    "     \"rescale_emb\": True,\n",
    "     \"inp_emb_bias\": True,\n",
    "     \"normalize_out\": True,\n",
    "     \"share_emb\": False,\n",
    "     \"replace\": 0,\n",
    "    \n",
    "     \"relu_dropout\": 0.1,\n",
    "     \"res_dropout\": 0.1,\n",
    "     \"attn_dropout\": 0.1,\n",
    "     \"label_smoothing\": 0.1,\n",
    "    \n",
    "     \"translator\": \"ingraph\",\n",
    "     \"beam_size\": 4,\n",
    "     \"beam_spread\": 3,\n",
    "     \"len_alpha\": 0.6,\n",
    "     \"attn_beta\": 0,\n",
    "    \n",
    "     \"dec1_attn_mode\": \"rdo_and_emb\",\n",
    "     \"share_loss\": False,\n",
    "     \"decoder2_name\": \"decoder2\",\n",
    "     \"max_ctx_sents\": 3,\n",
    "     \"use_dst_ctx\": True,\n",
    "}\n",
    "\n",
    "model = tr.CADecModel('mod', inp_voc, out_voc, inference_mode='lazy', **hp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_ckpt = # insert path to the final checkpoint\n",
    "var_list = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)\n",
    "lib.train.saveload.load(path_to_ckpt, var_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate <a name=\"translate\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load test set\n",
    "path_to_testset = # path to your data\n",
    "test_src = open(path_to_testset + 'test.src').readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are groups of consecutive sentences in our test set (sentences are separated with the `_eos` token):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"someone broke into my house last night and stole that ugly shirt . _eos yeah ... _eos i don 't believe that . _eos oh , it 's bo `xy on me , dude .\\n\",\n",
       " 'is she in danger ? _eos no . _eos things are different there . _eos different ?\\n',\n",
       " \"it can 't be you . _eos but he 's not dead . _eos well , he 'll have to be , right ? _eos otherwise , he 'll tell them the truth .\\n\"]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_src[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To translate group of sentences, CADec has to translate one sentence at a time, relying on previously generated translations (starting from baseline translation for the first sentence, which does not have context)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def num_sents(text):\n",
    "    return len(text.split(' _eos '))\n",
    "\n",
    "def get_first_n_sents(lines, N):\n",
    "    \"\"\" Take groups of first N sentences from ' _eos '-separated fragments \"\"\"\n",
    "    return [' _eos '.join(line.split(' _eos ')[:N]) for line in lines]\n",
    "\n",
    "def expand_groups(groups, sents):\n",
    "    \"\"\"Expand groups of sentences with new sentences\"\"\"\n",
    "    assert len(groups) == len(sents), \"Numbers of groups and new sentences do not match\"\n",
    "    return [group.strip() + ' _eos ' + sent for group, sent in zip(groups, sents)]\n",
    "\n",
    "def make_cadec_batch_data(src_lines, dst_lines):\n",
    "    \"\"\"\n",
    "    src_lines contain groups of N sentences, last of which is to be translated (' _eos '-separated)\n",
    "    dst_lines contain translations of the first N-1 sentences from src_lines (' _eos '-separated)\n",
    "    \"\"\"\n",
    "    assert len(src_lines) == len(dst_lines), \"Different number of text fragments\"\n",
    "    batch = []\n",
    "    for src, dst in zip(src_lines, dst_lines):\n",
    "        assert num_sents(src) == num_sents(dst) + 1\n",
    "        src_elems = src.strip().split(' _eos ')\n",
    "        batch.append(' _eos '.join([src_elems[-1]] + src_elems[:-1]) + ' _eos_eos ' + dst)\n",
    "    return batch\n",
    "\n",
    "def translate_batch(src_lines):\n",
    "    # produce translation of first sentences in groups using base model\n",
    "    current_translations = model.model1.translate_lines(get_first_n_sents(src_lines, N=1))\n",
    "    # consecutively translate sentences using previously translated context sentences\n",
    "    for iter_ in range(1, num_sents(src_lines[0])):\n",
    "        batch_data = make_cadec_batch_data(src_lines=get_first_n_sents(src_lines, iter_ + 1),\n",
    "                                           dst_lines=current_translations)\n",
    "        new_translations = model.translate_lines(batch_data)\n",
    "        current_translations = expand_groups(current_translations, new_translations)\n",
    "    return current_translations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new batch sample\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['кто-то вломился в мой дом прошлой ночью и украл эту урод `ливую рубашку . _eos да ... _eos я в это не верю . _eos о , это ужасно , чувак .',\n",
       " 'она в опасности ? _eos нет . _eos там все по-другому . _eos по-другому ?',\n",
       " 'это не можешь быть ты . _eos но он не умер . _eos ну , он должен быть , верно ? _eos иначе он расскажет им правду .']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translate_batch(test_src[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To translate a test set, just do this for a sequence of batches (50-100 sentences is ok)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Similar to the baseline, do not forget to unbpe your translations before evaluating BLEU score!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how different are translations generated by CADec from the ones generated by base model.\n",
    "\n",
    "Let's take one of the text examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is she in danger ? _eos no . _eos things are different there . _eos different ?\\n'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_src[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here what we get translating these sentences in isolation using baseline model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'она в опасности ? _eos нет . _eos там все по-другому . _eos другой ?'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' _eos '.join(model.model1.translate_lines(test_src[1].split(' _eos ')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Translations of the word `different` are not consistent, but have to be - in clarification questions we repeat the form of a word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__она в опасности ? \\_eos нет . \\_eos там все <span style=\"color:blue\">по-другому</span> . \\_eos <span style=\"color:red\">другой</span> ?__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the translation using CADec:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['она в опасности ? _eos нет . _eos там все по-другому . _eos по-другому ?']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translate_batch(test_src[1:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This translation is consistent!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__она в опасности ? \\_eos нет . \\_eos там все <span style=\"color:blue\">по-другому</span> . \\_eos <span style=\"color:blue\">по-другому</span> ?__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}