Multilingual Embeddings - 1. Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilingual Embeddings - 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word embeddings are one of the main contributors to the success of deep learning in Natural Language Processing. They help models capitalize on semantic knowledge that cannot be learned from their training data, but that can be inferred from other corpora. Even if your training data contains only one instance of the words _cat_ and _dog_, word embeddings trained on a different corpus can tell the model the two words refer to similar creatures. However, because word embeddings are usually trained on monolingual corpora, they cannot be compared across languages. Pretrained monolingual embeddings may tell you that _cat_ is semantically similar to _dog_, but not that English _cat_ is similar to French _chat_. This is where multilingual embeddings come in. In this and the following two notebooks, we will explore the concept of multilingual embeddings and how they can be used in practice. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monolingual word embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most people will be familiar with monolingual embeddings. When you're working with one of the major Western languages, and English in particular, you can easily get several sets of word embeddings. These have typically been obtained with different methods: there is word2vec, GloVe, FastText, etc. In this notebook, we'll work with the well-known FastText embeddings. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to work with the FastText pretrained word embeddings you can download from [Github](https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md). We download the pretrained embeddings for Dutch and French, store them in the directory FASTTEXT_PATH, and load them with [Gensim](https://radimrehurek.com/gensim/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from gensim.models import KeyedVectors\n",
    "\n",
    "FASTTEXT_PATH = os.path.expanduser(\"~/data/fastText/\")\n",
    "\n",
    "dutch_embedding_path = os.path.join(FASTTEXT_PATH, \"nl/wiki.nl.vec\")\n",
    "french_embedding_path = os.path.join(FASTTEXT_PATH, \"fr/wiki.fr.vec\")\n",
    "\n",
    "nl_model = KeyedVectors.load_word2vec_format(dutch_embedding_path)\n",
    "fr_model = KeyedVectors.load_word2vec_format(french_embedding_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check if everything works as expected. In the Dutch monolingual space, the word _kat_ (cat) has as nearest neighbors _katten_ (cats), _huiskat_ (house cat), _huiskatten_ (house cats) and _hond_ (dog). In the French monolingual space, the word _chat_ (cat) has as nearest neighbors _chats_ (katten), _chien_ (dog), _félin_ (feline) and _chatte_ (female cat). That all looks fine. \n",
    "\n",
    "Now, if we take the embedding of the Dutch word _kat_ and look for its nearest neighbors in the French monolingual space, we get a list of foreign words that have nothing to do with cats. The same thing happens if we take the French embedding of _chat_ and look for its neighbors in the Dutch space. So far so good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dutch words most similar to 'kat' (cat)\n",
      "[('katten', 0.6762495040893555), ('huiskat', 0.6694309711456299), ('huiskatten', 0.6053364276885986), ('hond', 0.6051813364028931), ('rexkatten', 0.6009929180145264), ('konijn', 0.5963096618652344), ('lapjeskat', 0.5929849147796631), ('viskat', 0.5845541954040527), ('konijntje', 0.5832188129425049), ('wezelkat', 0.5829440355300903)]\n",
      "\n",
      "French words most similar to 'chat' (cat)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('chats', 0.712058424949646), ('chien', 0.663030207157135), ('félin', 0.6281734108924866), ('chatte', 0.6235890984535217), ('lapin', 0.6211234927177429), ('viverrin', 0.5997868776321411), ('chat»', 0.5940316915512085), ('chaton', 0.5848442912101746), ('miaule', 0.583796501159668), ('blaireau', 0.5823991894721985)]\n",
      "\n",
      "French words most similar to Dutch 'kat' (cat)\n",
      "[('bhale', 0.24561069905757904), ('şale', 0.24260175228118896), ('jugaad', 0.24040547013282776), ('siddique', 0.23643596470355988), ('hansraj', 0.23435617983341217), ('sanjay', 0.2309851050376892), ('manoj', 0.2307053506374359), ('gwalior', 0.2288607954978943), ('jugântar', 0.2285393476486206), ('maxil', 0.22787001729011536)]\n",
      "\n",
      "Dutch words most similar to French 'chat' (cat)\n",
      "[('sourds', 0.2693006694316864), ('sourdis', 0.2681083679199219), ('bussotti', 0.2659595012664795), ('protti', 0.26567795872688293), ('perls', 0.258177250623703), ('würtz', 0.2545265555381775), ('proksch', 0.2543639838695526), ('dutrochet', 0.2540545165538788), ('cecchi', 0.25333812832832336), ('tecchi', 0.25160640478134155)]\n"
     ]
    }
   ],
   "source": [
    "print(\"Dutch words most similar to 'kat' (cat)\")\n",
    "print(nl_model.most_similar(\"kat\"))\n",
    "print(\"\")\n",
    "print(\"French words most similar to 'chat' (cat)\")\n",
    "print(fr_model.most_similar(\"chat\"))\n",
    "print(\"\")\n",
    "print(\"French words most similar to Dutch 'kat' (cat)\")\n",
    "print(fr_model.similar_by_vector(nl_model.get_vector(\"kat\")))\n",
    "print(\"\")\n",
    "print(\"Dutch words most similar to French 'chat' (cat)\")\n",
    "print(nl_model.similar_by_vector(fr_model.get_vector(\"chat\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From monolingual to cross-lingual word embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "People have come up with many ways to obtain cross-lingual embeddings. Sebastian Ruder has written [an excellent blog post](http://ruder.io/cross-lingual-embeddings/) about this topic. One common way to get cross-lingual embeddings is to simply train them on a multilingual corpus. Unfortunately such corpora can be hard to collect, and very few such embeddings are available for download. Therefore it is more common to start from two (or more) monolingual embedding spaces and find a transformation that will project one space into the other. This is the approach taken by Samuel L. Smith and colleagues in [their ICLR paper from 2017](https://arxiv.org/abs/1702.03859). Their code is available on [Github](https://github.com/Babylonpartners/fastText_multilingual). Let's start by downloading it.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'fastText_multilingual' already exists and is not an empty directory.\r\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/Babylonpartners/fastText_multilingual.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now reload the Dutch and French monolingual word embeddings with the fastText_multingual package, so that we can use its multilingual functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reading word vectors from /Users/yvespeirsman/data/fastText/nl/wiki.nl.vec\n",
      "reading word vectors from /Users/yvespeirsman/data/fastText/fr/wiki.fr.vec\n"
     ]
    }
   ],
   "source": [
    "from fastText_multilingual.fasttext import FastVector\n",
    "\n",
    "nl_dictionary = FastVector(vector_file=dutch_embedding_path)\n",
    "fr_dictionary = FastVector(vector_file=french_embedding_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `translate_nearest_neighbour` method is similar to Gensim's `similar_by_vector`: given an embedding matrix, it finds the embedding that is most similar to another given embedding. With this method we can replicate the \"translation\" experiment we did above, by looking up the French nearest neighbor to Dutch _kat_, and the Dutch nearest neighbor to French _chat_. The results are indeed the same nonsense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "French translation of Dutch 'kat'\n",
      "bhale\n",
      "\n",
      "Dutch translation of French 'chat'\n",
      "sourds\n"
     ]
    }
   ],
   "source": [
    "print(\"French translation of Dutch 'kat'\")\n",
    "print(fr_dictionary.translate_nearest_neighbour(nl_dictionary[\"kat\"]))\n",
    "print(\"\")\n",
    "print(\"Dutch translation of French 'chat'\")\n",
    "print(nl_dictionary.translate_nearest_neighbour(fr_dictionary[\"chat\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Together with the Python code, fastText_multilingual also contains the alignment matrices that Smith et al. learned. These alignment matrices can be used to project the monolingual FastText embeddings of 78 languages in one and the same embedding space, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "nl_dictionary.apply_transform('fastText_multilingual/alignment_matrices/nl.txt')\n",
    "fr_dictionary.apply_transform('fastText_multilingual/alignment_matrices/fr.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we now rerun the translation experiment above, we suddenly find Dutch _kat_ and French _chat_ translate into each other! Thanks to the projection we just applied, their embeddings have been aligned and ended up very close to each other in our newly created multilingual embedding space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "French translation of Dutch 'kat'\n",
      "chat\n",
      "\n",
      "Dutch translation of French 'chat'\n",
      "kat\n"
     ]
    }
   ],
   "source": [
    "print(\"French translation of Dutch 'kat'\")\n",
    "print(fr_dictionary.translate_nearest_neighbour(nl_dictionary[\"kat\"]))\n",
    "print(\"\")\n",
    "print(\"Dutch translation of French 'chat'\")\n",
    "print(nl_dictionary.translate_nearest_neighbour(fr_dictionary[\"chat\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word translation\n",
    "\n",
    "One of the obvious applications of these cross-lingual word embeddings is simple word translation. This is typically also a main task to test their effectiveness in research papers. To evaluate our newly obtained cross-lingual embeddings, we're going to compare their translations to a those in a translation dictionary. Unfortunately, dowloadable Dutch-French translation dictionaries are hard to find, so we'll use a hack to create our own. We'll download the Dutch-English and English-French translation dictionaries from Facebook's Muse, a project we'll come back to later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-08-10 11:40:33--  https://s3.amazonaws.com/arrival/dictionaries/nl-en.txt\n",
      "Resolving s3.amazonaws.com... 52.216.165.237\n",
      "Connecting to s3.amazonaws.com|52.216.165.237|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1560709 (1.5M) [text/plain]\n",
      "Saving to: '/tmp/nl-en.txt'\n",
      "\n",
      "/tmp/nl-en.txt      100%[=====================>]   1.49M   958KB/s   in 1.6s   \n",
      "\n",
      "2018-08-10 11:40:35 (958 KB/s) - '/tmp/nl-en.txt' saved [1560709/1560709]\n",
      "\n",
      "--2018-08-10 11:40:35--  https://s3.amazonaws.com/arrival/dictionaries/en-fr.txt\n",
      "Resolving s3.amazonaws.com... 52.216.165.237\n",
      "Connecting to s3.amazonaws.com|52.216.165.237|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1857759 (1.8M) [text/plain]\n",
      "Saving to: '/tmp/en-fr.txt'\n",
      "\n",
      "/tmp/en-fr.txt      100%[=====================>]   1.77M  1.19MB/s   in 1.5s   \n",
      "\n",
      "2018-08-10 11:40:37 (1.19 MB/s) - '/tmp/en-fr.txt' saved [1857759/1857759]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://s3.amazonaws.com/arrival/dictionaries/nl-en.txt -O /tmp/nl-en.txt\n",
    "!wget https://s3.amazonaws.com/arrival/dictionaries/en-fr.txt -O /tmp/en-fr.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then merge the Dutch-English and English-French translation dictionaries into a Dutch-French dictionary. To do this, we take all Dutch words that have just one English translation (to avoid problems with ambiguity), and take the French translations of this English term as the French translations of the original Dutch word. This is obviously a hack, but it should work well enough for our purposes. The collected French translations of _fiets_ (bike) and _auto_ (car) indeed show it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception ignored in: <object repr() failed>\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 887, in __del__\n",
      "    self.close()\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm_notebook.py\", line 210, in close\n",
      "    super(tqdm_notebook, self).close(*args, **kwargs)\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 1098, in close\n",
      "    if self.disable:\n",
      "AttributeError: 'tqdm_notebook' object has no attribute 'disable'\n",
      "Exception ignored in: <object repr() failed>\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 887, in __del__\n",
      "    self.close()\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm_notebook.py\", line 210, in close\n",
      "    super(tqdm_notebook, self).close(*args, **kwargs)\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 1098, in close\n",
      "    if self.disable:\n",
      "AttributeError: 'tqdm_notebook' object has no attribute 'disable'\n",
      "Exception ignored in: <object repr() failed>\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 887, in __del__\n",
      "    self.close()\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm_notebook.py\", line 210, in close\n",
      "    super(tqdm_notebook, self).close(*args, **kwargs)\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 1098, in close\n",
      "    if self.disable:\n",
      "AttributeError: 'tqdm_notebook' object has no attribute 'disable'\n",
      "Exception ignored in: <object repr() failed>\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 887, in __del__\n",
      "    self.close()\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm_notebook.py\", line 210, in close\n",
      "    super(tqdm_notebook, self).close(*args, **kwargs)\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 1098, in close\n",
      "    if self.disable:\n",
      "AttributeError: 'tqdm_notebook' object has no attribute 'disable'\n",
      "Exception ignored in: <object repr() failed>\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 887, in __del__\n",
      "    self.close()\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm_notebook.py\", line 210, in close\n",
      "    super(tqdm_notebook, self).close(*args, **kwargs)\n",
      "  File \"/Users/yvespeirsman/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py\", line 1098, in close\n",
      "    if self.disable:\n",
      "AttributeError: 'tqdm_notebook' object has no attribute 'disable'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "French translations of Dutch 'fiets' (bike): ['bicyclette', 'vélo']\n",
      "French translations of Dutch 'auto' (car): ['voitures', 'voiture']\n"
     ]
    }
   ],
   "source": [
    "def read_translations(dictionary_file): \n",
    "    translations = {}\n",
    "    with open(dictionary_file) as i:\n",
    "        for line in i:\n",
    "            line = line.strip().split()\n",
    "            if len(line) == 2:\n",
    "                source, target = line\n",
    "                if source not in translations:\n",
    "                    translations[source] = []\n",
    "                translations[source].append(target)\n",
    "    return translations            \n",
    "    \n",
    "nl_en = read_translations(\"/tmp/nl-en.txt\")\n",
    "en_fr = read_translations(\"/tmp/en-fr.txt\")\n",
    "\n",
    "nl_fr = {}\n",
    "for word in nl_en: \n",
    "    if len(nl_en[word]) == 1 and nl_en[word][0] in en_fr:\n",
    "        nl_fr[word] = en_fr[nl_en[word][0]]\n",
    "        \n",
    "print(\"French translations of Dutch 'fiets' (bike):\", nl_fr[\"fiets\"])\n",
    "print(\"French translations of Dutch 'auto' (car):\", nl_fr[\"auto\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test the translations, we're going to restrict ourselves to some of the most frequent words in Dutch and French. We do this because the embeddings of infrequent words can often be unreliable. [Luminoso's wordfreq library](https://github.com/LuminosoInsight/wordfreq) is a great tool for looking up word frequencies. We use it to find the most frequent 1,000 words for Dutch and French, and then take the items in our dictionary where the Dutch target word is among the most frequent 1,000 Dutch words, and at least one of its French translations is among the most frequent 1,000 French words. These words must also be present in the multilingual embedding space. This results in a translation dictionary with 332 items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of filtered dictionary: 332\n"
     ]
    }
   ],
   "source": [
    "from wordfreq import top_n_list\n",
    "\n",
    "dutch_frequent_words = set(top_n_list('nl', 1000))\n",
    "french_frequent_words = set(top_n_list('fr', 1000))\n",
    "\n",
    "nl_fr_filtered = {}\n",
    "for word in nl_fr:\n",
    "    if word in dutch_frequent_words and word in nl_dictionary:\n",
    "        translations = [w for w in nl_fr[word] if w in french_frequent_words and w in fr_dictionary]\n",
    "        if len(translations) > 0:\n",
    "            nl_fr_filtered[word] = translations\n",
    "            \n",
    "print(\"Size of filtered dictionary:\", len(nl_fr_filtered))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fasttext_multilingual gives us two methods for word translation. The first, `translate_nearest_neighbour`, simply finds the nearest neighbor to a source embedding. This method is not ideal, however. In practice, it does not deal well with \"hubs\": target words that are nearest neighbours to many different source words. The second method, `translate_inverted_softmax`, therefore introduces the \"inverted softmax\". This inverted softmax estimates the probability that a candidate target word translates into the source word rather than the other way around. \n",
    "\n",
    "We test both methods on a small random set of words from our Dutch-French dictionary. This is because the translation process takes quite a bit of time, particularly with the inverted softmax. The simple method gives us 45% correct translations, the method with the inverted softmax gets 55% of the translations right. In contrat to the first, it is able to translation Dutch _link_ (\"link\") to French _lien_, and _eeuw_ (\"century\") to _siècle_. This difference is obviously not statistically significant with such a small test set, but it suggests the inverted softmax might indeed deal with frequent translations better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source: help Translations: ['aider', 'aide'] Neighbor: wikireader/moyen\n",
      "Source: seizoen Translations: ['saison'] Neighbor: saison\n",
      "Source: nog Translations: ['encore', 'toujours'] Neighbor: encore\n",
      "Source: speelt Translations: ['joue'] Neighbor: joue\n",
      "Source: achter Translations: ['derrière'] Neighbor: derrière\n",
      "Source: nationale Translations: ['national', 'nationale'] Neighbor: nationale\n",
      "Source: oktober Translations: ['octobre'] Neighbor: mai\n",
      "Source: maakt Translations: ['rend'] Neighbor: constitue\n",
      "Source: thuis Translations: ['maison'] Neighbor: domicile…\n",
      "Source: link Translations: ['lien'] Neighbor: source/référence\n",
      "Source: liever Translations: ['plutôt'] Neighbor: préférerais\n",
      "Source: helft Translations: ['moitié', 'demi'] Neighbor: moitié\n",
      "Source: werden Translations: ['devenu'] Neighbor: furent\n",
      "Source: wel Translations: ['bien'] Neighbor: effectivement\n",
      "Source: eeuw Translations: ['siècle'] Neighbor: xvsiècle\n",
      "Source: geschiedenis Translations: ['histoire'] Neighbor: histoire\n",
      "Source: vijf Translations: ['cinq'] Neighbor: cinq\n",
      "Source: situatie Translations: ['situation'] Neighbor: situation\n",
      "Source: klopt Translations: ['vrai'] Neighbor: ۷\n",
      "Source: eerste Translations: ['première', 'premières', 'premier', 'premiers'] Neighbor: deuxième\n",
      "Translation accuracy: 0.45\n",
      "Source: help Translations: ['aider', 'aide'] Neighbor: wikireader/moyen\n",
      "Source: seizoen Translations: ['saison'] Neighbor: saison\n",
      "Source: nog Translations: ['encore', 'toujours'] Neighbor: encore\n",
      "Source: speelt Translations: ['joue'] Neighbor: joue\n",
      "Source: achter Translations: ['derrière'] Neighbor: derrière\n",
      "Source: nationale Translations: ['national', 'nationale'] Neighbor: nationale\n",
      "Source: oktober Translations: ['octobre'] Neighbor: mai\n",
      "Source: maakt Translations: ['rend'] Neighbor: constitue\n",
      "Source: thuis Translations: ['maison'] Neighbor: domicile…\n",
      "Source: link Translations: ['lien'] Neighbor: lien\n",
      "Source: liever Translations: ['plutôt'] Neighbor: préfére\n",
      "Source: helft Translations: ['moitié', 'demi'] Neighbor: moitié\n",
      "Source: werden Translations: ['devenu'] Neighbor: furent\n",
      "Source: wel Translations: ['bien'] Neighbor: effectivement\n",
      "Source: eeuw Translations: ['siècle'] Neighbor: siècle\n",
      "Source: geschiedenis Translations: ['histoire'] Neighbor: histoire\n",
      "Source: vijf Translations: ['cinq'] Neighbor: cinq\n",
      "Source: situatie Translations: ['situation'] Neighbor: situation\n",
      "Source: klopt Translations: ['vrai'] Neighbor: effectivement\n",
      "Source: eerste Translations: ['première', 'premières', 'premier', 'premiers'] Neighbor: deuxième\n",
      "Translation accuracy with inverted softmax: 0.55\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "random.seed(42)\n",
    "\n",
    "def test_translation(source_words, target_words, source_embeddings, target_embeddings, inverted_softmax=False):\n",
    "    correct, total = 0, 0\n",
    "    for (source_word, translations) in zip(source_words, target_words):\n",
    "        source_embedding = source_embeddings[source_word]\n",
    "        if inverted_softmax: \n",
    "            translation = target_embeddings.translate_inverted_softmax(source_embedding, target_embeddings, 1000)\n",
    "        else:\n",
    "            translation = target_embeddings.translate_nearest_neighbour(source_embedding)\n",
    "        total += 1\n",
    "        if translation in translations:\n",
    "            correct += 1\n",
    "        print(\"Source:\", source_word, \"Translations:\", translations, \"Neighbor:\", translation)\n",
    "\n",
    "    return correct/total\n",
    "\n",
    "source_words = random.sample(nl_fr_filtered.keys(), 20)\n",
    "target_words = [nl_fr_filtered[word] for word in source_words]\n",
    "\n",
    "print(\"Translation accuracy:\", \n",
    "      test_translation(source_words, target_words, nl_dictionary, fr_dictionary))\n",
    "print(\"Translation accuracy with inverted softmax:\", \n",
    "      test_translation(source_words, target_words, nl_dictionary, fr_dictionary, inverted_softmax=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## False friends"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another nice application of these cross-lingual embeddings is their ability to identify false friends: words that look the same in two languages, but refer to different things. Such words are often problematic for foreign language learners. \n",
    "\n",
    "To test this ability, we again focus on the most frequent words in Dutch and French, according to the wordfreq library. We get the most frequent 10,000 words for Dutch and French, and then take the intersection between the two sets. This gives us 1276 words that occur pretty frequently in both French and Dutch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Dutch words: 10000\n",
      "Number of French words: 10000\n",
      "Number of shared words: 1276\n"
     ]
    }
   ],
   "source": [
    "from wordfreq import top_n_list\n",
    "\n",
    "dutch_words = set(top_n_list('nl', 10000))\n",
    "french_words = set(top_n_list('fr', 10000))\n",
    "\n",
    "shared_words = dutch_words & french_words\n",
    "\n",
    "print(\"Number of Dutch words:\", len(dutch_words))\n",
    "print(\"Number of French words:\", len(french_words))\n",
    "print(\"Number of shared words:\", len(shared_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of these words, we now calculate the similarity between their Dutch word embedding and their French word embedding. The idea is that words that have the same meaning in the two languages should have very similar word embeddings. False friends, words with very different meanings, should have very different word embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarities = []\n",
    "\n",
    "for word in shared_words:\n",
    "    if word in nl_dictionary and word in fr_dictionary and len(word) > 3:\n",
    "        similarity = FastVector.cosine_similarity(nl_dictionary[word], fr_dictionary[word])\n",
    "        similarities.append((similarity, word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test this hypothesis. We sort the similarities from small to large and take a look at the words at the two ends of the sorted list. \n",
    "\n",
    "The words with the highest cosine similarity are indeed all words with the same meaning in the two languages. They are often loan words from French into Dutch (such as _actrice_ \"actress\", _documentaire_ \"documentary\" or _restaurants_ \"restaurants), proper nouns that refer to the same entity (such as _Manchester_) or words with a shared historical origin (such as _biologie_ \"biology\" and _psychologie_ \"psychology\").\n",
    "\n",
    "The words with the lowest cosine similarity are indeed so-called false friends: despite their shared form in Dutch and French, they refer to completely different things. _Vaste_ means \"fixed\" in Dutch, but \"wide\" in French. _Boom_ means \"tree\" in Dutch, but \"boom\" in French. _Verre_ means \"far\" in Dutch and \"glass\" in French. _Lever_ means \"liver\" in Dutch, but \"raise\" in French. And so on. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True friends (high cosine similarity)\n",
      "[(0.7875397109131733, 'actrice'), (0.7902184128816384, 'documentaire'), (0.7904528410190746, 'restaurants'), (0.7922850597730441, 'auteurs'), (0.7937544526088117, 'concert'), (0.7945851950088083, 'interview'), (0.7980688657076762, 'instrument'), (0.8031223692726318, 'biologie'), (0.8042250198180378, 'ambassadeur'), (0.8072858739487832, 'islam'), (0.8074252681067334, 'manchester'), (0.8102706474813349, 'commandant'), (0.8124044204929419, 'album'), (0.8134099892600042, 'technologie'), (0.8135201369227627, 'officier'), (0.814204860407724, 'psychologie'), (0.8163966606522406, 'fans'), (0.8252393914782329, 'films'), (0.8326753383612543, 'acteur'), (0.8365625218234078, 'film')]\n",
      "\n",
      "False friends (low cosine similarity)\n",
      "[(0.10846536918019045, 'vaste'), (0.13416176852198267, 'boom'), (0.1402775948027539, 'verre'), (0.1505242316510172, 'lever'), (0.15905747382968102, 'onze'), (0.16334072302175257, 'arme'), (0.1694520705878885, 'tien'), (0.17851932912044016, 'trad'), (0.18798783228516985, 'appels'), (0.18993067791295393, 'aller'), (0.19564612512171123, 'dans'), (0.2157812793839099, 'vers'), (0.21612249298374175, 'dure'), (0.21924987879471217, 'vitesse'), (0.22956338485476627, 'over'), (0.23079769666223593, 'verte'), (0.23106403395856018, 'indien'), (0.23153370577178273, 'appel'), (0.23480694270360247, 'vies'), (0.23828676003392493, 'branche')]\n"
     ]
    }
   ],
   "source": [
    "similarities.sort()\n",
    "\n",
    "print(\"True friends (high cosine similarity)\")\n",
    "print(similarities[-20:])\n",
    "print(\"\")\n",
    "print(\"False friends (low cosine similarity)\")\n",
    "print(similarities[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Monolingual word embeddings can be transformed to multilingual word embeddings quite effectively. Our simple experiments with word translation and false friend identification showed that projecting monolingual word embeddings into a multilingual space indeed leads to similar embeddings for semantically similar words from different languages. While the experiments in this notebook were toy examples, we'll move to more practical applications in the next two notebooks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}