8-2-Word2vec-(gensim).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec (gensim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on [Dive Into NLTK, Part X: Play with Word2Vec Models based on NLTK Corpus by TextMiner](http://textminingonline.com/dive-into-nltk-part-x-play-with-word2vec-models-based-on-nltk-corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Exploring the `gutenburg` corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Project Gutenberg (PG) is a volunteer effort to digitize and archive cultural works. Most of the items in its collection are full texts of public domain books."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Project Gutenberg Selections http://gutenberg.net/  This corpus contains etexts from from Project Gutenberg, by the following authors:  * Jane Austen (3) * William Blake (2) * Thornton W. Burgess * Sarah Cone Bryant * Lewis Carroll * G. K. Chesterton (3) * Maria Edgeworth * King James Bible * Herman Melville * John Milton * William Shakespeare (3) * Walt Whitman  The beginning of the body of each book could not be identified automatically, so the semi-generic header of each file has been removed, and included below. Some source files ended with a line \"End of The Project Gutenberg Etext...\", and this has been deleted.  Information about Project Gutenberg (one page)  We produce about two million dollars for each hour we work.  The fifty hours is one conservative estimate for how long it we take to get any etext selected, entered, proofread, edited, copyright searched and analyzed, the copyright letters written, etc.  This projected audience is one hundred million readers.  If our value per text is nominally estimated at one dollar, then we produce 2 million dollars per hour this year we, will have to do four text files per month:  thus upping our productivity from one million. The Goal of Project Gutenberg is to Give Away One Trillion Etext Files by the December 31, 2001.  [10,000 x 100,000,000=Trillion] This is ten thousand titles each to one hundred million readers, which is 10% of the expected number of computer users by the end of the year 2001.  We need your donations more than ever!  All donations should be made to \"Project Gutenberg/IBC\", and are tax deductible to the extent allowable by law (\"IBC\" is Illinois Benedictine College).  (Subscriptions to our paper newsletter go to IBC, too)  For these and other matters, please mail to:  Project Gutenberg P. O. Box  2782 Champaign, IL 61825  When all other email fails try our Michael S. Hart, Executive Director: hart@vmd.cso.uiuc.edu (internet)   hart@uiucvmd   (bitnet)  We would prefer to send you this information by email (Internet, Bitnet, Compuserve, ATTMAIL or MCImail).  ****** If you have an FTP program (or emulator), please FTP directly to the Project Gutenberg archives: [Mac users, do NOT point and click. . .type]  ftp mrcnext.cso.uiuc.edu login:  anonymous password:  your@login cd etext/etext91 or cd etext92 or cd etext93 [for new books]  [now also in cd etext/etext93] or cd etext/articles [get suggest gut for more information] dir [to see files] get or mget [to get files. . .set bin for zip files] get INDEX100.GUT get INDEX200.GUT for a list of books and get NEW.GUT for general information and mget GUT* for newsletters.  **Information prepared by the Project Gutenberg legal advisor** (Three Pages)   ***START**THE SMALL PRINT!**FOR PUBLIC DOMAIN ETEXTS**START*** Why is this \"Small Print!\" statement here?  You know: lawyers. They tell us you might sue us if there is something wrong with your copy of this etext, even if you got it for free from someone other than us, and even if what\\'s wrong is not our fault.  So, among other things, this \"Small Print!\" statement disclaims most of our liability to you.  It also tells you how you can distribute copies of this etext if you want to.  *BEFORE!* YOU USE OR READ THIS ETEXT By using or reading any part of this PROJECT GUTENBERG-tm etext, you indicate that you understand, agree to and accept this \"Small Print!\" statement.  If you do not, you can receive a refund of the money (if any) you paid for this etext by sending a request within 30 days of receiving it to the person you got it from.  If you received this etext on a physical medium (such as a disk), you must return it with your request.  ABOUT PROJECT GUTENBERG-TM ETEXTS This PROJECT GUTENBERG-tm etext, like most PROJECT GUTENBERG- tm etexts, is a \"public domain\" work distributed by Professor Michael S. Hart through the Project Gutenberg Association at Illinois Benedictine College (the \"Project\").  Among other things, this means that no one owns a United States copyright on or for this work, so the Project (and you!) can copy and distribute it in the United States without permission and without paying copyright royalties.  Special rules, set forth below, apply if you wish to copy and distribute this etext under the Project\\'s \"PROJECT GUTENBERG\" trademark.  To create these etexts, the Project expends considerable efforts to identify, transcribe and proofread public domain works.  Despite these efforts, the Project\\'s etexts and any medium they may be on may contain \"Defects\".  Among other things, Defects may take the form of incomplete, inaccurate or corrupt data, transcription errors, a copyright or other intellectual property infringement, a defective or damaged disk or other etext medium, a computer virus, or computer codes that damage or cannot be read by your equipment.  LIMITED WARRANTY; DISCLAIMER OF DAMAGES But for the \"Right of Replacement or Refund\" described below, [1] the Project (and any other party you may receive this etext from as a PROJECT GUTENBERG-tm etext) disclaims all liability to you for damages, costs and expenses, including legal fees, and [2] YOU HAVE NO REMEDIES FOR NEGLIGENCE OR UNDER STRICT LIABILITY, OR FOR BREACH OF WARRANTY OR CONTRACT, INCLUDING BUT NOT LIMITED TO INDIRECT, CONSEQUENTIAL, PUNITIVE OR INCIDENTAL DAMAGES, EVEN IF YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH DAMAGES.  If you discover a Defect in this etext within 90 days of receiving it, you can receive a refund of the money (if any) you paid for it by sending an explanatory note within that time to the person you received it from.  If you received it on a physical medium, you must return it with your note, and such person may choose to alternatively give you a replacement copy.  If you received it electronically, such person may choose to alternatively give you a second opportunity to receive it electronically.  THIS ETEXT IS OTHERWISE PROVIDED TO YOU \"AS-IS\".  NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, ARE MADE TO YOU AS TO THE ETEXT OR ANY MEDIUM IT MAY BE ON, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.  Some states do not allow disclaimers of implied warranties or the exclusion or limitation of consequential damages, so the above disclaimers and exclusions may not apply to you, and you may have other legal rights.  INDEMNITY You will indemnify and hold the Project, its directors, officers, members and agents harmless from all liability, cost and expense, including legal fees, that arise directly or indirectly from any of the following that you do or cause: [1] distribution of this etext, [2] alteration, modification, or addition to the etext, or [3] any Defect.  DISTRIBUTION UNDER \"PROJECT GUTENBERG-tm\" You may distribute copies of this etext electronically, or by disk, book or any other medium if you either delete this \"Small Print!\" and all other references to Project Gutenberg, or:  [1]  Only give exact copies of it.  Among other things, this      requires that you do not remove, alter or modify the      etext or this \"small print!\" statement.  You may however,      if you wish, distribute this etext in machine readable      binary, compressed, mark-up, or proprietary form,      including any form resulting from conversion by word pro-      cessing or hypertext software, but only so long as      *EITHER*:       [*]  The etext, when displayed, is clearly readable, and           does *not* contain characters other than those           intended by the author of the work, although tilde           (~), asterisk (*) and underline (_) characters may           be used to convey punctuation intended by the           author, and additional characters may be used to           indicate hypertext links; OR       [*]  The etext may be readily converted by the reader at           no expense into plain ASCII, EBCDIC or equivalent           form by the program that displays the etext (as is           the case, for instance, with most word processors);           OR       [*]  You provide, or agree to also provide on request at           no additional cost, fee or expense, a copy of the           etext in its original plain ASCII form (or in EBCDIC           or other equivalent proprietary form).  [2]  Honor the etext refund and replacement provisions of this      \"Small Print!\" statement.  [3]  Pay a trademark license fee to the Project of 20% of the      net profits you derive calculated using the method you      already use to calculate your applicable taxes.  If you      don\\'t derive profits, no royalty is due.  Royalties are      payable to \"Project Gutenberg Association / Illinois      Benedictine College\" within the 60 days following each      date you prepare (or were legally required to prepare)      your annual (or equivalent periodic) tax return.  WHAT IF YOU *WANT* TO SEND MONEY EVEN IF YOU DON\\'T HAVE TO? The Project gratefully accepts contributions in money, time, scanning machines, OCR software, public domain etexts, royalty free copyright licenses, and every other sort of contribution you can think of.  Money should be paid to \"Project Gutenberg Association / Illinois Benedictine College\".  This \"Small Print!\" by Charles B. Kramer, Attorney Internet (72600.2026@compuserve.com); TEL: (212-254-5093) *END*THE SMALL PRINT! FOR PUBLIC DOMAIN ETEXTS*Ver.04.29.93*END* '"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import gutenberg\n",
    "gutenberg.readme().replace('\\n', ' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['austen-emma.txt',\n",
       " 'austen-persuasion.txt',\n",
       " 'austen-sense.txt',\n",
       " 'bible-kjv.txt',\n",
       " 'blake-poems.txt',\n",
       " 'bryant-stories.txt',\n",
       " 'burgess-busterbrown.txt',\n",
       " 'carroll-alice.txt',\n",
       " 'chesterton-ball.txt',\n",
       " 'chesterton-brown.txt',\n",
       " 'chesterton-thursday.txt',\n",
       " 'edgeworth-parents.txt',\n",
       " 'melville-moby_dick.txt',\n",
       " 'milton-paradise.txt',\n",
       " 'shakespeare-caesar.txt',\n",
       " 'shakespeare-hamlet.txt',\n",
       " 'shakespeare-macbeth.txt',\n",
       " 'whitman-leaves.txt']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gutenberg.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30103"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bible_kjv_sents = gutenberg.sents('bible-kjv.txt')\n",
    "len(bible_kjv_sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Implementing Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['in',\n",
       " 'the',\n",
       " 'beginning',\n",
       " 'god',\n",
       " 'created',\n",
       " 'the',\n",
       " 'heaven',\n",
       " 'and',\n",
       " 'the',\n",
       " 'earth']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from string import punctuation\n",
    "\n",
    "discard_punctuation_and_lowercased_sents = [[word.lower() for word in sent if word not in punctuation and word.isalpha()] \n",
    "                                            for sent in bible_kjv_sents]\n",
    "discard_punctuation_and_lowercased_sents[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5279"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models import Word2Vec\n",
    "\n",
    "bible_kjv_word2vec_model = word2vec.Word2Vec(discard_punctuation_and_lowercased_sents, min_count=5, size=200)\n",
    "bible_kjv_word2vec_model.save('bible_word2vec_gensim')\n",
    "# model = Word2Vec.load(fname) # To load a model\n",
    "word_vectors = bible_kjv_word2vec_model.wv\n",
    "del bible_kjv_word2vec_model # When we finish training the model, we can only delete it and keep the word vectors.\n",
    "word_vectors.save_word2vec_format('bible_word2vec_org', 'bible_word2vec_vocabulary')\n",
    "len(word_vectors.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('truth', 0.7849754095077515),\n",
       " ('hosts', 0.7561447620391846),\n",
       " ('lord', 0.7557143568992615),\n",
       " ('christ', 0.739787220954895),\n",
       " ('spirit', 0.7170284986495972),\n",
       " ('salvation', 0.7115912437438965),\n",
       " ('faith', 0.707336962223053),\n",
       " ('glory', 0.6949458122253418),\n",
       " ('hope', 0.6832618117332458),\n",
       " ('mercy', 0.6805936098098755)]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(['god']) # Most similar as in closest in the word graph. Word2vec is essentially about proportions of word occurrences in relations holding in general over large corpora of text. Consider word analogy ‘man is to woman as king is to X’ which was famously demonstrated in word2vec. The algorithm is able to come up with an answer queen, almost magically by simple vector differences. The main idea, called distributional hypothesis, is that similar words appear in similar contexts of words around them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('earth', 0.7416267991065979),\n",
       " ('heavens', 0.7133727073669434),\n",
       " ('sea', 0.6633787155151367)]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(['heaven'], topn=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('daughter', 0.6277151703834534)]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('queen', 0.9797199964523315)]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The `_cosmul` variant uses a slightly-different comparison when using multiple positive/negative examples (such as when asking about analogies). One paper has shown it does better:\n",
    "word_vectors.most_similar_cosmul(positive=['woman', 'king'], negative=['man'], topn=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7557145774555314"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.similarity('lord', 'god')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'food'"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.doesnt_match(\"lord god salvation food spirit\".split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'Word2VecKeyedVectors' object has no attribute 'score'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-59-33f7c0c837f1>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# Probability of a text under the model\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mword_vectors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mscore\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"In the beginning\"\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: 'Word2VecKeyedVectors' object has no attribute 'score'"
     ]
    }
   ],
   "source": [
    "# Probability of a text under the model\n",
    "# bible_kjv_word2vec_model.score([\"In the beginning\".split()]) # Doesn't work for 2 reasons: 1. I deleted the model. 2. It has only been implemented for models made with certain arguments as of the time of writing."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}