W8-2 Word Vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aerial-referral",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Preliminaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ahead-return",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def norm_vec(v):\n",
    "    mag = np.linalg.norm(v)\n",
    "    if mag == 0:\n",
    "        return v\n",
    "    return v / np.linalg.norm(v)\n",
    "\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "def normalize_rows(x):\n",
    "    return normalize(x, axis=1)\n",
    "\n",
    "def normalize_columns(x):\n",
    "    return normalize(x, axis=0)\n",
    "\n",
    "def check_float(potential_float):\n",
    "    try:\n",
    "        float(potential_float)\n",
    "        return True\n",
    "    except ValueError:\n",
    "        return False\n",
    "\n",
    "def round_if_float(v, prec=3):\n",
    "    if check_float(v):\n",
    "        return round(float(v), prec)\n",
    "    return v\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "def list_table(the_list, color_nums=False):\n",
    "    html = [\"<table style= 'border: 1px solid black; display:inline-block'>\"]\n",
    "    for row in the_list:\n",
    "        html.append(\"<tr>\")\n",
    "        for col in row:\n",
    "            if color_nums and check_float(col) and not float(col) == 0:\n",
    "                html.append(\"<td align='left' style='border: .5px solid gray; color: {1}; font-weight: bold'>{0}</td>\".format(round_if_float(col), color_nums))\n",
    "            else:\n",
    "                html.append(\"<td align='left' style='border: .5px solid gray;'>{0}</td>\".format(round_if_float(col)))\n",
    "        html.append(\"</tr>\")\n",
    "    html.append(\"</table>\")\n",
    "    return display(HTML(''.join(html)))\n",
    "\n",
    "def show_labeled_table(mat, col_names=None, row_names=None, nrows=10, ncols=10, color_nums=\"red\"):\n",
    "    sml = mat[:nrows, :ncols]\n",
    "    if col_names is not None:\n",
    "        sml = np.vstack([col_names[:ncols], sml])\n",
    "    if row_names is not None:\n",
    "        rnames = [[p] for p in row_names[:nrows]]\n",
    "        if col_names is not None:\n",
    "            new_col = np.array([[\"_\"]] + rnames)\n",
    "        else:\n",
    "            new_col = np.array(rnames)\n",
    "        sml = np.hstack((new_col, sml))\n",
    "    return list_table(sml, color_nums)\n",
    "\n",
    "def compute_doc_vector(tdoc, vocab):\n",
    "    return np.array([tdoc.count(w) for w in vocab])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "brutal-instrument",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Problems and solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "middle-offset",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There are a number of problems with our approach to getting document vectors. When our vocabulary is large, we will have many, many dimensions. This can get unwieldy.\n",
    "\n",
    "Even more problematic, many blocks of text will have a dot product of zero, even when they have a similar meaning, since they are using different words to say the same thing\n",
    "\n",
    "One way to think about the source of this problem is that we don't really have any way in which we have captured the meaning of words. For example, \"towards\" and \"closer\" probably have related meanings. But if one block of text uses \"towards\" and another \"closer,\" then our vectors for the two documents will have zero dot product."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "listed-drawing",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Word vectors to the rescue\n",
    "\n",
    "* A solution to both of these problems is to come up with a way of representing the words as vectors.\n",
    "* Words with similar meanings with will correspond to similar vectors.\n",
    "* We will get the vectors for documents by combining the vectors for words.\n",
    "* Coming up with good ways of finding the vectors for words is a place where a lot of recent work has focused."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "painted-memorabilia",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### A \"training\" or auxiliary corpus\n",
    "\n",
    "One way to do this, is to use some sort of auxiliary corpus to create a set of word vectors. Then apply those word vectors to your research data. That's what we'll do now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "monetary-browser",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Word vectors for the seasons"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "applied-angel",
   "metadata": {},
   "source": [
    "I went around the internet and collected a hundred or so text documents that use words that are related to the seasons. We'll use that to create some word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "frequent-radio",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Load the training corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adjacent-briefs",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "fname = 'corpora/seasons_training.txt'\n",
    "f = open(fname)\n",
    "raw = f.read().lower()\n",
    "whole_training_docs = re.findall(r\"<text>([\\s\\S]*?)</text>\", raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "basic-literacy",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(whole_training_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unavailable-irrigation",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So there are 196 training documents - essentially the text of web pages.\n",
    "\n",
    "We are going to split this up into paragraphs so that we have smaller contexts for words. We'll end up with over 3000 paragraphs.\n",
    "\n",
    "I'm also create a list called `para_names` that will contain a list of simple names for these paragraphs. You'll see where we'll use these below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "amazing-scanner",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_docs = []\n",
    "para_names = []\n",
    "for i, d in enumerate(whole_training_docs):\n",
    "    new_docs = d.split(\"\\n\\n\")\n",
    "    training_docs += new_docs\n",
    "    new_names = [\"d{}p{}\".format(i, p) for p in range(len(new_docs))]\n",
    "    para_names += new_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "valuable-rider",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(training_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "protective-carbon",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Tokenize the training documents\n",
    "\n",
    "Now we tokenize the 3000+ training documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "constant-catalog",
   "metadata": {},
   "outputs": [],
   "source": [
    "from seasons_module import seasons_tokenize\n",
    "tokenized_training_docs = []\n",
    "for doc in training_docs:\n",
    "    tdoc = seasons_tokenize(doc)\n",
    "    tokenized_training_docs.append(tdoc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "disciplinary-bunch",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## From tokenized training documents to word vectors\n",
    "\n",
    "So how dow we use these tokenized training documents to create words vectors?\n",
    "\n",
    "There are many ways. In particular there are increasingly sophisticated libraries that will just do this for us, and we will look at one of these later. But I want to go through one approach so you get the sense for what is going on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "labeled-montreal",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Create document vectors\n",
    "\n",
    "Create a document vector for each paragraph of the training corpus and put those vectors in a big matrix.\n",
    "\n",
    "We will use the same small vocabulary we used in the last notebook, just to keep things simple. (I'll just copy it over.)\n",
    "\n",
    "I'm going to drop any vectors that have all zeros as entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vanilla-southwest",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "vocab = ['towards', 'closer', 'orbits', 'spring', 'center', 'moves', 'point', 'farther', 'middle', 'ovally']\n",
    "\n",
    "def wfactor(tf):\n",
    "    if tf == 0:\n",
    "        result = 0\n",
    "    else:\n",
    "        result = (1 + np.log(tf))\n",
    "    return result\n",
    "\n",
    "\n",
    "def compute_doc_vector(token_list, vocab):\n",
    "    return np.array([wfactor(token_list.count(word)) for word in vocab])\n",
    "\n",
    "training_dt_matrix = np.zeros(len(vocab))\n",
    "\n",
    "for tdoc in tokenized_training_docs:\n",
    "    new_row = compute_doc_vector(tdoc, vocab)\n",
    "    if np.linalg.norm(new_row) == 0:\n",
    "        continue\n",
    "    training_dt_matrix = np.vstack([training_dt_matrix, new_row])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spiritual-privacy",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The result of this is a table with 10 columns (one for each term in the vocabulary) and 555 rows.\n",
    "\n",
    "The reason we have only 555 rows is that we dropped all of the rows for documents that didn't include any of the words in our vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "metric-infection",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_dt_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cloudy-answer",
   "metadata": {},
   "outputs": [],
   "source": [
    "show_labeled_table(training_dt_matrix, vocab, para_names, nrows=15, ncols=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chicken-disaster",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word vectors\n",
    "\n",
    "So now we've got this big table. The trick is that we can think of the *columns* as vectors that represent the meaning of words. In this case they will be 555 dimensional vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "crazy-continuity",
   "metadata": {},
   "outputs": [],
   "source": [
    "show_labeled_table(training_dt_matrix, vocab, para_names, nrows=15, ncols=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "processed-aerospace",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's transpose it, so the terms are on the rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hindu-function",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_td_matrix = training_dt_matrix.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "numerical-journey",
   "metadata": {},
   "outputs": [],
   "source": [
    "show_labeled_table(training_td_matrix, para_names, vocab, nrows=15, ncols=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collaborative-ecuador",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now we can compare two words using these word vectors. Let's look at a couple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "buried-samuel",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def get_word_vector(w, vocab, mat):\n",
    "    return norm_vec(mat[vocab.index(w)])\n",
    "\n",
    "def compare_word_vectors(w1, w2, vocab, mat):\n",
    "    return np.dot(get_word_vector(w1, vocab, mat), get_word_vector(w2, vocab, mat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "electric-morocco",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "compare_word_vectors(\"closer\", \"towards\", vocab, training_td_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "capable-sandwich",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "compare_word_vectors(\"closer\", \"farther\", vocab, training_td_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "charitable-whole",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Document vectors from word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "forward-democrat",
   "metadata": {},
   "source": [
    "We can use our word vectors to create document vectors for other documents, such as the ones we were working with in the last notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "varied-invitation",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_transcript_docs = {\n",
    "    \"d1\": \"That's because of the sun is in the center and the Earth moves around the sun and the Earth is like at one point in the winter\", \n",
    "    \"d2\": \"it's like farther away from the sun and towards the summer it's closer it's near, towards the sun.\",\n",
    "    \"d3\": \"The sun's in the middle  and the Earth kind of orbits around it.\",\n",
    "    \"d4\": \"And like say at one - it's probably more of an ovally type thing  In the winter, er probably this will be winter since it's further away\",\n",
    "    \"d5\": \"that's winter would be like, the Earth orbits around the sun .  Like summer is the closest to the sun\", \n",
    "    \"d6\": \"Spring is kind of a little further away, and then like Fall  is further away then spring but not as far as winter, and then winter is the furthest.\",\n",
    "    \"d7\": \"the sun doesn't, like the flashlight and the bulb, it hits summer, the lines like fade in , they get there closer, like quicker\",\n",
    "    \"d8\": \"And by the time they get there [winter], it fades and it's a lot colder for winter\"\n",
    "}\n",
    "\n",
    "transcript_doc_names = list(raw_transcript_docs.keys())\n",
    "tokenized_transcript_docs = [seasons_tokenize(doc) for doc in raw_transcript_docs.values()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "romance-respondent",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We take the vector for each word in the document and add them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "allied-asthma",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_doc_vector(doc, vocab, td_mat):\n",
    "    s = np.zeros(td_mat.shape[1])\n",
    "    for w in doc:\n",
    "        if w in vocab:\n",
    "            s = s + get_word_vector(w, vocab, td_mat)\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fitted-peter",
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 = get_doc_vector(tokenized_transcript_docs[0], vocab, training_td_matrix)\n",
    "v1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "least-isaac",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_folded_doc_vectors(docA, docB, vocab, mat):\n",
    "    v1 = get_doc_vector(tokenized_transcript_docs[transcript_doc_names.index(docA)], vocab, mat)\n",
    "    v2 = get_doc_vector(tokenized_transcript_docs[transcript_doc_names.index(docB)], vocab, mat)\n",
    "    return np.dot(norm_vec(v1), norm_vec(v2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stable-kuwait",
   "metadata": {},
   "outputs": [],
   "source": [
    "compare_folded_doc_vectors(\"d1\", \"d3\", vocab, training_td_matrix)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "316.59375px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}