W6-2 Vector Space Models.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Vector space models</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About vectors</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many ways to think about what a vector is. You can think of it as a list of numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "v1 = [3, 4]\n",
    "v2 = [5, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can think also think of a vector as a point in space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.grid(True)\n",
    "plt.axis([0, 10, 0, 10])\n",
    "plt.plot(3, 4, \"ro\")\n",
    "plt.plot(5, 2, \"ro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or as an arrow drawn from the origin to those points in space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_vec(v):\n",
    "    plt.arrow(0, 0, v[0], v[1], head_width=.2, head_length=.4, length_includes_head=True)\n",
    "    return\n",
    "plt.grid(True)\n",
    "plt.axis([0, 10, 0, 10])\n",
    "plot_vec(v1)\n",
    "plot_vec(v2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the magnitude (length) of a vector you use the **pythagorean theorem**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import sqrt\n",
    "magnitude_of_v1 = sqrt(3**2 + 4**2)\n",
    "magnitude_of_v1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closeness of two vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to want some way of measuring how close two vectors are.\n",
    "\n",
    "One way to do this is to use the pythagorean theorem again, this time to find the distance between the points at the end of the two vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distance_btwn_v1_and_v2 = sqrt((5 - 3)**2 + (2 - 4)**2)\n",
    "distance_btwn_v1_and_v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The dot product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More common though is to use a measurement known as the \"dot product.\"\n",
    "\n",
    "The dot product captures something like the overlap between two vectors. If the vectors are perpendicular to each other then the dot product is 0. If they point in the same direction then the dot product is the product of their lengths.\n",
    "\n",
    "Here are some ways of calcuting the dot product."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\vec{v}\\bullet\\vec{w}= {v_1}{w_1}+{v_2}{w_2}+{v_3}{w_3}+...$\n",
    "\n",
    "$\\vec{v}\\bullet\\vec{w}=|\\vec{v}||\\vec{w}|cos(\\theta)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In python, it looks like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1[0] * v2[0] + v1[1] * v2[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "numpy has a function to do this for you, `dot`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import dot\n",
    "dot(v1, v2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `dot` to compute the magnitude of a vector in a compact manner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1_mag = sqrt(dot(v1, v1))\n",
    "v1_mag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Normalizing\" vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalizing a vector, means converting the vector to an arrow in the same direction, but with a length of 1.\n",
    "\n",
    "To accomplish this, we divide each dimension of the vector by the length of the vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Yes, this is yet another meaning of the word \"normalize.\" There is even a third meaning that is relevant here. \"Normal\" is also a word used to mean \"perpendicular.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1_normalized = [v1[0] / v1_mag, v1[1] / v1_mag]\n",
    "v1_normalized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dot(v1_normalized, v1_normalized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is important because we often want to know the extent to which two vectors point in the same direction, regardless of how long each one is. You'll see why in a little bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectors in numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we try to add and multiply vectors without numpy?\n",
    "\n",
    "It doesn't do what we want it to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 + v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "3 * v1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With numpy, vectors behave like vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "v1 = np.array([3, 4])\n",
    "v2 = np.array([5, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 + v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "3 * v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1_mag = sqrt(dot(v1, v1))\n",
    "print(v1_mag)\n",
    "v1_mag = np.linalg.norm(v1)\n",
    "print(v1_mag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1_normalized = v1 / v1_mag\n",
    "print(v1_normalized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2_normalized = v2 / np.linalg.norm(v2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.grid(True)\n",
    "plt.axis([0, 3, 0, 3])\n",
    "plot_vec(v1_normalized)\n",
    "plot_vec(v2_normalized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting text to a vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to convert a block of text to a vector. To do that, we are going to first decide on an ordered list of words. We'll call this the \"vocabulary.\" Then, to convert a block of text, we'll count how many times each of these words appears in the block of text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t1 = \"now is the time for all good men to come to the aid of their country\"\n",
    "t1w = nltk.word_tokenize(t1)\n",
    "t2 = \"now is the time for all good women to come to the aid of their country\"\n",
    "t2w = nltk.word_tokenize(t2)\n",
    "t3 = \"is it time for the women to lead us all\"\n",
    "t3w = nltk.word_tokenize(t3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t1w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = sorted(list(set(t1w + t2w + t3w)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist = []\n",
    "for word in vocab:\n",
    "    mylist.append(t1w.count(word))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 = np.array([t1w.count(word) for word in vocab])\n",
    "print(v1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def norm_vec(v):\n",
    "    return v / np.linalg.norm(v)\n",
    "np.set_printoptions(precision=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 = norm_vec(v1)\n",
    "print(v1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2 = norm_vec(np.array([t2w.count(word) for word in vocab]))\n",
    "v3 = norm_vec(np.array([t3w.count(word) for word in vocab]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"dot product of v1 and v2 is \", dot(v1, v2))\n",
    "print(\"dot product of v1 and v3 is \", dot(v1, v3))\n",
    "print(\"dot product of v2 and v3 is \", dot(v1, v3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Squashing the vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "pca = PCA(n_components=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array([v1, v2, v3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_squashed = pca.fit_transform(X)\n",
    "X_squashed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = [v[0] for v in X_squashed]\n",
    "ys = [v[1] for v in X_squashed]\n",
    "plt.scatter(xs, ys)\n",
    "labels = [\"v1\", \"v2\", \"v3\"]\n",
    "for n, v in enumerate(X_squashed):\n",
    "    plt.annotate(labels[n], (v[0], v[1]), textcoords=\"offset points\", xytext=(5, 5,))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}