neural_nets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "matplotlib.rcParams['savefig.dpi'] = 2 * matplotlib.rcParams['savefig.dpi']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science in 30 minutes: Neural networks and `word2vec`\n",
    "\n",
    "We run a [data science fellowship](https://www.thedataincubator.com/fellowship.html?ref=ds30) and one of the most in demand topics in data science and machine-learning is neural networks.\n",
    "\n",
    "A _neural network_ is a method of machine learning which chains together a number of more basic models (_nodes_) at a number of _layers_, with each layer feeding into the next. To make this more clear, let's look at the most basic (1-node, 1-layer) neural network, the \"perceptron\".\n",
    "\n",
    "## Neural Net Basics: The perceptron\n",
    "The _perceptron_ is a linear decision boundary classifier\\* that trains by an iterative learning approach.\n",
    "\n",
    "The model works as follows:\n",
    "\n",
    "- **Input**: A data point. This point is transformed into an $n$-length \"feature vector\" $\\vec{v}$ $\\in R^n$, with each element describing the value of that particular feature.\n",
    "- **Output**: A classification, either -1 or 1.\n",
    "\n",
    "Here's an example of some randomly-generated data. In the below scatterplot, we've created some random data points in a 2-dimensional vector space and classified them relative to the line $y = x$ (points \"above\" are class 0, points \"below\" are class 1). \n",
    "\n",
    "\\*In fact, one can use [kernel methods](https://en.wikipedia.org/wiki/Kernel_perceptron) (much like with SVMs) to attempt nonlinear classification with perceptrons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "from numpy import random\n",
    "import numpy as np\n",
    "import matplotlib.pylab as plt\n",
    "from scipy.special import expit as sigmoid\n",
    "from itertools import izip\n",
    "\n",
    "SIZE = 100\n",
    "\n",
    "# Generate Training Data\n",
    "random.seed(42)\n",
    "training_x  = np.random.normal(size=[SIZE, 2])\n",
    "training_t = (training_x[:, 1] > training_x[:, 0]).astype(np.int)\n",
    "plt.scatter(training_x[:,0], training_x[:,1],  c=training_t)\n",
    "plt.xlabel(\"x[0]\")\n",
    "plt.ylabel(\"x[1]\")\n",
    "plt.title(\"Training Data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we're looking for is a model which correctly labels each of these points. Because the perceptron is linear, we can essentially think of the model as trying to draw the best possible line between the two classes of data points.\n",
    "\n",
    "As mentioned above, the basic perceptron is linear, which means we can represent our model as another $n$-length \"weight vector\" $\\vec{\\mathbf{w}}$ - in the image below with input vector $\\vec{v}$, we\n",
    " - compute the inner product $\\vec{\\mathbf{w}} \\cdot \\vec{v} := u$\n",
    " - calculate $f(u)$, where $f$ is an _activation function_ (in a perceptron, the Heaviside step function, as in the diagram below) and use this as our output predicted class.\n",
    " \n",
    "Activation functions typically map from the real number line into ${-1, 1}$ - the Heaviside step function does this discretely based on sign. Another commonly used function, the _[logistic sigmoid function](https://en.wikipedia.org/wiki/Logistic_function)_, maps into the continuous intervial (0, 1). \n",
    "\n",
    "It may also be instructive to think of this as matrix multiplication (with $\\vec{v}$ a $1 \\times n$ matrix and $\\vec{\\mathbf{w}}$ an $n \\times 1$ matrix)- when we chain together perceptrons for a neural net, we can in fact represent each layer of the model as a matrix.\n",
    "\n",
    "![Perceptron](http://i.stack.imgur.com/KUvpQ.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training: \n",
    "\n",
    "#### In theory\n",
    "Mathematically, whenever we perform supervised learning, we seek to minimize an _objective function_ which essentially measures how closely our model describes the training data. This objective function's input is our model's weights, which control exactly what the model is fitting to. In the case of the perceptron, those weights are our vector $\\vec{\\mathbf{w}}$.\n",
    "\n",
    "While the standard perceptron's training algorithm provably converges on linearly separable data, our function is not smooth (due to the use of the Heaviside function), and the proof is not immediate from standard convex optimization methods.\n",
    "\n",
    "Let's look at a slightly modified perceptron using the [logistic sigmoid function](http://mathworld.wolfram.com/SigmoidFunction.html) $\\sigma$ as the activation function instead of the Heaviside function, and derive our training method from there.  Note that our labels are now $\\{-1, 1\\}$.\n",
    "\n",
    "Here's our objective function: \n",
    "\n",
    "$$ E(\\vec{\\mathbf{w}}) = \\sum_{i=1}^{|T|} \\frac{1}{2} (t_i - f(u_i)) ^2 $$\n",
    "\n",
    "Where we take training samples from our training set $T$. We let $t_i$ be our gold standard datum for the $i$-th training sample, and let $x_i$ be the $i$-th sample's input. Then, $ u_i = u(\\vec{x}_i) = \\vec{\\mathbf{w}} \\cdot \\vec{x}_i$, and $f(u_i) = \\sigma (u_i)$ is the output given by our model with the current weights on this training sample. In otherwords, instead of the Heaviside function as our activation function $f$, we've switched to the logistic sigmoid. For simpler derivation later, let $f_i = f(u_i)$.\n",
    "\n",
    "As it turns out, this is the objective function for a very familiar model - [Logistic Regression](https://en.wikipedia.org/wiki/Logistic_regression). We can use the usual convex optimization methods - namely [Stochastic Gradient Descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) (SGD) - to train our model.\n",
    "\n",
    "To see this, let's take $\\frac{\\partial{E}}{\\partial{w_j}}$, the derivative of the objective with respect to the $j$-th index of the weight vector $\\vec{\\mathbf{w}}$. Noting that differentiation is linear and using the chain rule, it's easy to see that:\n",
    "\n",
    "$$ \\frac{\\partial{E}}{\\partial{w_j}} = \n",
    "        \\sum_{i} \\frac{\\partial{E}}{\\partial{f_i}} \\cdot \\frac{\\partial{f_i}}{\\partial{u_i}} \\cdot \\frac{\\partial{u_i}}{\\partial{w_j}} $$\n",
    "        \n",
    "Here's each of those terms, from the equation above (and noting $\\frac{d\\sigma}{dx} = \\sigma(x) ( 1 - \\sigma(x))$:\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\frac{\\partial{E}}{\\partial{f_i}} & = (t_i - f_i) \\\\\n",
    "   \\frac{\\partial{f_i}}{\\partial{u_i}} & =  (f_i)(1-f_i) \\\\\n",
    "   \\frac{\\partial{u_i}}{\\partial{w_j}} & = x_i\n",
    "\\end{align*}\n",
    "$$\n",
    "Because we're using SGD, we don't need to take the gradient with respect to the entire function - just the single example $x_i$ we plan to iteratively update $\\vec{\\mathbf{w}}$ with.\n",
    "\n",
    "Hence at each update phase, we update our weights in the direction of the local gradient as follows:\n",
    "\n",
    "$$ \\vec{\\mathbf{w_0}} = [0 ..... 0]\\\\\n",
    "\\vec{\\mathbf{w_{i+1}}} = \\vec{\\mathbf{w_i}} - \\eta  \\cdot (t_i - f_i)(f_i)(1 - f_i) \\vec{x}\n",
    "$$\n",
    "\n",
    "Where $\\eta$ is the _learning rate_, a parameter we choose (typically small, maybe .1 or .01) that determines the rate at which our weight vector updates after seeing each training example.\n",
    "\n",
    "#### In practice\n",
    "Let's show a low-dimensional example of the algorithm we just derived. We train the sigmoid perceptron by iterating over all the data multiple times, and updating our weight vector $\\vec{\\mathbf{w}}$ as we see each data point. We call 1 full iteration over the training set an _epoch_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "epochs = 1000\n",
    "weights = np.array([1., 0.])\n",
    "learning_rate = .001\n",
    "\n",
    "normalized_weight_history = []\n",
    "for epoch in xrange(epochs):\n",
    "    pred = sigmoid(np.dot(training_x, weights))\n",
    "    deriv = pred * (1 - pred) * (pred - training_t)\n",
    "    update = np.dot(deriv, training_x)\n",
    "    weights -= learning_rate * update\n",
    "    if epoch % 50 == 0:\n",
    "        normalized_weight_history += [weights / np.abs(weights).sum()]\n",
    "normalized_weight_history = np.array(normalized_weight_history)\n",
    "        \n",
    "preds = (sigmoid(np.dot(training_x, weights)) > .5).astype(int)\n",
    "\n",
    "print \"Mean Absolute Error:\", metrics.mean_absolute_error(training_t, preds)\n",
    "print \"Mean Squared Error:\", metrics.mean_squared_error(training_t, preds)\n",
    "print \"R^2:\", metrics.r2_score(training_t, preds)\n",
    "\n",
    "plt.scatter(\n",
    "    normalized_weight_history[:, 0],\n",
    "    normalized_weight_history[:, 1]\n",
    ")\n",
    "plt.xlim([-1, 1])\n",
    "plt.ylim([-1, 1])\n",
    "plt.xlabel(\"weight[0]\")\n",
    "plt.ylabel(\"weight[1]\")\n",
    "plt.title(\"Convergence of Weights\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exactly what we'd expect - a perfect fit! Now let's plot the separating hyperplane the model developed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#We rotate the resultant vector 90 degrees. \n",
    "m = -(weights[0])/(weights[1])\n",
    "#Since we didn't include a bias term, we start at (0,0)\n",
    "p1 = [0, 0]\n",
    "x2 = 1.3\n",
    "y2 = (x2 - p1[0])*m + p1[1]\n",
    "p2 = [x2, y2]\n",
    "\n",
    "def drawLine2P(x,y,xlims):\n",
    "    xrange = np.arange(xlims[0],xlims[1],0.1)\n",
    "    A = np.vstack([x, np.ones(len(x))]).T\n",
    "    k, b = np.linalg.lstsq(A, y)[0]\n",
    "    plt.plot(xrange, k*xrange + b, 'k')\n",
    "\n",
    "plt.scatter(training_x[:,0], training_x[:,1],  c=training_t)\n",
    "drawLine2P([p1[0], p2[0]], [p1[1], p2[1]], [-6, 6])\n",
    "plt.xlabel(\"x[0]\")\n",
    "plt.ylabel(\"x[1]\")\n",
    "plt.title(\"Training Data with Prediction\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Pitfalls:\n",
    "The perceptron is too simple of a model for more complex classification tasks. However, chaining perceptrons into a neural network allows for surprisingly good results across a wide variety of problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chaining Perceptrons into Neural Networks\n",
    "\n",
    "In the below diagram (a visual representation of a neural net), you can think of each \"hidden node\" as a perceptron that takes the average of the input layers' values as input and produces a unique prediction of its own:\n",
    "![A Neural Network](https://upload.wikimedia.org/wikipedia/commons/4/46/Colored_neural_network.svg)\n",
    "\n",
    "That said, there are a couple of important differences, mathematically speaking:\n",
    "\n",
    "1. As mentioned above, perceptrons typically use the Heaviside step function for activation. In neural nets, the logistic sigmoid function is typical. \n",
    "2. We derived the single node sigmoid training algorithm above - the SGD-based method we use to train a multi-layer neural network is commonly known as _backpropogation_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## word2vec\n",
    "As it turns out, the skip-gram and continuous-bag-of-words neural networks used in [word2vec](https://code.google.com/p/word2vec/) are simple neural networks that produce a suprisingly cool result on natural language data. \n",
    "\n",
    "word2vec is a library that, given a corpus of natural language text data (think Wikipedia articles), maps each word into a high-dimensional vector space. For those of you familiar with information retrieval theory, the concept is widely used in that field: a word's position in the vector space denotes how _similar_ it is to other nearby words. \n",
    "\n",
    "In this way we can create interesting clusters of related words, and, somewhat more compellingly, learn nontrivial associations between words.\n",
    "\n",
    "For example, consider the analogy, \n",
    "\n",
    "\"Man is to King as Woman is to \\_\\_\\_\\_\\_\\_\\_\"\n",
    "\n",
    "As an English speaker, you know the answer is \"Queen\". As it turns out, a well-trained word2vec vector space can also come to this conclusion by the following vector computation:\n",
    "\n",
    "` closest( (vector(\"king\") - vector(\"man\")) + vector(\"woman\") )`\n",
    "\n",
    "Where we measure closeness via cosine similarity. Let's take a look at a pre-trained `word2vec` model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The tool: `gensim`\n",
    "Gensim is a great python library for various topic-modelling and clustering tasks. The library recently added word2vec to its toolbox - it's a thin wrapper around the highly optimized [C implementation](https://code.google.com/p/word2vec/). \n",
    "\n",
    "To install gensim, you can simply run `conda install gensim` (if you're still on `pip` you can use this as well). You'll need:\n",
    "\n",
    "- at least `numpy/scipy` installed\n",
    "- ideally a C compiler so that you can use optimized word2vec training (as noted [here](https://radimrehurek.com/gensim/models/word2vec.html)).\n",
    "\n",
    "We'll also use the first [Pre-trained model](https://code.google.com/p/word2vec/#Pre-trained_word_and_phrase_vectors) (from the Google News Dataset), from the `word2vec` Google Code page above, to illustrate some of the classic analogies we mentioned above. Note these vectors are 1.5GB, so sit back and relax! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  The code - pre-trained model.\n",
    "\n",
    "First, let's look at the pre-trained model and see what fun analogies we can come up with.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "googlenews_model = gensim.models.word2vec.Word2Vec.load_word2vec_format(\"GoogleNews-vectors-negative300.bin\",\n",
    "                                                                        binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "googlenews_model.most_similar(\"king\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If asked to write down some words associated with royalty, you or I would probably write down similar lists. \n",
    "\n",
    "Clustering, however, is a long-solved problem - [LDA](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation), for example, could produce similar lists for us if trained on this dataset. What's particularly interesting about `word2vec` is the following sort of computation:\n",
    "\n",
    "$$ vec(king) - vec(man) + vec(woman) $$\n",
    "\n",
    "When we perform this vector operation, what words are closest to the resultant vector? Are they meaningful? Let's try! \n",
    "\n",
    "We'll start with the example we used above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# vec(king) - vec(man) + vec(woman)\n",
    "googlenews_model.most_similar(positive=[\"king\", \"woman\"], negative=[\"man\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note above that \"monarch\", a gender-neutral term, is also listed as quite similar. Does this hold when we subtract `vec(man)` without adding in `vec(woman)`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "googlenews_model.most_similar(positive=[\"king\"], negative=[\"man\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not quite - though \"kingdom\" is another interesting similarity when we think of \"man\" as referring to a human."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The layers\n",
    "As is typical of simple neural nets, CBOW has an _input_ layer, a _hidden_ layer, and an _output_ layer. A linear transformation (ie a matrix) governs the movement from input to hidden, and another handles movement from hidden to output - the above diagram labels these transformations $\\mathbf{W}$ and $\\mathbf{W'}$. In CBOW, we have\n",
    "\n",
    "- **input**: Four _one-hot encoded_\\* words - i.e. $1 x V$ vectors where $V$ is the size of our vocabulary, all values are 0 except for one.\n",
    "- $\\mathbf{W}$: A $V x N$ matrix that transitions into the hidden layer, and notably maps our encoded word into an $N$-dimensional (usually $N\\approx300$) vector space. In otherwords, after we've completed training this model, we'll /only/ care about this matrix and the embedding it creates for our vocabulary.\n",
    "- **hidden**: A $1 x N$ vector that represents the mapping of our input word into the desired vector space\n",
    "- $\\mathbf{W'}$: An $N x V$ matrix transitioning between the hidden layer and the output.\n",
    "- **output**: A single one-hot encoded word - the model's prediction for most likely to occur given the surrounding four-word context.\n",
    "\n",
    "\\* This isn't strictly true as word2vec actually uses a different, optimized tree structure - but the one-hot encoding is illustrative.\n",
    "\n",
    "Note that $\\mathbf{W}$ is the artifact we're looking for, as it transitions our encoded words into an interesting vector space. Hence, to develop $\\mathbf{W}$, we must simply train this model and recover the weight parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's try it!\n",
    "We decided to try this on some real-world data, in particular Yelp's [Academic Dataset](https://www.yelp.com/academic_dataset), to see if we could come up with anything interesting.\n",
    "\n",
    "Some questions we can ask via this analogy structure:\n",
    "\n",
    "- What's the closest thing to Pizza in France? \n",
    "- What is non-spicy Mexican food?\n",
    "- The tagine is emblematic of North African cuisine: is there anything similar in other geographies?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The dataset: The Yelp Academic Datset \n",
    "\n",
    "We host a version of [Yelp's Academic Dataset](https://www.yelp.com/academic_dataset) for our fellows. \n",
    "\n",
    "\n",
    "To follow along, you should\n",
    "- Install `gensim` as above.\n",
    "- Download the Yelp review data for yourself.\n",
    "\n",
    "First, we can query the Google News model as a baseline. In fact, if we wanted, we could build off of this model and simply continue training with the extra sentences provided by the Yelp Academic dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# vec('burrito') - vec('mexican') + vec('chinese')\n",
    "googlenews_model.most_similar(positive=[\"burrito\", \"chinese\"], negative=[\"mexican\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And another fun one: let's see the \"default\" burrito is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# vec('burrito') - vec('mexican')\n",
    "googlenews_model.most_similar(positive=[\"burrito\"], negative=[\"mexican\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Code: Training the model.\n",
    "Now that we've looked at a well-trained model, let's try some modelling of our own!\n",
    "\n",
    "Our text processing leaves something to be desired. However, one of the coolest things about `word2vec` is that even with the most trivial of input cleanings, we get interesting results, as you'll see below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#TODO: better text processing\n",
    "import simplejson as json\n",
    "reviews = []\n",
    "lines = open(\"yelp_train_academic_dataset_review.json\")\n",
    "for line in lines:\n",
    "    reviews.append(json.loads(line))\n",
    "reviews = [r['text'].lower().split(\" \") for r in reviews]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this took about an hour on our 16GB Digital Ocean instance - go grab lunch!\n",
    "yelp_model = gensim.models.word2vec.Word2Vec(reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can configure logs to send output to a file if you like - the [gensim docs](https://radimrehurek.com/gensim/tutorial.html) explain how to activate logging if that's something that interests you. Here's what training output looks like when using Spark's word2vec implementation:\n",
    "\n",
    "![Training output](progress.png)\n",
    "\n",
    "wordCount is how many words we've seen, and alpha is the learning rate - a useful optimization for training neural nets is to incrementally decrease the learning rate as time goes on and we converge toward the objective function's minimum. On our local setup with gensim (using a [16gb DigitalOcean instance](https://www.digitalocean.com/pricing/)), training on this dataset takes about 30 minutes.\n",
    "\n",
    "Once the model is trained, we can write it out to a file to load up later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "yelp_model.save(\"yelp_word2vec.model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying a Trained Model\n",
    "\n",
    "Once we have the model trained, we can begin to query the vector space for interesting results and similarities. `model.most_similar` lets us pass in a word to get the closest vectors to that word (if it exists in the dataset). Take a look below at some of the immediate results we found:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "yelp_model = gensim.models.word2vec.Word2Vec.load(\"yelp_word2vec.model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yelp_model.most_similar(\"chinese\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yelp_model.most_similar(\"mexican\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the above is more or less what we'd expect from LDA on this dataset.\n",
    "\n",
    "How does our model compare to Google News?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yelp_model.most_similar(positive=[\"chinese\", \"burrito\"], negative=[\"mexican\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not much here - vermicelli are noodles, beef/chicken might be getting to something - but this is an early hint that our dataset might not be big enough to effectively build an interesting vector space. And clearly we need to improve our text processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yelp_model.most_similar(\"pizza\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, we have some tokenizing issues, but more or less what we'd expect. Maybe we can try looking for pizza styles/places to avoid?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yelp_model.most_similar(positive=[\"pizza\", \"nyc\", \"bad\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note to self: Avoid Grimaldi's, and maybe Difara's! Maybe pizza in Brooklyn? (This is probably an artifact of the \"nyc\" vector's weight).\n",
    "\n",
    "## The dataset itself and future exploration\n",
    "\n",
    "This dataset itself has an issue we can't resolve with better parsing or parameter tweaking: it's too small to train a truly effective word2vec model. We got some fun and interesting results, but really, we'd want to run this on something closer to all of Yelp's restaurant review text to develop some really cool analogies. Of course, Yelp doesn't simply make all of its data available for free - however, Yelp is one of our hiring partners ;)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# References and useful links\n",
    "  \n",
    "http://www-personal.umich.edu/~ronxin/pdf/w2vexp.pdf - a good explanation of word2vec including a succinct overview of backpropogation, as well as `wevi`, the tool we used to visualize the algorithm and its embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## If you want more ...\n",
    "This is meant to just give you a brief guided tour of just a few topics in data science.\n",
    "\n",
    "If you enjoyed this and want to learn more about doing data science in industry, consider [applying](https://www.thedataincubator.com/fellowship.html#apply?ref=ds30) to be a fellow at [The Data Incubator](https://www.thedataincubator.com/?ref=ds30)\n",
    "\n",
    "If you would like to hire data scientists, introduce data science corporate training, or partner to bring The Data Incubator to your country, reach out [here](https://www.thedataincubator.com/hiring.html?ref=ds30).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}