Exploring TfIdfVectorizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring sklearn's TfIdfVectorizer\n",
    "\n",
    "`TfIdfVectorizer` is included in scikit-learn's `sklearn.feature_extraction.text` module. It's commonly used for creating a feature matrix from text data, and it's surprisingly easy to work with. `TfIdfVectorizer` makes it easy to fit and transform data at the same, and one can easily change the regex pattern, number of n-grams, minimum and maximum token frequency, and stopwords simply by tweaking the parameters at initialization.\n",
    "\n",
    "For the curious, `TfIdfVectorizer`'s documentation can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a test string from the first few paragraphs of a Wikipedia article on collaborative filtering\n",
    "test_string = 'Collaborative filtering (CF) is a technique used by recommender systems. Collaborative filtering has two senses, a narrow one and a more general one. In the newer, narrower sense, collaborative filtering is a method of making automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating). The underlying assumption of the collaborative filtering approach is that if a person A has the same opinion as a person B on an issue, A is more likely to have B\\'s opinion on a different issue than that of a randomly chosen person. For example, a collaborative filtering recommendation system for television tastes could make predictions about which television show a user should like given a partial list of that user\\'s tastes (likes or dislikes). Note that these predictions are specific to the user, but use information gleaned from many users. This differs from the simpler approach of giving an average (non-specific) score for each item of interest, for example based on its number of votes. In the more general sense, collaborative filtering is the process of filtering for information or patterns using techniques involving collaboration among multiple agents, viewpoints, data sources, etc. Applications of collaborative filtering typically involve very large data sets. Collaborative filtering methods have been applied to many different kinds of data including: sensing and monitoring data, such as in mineral exploration, environmental sensing over large areas or multiple sensors; financial data, such as financial service institutions that integrate many financial sources; or in electronic commerce and web applications where the focus is on user data, etc. The remainder of this discussion focuses on collaborative filtering for user data, although some of the methods and approaches may apply to the other major applications as well.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the string into sentences, which the vectorizer will consider as individual documents\n",
    "string_data = test_string.split('. ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'analyzer': 'word',\n",
       " 'binary': False,\n",
       " 'decode_error': 'strict',\n",
       " 'dtype': numpy.int64,\n",
       " 'encoding': 'utf-8',\n",
       " 'input': 'content',\n",
       " 'lowercase': True,\n",
       " 'max_df': 1.0,\n",
       " 'max_features': None,\n",
       " 'min_df': 1,\n",
       " 'ngram_range': (1, 1),\n",
       " 'norm': 'l2',\n",
       " 'preprocessor': None,\n",
       " 'smooth_idf': True,\n",
       " 'stop_words': None,\n",
       " 'strip_accents': None,\n",
       " 'sublinear_tf': False,\n",
       " 'token_pattern': '(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       " 'tokenizer': None,\n",
       " 'use_idf': True,\n",
       " 'vocabulary': None}"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize the vectorizer - the default is a standard 1-word vectorizer\n",
    "word_vectorizer = TfidfVectorizer()\n",
    "\n",
    "# Glimpse the vectorizer's default parameters\n",
    "word_vectorizer.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_word = word_vectorizer.fit_transform(string_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<11x149 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 240 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `fit_transform` rather than just `fit` tells the vectorizer to learn the vocabulary and frequency from the text data AND to return the feature matrix. Note that X_word is an 11 x 149 matrix, so there are 11 entries (sentences) and 149 features (words).\n",
    "\n",
    "We can make a nice scrollable list of all the words that were extracted from our text sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "about\n",
      "agents\n",
      "although\n",
      "among\n",
      "an\n",
      "and\n",
      "applications\n",
      "applied\n",
      "apply\n",
      "approach\n",
      "approaches\n",
      "are\n",
      "areas\n",
      "as\n",
      "assumption\n",
      "automatic\n",
      "average\n",
      "based\n",
      "been\n",
      "but\n",
      "by\n",
      "cf\n",
      "chosen\n",
      "collaborating\n",
      "collaboration\n",
      "collaborative\n",
      "collecting\n",
      "commerce\n",
      "could\n",
      "data\n",
      "different\n",
      "differs\n",
      "discussion\n",
      "dislikes\n",
      "each\n",
      "electronic\n",
      "environmental\n",
      "etc\n",
      "example\n",
      "exploration\n",
      "filtering\n",
      "financial\n",
      "focus\n",
      "focuses\n",
      "for\n",
      "from\n",
      "general\n",
      "given\n",
      "giving\n",
      "gleaned\n",
      "has\n",
      "have\n",
      "if\n",
      "in\n",
      "including\n",
      "information\n",
      "institutions\n",
      "integrate\n",
      "interest\n",
      "interests\n",
      "involve\n",
      "involving\n",
      "is\n",
      "issue\n",
      "item\n",
      "its\n",
      "kinds\n",
      "large\n",
      "like\n",
      "likely\n",
      "likes\n",
      "list\n",
      "major\n",
      "make\n",
      "making\n",
      "many\n",
      "may\n",
      "method\n",
      "methods\n",
      "mineral\n",
      "monitoring\n",
      "more\n",
      "multiple\n",
      "narrow\n",
      "narrower\n",
      "newer\n",
      "non\n",
      "note\n",
      "number\n",
      "of\n",
      "on\n",
      "one\n",
      "opinion\n",
      "or\n",
      "other\n",
      "over\n",
      "partial\n",
      "patterns\n",
      "person\n",
      "predictions\n",
      "preferences\n",
      "process\n",
      "randomly\n",
      "recommendation\n",
      "recommender\n",
      "remainder\n",
      "same\n",
      "score\n",
      "sense\n",
      "senses\n",
      "sensing\n",
      "sensors\n",
      "service\n",
      "sets\n",
      "should\n",
      "show\n",
      "simpler\n",
      "some\n",
      "sources\n",
      "specific\n",
      "such\n",
      "system\n",
      "systems\n",
      "taste\n",
      "tastes\n",
      "technique\n",
      "techniques\n",
      "television\n",
      "than\n",
      "that\n",
      "the\n",
      "these\n",
      "this\n",
      "to\n",
      "two\n",
      "typically\n",
      "underlying\n",
      "use\n",
      "used\n",
      "user\n",
      "users\n",
      "using\n",
      "very\n",
      "viewpoints\n",
      "votes\n",
      "web\n",
      "well\n",
      "where\n",
      "which\n"
     ]
    }
   ],
   "source": [
    "for word in word_vectorizer.get_feature_names():\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `.vocabulary_` attribute to look at the indices of each feature. Each feature (word) lives at an index in our feature matrix. For example, calling `X_word[0, 25]` will tell us the weighting of 'collaborative' within the first sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'collaborative': 25,\n",
       " 'filtering': 40,\n",
       " 'cf': 21,\n",
       " 'is': 62,\n",
       " 'technique': 125,\n",
       " 'used': 138,\n",
       " 'by': 20,\n",
       " 'recommender': 104,\n",
       " 'systems': 122,\n",
       " 'has': 50,\n",
       " 'two': 134,\n",
       " 'senses': 109,\n",
       " 'narrow': 83,\n",
       " 'one': 91,\n",
       " 'and': 5,\n",
       " 'more': 81,\n",
       " 'general': 46,\n",
       " 'in': 53,\n",
       " 'the': 130,\n",
       " 'newer': 85,\n",
       " 'narrower': 84,\n",
       " 'sense': 108,\n",
       " 'method': 77,\n",
       " 'of': 89,\n",
       " 'making': 74,\n",
       " 'automatic': 15,\n",
       " 'predictions': 99,\n",
       " 'about': 0,\n",
       " 'interests': 59,\n",
       " 'user': 139,\n",
       " 'collecting': 26,\n",
       " 'preferences': 100,\n",
       " 'or': 93,\n",
       " 'taste': 123,\n",
       " 'information': 55,\n",
       " 'from': 45,\n",
       " 'many': 75,\n",
       " 'users': 140,\n",
       " 'collaborating': 23,\n",
       " 'underlying': 136,\n",
       " 'assumption': 14,\n",
       " 'approach': 9,\n",
       " 'that': 129,\n",
       " 'if': 52,\n",
       " 'person': 98,\n",
       " 'same': 106,\n",
       " 'opinion': 92,\n",
       " 'as': 13,\n",
       " 'on': 90,\n",
       " 'an': 4,\n",
       " 'issue': 63,\n",
       " 'likely': 69,\n",
       " 'to': 133,\n",
       " 'have': 51,\n",
       " 'different': 30,\n",
       " 'than': 128,\n",
       " 'randomly': 102,\n",
       " 'chosen': 22,\n",
       " 'for': 44,\n",
       " 'example': 38,\n",
       " 'recommendation': 103,\n",
       " 'system': 121,\n",
       " 'television': 127,\n",
       " 'tastes': 124,\n",
       " 'could': 28,\n",
       " 'make': 73,\n",
       " 'which': 148,\n",
       " 'show': 115,\n",
       " 'should': 114,\n",
       " 'like': 68,\n",
       " 'given': 47,\n",
       " 'partial': 96,\n",
       " 'list': 71,\n",
       " 'likes': 70,\n",
       " 'dislikes': 33,\n",
       " 'note': 87,\n",
       " 'these': 131,\n",
       " 'are': 11,\n",
       " 'specific': 119,\n",
       " 'but': 19,\n",
       " 'use': 137,\n",
       " 'gleaned': 49,\n",
       " 'this': 132,\n",
       " 'differs': 31,\n",
       " 'simpler': 116,\n",
       " 'giving': 48,\n",
       " 'average': 16,\n",
       " 'non': 86,\n",
       " 'score': 107,\n",
       " 'each': 34,\n",
       " 'item': 64,\n",
       " 'interest': 58,\n",
       " 'based': 17,\n",
       " 'its': 65,\n",
       " 'number': 88,\n",
       " 'votes': 144,\n",
       " 'process': 101,\n",
       " 'patterns': 97,\n",
       " 'using': 141,\n",
       " 'techniques': 126,\n",
       " 'involving': 61,\n",
       " 'collaboration': 24,\n",
       " 'among': 3,\n",
       " 'multiple': 82,\n",
       " 'agents': 1,\n",
       " 'viewpoints': 143,\n",
       " 'data': 29,\n",
       " 'sources': 118,\n",
       " 'etc': 37,\n",
       " 'applications': 6,\n",
       " 'typically': 135,\n",
       " 'involve': 60,\n",
       " 'very': 142,\n",
       " 'large': 67,\n",
       " 'sets': 113,\n",
       " 'methods': 78,\n",
       " 'been': 18,\n",
       " 'applied': 7,\n",
       " 'kinds': 66,\n",
       " 'including': 54,\n",
       " 'sensing': 110,\n",
       " 'monitoring': 80,\n",
       " 'such': 120,\n",
       " 'mineral': 79,\n",
       " 'exploration': 39,\n",
       " 'environmental': 36,\n",
       " 'over': 95,\n",
       " 'areas': 12,\n",
       " 'sensors': 111,\n",
       " 'financial': 41,\n",
       " 'service': 112,\n",
       " 'institutions': 56,\n",
       " 'integrate': 57,\n",
       " 'electronic': 35,\n",
       " 'commerce': 27,\n",
       " 'web': 145,\n",
       " 'where': 147,\n",
       " 'focus': 42,\n",
       " 'remainder': 105,\n",
       " 'discussion': 32,\n",
       " 'focuses': 43,\n",
       " 'although': 2,\n",
       " 'some': 117,\n",
       " 'approaches': 10,\n",
       " 'may': 76,\n",
       " 'apply': 8,\n",
       " 'other': 94,\n",
       " 'major': 72,\n",
       " 'well': 146}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 : 0.1666622159077999\n",
      "1 : 0.13425908668519743\n",
      "2 : 0.09542555931338716\n",
      "3 : 0.06798342182634644\n",
      "4 : 0.08035560781786677\n",
      "5 : 0.0\n",
      "6 : 0.0\n",
      "7 : 0.09928901260456367\n",
      "8 : 0.16848148408815192\n",
      "9 : 0.05350060595361461\n",
      "10 : 0.09512697975165428\n"
     ]
    }
   ],
   "source": [
    "for entry in range(11):\n",
    "    print(entry, \":\", X_word[entry, 25])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a little easier to make sense of these weightings if we can look at the sentences. Let's print 0, 3, and 5 below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 : Collaborative filtering (CF) is a technique used by recommender systems\n",
      "3 : The underlying assumption of the collaborative filtering approach is that if a person A has the same opinion as a person B on an issue, A is more likely to have B's opinion on a different issue than that of a randomly chosen person\n",
      "5 : Note that these predictions are specific to the user, but use information gleaned from many users\n"
     ]
    }
   ],
   "source": [
    "for num in [0, 3, 5]:\n",
    "    print(num, ':', string_data[num])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that 0 has an especially high weighting (0.1666) for 'collaborative' because the sentence is short - only 9 words. This results in 'collaborative' being considered an important element of the sentence.\n",
    "\n",
    "3 is quite a bit longer than 0, and 'collaborative' only occurs one time. 'Collaborative' thus gets a lower weighting of 0.0679.\n",
    "\n",
    "5 has no occurrence of 'collaborative', and thus the feature weighting for 'collaborative' is 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove some stopwords. We can fit the input data again and specify the parameter `max_df = 8` to ignore any words that occur in 8 out of 11 of our sentences. (\"Df\" stands for \"document frequency\".)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'collaborative', 'filtering'}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectorizer_stop = TfidfVectorizer(max_df = 8)\n",
    "X_word_stop = word_vectorizer_stop.fit_transform(string_data)\n",
    "word_vectorizer_stop.stop_words_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the data is pulled from a Wikipedia article on collaborative filtering, these were the only two words to occur in at least 8 out of 11 sentences.\n",
    "\n",
    "An alternative to setting `max_df` (or `min_df`) is that we can pass a list of stopwords to make sure the vectorizer doesn't include them in its vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "set()"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reinitialize word_vectorizer_stop\n",
    "word_vectorizer_stop = TfidfVectorizer(stop_words = ['is', 'the', 'of', 'by', 'in', 'or', 'on', 'an', 'to'])\n",
    "X_word_stop = word_vectorizer_stop.fit_transform(string_data)\n",
    "\n",
    "word_vectorizer_stop.stop_words_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "frozenset({'an', 'by', 'in', 'is', 'of', 'on', 'or', 'the', 'to'})"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectorizer_stop.get_stop_words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `word_vectorizer_stop.stop_words_` returns an empty set, while `get_stop_words()` returns a frozenset of our list of stopwords. According to the documentation, `.stop_words_` is built from words that were excluded based on setting the parameters `max_df`, `min_df`, and `max_features`, not from setting the `stop_words` parameter.\n",
    "\n",
    "If you passed stopwords as a list rather than specifying any of those three parameters, then keep this in mind when trying to check which words were used as stopwords. Make sure to call `get_stop_words()` to check your stopwords, not `.stop_words_`.\n",
    "\n",
    "We can now see that the new vectorizer has excluded adding the stopwords to its vocabulary, while the old one still includes them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'of' in word_vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'of' in word_vectorizer_stop.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make a vectorizer that will build 2-grams of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'collaborative filtering': 36,\n",
       " 'filtering cf': 58,\n",
       " 'cf is': 33,\n",
       " 'is technique': 103,\n",
       " 'technique used': 200,\n",
       " 'used by': 233,\n",
       " 'by recommender': 32,\n",
       " 'recommender systems': 177,\n",
       " 'filtering has': 60,\n",
       " 'has two': 83,\n",
       " 'two senses': 229,\n",
       " 'senses narrow': 182,\n",
       " 'narrow one': 133,\n",
       " 'one and': 156,\n",
       " 'and more': 9,\n",
       " 'more general': 129,\n",
       " 'general one': 77,\n",
       " 'in the': 89,\n",
       " 'the newer': 215,\n",
       " 'newer narrower': 135,\n",
       " 'narrower sense': 134,\n",
       " 'sense collaborative': 181,\n",
       " 'filtering is': 61,\n",
       " 'is method': 100,\n",
       " 'method of': 124,\n",
       " 'of making': 144,\n",
       " 'making automatic': 119,\n",
       " 'automatic predictions': 26,\n",
       " 'predictions filtering': 172,\n",
       " 'filtering about': 56,\n",
       " 'about the': 0,\n",
       " 'the interests': 212,\n",
       " 'interests of': 97,\n",
       " 'of user': 149,\n",
       " 'user by': 235,\n",
       " 'by collecting': 31,\n",
       " 'collecting preferences': 37,\n",
       " 'preferences or': 173,\n",
       " 'or taste': 163,\n",
       " 'taste information': 197,\n",
       " 'information from': 91,\n",
       " 'from many': 75,\n",
       " 'many users': 122,\n",
       " 'users collaborating': 239,\n",
       " 'the underlying': 221,\n",
       " 'underlying assumption': 231,\n",
       " 'assumption of': 25,\n",
       " 'of the': 147,\n",
       " 'the collaborative': 210,\n",
       " 'filtering approach': 57,\n",
       " 'approach is': 16,\n",
       " 'is that': 104,\n",
       " 'that if': 205,\n",
       " 'if person': 86,\n",
       " 'person has': 168,\n",
       " 'has the': 82,\n",
       " 'the same': 219,\n",
       " 'same opinion': 179,\n",
       " 'opinion as': 157,\n",
       " 'as person': 23,\n",
       " 'person on': 169,\n",
       " 'on an': 151,\n",
       " 'an issue': 6,\n",
       " 'issue is': 106,\n",
       " 'is more': 101,\n",
       " 'more likely': 130,\n",
       " 'likely to': 114,\n",
       " 'to have': 226,\n",
       " 'have opinion': 85,\n",
       " 'opinion on': 158,\n",
       " 'on different': 153,\n",
       " 'different issue': 46,\n",
       " 'issue than': 107,\n",
       " 'than that': 204,\n",
       " 'that of': 207,\n",
       " 'of randomly': 145,\n",
       " 'randomly chosen': 175,\n",
       " 'chosen person': 34,\n",
       " 'for example': 71,\n",
       " 'example collaborative': 54,\n",
       " 'filtering recommendation': 63,\n",
       " 'recommendation system': 176,\n",
       " 'system for': 196,\n",
       " 'for television': 73,\n",
       " 'television tastes': 203,\n",
       " 'tastes could': 198,\n",
       " 'could make': 39,\n",
       " 'make predictions': 118,\n",
       " 'predictions about': 170,\n",
       " 'about which': 1,\n",
       " 'which television': 245,\n",
       " 'television show': 202,\n",
       " 'show user': 188,\n",
       " 'user should': 237,\n",
       " 'should like': 187,\n",
       " 'like given': 113,\n",
       " 'given partial': 79,\n",
       " 'partial list': 166,\n",
       " 'list of': 116,\n",
       " 'of that': 146,\n",
       " 'that user': 209,\n",
       " 'user tastes': 238,\n",
       " 'tastes likes': 199,\n",
       " 'likes or': 115,\n",
       " 'or dislikes': 159,\n",
       " 'note that': 137,\n",
       " 'that these': 208,\n",
       " 'these predictions': 223,\n",
       " 'predictions are': 171,\n",
       " 'are specific': 19,\n",
       " 'specific to': 194,\n",
       " 'to the': 228,\n",
       " 'the user': 222,\n",
       " 'user but': 234,\n",
       " 'but use': 30,\n",
       " 'use information': 232,\n",
       " 'information gleaned': 92,\n",
       " 'gleaned from': 81,\n",
       " 'this differs': 224,\n",
       " 'differs from': 48,\n",
       " 'from the': 76,\n",
       " 'the simpler': 220,\n",
       " 'simpler approach': 189,\n",
       " 'approach of': 17,\n",
       " 'of giving': 142,\n",
       " 'giving an': 80,\n",
       " 'an average': 5,\n",
       " 'average non': 27,\n",
       " 'non specific': 136,\n",
       " 'specific score': 193,\n",
       " 'score for': 180,\n",
       " 'for each': 70,\n",
       " 'each item': 50,\n",
       " 'item of': 108,\n",
       " 'of interest': 143,\n",
       " 'interest for': 96,\n",
       " 'example based': 53,\n",
       " 'based on': 28,\n",
       " 'on its': 154,\n",
       " 'its number': 109,\n",
       " 'number of': 138,\n",
       " 'of votes': 150,\n",
       " 'the more': 214,\n",
       " 'general sense': 78,\n",
       " 'is the': 105,\n",
       " 'the process': 217,\n",
       " 'process of': 174,\n",
       " 'of filtering': 141,\n",
       " 'filtering for': 59,\n",
       " 'for information': 72,\n",
       " 'information or': 93,\n",
       " 'or patterns': 162,\n",
       " 'patterns using': 167,\n",
       " 'using techniques': 240,\n",
       " 'techniques involving': 201,\n",
       " 'involving collaboration': 99,\n",
       " 'collaboration among': 35,\n",
       " 'among multiple': 4,\n",
       " 'multiple agents': 131,\n",
       " 'agents viewpoints': 2,\n",
       " 'viewpoints data': 242,\n",
       " 'data sources': 44,\n",
       " 'sources etc': 191,\n",
       " 'applications of': 12,\n",
       " 'of collaborative': 139,\n",
       " 'filtering typically': 64,\n",
       " 'typically involve': 230,\n",
       " 'involve very': 98,\n",
       " 'very large': 241,\n",
       " 'large data': 112,\n",
       " 'data sets': 43,\n",
       " 'filtering methods': 62,\n",
       " 'methods have': 126,\n",
       " 'have been': 84,\n",
       " 'been applied': 29,\n",
       " 'applied to': 14,\n",
       " 'to many': 227,\n",
       " 'many different': 120,\n",
       " 'different kinds': 47,\n",
       " 'kinds of': 110,\n",
       " 'of data': 140,\n",
       " 'data including': 42,\n",
       " 'including sensing': 90,\n",
       " 'sensing and': 183,\n",
       " 'and monitoring': 8,\n",
       " 'monitoring data': 128,\n",
       " 'data such': 45,\n",
       " 'such as': 195,\n",
       " 'as in': 22,\n",
       " 'in mineral': 88,\n",
       " 'mineral exploration': 127,\n",
       " 'exploration environmental': 55,\n",
       " 'environmental sensing': 52,\n",
       " 'sensing over': 184,\n",
       " 'over large': 165,\n",
       " 'large areas': 111,\n",
       " 'areas or': 20,\n",
       " 'or multiple': 161,\n",
       " 'multiple sensors': 132,\n",
       " 'sensors financial': 185,\n",
       " 'financial data': 65,\n",
       " 'as financial': 21,\n",
       " 'financial service': 66,\n",
       " 'service institutions': 186,\n",
       " 'institutions that': 94,\n",
       " 'that integrate': 206,\n",
       " 'integrate many': 95,\n",
       " 'many financial': 121,\n",
       " 'financial sources': 67,\n",
       " 'sources or': 192,\n",
       " 'or in': 160,\n",
       " 'in electronic': 87,\n",
       " 'electronic commerce': 51,\n",
       " 'commerce and': 38,\n",
       " 'and web': 10,\n",
       " 'web applications': 243,\n",
       " 'applications where': 13,\n",
       " 'where the': 244,\n",
       " 'the focus': 211,\n",
       " 'focus is': 68,\n",
       " 'is on': 102,\n",
       " 'on user': 155,\n",
       " 'user data': 236,\n",
       " 'data etc': 41,\n",
       " 'the remainder': 218,\n",
       " 'remainder of': 178,\n",
       " 'of this': 148,\n",
       " 'this discussion': 225,\n",
       " 'discussion focuses': 49,\n",
       " 'focuses on': 69,\n",
       " 'on collaborative': 152,\n",
       " 'for user': 74,\n",
       " 'data although': 40,\n",
       " 'although some': 3,\n",
       " 'some of': 190,\n",
       " 'the methods': 213,\n",
       " 'methods and': 125,\n",
       " 'and approaches': 7,\n",
       " 'approaches may': 18,\n",
       " 'may apply': 123,\n",
       " 'apply to': 15,\n",
       " 'the other': 216,\n",
       " 'other major': 164,\n",
       " 'major applications': 117,\n",
       " 'applications as': 11,\n",
       " 'as well': 24}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2_vectorizer = TfidfVectorizer(ngram_range = (2, 2))\n",
    "X_word2 = word2_vectorizer.fit_transform(string_data)\n",
    "word2_vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A glimpse inside our feature matrix shows that sentence 0 has occurrences of terms 36, 58, and 33. The original sentence was \"Collaborative filtering (CF) is a technique used by recommender systems.\"\n",
    "\n",
    "Taking a look at the top of our vocabulary list above shows that these terms are:\n",
    "\n",
    "'collaborative filtering': 36\n",
    "'filtering cf': 58\n",
    "'cf is': 33\n",
    "\n",
    "So it's clear that the vectorizer is taking words that occur next to each other and pairing them. We can also see that 'collaborative filtering' (36) is downweighted compared to other word pairs. This is because the other word pairs are unique to this sentence, while 'collaborative filtering' is not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 36)\t0.15805741475500926\n",
      "  (0, 58)\t0.37321343063433804\n",
      "  (0, 33)\t0.37321343063433804\n",
      "  (0, 103)\t0.37321343063433804\n",
      "  (0, 200)\t0.37321343063433804\n",
      "  (0, 233)\t0.37321343063433804\n",
      "  (0, 32)\t0.37321343063433804\n",
      "  (0, 177)\t0.37321343063433804\n"
     ]
    }
   ],
   "source": [
    "print(X_word2[0, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, as we know from our stopwords experiments above, 'collaborative filtering' appears in 9 out of 11 sentences, and is weighted with varying significance in each sentence. (Generally speaking: shorter sentence, higher weighting.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 0)\t0.15805741475500926\n",
      "  (1, 0)\t0.14187939579983094\n",
      "  (2, 0)\t0.08332488626530703\n",
      "  (3, 0)\t0.07272669729102695\n",
      "  (4, 0)\t0.08163961550995026\n",
      "  (7, 0)\t0.08675073761032723\n",
      "  (8, 0)\t0.14808056479624188\n",
      "  (9, 0)\t0.05517764812255034\n",
      "  (10, 0)\t0.08452882670491463\n"
     ]
    }
   ],
   "source": [
    "print(X_word2[:, 36])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, a character vectorizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'c': 11,\n",
       " 'o': 23,\n",
       " 'l': 20,\n",
       " 'a': 9,\n",
       " 'b': 10,\n",
       " 'r': 26,\n",
       " 't': 28,\n",
       " 'i': 17,\n",
       " 'v': 30,\n",
       " 'e': 13,\n",
       " ' ': 0,\n",
       " 'f': 14,\n",
       " 'n': 22,\n",
       " 'g': 15,\n",
       " '(': 2,\n",
       " ')': 3,\n",
       " 's': 27,\n",
       " 'h': 16,\n",
       " 'q': 25,\n",
       " 'u': 29,\n",
       " 'd': 12,\n",
       " 'y': 33,\n",
       " 'm': 21,\n",
       " 'w': 31,\n",
       " ',': 4,\n",
       " 'k': 19,\n",
       " 'p': 24,\n",
       " \"'\": 1,\n",
       " 'x': 32,\n",
       " '-': 5,\n",
       " ':': 7,\n",
       " ';': 8,\n",
       " 'j': 18,\n",
       " '.': 6}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_vectorizer = TfidfVectorizer(analyzer = 'char')\n",
    "X_char = char_vectorizer.fit_transform(string_data)\n",
    "char_vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's evident looking at this vocabulary that it would probably be appropriate to remove any symbol and space characters as stopwords if using a character vectorizer. Finally, let's see what a 2-character vectorizer looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'co': 65,\n",
       " 'ol': 177,\n",
       " 'll': 137,\n",
       " 'la': 133,\n",
       " 'ab': 34,\n",
       " 'bo': 54,\n",
       " 'or': 181,\n",
       " 'ra': 199,\n",
       " 'at': 46,\n",
       " 'ti': 235,\n",
       " 'iv': 127,\n",
       " 've': 253,\n",
       " 'e ': 74,\n",
       " ' f': 6,\n",
       " 'fi': 96,\n",
       " 'il': 118,\n",
       " 'lt': 139,\n",
       " 'te': 233,\n",
       " 'er': 86,\n",
       " 'ri': 203,\n",
       " 'in': 120,\n",
       " 'ng': 162,\n",
       " 'g ': 99,\n",
       " ' (': 0,\n",
       " '(c': 23,\n",
       " 'cf': 61,\n",
       " 'f)': 93,\n",
       " ') ': 27,\n",
       " ' i': 9,\n",
       " 'is': 125,\n",
       " 's ': 213,\n",
       " ' a': 1,\n",
       " 'a ': 32,\n",
       " ' t': 18,\n",
       " 'ec': 78,\n",
       " 'ch': 62,\n",
       " 'hn': 111,\n",
       " 'ni': 163,\n",
       " 'iq': 123,\n",
       " 'qu': 195,\n",
       " 'ue': 245,\n",
       " ' u': 19,\n",
       " 'us': 251,\n",
       " 'se': 219,\n",
       " 'ed': 79,\n",
       " 'd ': 68,\n",
       " ' b': 2,\n",
       " 'by': 56,\n",
       " 'y ': 263,\n",
       " ' r': 16,\n",
       " 're': 201,\n",
       " 'om': 178,\n",
       " 'mm': 149,\n",
       " 'me': 146,\n",
       " 'en': 85,\n",
       " 'nd': 159,\n",
       " 'de': 70,\n",
       " 'r ': 196,\n",
       " ' s': 17,\n",
       " 'sy': 228,\n",
       " 'ys': 266,\n",
       " 'st': 226,\n",
       " 'em': 84,\n",
       " 'ms': 152,\n",
       " ' h': 8,\n",
       " 'ha': 108,\n",
       " 'as': 45,\n",
       " 'tw': 241,\n",
       " 'wo': 259,\n",
       " 'o ': 171,\n",
       " 'ns': 166,\n",
       " 'es': 87,\n",
       " 's,': 215,\n",
       " ', ': 28,\n",
       " ' n': 13,\n",
       " 'na': 157,\n",
       " 'ar': 44,\n",
       " 'rr': 208,\n",
       " 'ro': 207,\n",
       " 'ow': 186,\n",
       " 'w ': 256,\n",
       " ' o': 14,\n",
       " 'on': 179,\n",
       " 'ne': 160,\n",
       " 'an': 42,\n",
       " ' m': 12,\n",
       " 'mo': 150,\n",
       " ' g': 7,\n",
       " 'ge': 102,\n",
       " 'al': 40,\n",
       " 'l ': 131,\n",
       " 'n ': 154,\n",
       " 'th': 234,\n",
       " 'he': 109,\n",
       " 'ew': 90,\n",
       " 'we': 257,\n",
       " 'r,': 198,\n",
       " 'e,': 75,\n",
       " ' c': 3,\n",
       " 'et': 88,\n",
       " 'ho': 112,\n",
       " 'od': 174,\n",
       " 'of': 175,\n",
       " 'f ': 92,\n",
       " 'ma': 144,\n",
       " 'ak': 39,\n",
       " 'ki': 130,\n",
       " 'au': 47,\n",
       " 'ut': 252,\n",
       " 'to': 236,\n",
       " 'ic': 114,\n",
       " 'c ': 57,\n",
       " ' p': 15,\n",
       " 'pr': 193,\n",
       " 'di': 71,\n",
       " 'ct': 66,\n",
       " 'io': 121,\n",
       " '(f': 24,\n",
       " 'g)': 100,\n",
       " 'ou': 184,\n",
       " 't ': 229,\n",
       " 'nt': 167,\n",
       " 'ts': 238,\n",
       " 'le': 135,\n",
       " 'ef': 81,\n",
       " 'fe': 94,\n",
       " 'nc': 158,\n",
       " 'ce': 60,\n",
       " 'ta': 231,\n",
       " 'nf': 161,\n",
       " 'fo': 97,\n",
       " 'rm': 205,\n",
       " 'fr': 98,\n",
       " 'm ': 143,\n",
       " 'ny': 170,\n",
       " 'rs': 209,\n",
       " 'un': 249,\n",
       " 'rl': 204,\n",
       " 'ly': 142,\n",
       " 'yi': 264,\n",
       " 'ss': 225,\n",
       " 'su': 227,\n",
       " 'um': 248,\n",
       " 'mp': 151,\n",
       " 'pt': 194,\n",
       " 'ap': 43,\n",
       " 'pp': 192,\n",
       " 'oa': 172,\n",
       " 'ac': 35,\n",
       " 'h ': 107,\n",
       " 'if': 116,\n",
       " 'pe': 188,\n",
       " 'so': 223,\n",
       " 'sa': 217,\n",
       " 'am': 41,\n",
       " 'op': 180,\n",
       " 'pi': 189,\n",
       " 'b ': 50,\n",
       " ' l': 11,\n",
       " 'li': 136,\n",
       " 'ik': 117,\n",
       " 'ke': 129,\n",
       " 'el': 83,\n",
       " 'av': 48,\n",
       " \"b'\": 51,\n",
       " \"'s\": 22,\n",
       " ' d': 4,\n",
       " 'ff': 95,\n",
       " 'do': 72,\n",
       " 'ml': 148,\n",
       " 'os': 182,\n",
       " ' e': 5,\n",
       " 'ex': 91,\n",
       " 'xa': 261,\n",
       " 'pl': 190,\n",
       " 'da': 69,\n",
       " 'ev': 89,\n",
       " 'vi': 254,\n",
       " 'si': 221,\n",
       " 'ul': 247,\n",
       " 'ld': 134,\n",
       " ' w': 21,\n",
       " 'wh': 258,\n",
       " 'hi': 110,\n",
       " 'sh': 220,\n",
       " 'gi': 104,\n",
       " 'pa': 187,\n",
       " 'rt': 210,\n",
       " 'ia': 113,\n",
       " \"r'\": 197,\n",
       " '(l': 25,\n",
       " 'sl': 222,\n",
       " 's)': 214,\n",
       " 'no': 165,\n",
       " 'ot': 183,\n",
       " 'sp': 224,\n",
       " 'ci': 63,\n",
       " 'bu': 55,\n",
       " 'gl': 105,\n",
       " 'ea': 76,\n",
       " 'im': 119,\n",
       " 'ag': 36,\n",
       " '(n': 26,\n",
       " 'n-': 156,\n",
       " '-s': 29,\n",
       " 'c)': 58,\n",
       " 'sc': 218,\n",
       " 'it': 126,\n",
       " 't,': 230,\n",
       " 'ba': 52,\n",
       " 'nu': 168,\n",
       " 'mb': 145,\n",
       " 'be': 53,\n",
       " ' v': 20,\n",
       " 'vo': 255,\n",
       " 'oc': 173,\n",
       " 'tt': 239,\n",
       " 'rn': 206,\n",
       " 'nv': 169,\n",
       " 'lv': 141,\n",
       " 'mu': 153,\n",
       " 'ip': 122,\n",
       " 'ie': 115,\n",
       " 'wp': 260,\n",
       " 'po': 191,\n",
       " 'oi': 176,\n",
       " 'ur': 250,\n",
       " 'rc': 200,\n",
       " 'tc': 232,\n",
       " 'ca': 59,\n",
       " 'ty': 242,\n",
       " 'yp': 265,\n",
       " 'ry': 212,\n",
       " 'rg': 202,\n",
       " 'ds': 73,\n",
       " 'ee': 80,\n",
       " ' k': 10,\n",
       " 'cl': 64,\n",
       " 'lu': 140,\n",
       " 'ud': 244,\n",
       " 'g:': 101,\n",
       " ': ': 30,\n",
       " 'a,': 33,\n",
       " 'uc': 243,\n",
       " 'mi': 147,\n",
       " 'xp': 262,\n",
       " 'lo': 138,\n",
       " 'n,': 155,\n",
       " 'ir': 124,\n",
       " 'nm': 164,\n",
       " 'ov': 185,\n",
       " 's;': 216,\n",
       " '; ': 31,\n",
       " 'rv': 211,\n",
       " 'tu': 240,\n",
       " 'eg': 82,\n",
       " 'gr': 106,\n",
       " 'tr': 237,\n",
       " 'eb': 77,\n",
       " 'cu': 67,\n",
       " 'ai': 37,\n",
       " 'ug': 246,\n",
       " 'gh': 103,\n",
       " 'ay': 49,\n",
       " 'aj': 38,\n",
       " 'jo': 128,\n",
       " 'l.': 132}"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now a 2-character vectorizer!\n",
    "\n",
    "char2_vectorizer = TfidfVectorizer(analyzer = 'char', ngram_range = (2, 2))\n",
    "X_char2 = char2_vectorizer.fit_transform(string_data)\n",
    "char2_vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<11x267 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 1054 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_char2"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}