tf_idf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3a6e9b9",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook illustrates some basic document handling using [Spacy] (https://spacy.io/). Spacy is fast, and powerful, but not completely trivial to understand. There are though lots of useful resources, and the documentation is excellent.\n",
    "\n",
    "**The first block of our code simply sets things up - most important here is the language model that we use.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d5a2314b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy #Our NLP tools\n",
    "from collections import Counter #We will use this to do simple counts of terms\n",
    "\n",
    "#Load a German language model to do German NLP - the models we use will influence our results a lot\n",
    "#nlp = spacy.load('de_core_news_md')\n",
    "#And an English language model for English\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7881385",
   "metadata": {},
   "source": [
    "Now we load a default list of stop words and print them out.\n",
    "\n",
    "Look through the list of stop words, and consider what issues they might cause if we are interested in spatial relationships?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c8a2d9dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "326\n",
      "{'such', 'here', 'take', 'those', 'only', 'namely', 'or', 'and', 'again', 'few', \"'m\", 'whereby', 'using', 'that', 'fifty', 'thence', 'herself', 'enough', 'something', 'toward', 'whenever', '‘re', 'during', 'n‘t', 'hereafter', 'various', 'meanwhile', 'his', 'doing', \"'ll\", 'had', 'make', 'too', 'each', 'under', \"'re\", 'else', 'almost', 'get', 'give', 'say', '‘m', 'seemed', 'over', 'very', 'noone', 'formerly', 'what', 'yourselves', 'since', 'themselves', 'three', '’ll', 'nevertheless', 'down', 'thereupon', 'behind', 'whither', 'yours', 'seems', \"'d\", 'done', 'by', '’s', 'nor', 'do', 'empty', 'there', 'mine', 'everywhere', 'because', 'anywhere', 'ten', 'six', 'not', 'neither', 'about', '‘ll', 'has', 'sometimes', 'whoever', 'became', 'off', 'besides', 'thru', 'elsewhere', 'least', 'but', 'me', 'sometime', 'out', 'hereby', 'amount', 'ourselves', \"'ve\", 'as', 'latter', 'eight', 'every', 'at', 'amongst', 'above', 'never', 'call', 'then', 'her', 'together', 'its', 'anything', 'everyone', '‘d', 'further', 'will', 'see', '‘ve', 'via', 'none', 'another', 'would', 'now', 'yourself', 'except', 'forty', 'nowhere', 'therefore', 'once', 'therein', 'although', 'whence', 'bottom', 'thereafter', '’m', 'any', 'is', '’d', 'either', 'was', \"n't\", 'everything', 'alone', 'sixty', 'on', 'go', 'first', 'along', 'nine', 'two', 'we', 'thereby', 'regarding', 'anyhow', 'others', 'being', 'somehow', 'been', 'fifteen', 'next', 'former', 'of', 'same', 'across', 'my', 'wherein', 'who', 'himself', 'him', 'ever', 'wherever', 'front', 'hundred', 'less', 'twelve', 'against', 'throughout', 'mostly', 'must', 'from', 'whatever', 'after', 'often', 'whereas', 'upon', 'anyone', 'be', 'five', 'quite', 'seeming', 'both', 'so', 'onto', 'made', 'whereupon', 'no', 'most', 'much', 'becoming', 'which', 'to', 'unless', 'whether', 'whom', 'hers', 'show', 'may', 'twenty', 'around', 'becomes', 'really', 'it', 'until', 'this', 'are', 'i', 'whose', 'a', 'these', '’ve', 'towards', 'other', 'without', 'part', 'into', 'why', 'did', 'should', 'our', 'four', 'they', 'well', 'more', 'among', 'myself', '’re', 'within', 'even', 'last', 'move', 'nobody', 'ca', 'an', \"'s\", 'for', 'am', 'does', 'always', 'please', 'hereupon', 'how', 'were', 'some', 'rather', 'can', 'with', 'though', 'thus', 'itself', 'them', 'seem', 'serious', 'indeed', 'per', 'cannot', 'due', 'could', 'just', 'before', 'whole', 'also', 'back', 'side', 'nothing', 'through', 'whereafter', 'afterwards', 'while', 'the', 'all', 'in', 'beyond', 'used', 'beforehand', 'below', 'put', 'latterly', 'become', 'moreover', 'n’t', 'name', 'many', 'when', 'perhaps', 'top', 'she', 'where', 'still', 'anyway', 'already', '‘s', 'own', 'if', 'us', 'keep', 'several', 'beside', 'full', 'hence', 'you', 'however', 're', 'your', 'eleven', 'otherwise', 'someone', 'one', 'third', 'might', 'yet', 'have', 'herein', 'somewhere', 'he', 'than', 'up', 'ours', 'their', 'between'}\n"
     ]
    }
   ],
   "source": [
    "# This block loads our stop words \n",
    "stopwords = nlp.Defaults.stop_words\n",
    "\n",
    "print(len(stopwords))\n",
    "print(stopwords)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36485a9d",
   "metadata": {},
   "source": [
    "This function calculates the term frequency for a given document. At the moment it removes stop words and punctuation and stores lemmas in the index. Experiment with changing this - what happens to the end results?\n",
    "\n",
    "`token.text` returns the raw string from a document\n",
    "\n",
    "`token.lemma_` returns the lemmatised string\n",
    "\n",
    "`token.is_stop` returns a `boolean` value - `True` if the text is a stop word\n",
    "\n",
    "`token.is_punct` returns a `boolean` value - `True` if the text is punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7aeb185c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tf(text):\n",
    "    doc = nlp(text)\n",
    "    n = len(doc)\n",
    "    \n",
    "    terms = [token.lemma_ for token in doc if not token.is_stop and not token.is_punct ]\n",
    "    tf = dict(Counter(terms))\n",
    "    \n",
    "    for term, count in tf.items():\n",
    "        tf[term] = count/n\n",
    "        \n",
    "    return tf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4ccd5ce",
   "metadata": {},
   "source": [
    "This function calculates the document frequency - that means we run it once over our whole collection, and it only changes if we add new documents. It is important that it is consistent with our term frequency. If you change the way we calculate tf, you need to change, and recalculate df."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5e7f1223",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "def df(texts):\n",
    "    df = dict()\n",
    "    ndocs = len(texts)\n",
    "    for text in texts:\n",
    "        doc = nlp(text)\n",
    "        terms = [token.lemma_ for token in doc if not token.is_stop and not token.is_punct ]\n",
    "        tf = set(terms)\n",
    "        for t in tf:\n",
    "            if t in df:\n",
    "                count = df[t]\n",
    "                count = count + 1\n",
    "            else:\n",
    "                count = 1\n",
    "            df[t] = count/ndocs\n",
    "    for term, count in df.items():\n",
    "        df[term] = math.log10(ndocs/(count + 1))\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a262fcd",
   "metadata": {},
   "source": [
    "Now we can calculate tf idf values for our corpus - that means we have a score for every word in every document, that can then be used in ranking. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9eec5dc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tfidf(corpus):\n",
    "    idf = df(list(corpus.values()))\n",
    "    results = {}\n",
    "    for id, text in corpus.items():                \n",
    "        t = tf(text)\n",
    "        scores = {}\n",
    "        for term in t:\n",
    "            scores[term] = t[term]*idf[term]\n",
    "        results[id] = scores\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9748c2b",
   "metadata": {},
   "source": [
    "This our search algorithm. We take a query, and the tfidf scores, and rank each document according to its score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2108bbad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def simpleSearch(query, weights):\n",
    "    q = nlp(query)\n",
    "    \n",
    "    #Iterate through each document and add its tf-idf score\n",
    "    results = {}\n",
    "    for id, scores in weights.items():\n",
    "        score = 0\n",
    "        for token in q:\n",
    "            if token.lemma_ in scores:\n",
    "                score = score + scores[token.lemma_]\n",
    "        results[id] = score\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2204d468",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.10036649796713228, 2: 0.20073299593426455, 3: 0.24087959512111748, 4: 0.08602842682897052, 5: 0.12043979756055874}\n"
     ]
    }
   ],
   "source": [
    "corpus = {1:\"the cat sat on the mat\",\n",
    "          2:\"the dog played with the cat\",\n",
    "          3:\"the cat bit the dog\",\n",
    "          4:\"the boy was playing with the dog\",\n",
    "          5:\"the girl saw the cat biting the dog far away\"}\n",
    "\n",
    "weights = tfidf(corpus)\n",
    "\n",
    "results = simpleSearch('cat dog', weights)\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f90f615",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}