2 - NMF Topic Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NMF Topic Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topic modelling aims to automatically discover the hidden thematic structure in a large corpus of text documents. One approach for topic modelling is to apply *matrix factorisation* methods, such as *Non-negative Matrix Factorisation (NMF)*. In this notebook we look at how to apply NMF using the *scikit-learn* library in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "import numpy as np\n",
    "from sklearn import decomposition\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "# settings for our plots later\n",
    "plt.style.use(\"ggplot\")\n",
    "matplotlib.rcParams.update({\"font.size\": 14})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying NMF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's load the TF-IDF normalised document-term matrix and list of terms that we stored earlier using *Joblib*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(A, terms, snippets) = joblib.load(\"articles-tfidf.pkl\")\n",
    "print(\"Loaded %d X %d document-term matrix\" % (A.shape[0], A.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key input parameter to NMF is the number of topics to generate *k*. For the moment, we will pre-specify a guessed value, for demonstration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another choice for NMF revolves around initialisation. Most commonly, NMF involves using random initialisation to populate the values in the factors W and H. Depending on the random seed that you use, you may get different results on the same dataset. Instead, using SVD-based initialisation provides more reliable results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the model, specifiying the initialization strategy and the number of topics to produce\n",
    "model = decomposition.NMF(init=\"nndsvd\", n_components=k) \n",
    "# apply the model and extract the two factor matrices\n",
    "W = model.fit_transform( A )\n",
    "H = model.components_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examining the Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NMF produces to factor matrices as its output: *W* and *H*.\n",
    "\n",
    "The *W* factor contains the document membership weights relative to each of the *k* topics. Each row corresponds to a single document, and each column correspond to a topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "W.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, for the first document, we see that it is strongly associated with one topic. However,  each document can be potentially associated with multiple topics to different degrees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# round to 2 decimal places for display purposes\n",
    "W[0,:].round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *H* factor contains the term weights relative to each of the *k* topics. In this case, each row corresponds to a topic, and each column corresponds to a unique term in the corpus vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, for the term \"brexit\", we see that it is strongly associated with a single topic. Again, in some cases each term can be associated with multiple topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "term_index = terms.index('brexit')\n",
    "# round to 2 decimal places for display purposes\n",
    "H[:,term_index].round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic Descriptors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top ranked terms from the *H* factor for each topic can give us an insight into the content of that topic. This is often called the *topic descriptor*. Let's define a function that extracts the descriptor for a specified topic: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_descriptor(terms, H, topic_index, top):\n",
    "    # reverse sort the values to sort the indices\n",
    "    top_indices = np.argsort(H[topic_index,:])[::-1]\n",
    "    # now get the terms corresponding to the top-ranked indices\n",
    "    top_terms = []\n",
    "    for term_index in top_indices[0:top]:\n",
    "        top_terms.append(terms[term_index])\n",
    "    return top_terms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now get a descriptor for each topic using the top ranked terms (e.g. top 10):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "descriptors = []\n",
    "for topic_index in range(k):\n",
    "    descriptors.append(get_descriptor( terms, H, topic_index, 10))\n",
    "    str_descriptor = \", \".join(descriptors[topic_index])\n",
    "    print(\"Topic %02d: %s\" % (topic_index+1, str_descriptor))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rankings above do not show the strength of association for the different terms. We can represent the distribution of the weights for the top terms in a topic using a *matplotlib* horizontal bar chart."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function to create a bar chart for the specified topic, based on the *H* factor from the current NMF model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_top_term_weights(terms, H, topic_index, top):\n",
    "    # get the top terms and their weights\n",
    "    top_indices = np.argsort(H[topic_index,:])[::-1]\n",
    "    top_terms = []\n",
    "    top_weights = []\n",
    "    for term_index in top_indices[0:top]:\n",
    "        top_terms.append(terms[term_index])\n",
    "        top_weights.append(H[topic_index,term_index])\n",
    "    # note we reverse the ordering for the plot\n",
    "    top_terms.reverse()\n",
    "    top_weights.reverse()\n",
    "    # create the plot\n",
    "    fig = plt.figure(figsize=(13,8))\n",
    "    # add the horizontal bar chart\n",
    "    ypos = np.arange(top)\n",
    "    ax = plt.barh(ypos, top_weights, align=\"center\", color=\"green\",tick_label=top_terms)\n",
    "    plt.xlabel(\"Term Weight\",fontsize=14)\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So for instance, for the 7th topic we can generate a plot with the top 15 terms using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_top_term_weights(terms, H, 6, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, the top 15 terms for the 4th topic in our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_top_term_weights(terms, H, 4, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Most Relevant Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the snippets for the top-ranked documents for each topic. We'll define a function to produce this ranking also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_top_snippets(all_snippets, W, topic_index, top):\n",
    "    # reverse sort the values to sort the indices\n",
    "    top_indices = np.argsort( W[:,topic_index] )[::-1]\n",
    "    # now get the snippets corresponding to the top-ranked indices\n",
    "    top_snippets = []\n",
    "    for doc_index in top_indices[0:top]:\n",
    "        top_snippets.append(all_snippets[doc_index])\n",
    "    return top_snippets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, for the first topic listed above, the top 10 documents are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_snippets = get_top_snippets(snippets, W, 0, 10)\n",
    "for i, snippet in enumerate(topic_snippets):\n",
    "    print(\"%02d. %s\" % ((i+1), snippet ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, for the second topic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_snippets = get_top_snippets(snippets, W, 1, 10)\n",
    "for i, snippet in enumerate(topic_snippets):\n",
    "    print(\"%02d. %s\" % ((i+1), snippet))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exporting the Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to keep this topic model for later user, we can save it using *joblib*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "joblib.dump((W,H,terms,snippets), \"articles-model-nmf-k%02d.pkl\" % k) "
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "a3dd349d56abd7701b2e4d88ab44696293e44ecedbdcf133fcb5722b3cee9765"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}