15_topic_modeling/05_lda_with_gensim.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Topic Modeling: Latent Dirichlet Allocation with gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim is a specialized NLP library with a fast LDA implementation and many additional features. We will also use it in the next chapter on word vectors (see the notebook lda_with_gensim for details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Imports & Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:27.604182Z",
     "start_time": "2021-04-25T21:52:27.602354Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.493469Z",
     "start_time": "2021-04-25T21:52:27.605558Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/scipy/sparse/sparsetools.py:21: DeprecationWarning: `scipy.sparse.sparsetools` is deprecated!\n",
      "scipy.sparse.sparsetools is a private module for scipy.sparse, and should not be used.\n",
      "  _deprecated()\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "\n",
    "# Visualization\n",
    "import seaborn as sns\n",
    "import pyLDAvis\n",
    "\n",
    "# sklearn for feature extraction & modeling\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "import joblib\n",
    "\n",
    "# gensim for alternative models\n",
    "from gensim.models import LdaModel\n",
    "from gensim.corpora import Dictionary\n",
    "from gensim.matutils import Sparse2Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.496699Z",
     "start_time": "2021-04-25T21:52:28.494499Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "sns.set_style('white')\n",
    "pyLDAvis.enable_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Load BBC data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.511553Z",
     "start_time": "2021-04-25T21:52:28.497754Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "# change to your data path if necessary\n",
    "DATA_DIR = Path('../data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.577125Z",
     "start_time": "2021-04-25T21:52:28.512861Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "path = DATA_DIR / 'bbc'\n",
    "files = path.glob('**/*.txt')\n",
    "doc_list = []\n",
    "for i, file in enumerate(files):\n",
    "    with open(str(file), encoding='latin1') as f:\n",
    "        topic = file.parts[-2]\n",
    "        lines = f.readlines()\n",
    "        heading = lines[0].strip()\n",
    "        body = ' '.join([l.strip() for l in lines[1:]])\n",
    "        doc_list.append([topic.capitalize(), heading, body])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Convert to DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.588113Z",
     "start_time": "2021-04-25T21:52:28.578305Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2225 entries, 0 to 2224\n",
      "Data columns (total 3 columns):\n",
      " #   Column   Non-Null Count  Dtype \n",
      "---  ------   --------------  ----- \n",
      " 0   topic    2225 non-null   object\n",
      " 1   heading  2225 non-null   object\n",
      " 2   article  2225 non-null   object\n",
      "dtypes: object(3)\n",
      "memory usage: 52.3+ KB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "docs = pd.DataFrame(doc_list, columns=['topic', 'heading', 'article'])\n",
    "docs.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Create Train & Test Sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.601178Z",
     "start_time": "2021-04-25T21:52:28.589242Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "train_docs, test_docs = train_test_split(docs, \n",
    "                                         stratify=docs.topic, \n",
    "                                         test_size=50, \n",
    "                                         random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.610787Z",
     "start_time": "2021-04-25T21:52:28.602193Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "((2175, 3), (50, 3))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_docs.shape, test_docs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:28.621570Z",
     "start_time": "2021-04-25T21:52:28.612418Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Sport            12\n",
       "Business         11\n",
       "Entertainment     9\n",
       "Tech              9\n",
       "Politics          9\n",
       "Name: topic, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(test_docs.topic).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorize train & test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:29.032778Z",
     "start_time": "2021-04-25T21:52:28.622630Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<2175x2000 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 179068 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(max_df=.2, \n",
    "                             min_df=3, \n",
    "                             stop_words='english', \n",
    "                             max_features=2000)\n",
    "\n",
    "train_dtm = vectorizer.fit_transform(train_docs.article)\n",
    "words = vectorizer.get_feature_names()\n",
    "train_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:29.046231Z",
     "start_time": "2021-04-25T21:52:29.033869Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<50x2000 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 4150 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dtm = vectorizer.transform(test_docs.article)\n",
    "test_dtm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## LDA with gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Using `CountVectorizer` Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:29.453031Z",
     "start_time": "2021-04-25T21:52:29.047292Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "max_df = .2\n",
    "min_df = 3\n",
    "max_features = 2000\n",
    "\n",
    "# used by sklearn: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/stop_words.py\n",
    "stop_words = pd.read_csv('http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words', \n",
    "                         header=None, \n",
    "                         squeeze=True).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:29.903869Z",
     "start_time": "2021-04-25T21:52:29.454952Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(max_df=max_df, \n",
    "                             min_df=min_df, \n",
    "                             stop_words='english', \n",
    "                             max_features=max_features)\n",
    "\n",
    "train_dtm = vectorizer.fit_transform(train_docs.article)\n",
    "test_dtm = vectorizer.transform(test_docs.article)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert sklearn DTM to gensim data structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It faciltiates the conversion of DTM produced by sklearn to gensim data structures as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:29.908984Z",
     "start_time": "2021-04-25T21:52:29.904928Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "train_corpus = Sparse2Corpus(train_dtm, documents_columns=False)\n",
    "test_corpus = Sparse2Corpus(test_dtm, documents_columns=False)\n",
    "id2word = pd.Series(vectorizer.get_feature_names()).to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Train Model & Review Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:32.023411Z",
     "start_time": "2021-04-25T21:52:29.910170Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "LdaModel(corpus=train_corpus, \n",
    "         num_topics=100, \n",
    "         id2word=None, \n",
    "         distributed=False, \n",
    "         chunksize=2000,                   # Number of documents to be used in each training chunk.\n",
    "         passes=1,                         # Number of passes through the corpus during training\n",
    "         update_every=1,                   # Number of docs to be iterated through for each update\n",
    "         alpha='symmetric', \n",
    "         eta=None,                         # a-priori belief on word probability\n",
    "         decay=0.5,                        # percentage of previous lambda forgotten when new document is examined\n",
    "         offset=1.0,                       # controls slow down of the first steps the first few iterations.\n",
    "         eval_every=10,                    # estimate log perplexity\n",
    "         iterations=50,                    # Maximum number of iterations through the corpus\n",
    "         gamma_threshold=0.001,            # Minimum change in the value of the gamma parameters to continue iterating\n",
    "         minimum_probability=0.01,         # Topics with a probability lower than this threshold will be filtered out\n",
    "         random_state=None, \n",
    "         ns_conf=None, \n",
    "         minimum_phi_value=0.01,           # if `per_word_topics` is True, represents lower bound on term probabilities\n",
    "         per_word_topics=False,            #  If True, compute a list of most likely topics for each word with phi values multiplied by word count\n",
    "         callbacks=None);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:32.026437Z",
     "start_time": "2021-04-25T21:52:32.024368Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "num_topics = 5\n",
    "topic_labels = ['Topic {}'.format(i) for i in range(1, num_topics+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:33.090786Z",
     "start_time": "2021-04-25T21:52:32.027511Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "lda_gensim = LdaModel(corpus=train_corpus,\n",
    "                      num_topics=num_topics,\n",
    "                      id2word=id2word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:33.095280Z",
     "start_time": "2021-04-25T21:52:33.091733Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0,\n",
       " '0.007*\"company\" + 0.005*\"party\" + 0.005*\"best\" + 0.004*\"says\" + 0.004*\"group\" + 0.004*\"film\" + 0.004*\"court\" + 0.004*\"games\" + 0.004*\"legal\" + 0.003*\"oil\"')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics = lda_gensim.print_topics()\n",
    "topics[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Evaluate Topic Coherence\n",
    "\n",
    "Topic Coherence measures whether the words in a topic tend to co-occur together. \n",
    "\n",
    "- It adds up a score for each distinct pair of top ranked words. \n",
    "- The score is the log of the probability that a document containing at least one instance of the higher-ranked word also contains at least one instance of the lower-ranked word.\n",
    "\n",
    "Large negative values indicate words that don't co-occur often; values closer to zero indicate that words tend to co-occur more often."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:33.165123Z",
     "start_time": "2021-04-25T21:52:33.096559Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "coherence = lda_gensim.top_topics(corpus=train_corpus, coherence='u_mass')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim permits topic coherence evaluation that produces the topic coherence and shows the most important words per topic: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:33.289360Z",
     "start_time": "2021-04-25T21:52:33.166273Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Topic 1           Topic 2         Topic 3          Topic 4        Topic 5         \n",
      "     prob      term    prob    term    prob     term    prob   term    prob     term\n",
      "0   1.10%    labour   0.91%  mobile   0.70%  company   0.77%   game   0.62%  england\n",
      "1   0.80%  election   0.64%   games   0.55%    party   0.69%  sales   0.56%     best\n",
      "2   0.75%     blair   0.57%     web   0.46%     best   0.55%  games   0.52%      win\n",
      "3   0.69%     brown   0.53%  search   0.42%     says   0.47%   deal   0.50%      won\n",
      "4   0.60%    howard   0.52%     net   0.41%    group   0.45%   good   0.41%     team\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAAEVCAYAAAAW1KgZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAY9klEQVR4nO3df2xd5X0/8Pe1TUaawNak+HrL3KCw9Mcf0ZCm0CFQjLx4AQJJljZi7aTNIQm0Y6JOt6GvOpSuYYMiDRWJVm3itDgZVcQoIYR41UrdUY+sqtStm7tpP5SqUUM2X2qjriM/msT4+weqUeR7CM2JfS/j9frrnuc89z6fHH3+eOfkyTmVycnJyQAAANO0NLoAAABoVsIyAAAUEJYBAKCAsAwAAAWEZQAAKCAsAwBAgYsSloeHh7Nq1ar09PRk586d085PTk7mz/7sz9LT05Nbb701//qv/3oxlgUAgBlVOixPTExk+/bt2bVrVwYHB3Pw4MEcPnz4nDnDw8M5cuRIvvrVr+a+++7Ln/7pn5ZdFgAAZlzpsDwyMpLFixens7Mzc+bMyerVqzM0NHTOnKGhoaxbty6VSiVXX311fvzjH+fFF18suzQAAMyo0mG5Vqulo6Nj6rharaZWq73unI6OjmlzAACg2bSV/YF6b8uuVCo/85x63ve+92XRokUXXtxFMDmZvIFS3xJci9e4Fq9xLV7jWrzGtXiNa/Ea1+I1rsVrmuFaHDt2LN/61rfqnisdljs6OjI6Ojp1XKvV0t7e/rpzRkdHp82pZ9GiRdm3b1/ZEku78v8NNrqEpnDkU6sbXUJT0Rev0hfn0hev0hfn0hev0hfn0hevaoa+WL9+feG50tswli1bliNHjuTo0aM5ffp0BgcH093dfc6c7u7u7N+/P5OTk/mnf/qnXHbZZW8oLAMAQCOVvrPc1taWbdu2ZfPmzZmYmMj73//+LF26NHv37k2SfPCDH0xXV1e+8Y1vpKenJ3Pnzs39999funAAAJhppcNyknR1daWrq+ucsQ9+8INTnyuVSj7xiU9cjKUAAGDWeIMfAAAUEJYBAKCAsAwAAAWEZQAAKCAsAwBAAWEZAAAKCMsAAFBAWAYAgALCMgAAFBCWAQCggLAMAAAFhGUAACggLAMAQAFhGQAACgjLAABQQFgGAIACwjIAABQQlgEAoEBbmS//6Ec/ytatW3Ps2LEsWrQoDz/8cH7+539+2rzu7u7MmzcvLS0taW1tzb59+8osCwAAs6LUneWdO3fm2muvzVe/+tVce+212blzZ+Hc3bt35+mnnxaUAQB40ygVloeGhrJu3bokybp16/K1r33tYtQEAABNoVRYHh8fT3t7e5Kkvb09L730UuHcTZs2Zf369Xn88cfLLAkAALPmvHuWe3t7MzY2Nm28r6/vDS+yd+/eVKvVjI+PZ+PGjVmyZEmWL1/+MxUKAACz7bxheWBgoPDcwoUL8+KLL6a9vT0vvvhiFixYUHdetVqdmt/T05ORkRFhGQCApldqG0Z3d3f279+fJNm/f39+4zd+Y9qcEydO5OWXX576fOjQoSxdurTMsgAAMCtKheU77rgjhw4dym/+5m/m0KFDueOOO5IktVotW7ZsSfLqvuYPfehDWbNmTTZs2JCurq6sWLGifOUAADDDSj1n+e1vf3t27949bbxaraa/vz9J0tnZmQMHDpRZBgAAGsIb/AAAoICwDAAABYRlAAAoICwDAEABYRkAAAoIywAAUEBYBgCAAsIyAAAUEJYBAKCAsAwAAAWEZQAAKCAsAwBAAWEZAAAKCMsAAFBAWAYAgALCMgAAFBCWAQCggLAMAAAFSoXlr3zlK1m9enXe85735Lvf/W7hvOHh4axatSo9PT3ZuXNnmSUBAGDWlArL73rXu/LII49k+fLlhXMmJiayffv27Nq1K4ODgzl48GAOHz5cZlkAAJgVbWW+fNVVV513zsjISBYvXpzOzs4kyerVqzM0NJRf+ZVfKbM0AADMuBnfs1yr1dLR0TF1XK1WU6vVZnpZAAAo7bx3lnt7ezM2NjZtvK+vLytXrjzvApOTk9PGKpXKGywPAAAa57xheWBgoNQCHR0dGR0dnTqu1Wppb28v9ZsAADAbZnwbxrJly3LkyJEcPXo0p0+fzuDgYLq7u2d6WQAAKK1UWH722WezYsWKfOc738mdd96ZTZs2JXn17vGWLVuSJG1tbdm2bVs2b96cm2++OTfddFOWLl1avnIAAJhhpZ6G0dPTk56enmnj1Wo1/f39U8ddXV3p6uoqsxQAAMw6b/ADAIACwjIAABQQlgEAoICwDAAABYRlAAAoICwDAEABYRkAAAoIywAAUEBYBgCAAsIyAAAUEJYBAKCAsAwAAAWEZQAAKCAsAwBAAWEZAAAKCMsAAFBAWAYAgALCMgAAFGgr8+WvfOUr+cxnPpPvfe97eeKJJ7Js2bK687q7uzNv3ry0tLSktbU1+/btK7MsAADMilJh+V3velceeeSRfOITnzjv3N27d2fBggVllmuIU2cmcuRTqxtdRlM4dWYil17S2ugyAABmTamwfNVVV12sOpqWcPga1wIAeKuZtT3LmzZtyvr16/P444/P1pIAAFDKee8s9/b2ZmxsbNp4X19fVq5c+YYW2bt3b6rVasbHx7Nx48YsWbIky5cv/9mrBQCAWXTesDwwMFB6kWq1miRZuHBhenp6MjIyIiwDAND0ZnwbxokTJ/Lyyy9PfT506FCWLl0608sCAEBppcLys88+mxUrVuQ73/lO7rzzzmzatClJUqvVsmXLliTJ+Ph4PvShD2XNmjXZsGFDurq6smLFivKVAwDADCv1NIyenp709PRMG69Wq+nv70+SdHZ25sCBA2WWAQCAhvAGPwAAKCAsAwBAAWEZAAAKCMsAAFBAWAYAgALCMgAAFBCWAQCggLAMAAAFhGUAACggLAMAQAFhGQAACgjLAABQQFgGAIACwjIAABQQlgEAoICwDAAABYRlAAAoICwDAECBUmH5wQcfzI033phbb701d911V3784x/XnTc8PJxVq1alp6cnO3fuLLMkAADMmlJh+brrrsvBgwfzzDPP5Morr8yOHTumzZmYmMj27duza9euDA4O5uDBgzl8+HCZZQEAYFaUCsvXX3992trakiRXX311RkdHp80ZGRnJ4sWL09nZmTlz5mT16tUZGhoqsywAAMyKi7Zn+cknn8yKFSumjddqtXR0dEwdV6vV1Gq1i7UsAADMmLbzTejt7c3Y2Ni08b6+vqxcuTJJ8rnPfS6tra1Zs2bNtHmTk5PTxiqVyoXUCgAAs+q8YXlgYOB1zz/11FN57rnnMjAwUDcEd3R0nLM9o1arpb29/WevFAAAZtl5w/LrGR4eTn9/fx577LHMnTu37pxly5blyJEjOXr0aKrVagYHB/PQQw+VWRYa7tSZiRz51OpGl9EUTp2ZyKWXtDa6DACYEaX2LN933305fvx4Nm7cmLVr12bbtm1JXr17vGXLliRJW1tbtm3bls2bN+fmm2/OTTfdlKVLl5avHBpIOHyNawHA/2Wl7iw/++yzdcer1Wr6+/unjru6utLV1VVmKQAAmHXe4AcAAAWEZQAAKCAsAwBAAWEZAAAKCMsAAFBAWAYAgALCMgAAFBCWAQCggLAMAAAFhGUAACggLAMAQAFhGQAACgjLAABQQFgGAIACwjIAABQQlgEAoICwDAAABYRlAAAo0Fbmyw8++GD+9m//Npdcckne+c535oEHHsjll18+bV53d3fmzZuXlpaWtLa2Zt++fWWWBQCAWVHqzvJ1112XgwcP5plnnsmVV16ZHTt2FM7dvXt3nn76aUEZAIA3jVJh+frrr09b26s3p6+++uqMjo5elKIAAKAZXLQ9y08++WRWrFhReH7Tpk1Zv359Hn/88Yu1JAAAzKjz7lnu7e3N2NjYtPG+vr6sXLkySfK5z30ura2tWbNmTd3f2Lt3b6rVasbHx7Nx48YsWbIky5cvL1k6AADMrPOG5YGBgdc9/9RTT+W5557LwMBAKpVK3TnVajVJsnDhwvT09GRkZERYBgCg6ZV6Gsbw8HD6+/vz2GOPZe7cuXXnnDhxIq+88krmz5+fEydO5NChQ/n93//9MssCNKVTZyZy5FOrG11GUzh1ZiKXXtLa6DIASisVlu+7776cPn06GzduTJL86q/+arZv355arZZ77703/f39GR8fz1133ZUkmZiYyC233PK6e5sB3qyEw9e4FsD/FaXC8rPPPlt3vFqtpr+/P0nS2dmZAwcOlFkGAAAawhv8AACggLAMAAAFhGUAACggLAMAQAFhGQAACgjLAABQQFgGAIACwjIAABQo9VISAOD1eQ36a7wGnTcjd5YBYAYJh69xLXgzEpYBAKCAsAwAAAWEZQAAKCAsAwBAAWEZAAAKCMsAAFBAWAYAgALCMgAAFCgVlh9++OHceuutWbt2bW6//fbUarW684aHh7Nq1ar09PRk586dZZYEAIBZUyosb968Oc8880yefvrp3HDDDfnsZz87bc7ExES2b9+eXbt2ZXBwMAcPHszhw4fLLAsAALOiVFieP3/+1OeTJ0+mUqlMmzMyMpLFixens7Mzc+bMyerVqzM0NFRmWQAAmBVtZX/g05/+dPbv35/LLrsse/bsmXa+Vqulo6Nj6rharWZkZKTssgAAMOPOG5Z7e3szNjY2bbyvry8rV67M1q1bs3Xr1uzYsSOPPfZY7r777nPmTU5OTvtuvTvQAADQbM4blgcGBt7QD91yyy258847p4Xljo6OjI6OTh3XarW0t7f/bFUCAEADlNqzfOTIkanPX//617NkyZJpc5YtW5YjR47k6NGjOX36dAYHB9Pd3V1mWQAAmBWl9iw/9NBD+f73v59KpZJFixblk5/8ZJJX7x7fe++96e/vT1tbW7Zt25bNmzdnYmIi73//+7N06dKLUjwAAMykUmH5kUceqTterVbT398/ddzV1ZWurq4ySwEAwKzzBj8AACggLAMAQAFhGQAACgjLAABQQFgGAIACwjIAABQQlgEAoICwDAAABYRlAAAoICwDAEABYRkAAAoIywAAUEBYBgCAAsIyAAAUEJYBAKCAsAwAAAWEZQAAKCAsAwBAgbYyX3744YczNDSUlpaWLFy4MA888ECq1eq0ed3d3Zk3b15aWlrS2tqaffv2lVkWAABmRamwvHnz5vT19SVJ9uzZk89+9rPZvn173bm7d+/OggULyiwHAACzqtQ2jPnz5099PnnyZCqVSumCAACgWZS6s5wkn/70p7N///5cdtll2bNnT+G8TZs2pVKp5Lbbbsttt91WdlkAAJhx5w3Lvb29GRsbmzbe19eXlStXZuvWrdm6dWt27NiRxx57LHffffe0uXv37k21Ws34+Hg2btyYJUuWZPny5RfnTwAAADPkvGF5YGDgDf3QLbfckjvvvLNuWP7pf/pbuHBhenp6MjIyIiwDAND0Su1ZPnLkyNTnr3/961myZMm0OSdOnMjLL7889fnQoUNZunRpmWUBAGBWlNqz/NBDD+X73/9+KpVKFi1alE9+8pNJklqtlnvvvTf9/f0ZHx/PXXfdlSSZmJjILbfckhUrVpSvHAAAZlipsPzII4/UHa9Wq+nv70+SdHZ25sCBA2WWAQCAhvAGPwAAKCAsAwBAAWEZAAAKCMsAAFBAWAYAgALCMgAAFBCWAQCggLAMAAAFhGUAACggLAMAQAFhGQAACgjLAABQQFgGAIACbY0uAADgrebUmYkc+dTqRpfRFE6dmcill7Q2uoxC7iwDAMyyZg6Hs63Zr4WwDAAABYRlAAAoICwDAECBixKWv/CFL+Td7353Xnrppbrnh4eHs2rVqvT09GTnzp0XY0kAAJhxpcPyf//3f+fv//7v80u/9Et1z09MTGT79u3ZtWtXBgcHc/DgwRw+fLjssgAAMONKh+UHHnggf/zHf5xKpVL3/MjISBYvXpzOzs7MmTMnq1evztDQUNllAQBgxpUKy0NDQ2lvb8973vOewjm1Wi0dHR1Tx9VqNbVarcyyAAAwK877UpLe3t6MjY1NG+/r68uOHTvyxS9+8XW/Pzk5OW2s6C40AAA0k/OG5YGBgbrj//Ef/5EXXngha9euTZKMjo5m/fr1eeKJJ3LFFVdMzevo6Mjo6OjUca1WS3t7+xsq7tixY1m/fv0bmgsAABfi2LFjhecqk/Vu/V6A7u7ufPnLX86CBQvOGT979mxWrVqVgYGBVKvVfOADH8hDDz2UpUuXXoxlAQBgxszIc5ZrtVq2bNmSJGlra8u2bduyefPm3HzzzbnpppsEZQAA3hQu2p1lAAD4v8Yb/AAAoICwDAAABYRlAAAoICwDAEABYRkAAAoIy28STz75ZKNLoIG+973v5Zvf/GaOHz9+zvjw8HCDKqIZjIyMZGRkJEly+PDhPProo/nGN77R4KpoNvfcc0+jS6DJfPvb386jjz6a559/vtGlvCl4dNybxA033JDnnnuu0WXQAHv27MmXvvSlXHXVVfn3f//3fPzjH8/KlSuTJL/1W7+Vp556qsEV0gif+cxnMjw8nLNnz+a6667LP//zP+eaa67JN7/5zVx//fX5yEc+0ugSaYAPf/jD08a+9a1v5X3ve1+S5POf//xsl0QT+MAHPpAvf/nLSZK/+qu/ype+9KX09PTk+eefT3d3d+64444GV9jczvu6a2bPrbfeWnhubGxsFiuhmTzxxBPZt29f5s2blxdeeCF33313jh07lt/7vd+Lv+u+df3N3/xN9u/fn9OnT+e6667L8PBw5s+fn82bN2fDhg3C8ltUrVbLVVddlQ0bNqRSqWRycjL/8i//kttvv73RpdFAZ8+enfr8+OOP59FHH82CBQty++2357bbbhOWz0NYbiLj4+P5whe+kMsvv/yc8cnJyfz2b/92g6qi0SYmJjJv3rwkyS//8i/nL//yL3P33Xfnv/7rv4Tlt7DW1ta0trZm7ty5eec735n58+cnSS699NK0tNhh91b15JNPZs+ePfn85z+fe+65J+9973vzcz/3c7nmmmsaXRoN9Morr+R//ud/8sorr2RycjILFixIkrztbW9La2trg6trfsJyE7nhhhty/PjxvPe975127qf/hMZbzzve8Y7827/921RfzJs3Lzt27MjHP/7x/Od//meDq6NRLrnkkpw8eTJz587Nvn37psb/93//V1h+C2tpaUlvb29uvPHG3H///XnHO96RiYmJRpdFg7388stZv359JicnU6lU8sMf/jBXXHFFjh8/7qbLG2DPMjS50dHRtLa25oorrph27h/+4R/ya7/2aw2oikY7ffp05syZM238pZdeyg9/+MO8+93vbkBVNJvnnnsu//iP/5iPfexjjS6FJnTy5MmMjY2ls7Oz0aU0NWEZAAAK+Lc6AAAoICwDAEABYbkJHT16ND/5yU+mjk+dOpUXXnihgRXRDPQF9egL6tEX1KMvLoyw3IQ++tGPplKpTB23tLTkox/9aAMrohnoC+rRF9SjL6hHX1wYYbkJTUxMnPO/3OfMmZMzZ840sCKagb6gHn1BPfqCevTFhRGWm9CCBQsyNDQ0dfy1r30tb3/72xtYEc1AX1CPvqAefUE9+uLCeHRcE/rBD36QP/qjP8qLL76YycnJ/OIv/mIefPDBLF68uNGl0UD6gnr0BfXoC+rRFxdGWG5iP32zzk9fYwuJvqA+fUE9+oJ69MXPxuuum8jTTz+dtWvX5tFHH617fuPGjbNcEc1AX1CPvqAefUE9+qIcYbmJnDx5Msmrf+ODn9IX1KMvqEdfUI++KMc2DAAAKOBpGE3o6NGj+fCHP5xf//Vfz7XXXpuPfOQjOXr0aKPLosH0BfXoC+rRF9SjLy6MsNyE/vAP/zA33nhjnn/++fzd3/1dbrzxxnzsYx9rdFk0mL6gHn1BPfqCevTFhRGWm9Dk5GTWrVuXtra2tLW1Ze3atee8cYe3Jn1BPfqCevQF9eiLC2PPchP6i7/4i1x++eW5+eabU6lU8td//dc5ffp0fud3fidJ8gu/8AuNLZCG0BfUoy+oR19Qj764MMJyE+ru7i48V6lUznn7Dm8d+oJ69AX16Avq0RcXRlgGAIACnrPchM6cOZO9e/fm29/+dpLkmmuuyW233ZZLLrmkwZXRSPqCevQF9egL6tEXF8ad5Sb0J3/yJzl79mzWrVuXJDlw4EBaWlry53/+540tjIbSF9SjL6hHX1CPvrgw7iw3kbNnz6atrS3f/e53c+DAganxa6+9NmvWrGlgZTSSvqAefUE9+oJ69EU5Hh3XRDZs2JAkaW1tzQ9+8IOp8aNHj6a1tbVRZdFg+oJ69AX16Avq0RfluLPcRH66I+aee+7J7/7u76azszNJcuzYsdx///2NLI0G0hfUoy+oR19Qj74ox57lJrJixYps3LgxSXLq1KlMTEzkbW97W37yk5/k0ksvnTrHW4u+oB59QT36gnr0RTnuLDeRV155JcePHz9n7MSJE0kybZy3Dn1BPfqCevQF9eiLcoTlJnLFFVfkD/7gDxpdBk1GX1CPvqAefUE9+qIc/8GvidgRQz36gnr0BfXoC+rRF+XYs9xEfvSjH3kvO9PoC+rRF9SjL6hHX5QjLAMAQAHbMAAAoICwDAAABYRlAAAoICwDAEABYRkAAAr8f961gyNjhHp0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "topic_coherence = []\n",
    "topic_words = pd.DataFrame()\n",
    "for t in range(len(coherence)):\n",
    "    label = topic_labels[t]\n",
    "    topic_coherence.append(coherence[t][1])\n",
    "    df = pd.DataFrame(coherence[t][0], columns=[(label, 'prob'), (label, 'term')])\n",
    "    df[(label, 'prob')] = df[(label, 'prob')].apply(lambda x: '{:.2%}'.format(x))\n",
    "    topic_words = pd.concat([topic_words, df], axis=1)\n",
    "                      \n",
    "topic_words.columns = pd.MultiIndex.from_tuples(topic_words.columns)\n",
    "pd.set_option('expand_frame_repr', False)\n",
    "topic_words.head().to_csv('topic_words.csv', index=False)\n",
    "print(topic_words.head())\n",
    "\n",
    "pd.Series(topic_coherence, index=topic_labels).plot.bar(figsize=(12,4));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Using `gensim` `Dictionary` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:35.040648Z",
     "start_time": "2021-04-25T21:52:33.290356Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "docs = [d.split() for d in train_docs.article.tolist()]\n",
    "docs = [[t for t in doc if t not in stop_words] for doc in docs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:35.533850Z",
     "start_time": "2021-04-25T21:52:35.041563Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "dictionary = Dictionary(docs)\n",
    "dictionary.filter_extremes(no_below=min_df, no_above=max_df, keep_n=max_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:35.665767Z",
     "start_time": "2021-04-25T21:52:35.534747Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "corpus = [dictionary.doc2bow(doc) for doc in docs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:35.669154Z",
     "start_time": "2021-04-25T21:52:35.666736Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique tokens: 2000\n",
      "Number of documents: 2175\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "print('Number of unique tokens: %d' % len(dictionary))\n",
    "print('Number of documents: %d' % len(corpus))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:35.680447Z",
     "start_time": "2021-04-25T21:52:35.670213Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "num_topics = 5\n",
    "chunksize = 500\n",
    "passes = 20\n",
    "iterations = 400\n",
    "eval_every = None # Don't evaluate model perplexity, takes too much time.\n",
    "\n",
    "temp = dictionary[0]  # This is only to \"load\" the dictionary.\n",
    "id2word = dictionary.id2token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:48.081649Z",
     "start_time": "2021-04-25T21:52:35.681435Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "model = LdaModel(corpus=corpus,\n",
    "                 id2word=id2word,\n",
    "                 chunksize=chunksize,\n",
    "                 alpha='auto',\n",
    "                 eta='auto',\n",
    "                 iterations=iterations,\n",
    "                 num_topics=num_topics,\n",
    "                 passes=passes, \n",
    "                 eval_every=eval_every)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:48.086261Z",
     "start_time": "2021-04-25T21:52:48.082553Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.012*\"best\" + 0.007*\"film\" + 0.007*\"won\" + 0.007*\"win\" + 0.006*\"game\" + 0.006*\"England\" + 0.005*\"play\" + 0.005*\"good\" + 0.005*\"think\" + 0.005*\"second\"'),\n",
       " (1,\n",
       "  '0.021*\"Labour\" + 0.017*\"Blair\" + 0.015*\"election\" + 0.013*\"Brown\" + 0.012*\"Lord\" + 0.012*\"party\" + 0.011*\"Howard\" + 0.011*\"Tory\" + 0.010*\"prime\" + 0.008*\"public\"'),\n",
       " (2,\n",
       "  '0.012*\"government\" + 0.005*\"European\" + 0.005*\"legal\" + 0.005*\"says\" + 0.005*\"public\" + 0.005*\"Ms\" + 0.005*\"money\" + 0.005*\"EU\" + 0.004*\"work\" + 0.004*\"law\"'),\n",
       " (3,\n",
       "  '0.010*\"mobile\" + 0.009*\"technology\" + 0.008*\"games\" + 0.008*\"music\" + 0.007*\"use\" + 0.007*\"users\" + 0.007*\"phone\" + 0.006*\"used\" + 0.006*\"net\" + 0.006*\"video\"'),\n",
       " (4,\n",
       "  '0.010*\"company\" + 0.009*\"sales\" + 0.009*\"market\" + 0.009*\"growth\" + 0.008*\"economic\" + 0.007*\"rise\" + 0.007*\"oil\" + 0.007*\"economy\" + 0.007*\"firm\" + 0.006*\"chief\"')]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.show_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating Topic Assignments on the Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:48.160381Z",
     "start_time": "2021-04-25T21:52:48.087439Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "docs_test = [d.split() for d in test_docs.article.tolist()]\n",
    "docs_test = [[t for t in doc if t not in stop_words] for doc in docs_test]\n",
    "\n",
    "test_dictionary = Dictionary(docs_test)\n",
    "test_dictionary.filter_extremes(no_below=min_df, no_above=max_df, keep_n=max_features)\n",
    "test_corpus = [dictionary.doc2bow(doc) for doc in docs_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:48.195832Z",
     "start_time": "2021-04-25T21:52:48.161468Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19.146658</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>0.139427</td>\n",
       "      <td>17.055145</td>\n",
       "      <td>0.093734</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.964511</td>\n",
       "      <td>3.631570</td>\n",
       "      <td>5.128225</td>\n",
       "      <td>157.635162</td>\n",
       "      <td>0.093728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.437624</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>38.566299</td>\n",
       "      <td>40.330196</td>\n",
       "      <td>0.093719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.141710</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>27.128258</td>\n",
       "      <td>9.319160</td>\n",
       "      <td>19.845629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>86.310379</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>2.145877</td>\n",
       "      <td>0.072351</td>\n",
       "      <td>2.894809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5.462916</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>11.964343</td>\n",
       "      <td>0.072351</td>\n",
       "      <td>39.932270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>16.709969</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>26.629995</td>\n",
       "      <td>0.072351</td>\n",
       "      <td>87.001869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>84.127693</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>0.139049</td>\n",
       "      <td>0.072351</td>\n",
       "      <td>0.093723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.141705</td>\n",
       "      <td>1.410880</td>\n",
       "      <td>0.139311</td>\n",
       "      <td>1.598531</td>\n",
       "      <td>134.170822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.141487</td>\n",
       "      <td>0.051314</td>\n",
       "      <td>12.392625</td>\n",
       "      <td>2.545936</td>\n",
       "      <td>30.358202</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0         1          2           3           4\n",
       "0  19.146658  0.051314   0.139427   17.055145    0.093734\n",
       "1   2.964511  3.631570   5.128225  157.635162    0.093728\n",
       "2   2.437624  0.051314  38.566299   40.330196    0.093719\n",
       "3   0.141710  0.051314  27.128258    9.319160   19.845629\n",
       "4  86.310379  0.051314   2.145877    0.072351    2.894809\n",
       "5   5.462916  0.051314  11.964343    0.072351   39.932270\n",
       "6  16.709969  0.051314  26.629995    0.072351   87.001869\n",
       "7  84.127693  0.051314   0.139049    0.072351    0.093723\n",
       "8   0.141705  1.410880   0.139311    1.598531  134.170822\n",
       "9   0.141487  0.051314  12.392625    2.545936   30.358202"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gamma, _ = model.inference(test_corpus)\n",
    "topic_scores = pd.DataFrame(gamma)\n",
    "topic_scores.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:48.206896Z",
     "start_time": "2021-04-25T21:52:48.196950Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.524763</td>\n",
       "      <td>0.001406</td>\n",
       "      <td>0.003821</td>\n",
       "      <td>0.467440</td>\n",
       "      <td>0.002569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.017495</td>\n",
       "      <td>0.021431</td>\n",
       "      <td>0.030263</td>\n",
       "      <td>0.930258</td>\n",
       "      <td>0.000553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.029917</td>\n",
       "      <td>0.000630</td>\n",
       "      <td>0.473327</td>\n",
       "      <td>0.494976</td>\n",
       "      <td>0.001150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.002509</td>\n",
       "      <td>0.000908</td>\n",
       "      <td>0.480265</td>\n",
       "      <td>0.164982</td>\n",
       "      <td>0.351337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.943543</td>\n",
       "      <td>0.000561</td>\n",
       "      <td>0.023459</td>\n",
       "      <td>0.000791</td>\n",
       "      <td>0.031646</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4\n",
       "0  0.524763  0.001406  0.003821  0.467440  0.002569\n",
       "1  0.017495  0.021431  0.030263  0.930258  0.000553\n",
       "2  0.029917  0.000630  0.473327  0.494976  0.001150\n",
       "3  0.002509  0.000908  0.480265  0.164982  0.351337\n",
       "4  0.943543  0.000561  0.023459  0.000791  0.031646"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_probabilities = topic_scores.div(topic_scores.sum(axis=1), axis=0)\n",
    "topic_probabilities.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:48.213242Z",
     "start_time": "2021-04-25T21:52:48.208264Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    3\n",
       "2    3\n",
       "3    2\n",
       "4    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_probabilities.idxmax(axis=1).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-25T21:52:48.375108Z",
     "start_time": "2021-04-25T21:52:48.214357Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaYAAAEKCAYAAABZr/GWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAneUlEQVR4nO3de1iUdeL//+eAIgioYOGhXA1PmZp5ykjSdD0lInhEt8wsU+uTVOqmeOwyk+xj9lHaStdMSyV2FcVju+aJzRTPlkmam0dUzFARFJFxvn/4c35aKKeB+5b79eia63KGe+55jQkv3u/7PfdtczgcDkREREzCzegAIiIit1IxiYiIqaiYRETEVFRMIiJiKiomERExFRWTiIiYShmjA4iISOkQFRXFpk2bqFy5MqtWrQJg2rRpbNy4kbJly/KnP/2J6OhoKlSocNf9aMQkIiIu0bNnT+bOnXvbY61bt2bVqlWsXLmSWrVqMXv27Dz3o2ISERGXaNmyJRUrVrztseDgYMqUuTE599hjj3HmzJk896OpPJFCysoxOkHx8CyFPxXue+EroyMUi3Pz+xV5H15NX8v3tvPHPEVcXJzzfkREBBEREfl+/tKlS3nmmWfy3K4U/hMUEZF8s+V/4qygRXSrTz75BHd3d7p3757ntiomERErs9mK/SWWLVvGpk2bmD9/PrZ8vJ6KSUTEygowYiqMxMRE/v73v7Nw4UK8vLzy9RwVk4iIlblwxDRixAi2b9/O+fPnadOmDcOHD2fOnDlkZ2czaNAgAJo0acLkyZPvuh8Vk4iIlbm5u2xXM2bM+MNjffr0KfB+VEwiIlZWzFN5haFiEhGxshJY/FBQKiYRESvTiElERExFIyYRETEVjZhERMRUXLgqz1VUTCIiVqYRk4iImIqbjjGJiIiZaMQkIiKmolV5IiJiKlr8ICIipqKpPBEpiInjo0jcvAl//8rEJ6wyOo78zswXH6fTY9U5l57FU+O/BqCStwdzX3mSP93nzfFzmbz08RYuXr5mcNK7MOFUnvmqspAaNGhAWFgY3bt3p0ePHuzevbtQ+4mNjWX58uWuDSdSSGHhPflk9lyjY8gdfPXtESI+2HzbY6+HNCAxOZXHx6wmMTmV10MeMShdPtnc8n8rIaWmmDw9PUlISGDFihWMGDEi19Ov50f//v0JDw93bTiRQmreoiUVKlY0OobcwdZDv3I+M/u2x55p+gBx3x4BIO7bI3Rt9oAR0fLPZsv/rYSUmmK6VUZGBhUqVAAgKSmJoUOHOr82efJk4uPjAZg+fTpdu3YlNDSUadOmARATE8Nnn30GwIABA/jf//1fevfuTefOndm5cycAdrudadOm0atXL0JDQ/nqq68AOHv2LM8++yxhYWF069aNnTt3YrfbGTNmDN26dSM0NJT58+eX1F+DiBjg/oqepF7MAiD1Yhb3VfA0OFEeTDhiKjXHmLKysggLC+Pq1av8+uuvLFiw4K7bX7hwgXXr1vH1119js9lIT0/PdTu73c6SJUvYvHkzH330EfPnz2fJkiX4+vqydOlSsrOz6devH61bt2bdunUEBwfzyiuvYLfbuXLlCsnJyaSmprJq1Y3jA3d6HRERQ2hVXvG5OZUHsGfPHkaPHu0sg9z4+PhQrlw5xo0bx9NPP83TTz+d63YdO3YEoGHDhqSkpACwZcsWDh48yL/+9S8ALl26xLFjx2jcuDFjx44lJyeHDh060KBBA2rUqMGJEyd45513aNu2LcHBwS581yJiNr9ezKLK/zdqqlLRk3PpWUZHujsTrsozXyIXaNq0KefPnyctLQ13d3euX7/u/NrVq1cBKFOmDEuWLKFz58588803DB48ONd9eXh4AODm5obdbgfA4XAwfvx4EhISSEhIYMOGDQQHB9OyZUsWLlxIlSpVeOutt1i+fDkVK1YkISGBxx9/nMWLFzNu3LhifvciYqSv96YQEfwQABHBD7F2T4rBifKgY0wl47///S92u51KlSrxwAMP8N///pfs7GwuXbrE1q1bAcjMzOTSpUu0bduWsWPH8tNPP+V7/8HBwcTGxnLt2o0loEeOHOHy5cukpKRQuXJl+vbtS69evfjxxx9JS0vD4XDQuXNnXn/9dQ4cOFAs71lKp9GjRvD8X/px7OgROrZvQ/zSfxodSW4xZ1gQX4/vQJ2qFfh+RneebRPIzFXJPN2wKtvfC+HphlWZuTrZ6Jh3p2NMxefmMSa4MaKZNm0a7u7uVKtWjS5duhAaGkqtWrV45JEbSzczMzN59dVXnSOoqKiofL9Wnz59SElJoWfPnjgcDvz8/Pj444/Zvn07n332GWXKlKF8+fJMmzaNs2fPEhUV5Ry1jRgxwsXvXEqzadMLt7pUSsaQT7fm+njP9zeWcJIiMOHnmGwOh8NhdAiRe1FWjtEJiodnqfl19f933wtfGR2hWJyb36/I+/Dqkf/PyV1ZlvshD1crhf8ERUQkv2xu5juio2ISEbEwmwmn8lRMIiJWZr5eUjGJiFiZRkwiImIqKiYRETEVNy1+EBERUzHfgEnFJCJiZZrKExERUzFjMZlvclFEREqMzWbL9y0vUVFRBAUF0a1bN+djFy5cYNCgQXTq1IlBgwZx8eLFPPejYhIRsTBXFlPPnj2ZO/f2UxzNmTOHoKAg/v3vfxMUFMScOXPy3I+KSUTEwmxutnzf8tKyZUsqVqx422Pr168nPDwcgPDwcL755ps896NjTCIiFlaQY0xxcXHExcU570dERBAREXHX5/z2228EBAQAEBAQQFpaWp6vo2ISEbGwghRTRETfPIvIFTSVJyJiZbYC3AqhcuXKnD17FoCzZ8/i7++f53NUTCIiFubKxQ+5ad++PcuXLwdg+fLl/PnPf847ky4UaC4DFu0zOkKx+PLZJkZHkHw6dSHb6AguV72Sh9ERTKvqy0vyve2Zv/e+69dHjBjB9u3bOX/+PJUrV2b48OF06NCBN954g9OnT1OtWjVmzpxJpUqV7rofHWMSEbEwV54rb8aMGbk+vmDBggLtR8UkImJl5jvxg4pJRMTKzHhKIhWTiIiFqZhERMRUVEwiImIq+TnVUElTMYmIWJhGTCIiYioqJhERMRUVk4iImIv5eknFJCJiZRoxiYiIqbhpVZ6IiJiJRkwiImIqJuwlFZOIiJVpxCQiIqZiwl5SMVlRVd9yvBZc03k/wNeDpfvO8K+D5wxMJVZxNvUM0W+PJS3tHDabG93Ce9O733NGx7IsSy1+aNCgAfXq1XPeDwkJYciQIXfcPikpibJly9KsWbMCvU5ycjJnz56lbdu2d93uhx9+ICEhgfHjxxdo/67y6aefMmzYMENe+/fOXLrK+LWHgBu/Lc3q8Qg7T140OJVYhbu7O6+8Pop6Dz/C5cxMhg6MoMXjQdQKrG10NEuyVDF5enqSkJCQ7+23b99O+fLlC1RMOTk5JCcns3///jyLqXHjxjRu3Djf+3a12bNnm6aYbtWwig9nM7L5LfOa0VHEIirfdz+V77sfgPLe3vyp1kOc+zVVxWQQTeUB7du3Jzw8nI0bN5KTk8P//d//Ua5cOb766ivc3NxYsWIFEyZMIDAwkEmTJnHq1CkAxo4dS/PmzYmJieHs2bOkpKTg5+fHrl27yMrKYteuXQwdOpQHH3yQqVOnkpWVhaenJ1OnTiUwMJCkpCTmzZvH7NmziYmJ4dSpU5w8eZJTp04xcOBAnn/+eU6ePMngwYNp3rw5+/bto379+vTq1YtZs2aRlpbG9OnTefTRR7l8+TLvvPMOhw4dwm6389prr9GhQwfi4+PZsGEDV65c4cSJE3To0IG33nqL6dOnk5WVRVhYGHXq1OGDDz4o6b/2O3qilh9bj543OoZY1JlTKRw+9BMNGj5qdBTLstTih5s/iG8aOnQoXbt2BcDPz49ly5axaNEi5s2bx7vvvku/fv0oX748L730EgAjR45k4MCBtGjRglOnTvHSSy+xdu1aAH788UcWL16Mp6cn8fHx7N+/n4kTJwKQkZHBwoULKVOmDN999x0ffvghMTExf8h35MgRvvjiCzIyMnjmmWfo378/AMePH2fmzJnUrVuX3r17s3LlSmJjY1m/fj2ffvopH3/8MZ9++ilPPPEE0dHRpKen06dPH5588kngxtTi8uXL8fDwoEuXLgwYMIBRo0axaNGiAo0gS4K7m41mD1TgH3tPGx1FLOjK5ctMHPMm//PmaLx9fIyOY1mWKqa7TeV16tQJgEaNGrFu3bpct/nuu+84fPiw835GRgYZGRnAjVGXp6dnrs+7dOkSo0eP5tixY9hsNq5dy32Kqm3btnh4eODv74+/vz+//fYbAA8++CD169cHoE6dOgQFBWGz2ahfvz4pKSkAfPvtt2zYsIF58+YBcPXqVU6fvvHDPSgoCF9fXwBq165NSkoK1apVu8PfkrGaVPfl6PkrpGflGB1FLCYn5xoTx7xJhy4htGnXweg4lmbCXjJmVV7ZsmUBcHNzw26357rN9evXiYuLy7WAvLy87rjvmTNn0qpVK/72t79x8uRJnn/++Vy38/DwcP7Z3d2dnJycPzzu5ubmvG+z2W7LOmvWLAIDA2/b5759+/6w3zu9PzMIqllJ03hS4hwOB+9PmUTNWoH0/ctAo+NYnhkXP7gZHeAmb29vMjMznfeDg4NZuHCh835ycnK+nnfp0iWqVKkCwLJly4ol681sDocDgAMHDuT5nDJlytxx9GYED3cbDav5svOEVuNJydq/bw/r1q5kz64kBj/Xm8HP9WbblkSjY1mWzWbL962kFFsx3TzGdPM2ffr0u27frl071q1bR1hYGDt37mTcuHHs37+f0NBQunbtSmxsbK7Pa9WqFYcPHyYsLIw1a9YwePBgZsyYQb9+/YpttPLqq6+Sk5ND9+7d6datGzNnzszzOX379qV79+6MHDmyWDIVVLbdwatLfuTKtetGRxGLafxYMzYm/cBni+KZu3AJcxcu4YnWbYyOZVk2W/5vJZbJcfPXfjGFAYv2GR2hWHz5bBOjI0g+nbqQbXQEl6teySPvjSyq5bub8r3tjnFPF1uOW+nMDyIiFqbFDyIiYiqWWi4uIiLmZ8ZVeSomERELM+GAScUkImJlmsoTERFTcWUvzZ8/n3/+85/YbDbq1atHdHQ05cqVK/B+TPMBWxERKXmu+oBtamoqX3zxBUuXLmXVqlXY7XZWr15dqEwaMYmIWJgrp/LsdjtZWVmUKVOGrKwsAgICCrUfFZOIiIUVZFVeXFwccXFxzvsRERFEREQAUKVKFV588UXatWtHuXLlaN26NcHBwYXKpGISEbGwggyYbi2i37t48SLr169n/fr1+Pr68vrrr5OQkHDb5Y/yS8eYREQszFXHmL777jsefPBB/P39KVu2LJ06dWLPnj2FyqRiEhGxMFedxLV69ers27ePK1eu4HA42Lp1K7Vr1y5UJk3liYhYmJuLFj80adKEzp0706NHD8qUKUODBg3uOO2XFxWTiIiFufKURJGRkURGRhZ5PyomERELM+Gp8lRMIiJWplMSSZ5K6wX1SuMFEKeFNDA6QrHQRfWsxYS9pGISEbEyG+ZrJhWTiIiF6RiTiIiYii4UKCIipuKqzzG5kopJRMTCTNhLKiYRESvTcnERETEVE/aSiklExMrcTdhMKiYREQvTVJ6IiJiKCVeLq5hERKxMIyYRETEVE/ZS3lewHT16NOnp6c77Fy9eJCoqqlhDiYhIyXDVpdVdKc8R08GDB6lQoYLzfsWKFUlOTi7WUCIiUjLcTXiQKc8R0/Xr17l48aLz/oULF7Db7cUaSkRESoatALeSkueI6cUXX6Rfv3507twZgK+//pphw4YVezARESl+9+S58sLDw2nUqBHbtm3D4XDw0UcfUadOnZLIJiIixcyEvXTnYsrIyMDHx4cLFy5w33330a1bN+fXLly4QKVKlUoin0i+VPUtx2vBNZ33A3w9WLrvDP86eM7AVEV3NvUM0W+PJS3tHDabG93Ce9O733NGx5JS5J5aLj5y5Ehmz55Nz549sdlsOBwO59dsNhvr168vkYCu1KBBA+rVq4fdbicwMJBp06bh5eWV67bx8fHs37+fiRMnEhsbi5eXF+Hh4cTHx9O6dWuqVKkCwLhx4xg0aJBGkQY7c+kq49ceAm78BjirxyPsPHkxj2eZn7u7O6+8Pop6Dz/C5cxMhg6MoMXjQdQKrG10NCklTNhLdy6m2bNnA7Bhw4YSC1PcPD09SUhIAG4U71dffcWgQYPyfF7//v2df162bBl169Z1FtO7775bPGGl0BpW8eFsRja/ZV4zOkqRVb7vfirfdz8A5b29+VOthzj3a6qKSVzGjKvy8vUB23//+9/s2rULm81GixYt6NChQ3HnKnYtWrTg4MGDXLhwgbFjx3LixAm8vLyYPHkyDz/88G3bxsTEUL58eR544AH279/PqFGj8PT0JC4ujpdffpm33nqLxo0bk5iYyIcffojdbsfPz48FCxawfft2Z3nZbDYWLlyIj4+PEW/ZMp6o5cfWo+eNjuFyZ06lcPjQTzRo+KjRUaQUuaem8m56++23OX78OCEhIQDExsayZcsWJk2aVOzhiktOTg6JiYk89dRTxMTE8Mgjj/Dxxx+zdetWRo8e7RxV/V6XLl1YtGiRs4hulZaWxoQJE1i4cCE1atTgwoULAMybN4+JEyfSvHlzMjMzKVeuXHG/PUtzd7PR7IEK/GPvaaOjuNSVy5eZOOZN/ufN0XjrFxtxoTw/M2SAPItpx44drFq1ytmqPXr0IDQ0tNiDFYesrCzCwsKAGyOm3r1707dvX2JiYgAICgriwoULXLp0qcD73rt3Ly1atKBGjRoAzsUhzZo147333iM0NJROnTrh7e3tmjcjuWpS3Zej56+QnpVjdBSXycm5xsQxb9KhSwht2t37sxViLvfkiOmhhx7i1KlTPPDAAwCcPn2a+vXrF3uw4nDrMaabbl3UcVNh/kc5HI5cnzdkyBDatm3L5s2b6du3L59//jm1a+v4QHEJqlmpVE3jORwO3p8yiZq1Aun7l4FGx5FSyISHmPIexV24cIGuXbsyYMAABgwYQEhICGlpaQwbNqxUfNC2ZcuWrFixAoCkpCT8/PzuegzI29ubzMzMPzzetGlTduzYwYkTJwCcU3nHjx+nfv36DBkyhEaNGnHkyBHXvwkBwMPdRsNqvuw8ce+vxrtp/749rFu7kj27khj8XG8GP9ebbVsSjY4lpYi7my3ft5KS54gpMjKyJHIY5rXXXiMqKorQ0FC8vLx477337rp9jx49mDRpknPxw03+/v5MnjyZ4cOHc/36dSpXrsznn3/OggULSEpKws3NjTp16tCmTZvifkuWlW138OqSH42O4VKNH2vGxqQfjI4hpZgZR0w2R25zWb9z7tw5fvjhxjfHo48+SuXKlYs9mJQuAxbtMzqCy00LaWB0hGJRvZKH0RGkBL21+mC+t30/pGQO4+Q5lbdmzRr69OnD119/zdq1a51/FhGRe5+bzZbvW0nJcyrv008/ZcmSJc5RUlpaGi+88AJdunQp9nAiIlK8XLlcPD09nfHjx3Po0CFsNhtTp06ladOmBd5PnsXkcDhum7qrVKlSrivZRETk3uPKgdC7777LU089xaxZs8jOziYrK6tQ+8mzmJ566ileeukl5wds16xZowP4IiKlhKtW22VkZLBjxw7nAjIPDw88PAp3vDLPYrLZbERERLBr1y4cDgcRERHs3bu3UC8mIiLmUpBeiouLu201ckREBBEREQCcOHECf39/oqKi+Omnn2jYsCHjxo2jfPnyBc6U56q8Hj16sGzZstseCw0NZeXKlQV+MbEurcq7d2hVnrVMXnc439tO7Hjnqyj88MMPREREEBsbS5MmTZgyZQo+Pj688cYbBc50xxHT4sWLiY2N5cSJE7edgigzM5NmzZoV+IVERMR8XHWMqWrVqlStWpUmTZoAN84tOmfOnELt647FFBoaSps2bZgxYwYjR450Pu7t7a2LBIqIlBKu+oDt/fffT9WqVfnll18IDAxk69athT792h2LydfXF19fX2bMmFHooCIiYm42XLcsb8KECYwaNYpr165Ro0YNoqOjC7WffF2PSURESqcyLvwgU4MGDYiPjy/yflRMIiIWdk9e9kJEREovM57EVcUkImJhJhwwqZhERKysJE/Oml8qJhERC3N35VlcXUTFJCJiYW4uXC7uKiomk/Fq+prREYrFlT0fGR1BRHJhwpk8FZOIiJVpVZ6IiJiKFj+IiIipmLCXVEwiIlbmqgsFupKKSUTEwky4WlzFJCJiZTpXnoiImIr5aknFJCJiaVqVJyIipmK+WlIxiYhYmptW5YmIiJloVZ6IiJiKVuWJiIipmK+WVEwiIpamEZOIiJiKu4pJRETMxHy1pGKyjE8nPcszbRrxa9olWvSZCsDUN8Lp2qYR2dfsHDl5jiGTFnIx44rBSUWkJJlwwGTKlYKG+uSTTwgJCSE0NJSwsDD27dtX5H0mJSWxe/duF6QrvC9XbiPsf/5222Prt/1E8z5TeTwimp+PneWvL3YyKJ2IGMUNW75vJUUjplvs2bOHTZs2sWzZMjw8PEhLS+PatWtF2mdOTg7bt2+nfPnyNGvWzEVJC27L7v/yp2r+tz22fttPzj9v/+EIPTo0LelYImIwM46YVEy3+PXXX/Hz88PDwwMAf/8bP8jbt2/PM888Q1JSEgAffPABNWvWJCUlhbFjx5KWloa/vz/R0dFUr16dMWPGULFiRQ4cOEClSpXYvXs3bm5urFixggkTJtCiRQvD3uOdPB8WxJJ/GzuqE5GSZzPhUSZN5d2idevWnD59ms6dO/P222+zfft259d8fHxYsmQJzz33HFOn3jhG88477xAeHs7KlSsJDQ1lypQpzu2PHj3K/PnziYmJoV+/frzwwgskJCSYspTeeqkzdvt1vlqzw+goIlLC3G22fN9KiorpFt7e3sTHxzN58mT8/f158803iY+PB6Bbt24AhISEsHfvXuDG1N/Nx8PCwti1a5dzX126dMHd3b1k30AhPBvaiq5tGvHCuPlGRxERA9hs+b+VFE3l/Y67uzutWrWiVatW1KtXj+XLl+f7ubd+UM3Ly6sY0rlWxycbMPKFDnQaPJMrWUU7liYi9yYzHmPSiOkWv/zyC0ePHnXeT05Opnr16gCsXbsWgDVr1tC06Y1FAk2bNmX16tUArFy5kubNm+e6X29vbzIzM4sxed4WRL/ApgUjqVezCoe/foeB4UF8OLovvuU9WfXJa2z7agyzxvUzNKOIlDxbAf7LD7vdTnh4OEOHDi10Jo2YbnH58mWmTJlCeno67u7u1KxZk8mTJ7Np0yays7Pp06cP169fZ8aMGQCMHz+esWPH8tlnnzkXP+SmXbt2REZGsn79esMWPwyMmv+HxxYs31riOUTEXFx91YsvvviC2rVrk5GRUeh92BwOh8OFmUql9u3bs2TJEucqveLk1fS1Yn8NI1zZ85HREUQkFxsP/pbvbdvVr3zXr585c4bRo0czbNgw5s+fz+zZswuVSSMmERELK8hy8bi4OOLi4pz3IyIiiIiIcN6fOnUqf/3rX4t86ELFlA8bNmwwOoKISLEoyFTe74voVhs3bsTf359GjRo5P/NZWComERELc9UHbHfv3s2GDRtITEzk6tWrZGRkMGrUKKZPn17wTDrGZC46xiQiJenbn8/ne9vgun752i4pKYl58+bpGJOIiBScCT/GpGISEbGy4jjV0M2TFBSWiklExMpMOGRSMYmIWJgZzy6uYhIRsTAznitPxSQiYmEm7CUVk4iIpZmwmVRMIiIW5mbCuTwVk4iIhZmvllRMIiLWZsJmUjGJiFiYlouLiIipmPAQk4rJbM7v0MlOxVgDFu0zOoLL/T2iidERioWnC36Cq5hERMRUNJUnIiKmohGTiIiYigl7ScUkImJpJmwmFZOIiIXpGJOIiJiKm/l6ScUkImJpKiYRETETTeWJiIipaLm4iIiYigl7ScUkImJpJmwmFZOIiIXpQoEiImIq5qslFZOIiLWZsJlUTCIiFqbl4iIiYiomPMSkYhIRsTIVk5jCxPFRJG7ehL9/ZeITVhkdRyymqm85Xguu6bwf4OvB0n1n+NfBcwamKrp79fvKjFN5bkYHMLPz588TFhZGWFgYrVu35qmnnnLez87OzvP5SUlJDB06tASSFkxYeE8+mT3X6BhiUWcuXWX82kOMX3uICV8f4mrOdXaevGh0rCK7V7+vbLb830qKRkx34efnR0JCAgAxMTGUL1+el156yeBURde8RUtSUk4aHUOEhlV8OJuRzW+Z14yOUmT36veV+cZLKqYC279/P++99x6XL1/Gz8+P6OhoAgICOHbsGJMmTSItLQ13d3dmzpwJwOXLl4mMjOTQoUM0bNiQ6dOnYzPjpK6IAZ6o5cfWo+eNjmFprvpxdPr0ad566y3OnTuHm5sbffv2ZeDAgYXal4qpABwOB1OmTOHjjz/G39+fNWvW8OGHHxIdHc2oUaMYMmQIHTt25OrVq1y/fp3Tp09z4MABVq9eTUBAAP3792fXrl20aNHC6LciYjh3NxvNHqjAP/aeNjqKxbmmmdzd3RkzZgwNGzYkIyODXr160bp1a+rUqVPgfamYCiA7O5tDhw4xaNAgAK5fv879999PRkYGqampdOzYEYBy5co5n/Poo49StWpVAB5++GFSUlJUTCJAk+q+HD1/hfSsHKOjWJqrLhQYEBBAQEAAAD4+PgQGBpKamqpiKm4Oh4O6desSFxd32+MZGRl3fI6Hh4fzz+7u7tjt9mLLJ3IvCapZSdN4JlCQqby4uLjbfv5FREQQERHxh+1OnjxJcnIyTZo0KVQmrcorAA8PD9LS0tizZw8A165d4+eff8bHx4eqVavyzTffADdGVleuXDEy6l2NHjWC5//Sj2NHj9CxfRvil/7T6EhiMR7uNhpW82XniXt/Nd5N9+r3la0A/0VERBAfH++85VZKmZmZREZGMnbsWHx8fAqVSSOmAnBzc2PWrFlMmTKFS5cuYbfbGThwIHXr1uX9999n4sSJzJw5k7JlyzoXP5jRtOkzjI4gFpdtd/Dqkh+NjuFS9+z3lQvXYl27do3IyEhCQ0Pp1KlT4SM5HA6H62JJUZXW6XZP/Qp0zxiwaJ/REVzu7xGFm1IyO1d8X6Wm53+pfpUKZe/4NYfDwejRo6lYsSLjxo0rUiZN5YmIWJirPmC7a9cuEhIS2LZtm/NEBJs3by5UJv0eKyJiYa76XGWLFi04ePCgS/alYhIRsTAzftxfxSQiYmFmPBGNiklExMLMeHZxFZOIiIVpxCQiIqaiYhIREVPRVJ6IiJiKRkwiImIqJuwlFZOIiKWZsJlUTCIiFqZjTCIiYiquulCgK6mYRESsTMUkIiJmoqk8ERExFTMuF9eFAkVExFR0oUARETEVFZOIiJiKiklERExFxSQiIqaiYhIREVNRMYmIiKmomERExFRUTBaUmJhI586d6dixI3PmzDE6jktERUURFBREt27djI7iUqdPn2bAgAE888wzhISEsGDBAqMjFdnVq1fp3bs33bt3JyQkhFmzZhkdyaXsdjvh4eEMHTrU6Cj3LBWTxdjtdiZPnszcuXNZvXo1q1at4vDhw0bHKrKePXsyd+5co2O4nLu7O2PGjGHt2rXExcWxePHie/7/l4eHBwsWLGDFihUsX76c//znP+zdu9foWC7zxRdfULt2baNj3NNUTBbz/fffU7NmTWrUqIGHhwchISGsX7/e6FhF1rJlSypWrGh0DJcLCAigYcOGAPj4+BAYGEhqaqrBqYrGZrPh7e0NQE5ODjk5OdjMeF6cQjhz5gybNm2id+/eRke5p6mYLCY1NZWqVas671epUuWe/0FnFSdPniQ5OZkmTZoYHaXI7HY7YWFhPPnkkzz55JOl4j0BTJ06lb/+9a+4uelHa1Hob89icjs1Ymn5bbU0y8zMJDIykrFjx+Lj42N0nCJzd3cnISGBzZs38/3333Po0CGjIxXZxo0b8ff3p1GjRkZHuefp7OIWU7VqVc6cOeO8n5qaSkBAgIGJJC/Xrl0jMjKS0NBQOnXqZHQcl6pQoQKtWrXiP//5D/Xq1TM6TpHs3r2bDRs2kJiYyNWrV8nIyGDUqFFMnz7d6Gj3HI2YLKZx48YcPXqUEydOkJ2dzerVq2nfvr3RseQOHA4H48aNIzAwkEGDBhkdxyXS0tJIT08HICsri++++47AwECDUxXdyJEjSUxMZMOGDcyYMYMnnnhCpVRIGjFZTJkyZZg4cSKDBw/GbrfTq1cv6tata3SsIhsxYgTbt2/n/PnztGnThuHDh9OnTx+jYxXZrl27SEhIoF69eoSFhQE33mvbtm0NTlZ4Z8+eZcyYMdjtdhwOB126dKFdu3ZGxxIT0fWYRETEVDSVJyIipqJiEhERU1ExiYiIqaiYRETEVFRMIiJiKiomERNKSkpynp16/fr1dz0LfHp6OosWLSrwa8TExPDZZ58VOqNIcVExiZQgu91e4Of8+c9/ZsiQIXf8enp6OrGxsUWJJWIq+oCtiIucPHmSwYMH06RJEw4cOMBDDz3EtGnTCAkJoWfPnmzZsoXnnnuOihUrEhMTQ3Z2NjVq1CA6Ohpvb28SExOZOnUqfn5+zjOKA8THx7N//34mTpzIuXPnmDRpEidOnADg7bff5ssvv+T48ePOk6KOHj2auXPnsnbtWrKzs+nYsSORkZEAfPLJJyxfvpxq1arh7+9/2+uImIWKScSFjhw5wrvvvkvz5s2Jiopi8eLFAJQrV47Y2FjS0tIYPnw4n3/+OeXLl2fOnDl8/vnnvPzyy0yYMIEFCxZQs2ZN3njjjVz3P2XKFFq2bMnf/vY37HY7ly9fZuTIkfz8888kJCQA8O2333Ls2DGWLFmCw+HglVdeYceOHXh5ebFmzRqWL1+O3W6nR48eKiYxJRWTiAtVq1aN5s2bA9C9e3e+/PJLALp27QrAvn37OHz4MP379wdunKD1scce45dffuHBBx+kVq1azuf+4x//+MP+t23bxvvvvw/cOEO3r68vFy9evG2bLVu2sGXLFsLDwwG4fPkyR48eJTMzkw4dOuDl5QWgcySKaamYRFzo95cQuXn/Zhk4HA5at27NjBkzbtsuOTnZZZcfcTgcDBkyhH79+t32+Pz583WJE7knaPGDiAudOnWKPXv2ALB69Wrn6Ommxx57jN27d3Ps2DEArly5wpEjRwgMDOTkyZMcP37c+dzcBAUFOacH7XY7GRkZeHt7k5mZ6dwmODiYpUuXOh9LTU3lt99+o2XLlqxbt46srCwyMjLYuHGja9+8iIuomERcqHbt2ixbtozQ0FAuXrzonLK7yd/fn+joaEaMGEFoaCh9+/bll19+oVy5ckyePJkhQ4bQv39/qlevnuv+x40bR1JSEqGhofTs2ZOff/4ZPz8/mjVrRrdu3Zg2bRrBwcF069aNfv36ERoaSmRkJJmZmTRs2JCuXbsSFhZGZGTkH0pTxCx0dnERFzl58iTDhg1j1apVRkcRuadpxCQiIqaiEZOIiJiKRkwiImIqKiYRETEVFZOIiJiKiklERExFxSQiIqby/wBlFayFeeSougAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions = test_docs.topic.to_frame('topic').assign(predicted=topic_probabilities.idxmax(axis=1).values)\n",
    "heatmap_data = predictions.groupby('topic').predicted.value_counts().unstack()\n",
    "sns.heatmap(heatmap_data, annot=True, cmap='Blues');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "- pyLDAvis: \n",
    "    - [Talk by the Author](https://speakerdeck.com/bmabey/visualizing-topic-models) and [Paper by (original) Author](http://www.aclweb.org/anthology/W14-3110)\n",
    "    - [Documentation](http://pyldavis.readthedocs.io/en/latest/index.html)\n",
    "- LDA:\n",
    "    - [David Blei Homepage @ Columbia](http://www.cs.columbia.edu/~blei/)\n",
    "    - [Introductory Paper](http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf) and [more technical review paper](http://www.cs.columbia.edu/~blei/papers/BleiLafferty2009.pdf)\n",
    "    - [Blei Lab @ GitHub](https://github.com/Blei-Lab)\n",
    "    \n",
    "- Topic Coherence:\n",
    "    - [Exploring Topic Coherence over many models and many topics](https://www.aclweb.org/anthology/D/D12/D12-1087.pdf)\n",
    "    - [Paper on various Methods](http://www.aclweb.org/anthology/N10-1012)\n",
    "    - [Blog Post - Overview](http://qpleple.com/topic-coherence-to-evaluate-topic-models/)\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "name": "_merged",
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "203.153px",
    "left": "69.9915px",
    "right": "1064px",
    "top": "66.3352px",
    "width": "302px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}