topic-modeling-with-colab-gensim-mallet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling with Google Colab, Gensim and Mallet\n",
    "\n",
    "This notebook implements [Gensim](https://radimrehurek.com/gensim/) and [Mallet](http://mallet.cs.umass.edu/index.php) for topic modeling using the [Google Colab](https://colab.research.google.com/) platform. The README is available at the [Colab + Gensim + Mallet Github repository](https://github.com/polsci/colab-gensim-mallet)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upgrade Gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade gensim==3.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Java"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os       #importing os to set environment variable\n",
    "def install_java():\n",
    "  !apt-get install -y openjdk-8-jdk-headless -qq > /dev/null      #install openjdk\n",
    "  os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"     #set environment variable\n",
    "  !java -version       #check java version\n",
    "install_java()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Mallet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget http://mallet.cs.umass.edu/dist/mallet-2.0.8.zip\n",
    "!unzip mallet-2.0.8.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload and extract corpus\n",
    "\n",
    "Upload a zip file with your corpus. The zip file of the corpus should contain a single directory containing .txt files. It appears that you need to rerun the cell if you don't select the file within a set time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "from google.colab import files\n",
    "\n",
    "uploaded = files.upload()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This assumes you have uploaded a file above! This will output a directory listing as well so you can see your uploaded file and the directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_zip_file = list(uploaded.keys())[0]\n",
    "\n",
    "print ('Extracting',path_to_zip_file)\n",
    "\n",
    "with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n",
    "  zip_ref.extractall('.')\n",
    "\n",
    "print()\n",
    "print('Here is a directory listing (you should see a directory with your corpus):')\n",
    "!ls -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import required libraries for topic modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "import gensim.corpora as corpora\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.models.wrappers import LdaMallet\n",
    "from gensim.models.coherencemodel import CoherenceModel\n",
    "from gensim import similarities\n",
    "\n",
    "import os.path\n",
    "import re\n",
    "import glob\n",
    "\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the path to the Mallet binary and set the path to the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['MALLET_HOME'] = '/content/mallet-2.0.8'\n",
    "mallet_path = '/content/mallet-2.0.8/bin/mallet' # you should NOT need to change this \n",
    "corpus_path = 'transcripts' # you need to change this path to the directory containing your corpus of .txt files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions to load and preprocess the corpus and create the document-term matrix\n",
    "\n",
    "The following cell contains functions to load a corpus from a directory of text files, preprocess the corpus and create the bag of words document-term matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data_from_dir(path):\n",
    "    file_list = glob.glob(path + '/*.txt')\n",
    "\n",
    "    # create document list:\n",
    "    documents_list = []\n",
    "    source_list = []\n",
    "    for filename in file_list:\n",
    "        with open(filename, 'r', encoding='utf8') as f:\n",
    "            text = f.read()\n",
    "            f.close()\n",
    "            documents_list.append(text)\n",
    "            source_list.append(os.path.basename(filename))\n",
    "    print(\"Total Number of Documents:\",len(documents_list))\n",
    "    return documents_list, source_list\n",
    "\n",
    "def preprocess_data(doc_set,extra_stopwords = {}):\n",
    "    # adapted from https://www.datacamp.com/community/tutorials/discovering-hidden-topics-python\n",
    "    # replace all newlines or multiple sequences of spaces with a standard space\n",
    "    doc_set = [re.sub('\\s+', ' ', doc) for doc in doc_set]\n",
    "    # initialize regex tokenizer\n",
    "    tokenizer = RegexpTokenizer(r'\\w+')\n",
    "    # create English stop words list\n",
    "    en_stop = set(stopwords.words('english'))\n",
    "    # add any extra stopwords\n",
    "    if (len(extra_stopwords) > 0):\n",
    "        en_stop = en_stop.union(extra_stopwords)\n",
    "    \n",
    "    # list for tokenized documents in loop\n",
    "    texts = []\n",
    "    # loop through document list\n",
    "    for i in doc_set:\n",
    "        # clean and tokenize document string\n",
    "        raw = i.lower()\n",
    "        tokens = tokenizer.tokenize(raw)\n",
    "        # remove stop words from tokens\n",
    "        stopped_tokens = [i for i in tokens if not i in en_stop]\n",
    "        # add tokens to list\n",
    "        texts.append(stopped_tokens)\n",
    "    return texts\n",
    "\n",
    "def prepare_corpus(doc_clean):\n",
    "    # adapted from https://www.datacamp.com/community/tutorials/discovering-hidden-topics-python\n",
    "    # Creating the term dictionary of our courpus, where every unique term is assigned an index. dictionary = corpora.Dictionary(doc_clean)\n",
    "    dictionary = corpora.Dictionary(doc_clean)\n",
    "    \n",
    "    dictionary.filter_extremes(no_below=5, no_above=0.5)\n",
    "    # Converting list of documents (corpus) into Document Term Matrix using dictionary prepared above.\n",
    "    doc_term_matrix = [dictionary.doc2bow(doc) for doc in doc_clean]\n",
    "    # generate LDA model\n",
    "    return dictionary,doc_term_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and pre-process the corpus\n",
    "Load the corpus, preprocess with additional stop words and output dictionary and document-term matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adjust the path below to wherever you have the transcripts2018 folder\n",
    "document_list, source_list = load_data_from_dir(corpus_path)\n",
    "\n",
    "# I've added extra stopwords here in addition to NLTK's stopword list - you could look at adding others.\n",
    "doc_clean = preprocess_data(document_list,{'laughter','applause'})\n",
    "dictionary, doc_term_matrix = prepare_corpus(doc_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LDA model with 30 topics\n",
    "The following cell sets the number of topics we are training the model for. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_topics=30 # adjust this to alter the number of topics\n",
    "words=20 #adjust this to alter the number of words output for the topic below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell runs LDA using Mallet from Gensim using the number_of_topics specified above. This might take a few minutes! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ldamallet30 = LdaMallet(mallet_path, corpus=doc_term_matrix, num_topics=number_of_topics, id2word=dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell outputs the topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ldamallet30.show_topics(num_topics=number_of_topics,num_words=words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert to Gensim model format\n",
    "Convert the Mallet model to Gensim format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gensimmodel30 = gensim.models.wrappers.ldamallet.malletmodel2ldamodel(ldamallet30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a coherence score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coherencemodel = CoherenceModel(model=gensimmodel30, texts=doc_clean, dictionary=dictionary, coherence='c_v')\n",
    "print (coherencemodel.get_coherence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get id for specific videos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lookup_doc_id = source_list.index('2017-09-20-zeynep_tufekci_we_re_building_a_dystopia_just_to_make_people_click_on_ads.txt')\n",
    "print('Document ID from lookup:', lookup_doc_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preview a document\n",
    "\n",
    "Preview a document - you can change the doc_id to view another document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_id = lookup_doc_id # index of document to explore - this can be an id number or set to lookup_doc_id\n",
    "print(re.sub('\\s+', ' ', document_list[doc_id])) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output the distribution of topics for the document\n",
    "\n",
    "The next cell outputs the distribution of topics on the document specified above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_topics = gensimmodel30.get_document_topics(doc_term_matrix[doc_id])\n",
    "document_topics = sorted(document_topics, key=lambda x: x[1], reverse=True) # sorts document topics\n",
    "\n",
    "for topic, prop in document_topics:\n",
    "    topic_words = [word[0] for word in gensimmodel30.show_topic(topic, 10)]\n",
    "    print (\"%.2f\" % prop, topic, topic_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find similar documents\n",
    "This will find the 5 most similar documents to the document specified above based on their topic distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gensimmodel30[doc_term_matrix] below represents the documents in the corpus in LDA vector space\n",
    "lda_index = similarities.MatrixSimilarity(gensimmodel30[doc_term_matrix])\n",
    "\n",
    "# query for our doc_id from above\n",
    "similarity_index = lda_index[gensimmodel30[doc_term_matrix[doc_id]]]\n",
    "\n",
    "# Sort the similarity index\n",
    "similarity_index = sorted(enumerate(similarity_index), key=lambda item: -item[1])\n",
    "\n",
    "for i in range(1,6): \n",
    "    document_id, similarity_score = similarity_index[i]\n",
    "\n",
    "    print('Document Index:',document_id)\n",
    "    print('Document:', source_list[document_id])\n",
    "    print('Similarity Score:',similarity_score)\n",
    "    \n",
    "    print(re.sub('\\s+', ' ', document_list[document_id][:500]), '...') # preview first 500 characters\n",
    "\n",
    "    document_topics = gensimmodel30[doc_term_matrix[document_id]]\n",
    "    document_topics = sorted(document_topics, key=lambda x: x[1], reverse=True)\n",
    "    for topic, prop in document_topics:\n",
    "        topic_words = [word[0] for word in gensimmodel30.show_topic(topic, 10)]\n",
    "        print (\"%.2f\" % prop, topic, topic_words)\n",
    "    \n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}