2014-02-16-PLOS_Biology_Topics.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "PLOS Biology Topics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Ever wonder what topics are discussed in PLOS Biology articles?\n",
      "Here I will apply an implementation of [Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) (LDA)\n",
      "on a set of 1,754 PLOS Biology articles to work out what a possible collection of underlying topics could be.\n",
      "\n",
      "I first read about LDA in [Building Machine Learning Systems with Python](http://www.packtpub.com/building-machine-learning-systems-with-python/book)\n",
      "co-authored by [Luis Coelho](https://twitter.com/luispedrocoelho).\n",
      "\n",
      "LDA seems to have been first described by [Blei *et al.*](http://jmlr.org/papers/volume3/blei03a/blei03a.pdf) and\n",
      "I will use the implementation provided by\n",
      "[gensim](http://radimrehurek.com/gensim/wiki.html#latent-dirichlet-allocation) which was written by\n",
      "[Radim \u0158eh\u016f\u0159ek](https://twitter.com/RadimRehurek)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import gensim"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import plospy\n",
      "import os"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import nltk"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import cPickle as pickle"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "With the following lines of code we open, parse, and tokenize all 1,754 PLOS Biology articles in our collection.\n",
      "\n",
      "As this takes a bit of time and memory, I carried out all of these steps once and stored the resulting data structures to my hard disk for later reuse - see further below."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "all_names = [name for name in os.listdir('../plos/plos_biology/plos_biology_data') if '.dat' in name]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "article_bodies = []\n",
      "\n",
      "for name_i, name in enumerate(all_names):\n",
      "    docs = plospy.PlosXml('../plos/plos_biology/plos_biology_data/'+name)\n",
      "    for article in docs.docs:\n",
      "        article_bodies.append(article['body'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We have 1,754 PLOS Biology articles in our collection:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(article_bodies)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "punkt_param = nltk.tokenize.punkt.PunktParameters()\n",
      "punkt_param.abbrev_types = set(['et al', 'i.e', 'e.g', 'ref', 'c.f',\n",
      "                                'fig', 'Fig', 'Eq', 'eq', 'eqn', 'Eqn',\n",
      "                                'dr', 'Dr'])\n",
      "sentence_splitter = nltk.tokenize.punkt.PunktSentenceTokenizer(punkt_param)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sentences = []\n",
      "for body in article_bodies:\n",
      "    sentences.append(sentence_splitter.tokenize(body))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "articles = []\n",
      "for body in sentences:\n",
      "    this_article = []\n",
      "    for sentence in body:\n",
      "        this_article.append(nltk.tokenize.word_tokenize(sentence))\n",
      "    articles.append(this_article)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pickle.dump(articles, open('plos_biology_articles_tokenized.list', 'w'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "articles = pickle.load(open('plos_biology_articles_tokenized.list', 'r'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "is_stopword = lambda w: len(w) < 4 or w in nltk.corpus.stopwords.words('english')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Save each article as one list of tokens and filter out stopwords:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "articles_unfurled = []\n",
      "for article in articles:\n",
      "    this_article = []\n",
      "    for sentence in article:\n",
      "        this_article += [token.lower().encode('utf-8') for token in sentence if not is_stopword(token)]\n",
      "    articles_unfurled.append(this_article)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pickle.dump(articles_unfurled, open('plos_biology_articles_unfurled.list', 'w'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "articles_unfurled = pickle.load(open('plos_biology_articles_unfurled.list', 'r'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Dictionary and Corpus Creation"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Create a dictionary of all words (tokens) that appear in our collection of PLOS Biology articles\n",
      "and create a bag of words object for each article (`doc2bow`)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dictionary = gensim.corpora.Dictionary(articles_unfurled)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dictionary.save('plos_biology.dict')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dictionary = gensim.corpora.dictionary.Dictionary().load('plos_biology.dict')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I noticed that the word *figure* occurs rather frequently in these articles, so let us exclude this and any other words\n",
      "that appear in more than half of the articles in this data set ([thanks to Radim](https://twitter.com/RadimRehurek/status/436136774906044416)\n",
      "for pointing this out to me)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dictionary.filter_extremes()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "corpus = [dictionary.doc2bow(article) for article in articles_unfurled]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gensim.corpora.MmCorpus.serialize('plos_biology_corpus.mm', corpus)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "corpus = gensim.corpora.MmCorpus('plos_biology_corpus.mm')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model = gensim.models.ldamodel.LdaModel(corpus, id2word=dictionary, update_every=1, chunksize=100, passes=2, num_topics=20)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model.save('plos_biology.lda_model')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model = gensim.models.ldamodel.LdaModel.load('plos_biology.lda_model')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And these are the twenty topics we find in 1,754 PLOS Biology articles:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for topic_i, topic in enumerate(model.print_topics(20)):\n",
      "    print('topic # %d: %s\\n' % (topic_i+1, topic))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Topics with Lemmatized Tokens"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As we can notice, some of the tokens in the above topics are just singular and plural forms of the same word.\n",
      "\n",
      "Let us see what topics we find after lemmatizing all of our tokens."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nltk.stem import WordNetLemmatizer\n",
      "wnl = WordNetLemmatizer()\n",
      "articles_lemmatized = []\n",
      "for article in articles_unfurled:\n",
      "    articles_lemmatized.append([wnl.lemmatize(token) for token in article])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pickle.dump(articles_lemmatized, open('plos_biology_articles_lemmatized.list', 'w'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dictionary_lemmatized = gensim.corpora.Dictionary(articles_lemmatized)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dictionary_lemmatized.save('plos_biology_lemmatized.dict')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dictionary_lemmatized.filter_extremes()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "corpus_lemmatized = [dictionary_lemmatized.doc2bow(article) for article in articles_lemmatized]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gensim.corpora.MmCorpus.serialize('plos_biology_corpus_lemmatized.mm', corpus_lemmatized)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model_lemmatized = gensim.models.ldamodel.LdaModel(corpus_lemmatized, id2word=dictionary_lemmatized, update_every=1, chunksize=100, passes=2, num_topics=20)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for topic_i, topic in enumerate(model_lemmatized.print_topics(20)):\n",
      "    print('topic # %d: %s\\n' % (topic_i+1, topic))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}