W10-1 collocations revisited.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collocations and ngrams revisited"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting n-grams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt = \"now is the time for all good people to come to the aid of their country\".split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "list(nltk.bigrams(txt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(nltk.ngrams(txt, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We **could** include bigrams, or even trigrams, as dimensions in our vectors, and redo all of the analyses we have already done. But there are a couple of patterns:\n",
    "\n",
    "* There will be a lot of ngrams.\n",
    "* Most of them won't appear all that often.\n",
    "\n",
    "That means we generally want to be strategic about which ones to include. We want only the good ones (whatever that means).\n",
    "\n",
    "Good ngrams are also interesting for other reasons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the civil war dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wikipediaapi\n",
    "pages = [\n",
    "    \"American Civil War\",\n",
    "    \"Abraham Lincoln\",\n",
    "    \"Slavery in the United States\",\n",
    "    \"Slave states and free states\",\n",
    "    \"Emancipation Proclamation\",\n",
    "    \"Robert E. Lee\",\n",
    "    \"Ulysses S. Grant\",\n",
    "    \"Conclusion of the American Civil War\",\n",
    "    \"Origins of the American Civil War\",\n",
    "    \"Issues of the American Civil War\"\n",
    "]\n",
    "import re\n",
    "\n",
    "def underscorize(pagename):\n",
    "    return re.sub(\" \", \"_\", pagename)\n",
    "\n",
    "wiki_wiki = wikipediaapi.Wikipedia(\n",
    "        language='en',\n",
    "        extract_format=wikipediaapi.ExtractFormat.WIKI\n",
    ")\n",
    "page_dict = {}\n",
    "for page in pages:\n",
    "    pagename = underscorize(page)\n",
    "    print(pagename)\n",
    "    p_wiki = wiki_wiki.page(pagename)\n",
    "    page_text = p_wiki.text.split(\"\\n\")\n",
    "    page_paras = [para for para in page_text if len(para) > 1]\n",
    "    page_dict[pagename] = page_paras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wordify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_para(para):\n",
    "    sentences = nltk.sent_tokenize(para)\n",
    "    wordified_sentences = []\n",
    "    for sent in sentences:\n",
    "        wordified_sentence = nltk.word_tokenize(sent)\n",
    "        wordified_sentences.append(wordified_sentence)\n",
    "    return wordified_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wordified_sentences = []\n",
    "for pagename, page_paras in page_dict.items():\n",
    "    for para in page_paras:\n",
    "        wordified_sentences += process_para(para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wordified_sentences[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a bigram frequency distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigram_fdist = nltk.FreqDist()\n",
    "for sent in wordified_sentences:\n",
    "    bigram_fdist.update(nltk.bigrams(sent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigram_fdist.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using part of speech"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tagging a sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "sentence = \"now is the time for all good people to come to the aid of their country\"\n",
    "print(nltk.pos_tag(sentence.split()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "nltk.help.upenn_tagset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   Note that everything we did with word vectors, \n",
    "   you can do with tagged words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tag the entire civil war corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have to separate it into sentences before wordifying"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "def process_para2(para):\n",
    "    sentences = nltk.sent_tokenize(para)\n",
    "    tagged_sentences = []\n",
    "    for sent in sentences:\n",
    "        wordified_sentence = nltk.word_tokenize(sent)\n",
    "        tagged_sentence = nltk.pos_tag(wordified_sentence)\n",
    "        tagged_sentences.append(tagged_sentence)\n",
    "    return tagged_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagged_sentences = []\n",
    "for pagename, page_paras in page_dict.items():\n",
    "    for para in page_paras:\n",
    "        tagged_sentences += process_para2(para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tagged_sentences[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's again get the most frequent bigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigram_fdist2 = nltk.FreqDist()\n",
    "for sent in tagged_sentences:\n",
    "    bigram_fdist2.update(nltk.bigrams(sent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigram_fdist2.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's filter on part of speech"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want the first tag to be a noun or adjective, and the second tag to be a noun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_on_pos(bg):\n",
    "    tag1 = bg[0][0][1]\n",
    "    tag2 = bg[0][1][1]\n",
    "    return re.search(\"^N.*\", tag2) and ((re.search(\"^N.*\", tag1) or re.search(\"^J.*\", tag1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for bg in bigram_fdist2.most_common(100):\n",
    "    if filter_on_pos(bg):\n",
    "        print(bg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "328.3999938964844px",
    "width": "402.3999938964844px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}