Basic Natural Language Processing in Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Intro to Jupyter Notebooks\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Natural Language Processing in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to do some simple natural language processing tasks in Python.\n",
    "\n",
    "Steps:\n",
    "   \n",
    "1. Get some text\n",
    "2. Prepare our text\n",
    "3. Process our text\n",
    "4. Revel in what we've done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step One: Get some text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to select a text from the Gutenberg Project: https://www.gutenberg.org/. All texts at the Gutenberg Project are in the public domain and already prepared in plaintext (.txt file).\n",
    "\n",
    "For this exercise, we're going to use Mark Twain's \"Innocents Abroad,\" about his Grand Tour of Europe and the Mediterranean in 1867.\n",
    "\n",
    "In the Gutenberg search box, search for \"Mark Twain.\" In the list of results, select \"Innocents Abroad.\"\n",
    "\n",
    "_Take note of the url: https://www.gutenberg.org/ebooks/3176. We're going to need the number at the end later._\n",
    "\n",
    "Click on the file called \"Plain Text UTF-8.\" We also want to use plain text files in computational text analysis so that the computer is not confused by formatting. UTF-8 can encode any Unicode character and therefore, can handle text in most writing systems.\n",
    "\n",
    "This will open up the text file in a browser window (Mac). You'll see it is not easy to read on the screen, which is why we have ePub and other formats for reading on a Kindle or other devices.\n",
    "\n",
    "You'll see some preliminary information from the Gutenberg Project in the beginning, and at the very end, you'll see the Gutenberg license agreement. In our next step, we'll learn how to remove that information so that only Mark Twain's words appear in our text file.\n",
    "\n",
    "But wait, how do I download the file onto my computer? You actually don't have to. We're going to let our program retrieve the file for us.\n",
    "\n",
    "If you do want to download it, go back one page to the page with the different file formats. (If you get lost, it's the url above). (Mac) Right click the file called \"Plain Text UTF-8\" and select \"Save Link As ...\" When prompted, select OK and remember where you saved the file.\n",
    "\n",
    "Congratulations, you've now collected some data. Text = data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk  \n",
    "from gutenberg.acquire import load_etext \n",
    "from gutenberg.cleanup import strip_headers\n",
    "\n",
    "text = strip_headers(load_etext(3176)).strip()  # 3176 = Innocents Abroad\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's review what just happened.\n",
    "\n",
    "First, we imported the `nltk` (Natural Language Toolkit) Python package. You can read more about the Natural Language Toolkit here: http://www.nltk.org/. It includes a package called `gutenberg` that helps retrieve the text from the Gutenberg Project webpage and extract the extraneous information.\n",
    "\n",
    "Notice how we used the number from the end of the Gutenberg Project url. That number is telling the program to retrieve the plain-text version of \"Innocents Abroad.\"\n",
    "\n",
    "Rather than using the entire book, let's just use an excerpt to make things easier. In the next code block, I've reset the text variable to an excerpt, which includes an overview of Twain's itinerary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remember to wrap in triple quotes for multi-line text\n",
    "text = \"\"\"The undersigned will make an excursion as above during the coming\n",
    "     season, and begs to submit to you the following programme:\n",
    "\n",
    "       A first-class steamer, to be under his own command, and capable of\n",
    "     accommodating at least one hundred and fifty cabin passengers, will\n",
    "     be selected, in which will be taken a select company, numbering not\n",
    "     more than   three-fourths of the ship's capacity.  There is good\n",
    "     reason to believe that this company can be easily made up in this\n",
    "     immediate vicinity, of mutual friends and acquaintances.\n",
    "\n",
    "       The steamer will be provided with every necessary comfort,\n",
    "     including library and musical instruments.\n",
    "\n",
    "       An experienced physician will be on board.\n",
    "\n",
    "       Leaving New York about June 1st, a middle and pleasant route will\n",
    "     be taken across the Atlantic, and passing through the group of\n",
    "     Azores, St. Michael will be reached in about ten days.  A day or two\n",
    "     will be spent here, enjoying the fruit and wild scenery of these\n",
    "     islands, and the voyage continued, and Gibraltar reached in three or\n",
    "     four days.\n",
    "\n",
    "       A day or two will be spent here in looking over the wonderful\n",
    "     subterraneous fortifications, permission to visit these galleries\n",
    "     being readily obtained.\n",
    "\n",
    "       From Gibraltar, running along the coasts of Spain and France,\n",
    "     Marseilles will be reached in three days.  Here ample time will be\n",
    "     given not only to look over the city, which was founded six hundred\n",
    "     years before the Christian era, and its artificial port, the finest\n",
    "     of the kind in the Mediterranean, but to visit Paris during the\n",
    "     Great Exhibition; and the beautiful city of Lyons, lying\n",
    "     intermediate, from the heights of which, on a clear day, Mont Blanc\n",
    "     and the Alps can be distinctly seen.  Passengers who may wish to\n",
    "     extend the time at Paris can do so, and, passing down through\n",
    "     Switzerland, rejoin the steamer at Genoa.\n",
    "\n",
    "       From Marseilles to Genoa is a run of one night.  The excursionists\n",
    "     will have an opportunity to look over this, the “magnificent city of\n",
    "     palaces,” and visit the birthplace of Columbus, twelve miles off,\n",
    "     over a beautiful road built by Napoleon I.  From this point,\n",
    "     excursions may be made to Milan, Lakes Como and Maggiore, or to\n",
    "     Milan, Verona (famous for its extraordinary fortifications), Padua,\n",
    "     and Venice.  Or, if passengers desire to visit Parma (famous for\n",
    "     Correggio's frescoes) and Bologna, they can by rail go on to\n",
    "     Florence, and rejoin the steamer at Leghorn, thus spending about\n",
    "     three weeks amid the cities most famous for art in Italy.\n",
    "\n",
    "       From Genoa the run to Leghorn will be made along the coast in one\n",
    "     night, and time appropriated to this point in which to visit\n",
    "     Florence, its palaces and galleries; Pisa, its cathedral and\n",
    "     “Leaning Tower,” and Lucca and its baths, and Roman amphitheater;\n",
    "     Florence, the most remote, being distant by rail about sixty miles.\n",
    "\n",
    "       From Leghorn to Naples (calling at Civita Vecchia to land any who\n",
    "     may prefer to go to Rome from that point), the distance will be made\n",
    "     in about thirty-six hours; the route will lay along the coast of\n",
    "     Italy, close by Caprera, Elba, and Corsica.  Arrangements have been\n",
    "     made to take on board at Leghorn a pilot for Caprera, and, if\n",
    "     practicable, a call will be made there to visit the home of\n",
    "     Garibaldi.\n",
    "\n",
    "       Rome [by rail], Herculaneum, Pompeii, Vesuvius, Vergil's tomb, and\n",
    "     possibly the ruins of Paestum can be visited, as well as the\n",
    "     beautiful surroundings of Naples and its charming bay.\n",
    "\n",
    "       The next point of interest will be Palermo, the most beautiful\n",
    "     city of Sicily, which will be reached in one night from Naples.  A\n",
    "     day will be spent here, and leaving in the evening, the course will\n",
    "     be taken towards Athens.\n",
    "\n",
    "       Skirting along the north coast of Sicily, passing through the\n",
    "     group of Aeolian Isles, in sight of Stromboli and Vulcania, both\n",
    "     active volcanoes, through the Straits of Messina, with “Scylla” on\n",
    "     the one hand and “Charybdis” on the other, along the east coast of\n",
    "     Sicily, and in sight of Mount Etna, along the south coast of Italy,\n",
    "     the west and south coast of Greece, in sight of ancient Crete, up\n",
    "     Athens Gulf, and into the Piraeus, Athens will be reached in two and\n",
    "     a half or three days.  After tarrying here awhile, the Bay of\n",
    "     Salamis will be crossed, and a day given to Corinth, whence the\n",
    "     voyage will be continued to Constantinople, passing on the way\n",
    "     through the Grecian Archipelago, the Dardanelles, the Sea of\n",
    "     Marmora, and the mouth of the Golden Horn, and arriving in about\n",
    "     forty-eight hours from Athens.\n",
    "\n",
    "       After leaving Constantinople, the way will be taken out through\n",
    "     the beautiful Bosphorus, across the Black Sea to Sebastopol and\n",
    "     Balaklava, a run of about twenty-four hours.  Here it is proposed to\n",
    "     remain two days, visiting the harbors, fortifications, and\n",
    "     battlefields of the Crimea; thence back through the Bosphorus,\n",
    "     touching at Constantinople to take in any who may have preferred to\n",
    "     remain there; down through the Sea of Marmora and the Dardanelles,\n",
    "     along the coasts of ancient Troy and Lydia in Asia, to Smyrna, which\n",
    "     will be reached in two or two and a half days from Constantinople.\n",
    "     A sufficient stay will be made here to give opportunity of visiting\n",
    "     Ephesus, fifty miles distant by rail.\n",
    "\n",
    "       From Smyrna towards the Holy Land the course will lay through the\n",
    "     Grecian  Archipelago, close by the Isle of Patmos, along the coast\n",
    "     of Asia, ancient Pamphylia, and the Isle of Cyprus.  Beirut will be\n",
    "     reached in three days.  At Beirut time will be given to visit\n",
    "     Damascus; after which the steamer will proceed to Joppa.\n",
    "\n",
    "       From Joppa, Jerusalem, the River Jordan, the Sea of Tiberias,\n",
    "     Nazareth, Bethany, Bethlehem, and other points of interest in the\n",
    "     Holy Land can be visited, and here those who may have preferred to\n",
    "     make the journey from Beirut through the country, passing through\n",
    "     Damascus, Galilee, Capernaum, Samaria, and by the River Jordan and\n",
    "     Sea of Tiberias, can rejoin the steamer.\n",
    "\n",
    "       Leaving Joppa, the next point of interest to visit will be\n",
    "     Alexandria, which will be reached in twenty-four hours.  The ruins\n",
    "     of Caesar's Palace, Pompey's Pillar, Cleopatra's Needle, the\n",
    "     Catacombs, and ruins of ancient Alexandria will be found worth the\n",
    "     visit.  The journey to Cairo, one hundred and thirty miles by rail,\n",
    "     can be made in a few hours, and from which can be visited the site\n",
    "     of ancient Memphis, Joseph's Granaries, and the Pyramids.\n",
    "\n",
    "       From Alexandria the route will be taken homeward, calling at\n",
    "     Malta, Cagliari (in Sardinia), and Palma (in Majorca), all\n",
    "     magnificent harbors, with charming scenery, and abounding in fruits.\n",
    "\n",
    "       A day or two will be spent at each place, and leaving Parma in the\n",
    "     evening, Valencia in Spain will be reached the next morning.  A few\n",
    "     days will be spent in this, the finest city of Spain.\n",
    "\n",
    "       From Valencia, the homeward course will be continued, skirting\n",
    "     along the coast of Spain.  Alicant, Carthagena, Palos, and Malaga\n",
    "     will be passed but a mile or two distant, and Gibraltar reached in\n",
    "     about twenty-four hours.\n",
    "\n",
    "       A stay of one day will be made here, and the voyage continued to\n",
    "     Madeira, which will be reached in about three days.  Captain\n",
    "     Marryatt writes: “I do not know a spot on the globe which so much\n",
    "     astonishes and delights upon first arrival as Madeira.” A stay of\n",
    "     one or two days will be made here, which, if time permits, may be\n",
    "     extended, and passing on through the islands, and probably in sight\n",
    "     of the Peak of Teneriffe, a southern track will be taken, and the\n",
    "     Atlantic crossed within the latitudes of the northeast trade winds,\n",
    "     where mild and pleasant weather, and a smooth sea, can always be\n",
    "     expected.\n",
    "\n",
    "       A call will be made at Bermuda, which lies directly in this route\n",
    "     homeward, and will be reached in about ten days from Madeira, and\n",
    "     after spending a short time with our friends the Bermudians, the\n",
    "     final departure will be made for home, which will be reached in\n",
    "     about three days.\n",
    "\n",
    "       Already, applications have been received from parties in Europe\n",
    "     wishing to join the Excursion there.\n",
    "\n",
    "       The ship will at all times be a home, where the excursionists, if\n",
    "     sick, will be surrounded by kind friends, and have all possible\n",
    "     comfort and sympathy.\n",
    "\n",
    "       Should contagious sickness exist in any of the ports named in the\n",
    "     program, such ports will be passed, and others of interest\n",
    "     substituted.\n",
    "\n",
    "       The price of passage is fixed at $1,250, currency, for each adult\n",
    "     passenger.  Choice of rooms and of seats at the tables apportioned\n",
    "     in the order in which passages are engaged; and no passage\n",
    "     considered engaged until ten percent of the passage money is\n",
    "     deposited with the treasurer.\n",
    "\n",
    "       Passengers can remain on board of the steamer, at all ports, if\n",
    "     they desire, without additional expense, and all boating at the\n",
    "     expense of the ship.\n",
    "\n",
    "       All passages must be paid for when taken, in order that the most\n",
    "     perfect arrangements be made for starting at the appointed time.\n",
    "\n",
    "       Applications for passage must be approved by the committee before\n",
    "     tickets are issued, and can be made to the undersigned.\n",
    "\n",
    "       Articles of interest or curiosity, procured by the passengers\n",
    "     during the voyage, may be brought home in the steamer free of\n",
    "     charge.\n",
    "\n",
    "       Five dollars per day, in gold, it is believed, will be a fair\n",
    "     calculation to make for all traveling expenses onshore and at the\n",
    "     various points where passengers may wish to leave the steamer for\n",
    "     days at a time.\n",
    "\n",
    "       The trip can be extended, and the route changed, by unanimous vote\n",
    "     of the passengers.\n",
    "\n",
    "      CHAS.  C.  DUNCAN,  117 WALL STREET, NEW YORK  R.  R.  G******,\n",
    "     Treasurer\n",
    "\n",
    "      Committee on Applications  J.  T.  H*****, ESQ.  R.  R.  G*****,\n",
    "     ESQ.  C.  C.  Duncan\n",
    "\n",
    "      Committee on Selecting Steamer  CAPT.  W.  W.  S* * * *, Surveyor\n",
    "     for Board of Underwriters\n",
    "\n",
    "       C.  W.  C******, Consulting Engineer for U.S.  and Canada  J.  T.\n",
    "     H*****, Esq. C.  C.  DUNCAN\n",
    "\n",
    "       P.S.--The very beautiful and substantial side-wheel steamship\n",
    "     “Quaker City” has been chartered for the occasion, and will leave\n",
    "     New York June 8th.  Letters have been issued by the government\n",
    "     commending the party to courtesies abroad.\n",
    "\"\"\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('punkt')\n",
    "tokens = nltk.word_tokenize(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable `tokens` is a Python list, where each item in the list is a word or punctuation mark from our text. If we run the `print` command, we'll see what that looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tokens)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now the computer identify the part of speech of each word (\"POS tagging\"). Ignore punctuation for now. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('averaged_perceptron_tagger') # we'll have to install some additional modules from nltk\n",
    "tagged_tokens = nltk.pos_tag(tokens)\n",
    "print(tagged_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two or three letter code corresponds to a different part of speech. In addition to the common parts of speech (noun, verb, article, adjective, preposition, pronoun, adverb, conjunction, and interjection), the POS tagging also gives us many more tags for words like plurals, possessives, or proper nouns, or tense.  \n",
    "\n",
    "This model only works in English. If you are working in other languages, you will need to download a different tag set. Non-English tag set might add more tags for grammatical gender or case.\n",
    "\n",
    "We are using the Penn Treebank English tagset, and you can see what the tags mean here: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html\n",
    "\n",
    "You can also quickly look up the meaning of a code in your program:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('tagsets')\n",
    "nltk.help.upenn_tagset('NNP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This told us that the NNP tag refers to a proper noun. Sounds like it might be useful for named entity recognition tasks.\n",
    "\n",
    "Some of you might be wondering how NLTK handles two-word nouns (called bigrams) like \"New York.\"\n",
    "\n",
    "The fifth paragraph begins, \"\"Leaving New York about June 1st, ...\"\n",
    "\n",
    "Here's the results of our tokenization and tagging for that phrase: \"('Leaving', 'VBG'), ('New', 'NNP'), ('York', 'NNP'), ('about', 'IN'), ('June', 'NNP'), ('1st', 'CD'), (',', ',')\"\n",
    "\n",
    "Hmm, so \"New\" and \"York\" are separate tokens (words), but both are tagged as proper nouns. Is that just because they are both capitalized? Let's find out.\n",
    "\n",
    "Next we are going to identify named entities (persons or places), a process known as \"named entity recognition\" (NER). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('maxent_ne_chunker')\n",
    "nltk.download('words')\n",
    "named_entities = nltk.chunk.ne_chunk(tagged_tokens)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(named_entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Digging through the results file above, we see how it handled \"New York\":\n",
    "    \"Leaving/VBG\n",
    "  (GPE New/NNP York/NNP)\n",
    "  about/IN\n",
    "  June/NNP\n",
    "  1st/CD\n",
    "  ,/,\"\n",
    "\n",
    "Looks to me like it identified New York as a single place and gave it the tag \"GPE,\" which stands for Geo-Political Entity. It did so using a process called noun-phrase chunking, which we won't get into here. \n",
    "\n",
    "There's some errors too. \"St. Michael\" is tagged as \"PERSON\", but from the context, it is clear that it refers to a place. And \"Atlantic\" is tagged an an \"ORGANIZATION,\" when it is clear that it is the \"Atlantic Ocean.\" Perhaps the absence of the word \"Ocean\" fooled the tagger.\n",
    "\n",
    "And as we skim through, should we have expected it to pick up \"Cleopatra's Needle\" or \"Caesar's Palace\"?\n",
    "\n",
    "What about \"River Jordan\" and \"Black Sea\"?\n",
    "\n",
    "It was a hassle to scroll through looking for tokens that were tagged as placenames. How can we do it computationally?\n",
    "\n",
    "First we're going to figure out what kind of variable our named_entities variable is. Is it a list, dictionary, something else?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(named_entities))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm, that's not a variable type I recognize. It looks like a special variable type used within the nltk package. How do I extract just the placenames?\n",
    "\n",
    "The code below is traversing the named_entities tree and spitting out everything with the label \"GPE.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in named_entities.subtrees(filter=lambda x: x.label() == 'GPE'):\n",
    "     print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoa that was fast. Imagine how long it would take to manually extract every place name in the entire book?\n",
    "\n",
    "You'll see that it also picked up some other two-word placenames like \"Grecian Archipelago\" and \"Aeolian Isles.\"\n",
    "\n",
    "Hmm, but I also see some false positives -- anyone know where \"Christian\" is? I'm guessing that \"Garibaldi\" refers to the person. We'd have to dive into the text to be sure.\n",
    "\n",
    "So it's not perfect, but even having to do a little clean up, this certainly saved us a lot of time.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in named_entities.subtrees(filter=lambda x: x.label() == 'LOCATION'):\n",
    "     print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in named_entities.subtrees(filter=lambda x: x.label() == 'FACILITY'):\n",
    "     print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "named_entities2 = nltk.chunk.ne_chunk(tagged_tokens, binary=True)\n",
    "for i in named_entities2.subtrees(filter=lambda x: x.label() == 'NE'):\n",
    "     print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll see some names repeated. What if we wanted only unique names? \n",
    "\n",
    "Instead of printing out each named entity, we'll make a set, which acts much like a list, but only includes unique values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ne = set()\n",
    "for tree in named_entities2.subtrees(filter=lambda t: t.label() == 'NE'):\n",
    "    ne.add(' '.join([child[0] for child in tree.leaves()]))\n",
    "ne = list(ne)\n",
    "print(type(ne))\n",
    "print(ne)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's export it into a csv file so that we can put our places on a map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "with open('places.csv', 'w') as csvfile:\n",
    "    writer = csv.writer(csvfile)\n",
    "    for item in ne:\n",
    "        writer.writerow([item])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "import into Google Maps - see mistakes\n",
    "\n",
    "didn't pick up Troy in NER\n",
    "\n",
    "Smyrna - Greece or Turkey at the time?\n",
    "Galilee - adding Israel helps, but did not exist at the time\n",
    "Mont Blanc\n",
    "add country names or other details\n",
    "doesn't do well with terms like \"Holy Land\"\n",
    "Pompey's Pillar, when combined, Google went to Montana, not from context in Alexandria Egypt\n",
    "Cleopatra's Needle when combined went to London where it is now, but from context it was then in Alexandria and indeed was not moved to London until 1877, but if change to Cleo Needle, Alexandria, it puts in NYC\n",
    "Leghorn = Livorno\n",
    "Teneriffe went to Australia, not Canary Islands\n",
    "Sebastopol Calif not Crimea\n",
    "Civitavecchia usually spelled as one word, Twain spelled as two"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and then annotate place names with https://recogito.pelagios.org/ !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc1 = \"Sugar is bad to consume. My sister likes to have sugar, but not my father.\"\n",
    "doc2 = \"My father spends a lot of time driving my sister around to dance practice.\"\n",
    "doc3 = \"Doctors suggest that driving may cause increased stress and blood pressure.\"\n",
    "doc4 = \"Sometimes I feel pressure to perform well at school, but my father never seems to drive my sister to do better.\"\n",
    "doc5 = \"Health experts say that Sugar is not good for your lifestyle.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " # compile documents\n",
    "doc_complete = [doc1, doc2, doc3, doc4, doc5]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(doc_complete)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "nltk.download('wordnet')\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "import string\n",
    "stop = set(stopwords.words('english'))\n",
    "exclude = set(string.punctuation)\n",
    "lemma = WordNetLemmatizer()\n",
    "def clean(doc):\n",
    "    stop_free = \" \".join([i for i in doc.lower().split() if i not in stop])\n",
    "    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)\n",
    "    normalized = \" \".join(lemma.lemmatize(word) for word in punc_free.split())\n",
    "    return normalized\n",
    "\n",
    "doc_clean = [clean(doc).split() for doc in doc_complete]  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(doc_clean)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing Document-Term Matrix\n",
    "\n",
    "All the text documents combined is known as the corpus. To run any mathematical model on text corpus, it is a good practice to convert it into a matrix representation. LDA model looks for repeating term patterns in the entire DT matrix. Python provides many great libraries for text mining practices, “gensim” is one such clean and beautiful library to handle text data. It is scalable, robust and efficient. Following code shows how to convert a corpus into a document-term matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing Gensim\n",
    "import gensim\n",
    "from gensim import corpora\n",
    "\n",
    "# Creating the term dictionary of our courpus, where every unique term is assigned an index. \n",
    "dictionary = corpora.Dictionary(doc_clean)\n",
    "\n",
    "# Converting list of documents (corpus) into Document Term Matrix using dictionary prepared above.\n",
    "doc_term_matrix = [dictionary.doc2bow(doc) for doc in doc_clean]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "print(doc_term_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running LDA Model\n",
    "\n",
    "Next step is to create an object for LDA model and train it on Document-Term matrix. The training also requires few parameters as input which are explained in the above section. The gensim module allows both LDA model estimation from a training corpus and inference of topic distribution on new, unseen documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the object for LDA model using gensim library\n",
    "Lda = gensim.models.ldamodel.LdaModel\n",
    "\n",
    "# Running and Training LDA model on the document term matrix.\n",
    "ldamodel = Lda(doc_term_matrix, num_topics=2, id2word = dictionary, passes=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ldamodel.print_topics(num_topics=2, num_words=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each line is a topic with individual topic terms and weights. Topic1 can be termed as Bad Health, and Topic3 can be termed as Family."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}