W6-1 normalizing text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normalizing text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general situation here is this: \n",
    "\n",
    "* We have extracted the text as a string - a sequence of characters\n",
    "* Now we want to convert that into a list of tokens. These will, generally speaking, correspond to words.\n",
    "* We have to think, though, about what sequences of characters will count as instances of a token."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm going to get one of the seasons transcripts for us to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "csvfile = open('corpora/angelaprenew.csv')\n",
    "areader = csv.DictReader(csvfile, delimiter=\"\\t\")\n",
    "data_list = []\n",
    "for row in areader:\n",
    "    data_list.append(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's work with the transcript for **angela**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_list[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt = data_list[1][\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Tokenize the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def my_tokenizer(txt):\n",
    "    words = re.findall(\"(\\S*)\\s\", txt)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_tokenizer(txt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some issues at this point:\n",
    "\n",
    "* Do we want \"That's\" and \"that's\" to be the same token? If so, we should convert to lower case.\n",
    "* Some words have punctuation at the end: \"near,\" \"sun.\" Those will be treated as distinct tokens if we don't split it off.\n",
    "* I see some words with \"[]\" and \"<>\" around them.\n",
    "* We have tokens thare are some symbols: '//'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's get rid of the punctuation and the brackets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_tokenizer(txt):\n",
    "    words = re.findall(\"\\[?(\\S*?)[\\s,.\\]]\", txt)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_tokenizer(txt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get rid of tokens that don't have any content, and also convert to lower case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = my_tokenizer(txt)\n",
    "new_words = []\n",
    "for w in words:\n",
    "    if len(re.findall(\"\\w\", w)) > 0:\n",
    "           new_words.append(w.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(new_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For fun, let's compare to ntlk's default tokenizer.\n",
    "Note that it makes some different decisions than us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "print(nltk.word_tokenize(txt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we are treating all forms of a word as distinct. For example Angela says\n",
    "\"orbit\" and \"orbits.\" And people sometimes say \"toward\" and sometimes \"towards.\" \"Move and \"moves\" are additional examples.\n",
    "\n",
    "If we were counting up occurrences, we would currently put these into separate piles.\n",
    "But sometimes we will want to reduce each word to its root.\n",
    "\n",
    "This is hard. nltk includes multiple stemmers that we can use. It does some weird things.\n",
    "Note that it even produces some words that aren't real words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "porter = nltk.PorterStemmer()\n",
    "stemmed_words = [porter.stem(w) for w in new_words]\n",
    "print(stemmed_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One immediate improvement we can make is to simply revert any stemmed words that aren't real words.\n",
    "\n",
    "nltk's wordnet interface provides us with something that we can use like a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import wordnet as wn\n",
    "wn.synsets(\"dog\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wn.synsets(\"asdlfkj\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmed_words = []\n",
    "for w in new_words:\n",
    "    w_temp = porter.stem(w)\n",
    "    if len(wn.synsets(w_temp)) == 0:\n",
    "        w_temp = w\n",
    "    stemmed_words.append(w_temp)\n",
    "print(stemmed_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The wordnet lemmatizer is supposed to do both steps for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wnl = nltk.WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wstemmed_words = [wnl.lemmatize(w) for w in new_words]\n",
    "print(wstemmed_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's quite a bit of variability in what the stemmers do. \n",
    "Let's add a third stemmer, and look at what they all do with a list of test words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lancaster = nltk.LancasterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_words = [\"orbits\", \"books\", \"men\", \"women\", \"lying\", \"orbiting\", \"running\", \"run\", \"jumped\", \"ran\", \"jumps\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []\n",
    "for w in test_words:\n",
    "    results.append(porter.stem(w))\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []\n",
    "for w in test_words:\n",
    "    results.append(lancaster.stem(w))\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []\n",
    "for w in test_words:\n",
    "    results.append(wnl.lemmatize(w))\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}