Lab1_Analyzing_Bokmässan_Tweets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1 Analyzing Bokmässan Tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparatory task for the laboratory\n",
    " \n",
    "Before you get started with the lab, you need to familiarize yourself with the Jupyter environment.  Take a look at the `Lab 0` exercises 1-4:\n",
    "\n",
    "- [Lab 0 / Exercise 1: Notebook_Basics](Lab0_Ex1_Notebook_Basics.ipynb)\n",
    "- [Lab 0 / Exercise 2: Running Code](Lab0_Ex2_Running_Code.ipynb)\n",
    "- [Lab 0 / Exercise 3: Working with Markdown Cells](Lab0_Ex3_Working_with_Markdown_Cells.ipynb)\n",
    "- [Lab 0 / Exercise 4: Notebook Exercises](Lab0_Ex4_Notebook_Exercises.ipynb)\n",
    "\n",
    "To learn more detailed Python programming you can also study chapters 1-9 from the free online book [Python for Everybody](https://www.py4e.com/html3/), however it is not essential to completing the labs.\n",
    "\n",
    "Once you are familiar with Jupyter, we can now try out a simple analysis of Twitter data with a focus on data cleansing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, we need to load the relevant Python modules that we will use in the code later by running the next code cell. In this cell, we use the `import` command to load five modules:\n",
    "\n",
    "1. `pandas` - A collection of utilities to load and manipulate data tables.\n",
    "1. `textmining` - Functions for statistical text mining, focused on the bag-of-words model.\n",
    "1. `wordcloud` - A visualization module that generates wordclouds.\n",
    "1. `matplotlib` -  A 2D plotting library to create figures such as charts and plots.\n",
    "1. `sklearn` -  Scikit-learn, a library of machine learning algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================================================================\n",
      "Assignment: Lab 1 Analyzing Bokmässan Tweets\n",
      "OK, version v1.14.19\n",
      "=====================================================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to import the modules and set up some stuff\n",
    "import pandas as pd\n",
    "import textmining as tm\n",
    "import wordcloud\n",
    "import matplotlib\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "# the following two lines set up our visualization settings\n",
    "%matplotlib inline\n",
    "matplotlib.pyplot.rcParams['figure.figsize'] = [10, 6]\n",
    "# finally, the next couple of lines set up our auto-grader\n",
    "from client.api.notebook import Notebook\n",
    "ok = Notebook('lab1.ok')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import textmining as tm\n",
    "import wordcloud\n",
    "import matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "## Analysis of Twitter data from the book fair\n",
    "\n",
    "You have been hired as consultants by a book publisher who wants you to find out which themes and books have generated attention on Twitter during the 2016 book fair in Gothenburg.\n",
    "\n",
    "Your task is to find out if there is any topic that has been particularly hot on Twitter before the book fair and during the book fair and to present a proposal to the company on what themes seem to create debate. In this lab we focus on data preparation. In order to prepare data, it is important to understand data\n",
    "\n",
    "**Q1.1.** What do you think is distinctive Twitter data and how will this effect how we might want to pre-process the data?\n",
    "\n",
    "Provide your answer by editing the cell below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*\n",
    "\n",
    "*Hint: Double-click on this cell to edit it, then to exit edit-mode run the cell using the `Run` button in the toolbar above or press shift-return*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "## Data processing\n",
    "\n",
    "Often, the data to be analyzed must be cleansed before we can use it. Data cleansing can include tasks such as dealing with missing values or, as in our case, filtering out some parts of the raw text data. Data you have been provided with was collected from Twitter during the period May 2016 to September 2016 during the \"Book Fair\" event.\n",
    "\n",
    "The data can be found in the `lab1-data.tsv` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Loading data\n",
    "\n",
    "Start by loading data from a `.tsv` file. We can load this data from the Cloud using a URL that points to the file. A TSV (tab-separated variable) file where each line in the file corrsponds to a row of a table, and each cell in every row is delimited with a tab character.\n",
    "\n",
    "You can load data into a `DataFrame` object using `pandas` with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "bok_tweets = pd.read_csv(\n",
    "    \"https://s3.eu-west-2.amazonaws.com/uu-datamining-assets/lab1-data.tsv\", \n",
    "    encoding=\"utf-8\", sep=\"\\t\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "In Jupyter you can inspect variable `bok_tweets` by executing a cell with the variable name, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bok_tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "To inspect the column names, use the `columns` attribute.\n",
    "\n",
    "**Q1.2** Replace the ellipsis `...` with `bok_tweets.columns` to assign the column list to the variable named `columns` and run the cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = ...\n",
    "columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Checking your answers\n",
    "\n",
    "When providing some code answers, you can start using the built-in tests to check whether your work is correct.\n",
    "\n",
    "Try not to change the contents of the test cells. Running the following cell will test whether you have assigned `columns` correctly in Question 1.2. If you haven't, this test will tell you the correct answer or give you feedback. Resist the urge to just copy it, and instead try to adjust your expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test cell; do not change these! Just run them to auto-check your answer in the immediately preceding section!\n",
    "_ = ok.grade('q12')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the first few rows of a dataset, use the `head()`function with `3` as a parameter to tell the function to only load the first 3 records from `bok_tweets`. You can of course replace this with other numbers (try it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bok_tweets.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "To get some summary statistics on the dataset, use the `describe()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bok_tweets.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the shape of the dataset (length and width), use the `shape` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bok_tweets.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.3.** How many rows and columns are in the dataset? Provide your answer by replacing the ellipses `...` in the next cell with your answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_rows = ...\n",
    "number_of_columns = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q13')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get general information about the data set, such as how many values are not empty, use the `info()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bok_tweets.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.4.** How many tweets the dataset are mentions to another user (i.e. an `@`, or a mention, is when you include somebody's `@twittername` in the tweet? \n",
    "\n",
    "*Hint: The count of non-null objects in `info()` imply that of values present in a particular column.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_mentions = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q14')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.5.** Inspect the columns and contents of data. What part of data may be of interest for your analysis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "A collection of documents containing text is usually called a corpus. We can create a corpus by extracting just the `text` column of `bok_tweets`. Pandas let's us do this by simply indexing the column using the dot accessor to the column name.\n",
    "\n",
    "Run the next cell to extract the text from the data and create the corpus that you will work with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_corpus = bok_tweets.text\n",
    "tweets_corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Emojis\n",
    "\n",
    "On Twitter it is common to use emojis 👍 ✨ 🐫 🎉 🚀 🤘.\n",
    "\n",
    "When doing text analysis this can be useful because an emoji can contain a lot of information about what a person who wrote something means and what tone the text has. However, emojis may be problematic during analyses since coding of these is not necessarily compatible with the processing modules like NLTK.\n",
    "\n",
    "Unfortunately, emojis creates problems for the features we use in this lab 😭 and you will therefore need to filter out emojis from the raw data.\n",
    "\n",
    "Run the following cell that removes emojis from the `tweets` corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "encode2ascii = lambda x: x.encode('ascii', errors='ignore').decode('utf-8')\n",
    "clean_tweets = tweets_corpus.apply(encode2ascii)\n",
    "clean_tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.6.** How might removing emojis effect the quality of analysis? Explain your answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Remove URLs\n",
    "On Twitter, it is common to link to locations on the Web using URLs. It is often the case that commonly occuring parts of URLs will end up among the most frequent words. It is therefore important to filter them out.\n",
    "\n",
    "We can remove content that matches URL patterns with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_tweets = clean_tweets.str.replace(r'http\\S+', '')\n",
    "clean_tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.7** How might removing URLs effect the quality of analysis? Explain your answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Function for most frequent words\n",
    "\n",
    "We will look for the most frequent words several times during this lab after each pre-processing step in order to compare the affect of the pre-processing. We will do the same operations several times, so therefore we will create a couple of functions to help us with our analysis.\n",
    "\n",
    "#### What is a Term Document Matrix?\n",
    "\n",
    "First, we create a term-document matrix (TDM), which can also referred to as a document-term matrix (DTM). A TDM gives us a table of the number of instances of a word for each document in a corpus. You should start by creating a TDM that is a representation of each tweet in terms of a feature vector. The feature vector creates an element for each word (unless excluded in the pre-processing, see further below). Thus, each element in the feature vector represents a word contained in one of the tweets. In the TDM you create, each line corresponds to the text of a tweet where all words that are not filtered out of the tweet are saved in the corresponding elements in the feature vector.\n",
    "\n",
    "Our function `create_term_document_matrix()` to create a TDM is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_term_document_matrix(corpus, min_df=1):\n",
    "    cvec = CountVectorizer(min_df=min_df, stop_words=tm.stopwords)\n",
    "    tfmatrix = cvec.fit_transform(corpus)\n",
    "    return pd.DataFrame(data=tfmatrix.toarray(), columns=cvec.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can create a TDM for only the first three documents in our tweets corpus by using the `.head(3)` function on the `tweets_corpus`, similar to what we did at the beginning with `bok_tweets`.\n",
    "\n",
    "Replace the ellipsis with in the next cell with the documents we wish to pass to the `create_term_document_matrix()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_term_document_matrix( ... )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.8.** How many columns are created for our small TDM above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_tdm_columns = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q18')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the top words we will do a bit more work with our next function `plot_top_words()`. In this function we sum up each of the columns in the TDM, sort the word frequencies by counts, return the top sorted words list, and additionally plot these words in a nice bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_top_words(tweets, num_word_instances, top_words):\n",
    "    tdm_df = create_term_document_matrix(tweets, min_df=2)\n",
    "    word_frequencies = tdm_df[[x for x in tdm_df.columns if len(x) > 1]].sum()\n",
    "    sorted_words = word_frequencies.sort_values(ascending=False)\n",
    "    top_sorted_words = sorted_words[:num_word_instances]\n",
    "    top_sorted_words[:top_words].plot.bar()\n",
    "    return top_sorted_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After defining our own `plot_top_words()` function, we can use it by using the tweets corpus as input (be patient as it make take some time for the function to complete processing):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_words = plot_top_words(clean_tweets, 50, 30)\n",
    "top_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.9** How many times must a word occur in your corpus for the function to appear in the top words list output above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_occurences_to_make_top_50_words = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q19')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.10.** How many words does the function plot in the bar chart?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_words_plotted_in_bar_chart = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q110')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Lowercase\n",
    "\n",
    "The next step is to redo all the words in lowercase letters. You do this to avoid identiftyinhg the same words as different ones, when written in different cases. For example before transforming the whole corpus into lowercase letters, the word `Bokmaessan`and `bokmaessan` may be identified as different words.\n",
    "\n",
    "To change the `clean_tweets` corpus into lower case text, use the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_lowered = clean_tweets.str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell, write some code to plot the top words again with the lowered tweets corupus. Replace the ellipsis (`...`) with your own code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_words_lowered = ...\n",
    "top_words_lowered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.11.** What do you observe in the data after plotting the lowered tweets, and why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "In the next cell, you can use the code to compare your different lists with the most common words. The code creates a table using the `DataFrame` class, with the indexes of both top tweets corpuses as inputs.\n",
    "\n",
    "This code lets us preview the top 20 tweets, where the range given in the square brackets `[0:20]` defines which part of the top words lists are used. For example `[5:40]` will give you the 5th to 40th words in the list. You can try changing the range values if you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame({\n",
    "    'Top tweeted clean': top_words[0:20].index,\n",
    "    'Top tweeted lowered': top_words_lowered[0:20].index\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "You can use the following code to check the words in two top words lists they are identical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(top_words[0:20].index) - set(top_words_lowered[0:20].index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "If the lists of top words are identical, the cell will return only `set()`, otherwise it will list the words that are different."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.12.** Has the pre-processing of data you performed so far changed the list of the 20 most frequent words? Provide a reason for your observation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Small words\n",
    "\n",
    "Most small words are usually of limited importance, so let's strip those out. We can simply use a regular expression to identify words that are 3 letters long and keep them in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_low_no_small = tweets_lowered.str.findall('\\w{3,}').str.join(' ')  # short words (2 chars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_words_low_no_small = plot_top_words(tweets_low_no_small, 50, 30)\n",
    "top_words_low_no_small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.13.** Now after removing short words, how many times must a word occur in your corpus for the function to appear in the top words list output above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_occurences_to_make_top_50_words_short_words = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q113')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Stop Words\n",
    "\n",
    "Stop words are words of limited importance and are therefore not so interesting for your analysis. We use stop words as a reference so that we can filter out words that we do not want to analyze, for example prepositions and conjunctions.\n",
    "\n",
    "First, we can create a list of stopwords that we can use to remove from the most frequent word collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_stop_words = [\"och\", \"det\", \"att\", \"i\", \"en\", \"jag\", \"hon\", \n",
    "                \"som\", \"han\", \"paa\", \"den\", \"med\", \"var\", \"sig\", \n",
    "                \"foer\", \"saa\", \"till\", \"aer\", \"men\", \"ett\", \n",
    "                \"om\", \"hade\", \"de\", \"av\", \"icke\", \"mig\", \"du\", \n",
    "                \"henne\", \"daa\", \"sin\", \"nu\", \"har\", \"inte\", \n",
    "                \"hans\", \"honom\", \"skulle\", \"hennes\", \"daer\", \n",
    "                \"min\", \"man\", \"ej\", \"vid\", \"kunde\", \"naagot\", \n",
    "                \"fraan\", \"ut\", \"naer\", \"efter\", \"upp\", \"vi\", \n",
    "                \"dem\", \"vara\", \"vad\", \"oever\", \"aen\", \"dig\", \n",
    "                \"kan\", \"sina\", \"haer\", \"ha\", \"mot\", \"alla\", \n",
    "                \"under\", \"naagon\", \"eller\", \"allt\", \"mycket\", \n",
    "                \"sedan\", \"ju\", \"denna\", \"sjaelv\", \"detta\", \n",
    "                \"aat\", \"utan\", \"varit\", \"hur\", \"ingen\", \"mitt\", \n",
    "                \"ni\", \"bli\", \"blev\", \"oss\", \"din\", \"dessa\", \n",
    "                \"naagra\", \"deras\", \"blir\", \"mina\", \"samma\", \n",
    "                \"vilken\", \"er\", \"saadan\", \"vaar\", \"blivit\", \n",
    "                \"dess\", \"inom\", \"mellan\", \"saadant\", \"varfoer\", \n",
    "                \"varje\", \"vilka\", \"ditt\", \"vem\", \"vilket\", \n",
    "                \"sitta\", \"saadana\", \"vart\", \"dina\", \"vars\", \n",
    "                \"vaart\", \"vaara\", \"ert\", \"era\", \"vilka\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "Then we can define a function `remove_stopwods` that removes the stop words from a single document. We then use the `.apply()` function to apply the function across the whole of the `tweets_lowered_no_urls` corpus to delete these words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_stopwords = lambda x: ' '.join(y for y in x.split() if y not in my_stop_words)\n",
    "tweets_low_no_small_stopwords = tweets_low_no_small.apply(remove_stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_words_low_no_small_stopwords = plot_top_words(tweets_low_no_small_stopwords, 50, 30)\n",
    "top_words_low_no_small_stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1.14.** Now after removing stop words, how many times must a word occur in your corpus for the function to appear in the top words list output above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_occurences_to_make_top_50_words_short_stop_words = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q114')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.15.** What are the differences between the most frequent words compared to the previous pre-processed lists?\n",
    "\n",
    "*Hint: To help with your answer, read a little further to create a comparison table.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "Write some code to create a table showiung the top 20 words at each stage of pre-processing by comparing `top_words_lowered`, `top_words_low_no_small` and `top_words_low_no_small_stopwords` to help you answer this question.\n",
    "\n",
    "In the code cell below replace the ellipsis with your own code and run it to create your comparison table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Add your own stopwords\n",
    "\n",
    "Now you can choose to add your own stop words if you think there are words in the graph that are not so informative to determine what kind of topics discussed at the book fair. For example, you could remove `years` as represented in the text with` aar`. Write your own code in the cell below and run it to remove your own stop words:\n",
    "\n",
    "*Check the earlier example code that removes the initial list of stop words if you are not sure how to do this.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "more_stop_words = [ ... ]\n",
    "\n",
    "remove_more_stopwords = lambda x: ' '.join(y for y in x.split() if y not in more_stop_words)\n",
    "tweets_low_no_small_more_stopwords = tweets_low_no_small_stopwords.apply(remove_more_stopwords)\n",
    "top_words_no_small_more_stopwords = plot_top_words(tweets_low_no_small_more_stopwords, 50, 30)\n",
    "top_words_no_small_more_stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.16.** What stop words did you add and why? Did you notice any further problems?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = ok.grade('q116')  # if your extra stop words removed more, this test should pass!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Visualization of analysis and recommendation\n",
    "\n",
    "Now you will create a visualization that will help you convince the company why they should focus on this particular topic. A common way of visualizing commonly used words in a text is by using a word cloud.\n",
    "\n",
    "You create a word cloud using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "wordcloud = WordCloud(max_font_size=40)\n",
    "wordcloud.fit_words(top_words_no_small_more_stopwords.to_dict())\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "The code above creates a word cloud with words from the `top_words_no_small_more_stopwords` list. Run the next cell to generate a word cloud with the `top_words_low_no_small_stopwords` list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "wordcloud = WordCloud(max_font_size=40)\n",
    "wordcloud.fit_words(top_words_low_no_small_stopwords.to_dict())\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "### Compare your word clouds\n",
    "\n",
    "Create word clouds for at least two of your top words lists to compare how the pre-processing has affected the word clouds. You can also change the minimum frequency for a word to end up in the word cloud. If you think any words should be deleted, you can go back to an earlier step and add it to your stop word list and re-run the cells afterwards."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.17.** Are there any words that are not as informative that you removed to improve visualization? Explain why you removed any additional words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.18.** What theme would you recommend the book publisher to target next year? Explain your answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "**Q1.19.** Now that you have explored some Twitter data, what do you now think are the interesting characteristics of this kind of data? How does it affect how you must pre-process data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lang": "en"
   },
   "source": [
    "*Edit this cell to type your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "*When you are finished*, save your work in this notebook to upload to Studium.\n",
    "\n",
    "Choose **Save and Checkpoint** from the **File** menu, then choose **Download as HTML** from the **File** menu and save it to your computer (the filename saved should be `Lab1_Analyzing_Bokmässan_Tweets.html`).\n",
    "\n",
    "Then on the course page in *Studium* choose **Labb 1** and upload the HTML file and send the submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "nbTranslate": {
   "displayLangs": [],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "sv",
   "targetLang": "en",
   "useGoogleTranslate": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}