lecture-2-3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 2-3: Data analysis and visualization\n",
    "\n",
    "Despite the central place of data in scientific work, we often don't think much about how to properly organize or store it. Today we'll discuss the organizing principles of [tidy data](https://vita.had.co.nz/papers/tidy-data.pdf), an increasingly common approach to organizing data that makes it easy to understand and to analyze.\n",
    "\n",
    "For our examples below, we'll use the Python library `pandas`, which [stores data](https://pandas.pydata.org/pandas-docs/stable/getting_started/dsintro.html) in a structure called a `DataFrame`. As we'll see below, it's very easy to manipulate data in this format.\n",
    "\n",
    "### Example: Examining the `iris` dataset from `seaborn`\n",
    "\n",
    "As a first test, let's load a precompiled dataset called `iris` from the `seaborn` package, which we've used extensively for plotting. This dataset is stored in a \"tidy\" format. We'll start by just looking at the rough properties of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plot\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "df = sns.load_dataset(\"iris\")\n",
    "df.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Showing the relationships between variables\n",
    "\n",
    "As we can see from the column names, `iris` describes measurements from different species of iris flowers. We can look more closely to see how different variables are related. For example, what is the relationship between the length of the petals and sepals?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.jointplot('petal_length', 'sepal_length', data=df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting out subsets\n",
    "\n",
    "In the plot above, all species were included together. What happens if we look at just a single species -- is the relationship different? We can explore this by using `pandas` to selection out a subset of the data that corresponds to just a single species and repeating the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.unique(df.species))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_setosa = df[df.species == 'setosa']\n",
    "df_versicolor = df[df.species == 'versicolor']\n",
    "df_virginica = df[df.species == 'virginica']\n",
    "\n",
    "sns.jointplot('petal_length', 'sepal_length', data=df_setosa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing many variables at once\n",
    "\n",
    "We can also use `seaborn` to show how many variables in the data are related to one another. What other questions might you ask about this data? Because the data is stored in a simple format, it's easy to quickly analyze the data using `pandas`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(df, hue=\"species\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A quick test on another dataset\n",
    "\n",
    "The `planets` dataset contains information about recently discovered planets. We can use this to explore, for example, how the distance of the planet is related to the year in which it was discovered, or how methods of discovery have changed over time. See [here](https://jakevdp.github.io/PythonDataScienceHandbook/04.14-visualization-with-seaborn.html) for some more examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = sns.load_dataset(\"planets\")\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.jointplot('year', 'distance', data=df)\n",
    "plot.yscale('log')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.catplot('year', kind='count', aspect=4, data=df, hue='method')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Tidying the TB dataset\n",
    "\n",
    "Earlier, we saw an example of a dataset about rates of tuberculosis in different countries that was very hard to interpret and to work with. Here, we'll use `pandas` to load the data set and clean it for analysis. This exercise follows the description [here](http://www.jeannicholashould.com/tidy-data-in-python.html).\n",
    "\n",
    "FYI: `iso2` refers to the [two-digit country code](https://www.nationsonline.org/oneworld/country_code_list.htm) for different countries. This dataset was collected by the [WHO](https://www.who.int/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 252
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 757,
     "status": "ok",
     "timestamp": 1552455801507,
     "user": {
      "displayName": "John Barton",
      "photoUrl": "",
      "userId": "17550929711165105785"
     },
     "user_tz": 420
    },
    "id": "HsUzcJ2zg0RA",
    "outputId": "82fc5a46-2327-4673-841d-8b64089c2aec"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('https://raw.githubusercontent.com/hadley/tidy-data/master/data/tb.csv')\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A first step: 'melting' the dataset to extract the sex and age range\n",
    "# In this process all column names except for 'iso2' and 'year' are stored as 'sex_and_age'\n",
    "# All values are stored as 'cases'\n",
    "\n",
    "df = pd.melt(df, id_vars=[\"iso2\", \"year\"], value_name=\"cases\", var_name=\"sex_and_age\")\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parse the column names to extract sex, age upper and lower bounds\n",
    "tmp_df = df[\"sex_and_age\"].str.extract(\"(\\D)(\\d+)(\\d{2})\")\n",
    "\n",
    "# Add new columns for these variables\n",
    "tmp_df.columns = [\"sex\", \"age_lower\", \"age_upper\"]\n",
    "\n",
    "# Create a single `age` column using `age_lower` and `age_upper`\n",
    "tmp_df[\"age\"] = tmp_df[\"age_lower\"] + \"-\" + tmp_df[\"age_upper\"]\n",
    "\n",
    "# Merge the data frames together\n",
    "df = pd.concat([df, tmp_df], axis=1)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop unnecessary columns and rows\n",
    "df = df.drop(['sex_and_age', 'age_lower', 'age_upper'], axis=1)\n",
    "df = df.dropna()\n",
    "\n",
    "# Rename `iso2` to `country`\n",
    "df = df.rename(index=str, columns={'iso2': 'country'})\n",
    "\n",
    "# Sort the data frame\n",
    "df = df.sort_values([\"country\", \"year\", \"sex\", \"age\"], ascending=True)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examining the dataset\n",
    "\n",
    "Now, it is much easier for us to analyze the data. For example, we can check: how many cases of tuberculosis were there in the US among men? How do the number of cases vary over time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_US = df[(df.country == 'US') & (df.sex == 'm')]\n",
    "df_US.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GKRWMon9nNun"
   },
   "outputs": [],
   "source": [
    "sns.lineplot(x='year', y='cases', data=df_US, hue='age')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "21-temp.ipynb",
   "provenance": [
    {
     "file_id": "1FOPFBSIde067ut4LgM8M_1r3d1qXmJ9v",
     "timestamp": 1552455905846
    }
   ],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}