W142 Jupyter Notebook Tutorial -Revised 060719.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [W142] Jupyter Notebook Tutorial - Out of School Children Rate in Uganda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated Time: 90 Minutes\n",
    "\n",
    "Created by: Yidan Zhang\n",
    "\n",
    "\n",
    "Today we will be examining a dataset from The World Inequality Database on Education (WIDE) to explore the powerful influence of circumstances (such as wealth, gender, ethnicity, and location) have on children's opportunities for education and life.\n",
    "\n",
    "Please use this international dataset to begin to identify, through analysis and data visualizations,  the most marginalized children with regards to education in the country your group has chosen.\n",
    "\n",
    "This Notebook serves as an example and tutorial of some data visualization techniques, and you will be completing a similar Notebook for this week's assignment to tell us about gender inequality in the country you have selected. You may want to be in touch with your group members to each run different analyses that look at different markers of marginalization.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jupyter Notebook Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This webpage is a Jupyter notebook. We have created this specific notebook so that you can run code in order to perform data analysis and answer questions. Cells are the boxes that you can populate with code or text.\n",
    "\n",
    "To insert a cell, click the + button in the toolbar at the top of this webpage, or click insert.\n",
    "\n",
    "To select a cell type, click on the Cell in the tool bar, then cell type. Cell type \"code\" is for any cell containing codes, and the cell type \"markdown\" is for any cell containing texts, like this one. \n",
    "\n",
    "To edit a cell, double click it. \n",
    "\n",
    "To run a cell, click the cell and press Shift + Enter while the cell is selected or hit the ▶| Run button in the toolbar at the top. \n",
    "\n",
    "To download a Notebook as a pdf file, click on File in the toolbar, then Download as, then select pdf(.pdf). This will download the notebook to your computer in the pdf format. You can sumbit the pdf version of the assignment notebook.\n",
    "\n",
    "Python is the programming language that we will use in this lab. Although this lab will go over some basics, should you be more interested in learning Python feel free to check out the following resources:\n",
    "\n",
    "1. [Python Tutorial](https://docs.python.org/3.5/tutorial/): Introduction to Python from the creators of Python\n",
    "2. [Composing Programs](http://composingprograms.com/pages/11-getting-started.html): An introduction to programming with Python from CS 61A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this Notebook, we will go over simple techniques in Python and Matplotlib that you can use to perform data cleaning and generate graphs!\n",
    "\n",
    "\n",
    "First on our agenda is to import **<font color = ##008700>dependencies</font>** -- or packages in Python that add to the basic functions in Python. Kind of like accessorizing! For example, `matplotlib` allows us to generate graphs.\n",
    "\n",
    "The format is as follows: <code><font color = ##008700><b>from</b></font> (package) <font color = ##008700><b>import</b></font> (stuff)</code>, where the \"stuff\" we're importing can range from a specific function in that package to a whole library of functions, as is the case when we type <code><font color = ##008700><b>import</b></font> (package) <font color = ##008700><b>as</b></font> (name)</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Just run this cell by click the cell and press Shift + Enter while the cell is selected \n",
    "#or hit the ▶| Run button in the toolbar at the top\n",
    "\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import numpy as np\n",
    "from datascience import *\n",
    "import matplotlib.pyplot as plots\n",
    "import math as m\n",
    "import qgrid as q\n",
    "import pandas as pd\n",
    "import sys\n",
    "sys.path.insert(0, \"development\")\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from IPython.display import HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = #254117>Section 1: Data Cleaning</font>\n",
    "In this section, we will cover some essential functions that are useful for data cleaning. In order to begin filtering through information stored in a dataset, we will have to \"read in\" the information. Most of the time, information to be displayed as a table is stored as a `.csv` file which stands for **<font color = ##008700>comma separated values</font>**.\n",
    "\n",
    "The WIDE dataset we will be using is already downloaded to your Jupyter Folder\n",
    "\n",
    "To read in a file thats in your folder, we use the following command:\n",
    "\n",
    "`pd.read_csv('file_name.csv')`\n",
    "\n",
    "and in order to store it, we'll assign it a name or label. We'll begin by reading in the file that you'll be using for the remainder of the tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just run this code block! This code reads in our WIDE dataset and label it as \"wide\". \n",
    "\n",
    "wide = pd.read_csv('WIDE.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color = #254117>Table Manipulations</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lets look at the first 3 rows of our dataset by running this cell\n",
    "wide.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`yourdata.loc[]` is a function that helps us select certain rows, columns, or a combination of rows and columns from a larger dataset. The following are some examples of how it can be used. Make sure to substitute `yourdata` in the function with the name of your dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting the same disply as wide.head(3) with a different method by running this cell\n",
    "wide.loc[[0,1,2],:] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since we want a range of rows, we could also use the following:\n",
    "wide.loc[0:2,:] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now lets try to select columns by column name by running this cell\n",
    "wide.loc[:, \"country\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## We can print the list of countries included in this dataset by asking for unique values or the country column\n",
    "## Run this cell\n",
    "countrynames = wide[\"country\"].unique()\n",
    "countrynames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## print all column names: you should refer to the data note from the WIDE website to see what each column denotes\n",
    "## Run this cell\n",
    "wide.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now lets select our country of interest, and only display variables we might be interested in by running this cell\n",
    "\n",
    "# Here the markers of marginalization (or outcome variable) we included are OOSC at primary, lower secondary\n",
    "# and upper secondary level\n",
    "# The characteristics (or independent variable) we included are year, sex, wealth and location\n",
    "uganda = wide.loc[wide.country =='Uganda',['year','Sex','Wealth','Location','eduout_prim_m','eduout_lowsec_m',\n",
    "       'eduout_upsec_m']]\n",
    "uganda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might have noticed the \"NaN\" entries in the table. NaN stands for \"Not a Number,\" however, conceptually, NaN means missing value. Now that we have selected our outcome indicator and dependent variables, and narrowed down on the country, we need to handle the missing data before we do any other analysis or visualization. \n",
    "\n",
    "There are several different ways to handle missing data, but we will only demonstrate how to delete these values here. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get number of missing values in each column by running this cell\n",
    "uganda.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that many of the columns we selected contains missing values. We have to delete these before we move on to graphs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#First lets check the shape of our dataset\n",
    "uganda.shape "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means we currently have 1474 rows and 8 columns in the dataset for Uganda. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#drop all rows that contains a missing value by running this cell\n",
    "uganda.dropna(how ='any').shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that after deleting the missing values, we have only 41 rows left, comfirming that we deleted many missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Since we are confindent that missing values are deleted, \n",
    "## we assign this new dataframe a name and display it by running this cell\n",
    "uganda = uganda.dropna(how ='any')\n",
    "uganda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more detailed tutorial on data selection and how to handle missing values, check out the following videos on YouTube:\n",
    "\n",
    "*[Data Selection](https://www.youtube.com/watch?v=xvpNA7bC8cs)\n",
    "\n",
    "*[Missing Values](https://www.youtube.com/watch?v=fCMrO_VzeL8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center><font color = #254117>Congratulations! You've successfully cleaned the data you want to visualize!</font></center>\n",
    "\n",
    "<center>We will now move onto the visualization. Make sure you understand the data cleaning process as we will be using the cleaned data in the parts that follow.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = #254117>Section 2: Summary Statistics and Visualization </font>\n",
    "\n",
    "First lets explore some ways to look at the summary statistics of the entire population by different survey year, using the groupby function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets look at the overall mean out of school children rate at each level for Uganda, by survey year \n",
    "# Run this cell\n",
    "uganda.groupby('year').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can also filter the data by year first, and then compute the mean by running this cell. \n",
    "# This result should be the same as the lower right cell in the table above.\n",
    "uganda[uganda.year==2016].eduout_upsec_m.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot the information in the above table in a line graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Running this cell allows the plot to appear in the Notebook\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Run this cell\n",
    "uganda.groupby('year').mean().plot(kind='line')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = #254117>Disaggregate Data</font>\n",
    "Now we have seen the trend of OOSC in Uganda over the past three survey years, let's narrow down to the most recent year and explore the difference in OOSC for different genders, location and wealth level to find out the most marginalized children with regards to this indicator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, select only Uganda data from 2016 by running this cell\n",
    "uganda16 = uganda.loc[uganda.year == 2016,['Sex','Wealth','Location','eduout_prim_m','eduout_lowsec_m',\n",
    "       'eduout_upsec_m']]\n",
    "uganda16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color = #254117>Urban vs. Rural Locations</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## By running this cell,  you produce a graph of the OOSC at each level between rural and urban locations\n",
    "uganda16.groupby('Location').mean().plot(kind='bar').legend(bbox_to_anchor=(1, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we want to see the difference of OOSC by location and gender? We could create subplots and compare the genders side by side. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## plotting two subplots ax1 and ax2\n",
    "fig, (ax1,ax2) = plots.subplots(nrows = 1, ncols = 2,sharex=True, sharey=True)\n",
    "#use ax1 to plot girls\n",
    "uganda16[uganda16['Sex']=='Female'].groupby('Location').mean().plot(kind = 'bar', ax = ax1, legend = False)\n",
    "ax1.set_title(\"Female\")\n",
    "#use ax2 to plot boys\n",
    "uganda16[uganda16['Sex']=='Male'].groupby('Location').mean().plot(kind = 'bar', ax = ax2).legend(bbox_to_anchor=(1, 0.5))\n",
    "ax2.set_title(\"Male\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if you want to reorganize the bars so you have a subplot for rural and a subplot for urban?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## plotting two subplots ax1 and ax2\n",
    "fig, (ax1,ax2) = plots.subplots(nrows = 1, ncols = 2,sharex=True, sharey=True)\n",
    "#use ax1 to plot rural\n",
    "uganda16[uganda16['Location']=='Rural'].groupby('Sex').mean().plot(kind = 'bar', ax = ax1, legend = False)\n",
    "ax1.set_title(\"Rural\")\n",
    "#use ax2 to plot urban\n",
    "uganda16[uganda16['Location']=='Urban'].groupby('Sex').mean().plot(kind = 'bar', ax = ax2).legend(bbox_to_anchor=(1, 0.5))\n",
    "ax2.set_title(\"Urban\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color = #254117>Wealth</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## By running this cell,  you produce a graph of the OOSC at each level for the 5 wealth quitiles\n",
    "uganda16.groupby('Wealth').mean().plot(kind='bar').legend(bbox_to_anchor=(1, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, lets plot the average OOSC by wealth and gender, using the same method.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## plotting  two subplots ax1 and ax2\n",
    "fig, (ax1,ax2) = plots.subplots(nrows = 1, ncols = 2,sharex = True, sharey = True)\n",
    "#use ax1 to plot girls\n",
    "uganda16[uganda16['Sex']=='Female'].groupby('Wealth').mean().plot(kind = 'bar', ax = ax1, legend = False)\n",
    "ax1.set_title(\"Female\")\n",
    "#use ax2 to plot boys\n",
    "uganda16[uganda16['Sex']=='Male'].groupby('Wealth').mean().plot(kind = 'bar', ax = ax2).legend(bbox_to_anchor=(1.2, 0.5))\n",
    "ax2.set_title(\"Male\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets reorganizing the above OOSC plot by gender and wealth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## plotting  two subplots ax1 ... ax5\n",
    "fig, (ax1,ax2,ax3,ax4,ax5) = plots.subplots(nrows = 1, ncols = 5 ,sharex = True, sharey = True)\n",
    "#use each ax to plot a quintile\n",
    "uganda16[uganda16['Wealth']=='Quintile 1'].groupby('Sex').mean().plot(kind = 'bar', ax = ax1, legend = False)\n",
    "ax1.set_title(\"Quintile 1\")\n",
    "uganda16[uganda16['Wealth']=='Quintile 2'].groupby('Sex').mean().plot(kind = 'bar', ax = ax2, legend = False)\n",
    "ax2.set_title(\"Quintile 2\")\n",
    "uganda16[uganda16['Wealth']=='Quintile 3'].groupby('Sex').mean().plot(kind = 'bar', ax = ax3, legend = False)\n",
    "ax3.set_title(\"Quintile 3\")\n",
    "uganda16[uganda16['Wealth']=='Quintile 4'].groupby('Sex').mean().plot(kind = 'bar', ax = ax4, legend = False)\n",
    "ax4.set_title(\"Quintile 4\")\n",
    "uganda16[uganda16['Wealth']=='Quintile 5'].groupby('Sex').mean().plot(kind = 'bar', ax = ax5).legend(bbox_to_anchor=(1.2, 0.5))\n",
    "ax5.set_title(\"Quintile 5\")\n",
    "#remove overlaps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the above visualizations, we can see that the out of school children rate does not look so different between rural and urban locations when gender is not taking into consideration.\n",
    "\n",
    "When looking at the difference in gender and locations, we can see that within each gender, the OOSC for rural and urban still looks similar, especially at the lower and upper secondary levels. For female students, the OOSC rate at primary level is lower than male students, regardless of location. However, the OOSC rate at lower and upper secondary levels are higher for female, regardless of location. \n",
    "\n",
    "When looking at the difference in gender and wealth, we can see that the OOSC rate is lower for males at the upper secondary level, regardless of wealth. However, the OOSC rate is lower for girls for the primary level, for all wealth quintiles except for quintile 1. When we compare the OOSC rate for the two extremes in terms of wealth, we can see that there is a sizeable OOSC rate gap between girls who are in Quintile 1 and Quintile 5, at the primary and upper secondary level. The most substantial gender gap of OOSC rate occurs for students in Quintile 1 and Quintile 2, at the upper secondary level. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color = #254117>Good Job! You have successfully completed this tutorial, now is time to try these methods on your own and complete the Notebook assignment </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## <font color = #254117>Feedback</font>\n",
    "Thank you so much for completing the Notebook! The development team would appreciate your feedback to get a sense of how the notebook experience was! All feedback is confidential and will be used to guide future developers in creating a Notebook. We hope you learned a lot from this experience feel free to reach out to your GSI at (yidan.zhang@berkeley.edu) if you have any questions about the development or use of this Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}