2_handling_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling data with python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Author:** Ties de Kok ([Personal Website](https://www.tiesdekok.com))  \n",
    "**Last updated:** June 2020  \n",
    "**Conda Environment:** `LearnPythonForResearch`  \n",
    "**Python version:** Python 3.7   \n",
    "**Pandas version:** Pandas 1.0.4  \n",
    "**License:** MIT License  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Some features (like the ToC) will only work if you run it locally, use Binder, or use nbviewer by clicking this link: \n",
    "https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/2_handling_data.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Introduction*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting your data ready for analysis (i.e. \"data wrangling\") is usually the most time-consuming part of a project. For data wrangling tasks I recommend `Pandas` and `Numpy`.\n",
    "\n",
    "What is `Pandas`?\n",
    "\n",
    "> Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use  data structures and data analysis tools for the Python programming language.\n",
    "\n",
    "In other words, Pandas introduces a data structure (i.e. `dataframe`) that represents data as a table with columns and rows. Combining Python with `Pandas` yields a very powerful toolkit with which you can process any type of dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format of this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Pandas` library is massive and it is continuously expanding in functionality.  \n",
    "It is, therefore, impossible to keep it both comprehensive and cover everything in just one notebook.\n",
    "\n",
    "The goal of this notebook is to cover the basic functionality that I expect you to encounter for an average project. \n",
    "\n",
    "I have based parts on this notebook on a PyCon 2015 tutorial/talk by Brandon Rhodes. If you want to know more I highly recommend watching his talk and checking the accompanying GitHub page   \n",
    "(**updated note:** some parts of the talk use outdated syntax, but it is still a great video)\n",
    "  \n",
    "https://www.youtube.com/watch?v=5JnMutdy6Fw  \n",
    "https://github.com/brandon-rhodes/pycon-pandas-tutorial\n",
    "\n",
    "Another great resource is the `Python Data Science Handbook` from Jake VanderPlas, which you can read for free on his website:  \n",
    "https://jakevdp.github.io/PythonDataScienceHandbook/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Table of Contents* <a id='toc'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Import pandas](#import-pandas)   \n",
    "* [Create a dataframe](#create-dataframe)   \n",
    "* [Manipulate dataframe](#manipulate-dataframe)   \n",
    "* [Rename columns](#rename-columns)   \n",
    "* [View a dataframe using qgrid](#qgrid)   \n",
    "* [View (parts) of a dataframe using Pandas](#view-dataframe)   \n",
    "* [Dealing with datatypes](#datatypes)   \n",
    "* [Handling missing values](#missing-values)   \n",
    "* [Work with data in the dataframe](#work-with-data)   \n",
    "* [Combining dataframes](#combining-dataframes)   \n",
    "* [Group-by operations](#groupby)   \n",
    "* [Reshaping and Pivot Tables](#reshaping-pivot)   \n",
    "* [Dealing with dates](#dates)   \n",
    "* [Dealing with strings](#strings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Import Pandas</span><a id='import-pandas'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* it is usually a good idea to also import `numpy` when you use `pandas`, their functionality is quite intertwined.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas tends to throw a frequent warning about chained assignment. I prefer to disable that warning.  \n",
    "You can read more about it here: [link](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.mode.chained_assignment = None  # default='warn'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Path to our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = os.path.join(os.getcwd(), 'example_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  <span style=\"text-decoration: underline;\">Create a dataframe</span><a id='create-dataframe'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a dataframe in many ways. Below are a couple of situations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Load file from drive into Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For details on opening files such as Excel, CSV, Stata, SAS, HDF see the `1_opening_files` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto = pd.read_csv(os.path.join(data_path, 'auto_df.csv'), sep=';', index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Create new dataframe and pass data to it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pass many different types of data to the `pd.DataFrame()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   col1  col2\n",
       "0     1     5\n",
       "1     2     6\n",
       "2     3     7\n",
       "3     4     8"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {'col1': [1,2,3,4], 'col2': [5,6,7,8]}\n",
    "df = pd.DataFrame(data=d)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   col1  col2  col3  col4\n",
       "0     1     2     3     4\n",
       "1     5     6     7     8"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = [(1, 2 ,3 ,4), (5, 6, 7, 8)]\n",
    "df = pd.DataFrame(data=d, columns=['col1', 'col2', 'col3', 'col4'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Create dataframe from a dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also directly convert a dictionary to a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  col3  col4\n",
       "row1     1     2     3     4\n",
       "row2     5     6     7     8"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = {'row1': [1,2,3,4], 'row2': [5,6,7,8]}\n",
    "df = pd.DataFrame.from_dict(d, orient='index', columns=['col1', 'col2', 'col3', 'col4'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Manipulate dataframe</span><a id='manipulate-dataframe'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "      <th>col5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  col3  col4  col5\n",
       "row1     1     2     3     4    10\n",
       "row2     5     6     7     8    10"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['col5'] = [10, 10]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "      <th>col5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  col3  col4  col5\n",
       "row1     1     2     3     4    10\n",
       "row2     5     6     7     8    10\n",
       "row3    11    12    13    14    15"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['row3'] = [11, 12, 13, 14, 15]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverse the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row1</th>\n",
       "      <th>row2</th>\n",
       "      <th>row3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>col1</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>col2</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>col3</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>col4</th>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>col5</th>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      row1  row2  row3\n",
       "col1     1     5    11\n",
       "col2     2     6    12\n",
       "col3     3     7    13\n",
       "col4     4     8    14\n",
       "col5    10    10    15"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  col3  col4\n",
       "row1     1     2     3     4\n",
       "row2     5     6     7     8\n",
       "row3    11    12    13    14"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop('col5', axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  col3  col4\n",
       "row2     5     6     7     8\n",
       "row3    11    12    13    14"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop('row1', axis=0)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  col3  col4\n",
       "row2     5     6     7     8\n",
       "row3    11    12    13    14"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>col1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col2  col3  col4\n",
       "col1                  \n",
       "5        6     7     8\n",
       "11      12    13    14"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('col1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Pandas` also allows a multi-index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           col2  col3  col4\n",
       "     col1                  \n",
       "row2 5        6     7     8\n",
       "row3 11      12    13    14"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('col1', append=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reset index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert the index to a regular column using `reset_index()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>row2</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>row3</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  index  col1  col2  col3  col4\n",
       "0  row2     5     6     7     8\n",
       "1  row3    11    12    13    14"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Rename columns</span><a id='rename-columns'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can either manipulate `df.columns` directly or use `df.rename()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      column1  column2  column3  column4\n",
       "row2        5        6        7        8\n",
       "row3       11       12       13       14"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns = ['column1', 'column2', 'column3', 'column4']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      col1  col2  column3  column4\n",
       "row2     5     6        7        8\n",
       "row3    11    12       13       14"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.rename(columns={'column1' : 'col1', 'column2' : 'col2'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** The above creates a copy, it does not modify it in place!  \n",
    "We need to use either the `inplace=True` argument or assign it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.rename(columns={'col1' : 'column1', 'col2' : 'column2'})\n",
    "#or\n",
    "df.rename(columns={'col1' : 'column1', 'col2' : 'column2'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">View (parts) of a dataframe using `Pandas`</span><a id='view-dataframe'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigating your way around a dataframe is an important skill. The ability to sub-select parts of a dataframe is important for inspection purposes, analysis, exporting, and much more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View entire dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* Pandas will only show the top and bottom parts if the dataframe is large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AMC Concord</td>\n",
       "      <td>4099</td>\n",
       "      <td>22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2930</td>\n",
       "      <td>186</td>\n",
       "      <td>40</td>\n",
       "      <td>121</td>\n",
       "      <td>3.58</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMC Pacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>17</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11</td>\n",
       "      <td>3350</td>\n",
       "      <td>173</td>\n",
       "      <td>40</td>\n",
       "      <td>258</td>\n",
       "      <td>2.53</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12</td>\n",
       "      <td>2640</td>\n",
       "      <td>168</td>\n",
       "      <td>35</td>\n",
       "      <td>121</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>4816</td>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>16</td>\n",
       "      <td>3250</td>\n",
       "      <td>196</td>\n",
       "      <td>40</td>\n",
       "      <td>196</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Buick Electra</td>\n",
       "      <td>7827</td>\n",
       "      <td>15</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20</td>\n",
       "      <td>4080</td>\n",
       "      <td>222</td>\n",
       "      <td>43</td>\n",
       "      <td>350</td>\n",
       "      <td>2.41</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>VW Dasher</td>\n",
       "      <td>7140</td>\n",
       "      <td>23</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>12</td>\n",
       "      <td>2160</td>\n",
       "      <td>172</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.74</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>VW Diesel</td>\n",
       "      <td>5397</td>\n",
       "      <td>41</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>2040</td>\n",
       "      <td>155</td>\n",
       "      <td>35</td>\n",
       "      <td>90</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>VW Rabbit</td>\n",
       "      <td>4697</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>1930</td>\n",
       "      <td>155</td>\n",
       "      <td>35</td>\n",
       "      <td>89</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>VW Scirocco</td>\n",
       "      <td>6850</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>1990</td>\n",
       "      <td>156</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>Volvo 260</td>\n",
       "      <td>11995</td>\n",
       "      <td>17</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>14</td>\n",
       "      <td>3170</td>\n",
       "      <td>193</td>\n",
       "      <td>37</td>\n",
       "      <td>163</td>\n",
       "      <td>2.98</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>74 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "0     AMC Concord   4099   22    3.0       2.5     11    2930     186    40   \n",
       "1       AMC Pacer   4749   17    3.0       3.0     11    3350     173    40   \n",
       "2      AMC Spirit   3799   22    NaN       3.0     12    2640     168    35   \n",
       "3   Buick Century   4816   20    3.0       4.5     16    3250     196    40   \n",
       "4   Buick Electra   7827   15    4.0       4.0     20    4080     222    43   \n",
       "..            ...    ...  ...    ...       ...    ...     ...     ...   ...   \n",
       "69      VW Dasher   7140   23    4.0       2.5     12    2160     172    36   \n",
       "70      VW Diesel   5397   41    5.0       3.0     15    2040     155    35   \n",
       "71      VW Rabbit   4697   25    4.0       3.0     15    1930     155    35   \n",
       "72    VW Scirocco   6850   25    4.0       2.0     16    1990     156    36   \n",
       "73      Volvo 260  11995   17    5.0       2.5     14    3170     193    37   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "0            121        3.58  Domestic  \n",
       "1            258        2.53  Domestic  \n",
       "2            121        3.08  Domestic  \n",
       "3            196        2.93  Domestic  \n",
       "4            350        2.41  Domestic  \n",
       "..           ...         ...       ...  \n",
       "69            97        3.74   Foreign  \n",
       "70            90        3.78   Foreign  \n",
       "71            89        3.78   Foreign  \n",
       "72            97        3.78   Foreign  \n",
       "73           163        2.98   Foreign  \n",
       "\n",
       "[74 rows x 12 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto"
   ]
  },
  {
   "attachments": {
    "3aff5b53-5087-4c8e-b18b-e6583f5c52cc.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAFFCAYAAAD8adQeAAAgAElEQVR4Ae2d+48dx5Xf+X+sBd37B+SnhRgkPxj7g40E8SJGLGyAWPZqBU3MuyE2AhZYaJE4Mmyv17sWSWBHF14E8guSN84MYUd31trVw5QiixxnLd6BvaSefIs0aY7El8hrUjTFE5yqPt2nqqv79r3Tt6cf3wF6+lVddepbj0+d6sfdQfiDAlAACkABKAAFGqnAjkZaDaOhABSAAlAACkABAsRRCaAAFIACUAAKNFQBQLyhBQezoQAUgAJQAAoA4qgDUAAKQAEoAAUaqgAg3tCCg9lQAApAASgABQBx1AEoAAWgABSAAg1VABBvaMHBbCgABaAAFIACgDjqABSAAlAACkCBhirQCYhvrg2o3+/bZfeINhtaWI03e2NI/eVx47OBDEABKAAF6qJA6yHuAFxAXhJIbNxDSmGJYdXv03AjXMxZ1/m2Dtbyhxvj5WhgEuUrK72wFdtwtEKIG228AZujr1cHHC29cxSVpx0IBsq7MinHNJQ63O+TXz+cPHh5r8zEjIQK23ZhRIM4j9updUZGKjxcVDMnHGvn198Kbc5Kyra9AY0uBEJEZe7X50DIxR1y6l2673Y0dvTNb5PkxLuY+txyiG/SaDeDLqo8saBliMmFN6ThcrrATae/e0CDYEcqha5tiOz0wo+XdRhdf6M4nMq0SaPlms8yVAVxKWetJ6cd71u9pdMwHYx3Lh4QmbiSzseEdXTX5bLY7c21oeoEbR3Qdg7VoM90OttkZ1qFMY1i21zt3bCcJ0/ruFzckK3fuzCiouXJZR3Xg9oJY+vpYG3klK2Yaerp7hGNltODUglTxdrpa0NtPq6Htv7Gem+MkjbpXUdUTX1uN8RTnXkEyxwvuXCFESDJWl9ojo1otDvpkOQ0Q8BW6ATQs4KBK74ASOL11ybO2KNx7bDXj6MBDg9yEltMPGx/fK06J3pG57QNNg+cZx2fbcCxBxvSyjd8y/tcxgMarWloE6U6OrbFNMwovPYQlJ22vPSMCOdJabJle+ePILcexPmbP/5FXZnWNEpJ6W6PuJ3gouxpRLyZ5Rmov7XMUH5Z5tblyvNjOWFBHdA3VU/FQC9sKly+BhLLrOuOQdx25nlT3UUFTKAQKBgpPFnHkUpYXgsIdIWJA+ZsSBw5QTjdOH4iO6Uj6YkGCdg5L/EUnHft5tooul3A6eoRv2u3HTQkcRK55+2oVKWTY/5WTsWA4Hw4o2dtm9bEa3icOA9Womvj+GKjAuHjc1Vu5NuRtrtK2/LSyrY7bTOH1XUuL952n0trI/m1GsWD7rjOy/m6rPP7rXpB3O+fA31HSGfVb7Dq6TJbTH0GxOep48YjdaGovVIznW6mMr2KG0N9SiXJtUlfGwoYrijJoMNCPGVvVCl1OB07V8gY9HIizo+tsM55B6LRBSq8RFHqWqept71pLZtmoqMziKFokCONlOPRAyKz7zXqUjNRLDLfZucq32bn5PbsmPpjZnCyteM8OfUyKot46nJ7TN/+VGcoz9x6sa058fpCz5ZQ2XtBKtq1/aeuh76mZl/6h8hZMYOo+Jg1NZQnPlZ2fe4YxGXUmt2RFKkpKaB5I7AE4no0xmlLuglA0h7rNAvyG4ONT9JJ4tKjwlTliu3XNibX8pa+Pj4TX5cBcf+eLHdG/rE4sq1uaE2J7HMJ8oxAIF/OQEzqhX1QcLiWeOKS99jTWQ7fJtmq9cWv53xmdwSmg1mYxsWtzA4ZaR2wMV3HbNiyO71s2+p3Zvby9NpBbbKU32+l+qTtsNv0Cem+U/po6QP8/iE21VyftM2q6nO7IW48sLIfbHM7fClYXsedjQOrqFGFjkWlz4U9C9zyK3y44+NrxL7U9TGMw9eymUEbVZ5S5/mcNzJNhYlr/9Y3TNzxfXwLY1s2PGMSyFfAPrGC49IjcTlu1rFWztGKdvI7aF3GFRk0XzIZGqbrR37HP1/izblqrvI0IElmCeuT2/yyTPVJVRueUSdDZuT1D/pcVfW55RCX+7+6U9/ifdmMzt8pMAU3rgRcQR3Im8GFbmhcwdN2OU9M6tpkGqo/9Zg8nW5s0VPAXsNONRhVgf1r43viUZoyEJDRqew7+Te22jzJ+aruiccy+eXk7AegLhd6Wslhu865zg24kL3cTt2rcwsxYN5I9RO80h7EE3fKxe3o03VqXgMaeF1eeWrNZniKfXtVcMvWtyXVJ/kBFrrP7TrkgQcS9fqH8ZrM9nHYUJ+XxLuo+tx6iBtp9fvU0nkEyqfIoczKZgo3KjC/AfI5xyvlwtYQ55QtIIKefdAwW2GS8EllMbHpD9x4aaXy4NlnQR4NfLTdJo/JgCgBdIan7oQf0tjXJZivkg7qji6KkvMtejmeNoeNvXi/XFyddZ5LsrRgNOn6YWyOyscpszgvbp0omFD5wZx64A1W/XLSZaHrXvlW1TrG3PLUmuVpW6sccjvKro+pPqlS2902HvcFwgpdJ72+1C+nVP+gr11Qfe4ExCutD0gMCkABKAAFoEBFCgDiFQmNZKAAFIACUAAKlK0AIF62oogPCkABKAAFoEBFCgDiFQmNZKAAFIACUAAKlK0AIF62oogPCkABKAAFoEBFCgDiFQmNZKAAFIACUAAKlK0AIF62oogPCkABKAAFoEBFCgDiFQmNZKAAFIACUAAKlK0AIF62oogPCkABKAAFoEBFCtQe4pPJhLBAA9QB1AHUAdQB1IF0HQDEMUjAIAl1AHUAdQB1oKF1ABBvaMFhRJoekUITaII6gDrQtToAiAPiGIGjDqAOoA6gDjS0DgDiDS24ro02kV94WKgDqAOoA+k6AIgD4hiBow6gDqAOoA40tA4A4g0tOIxI0yNSaAJNUAdQB7pWBwBxQBwjcNQB1AHUAdSBhtYBQLyhBde10SbyCw8LdQB1AHUgXQcAcUAcI3DUAdQB1AHUgYbWgc5A/NCeHvV6PertOYTK2tDKilF4ehQOTaAJ6kC360D7Ib6+z8KbAV5biB+ifb0eLe0/4w0wztDqrh71dq3SmSB4M647uUpLkl9eZ13va9NbotWT7WgQZtAW5zvSUWtitvfRIU/XM/uXqNfzjjs6eee866voUK2NblnZY1EdV/nct16P8owH0Xn1kbV06u72a11FeWal0SbNQnU2zndU5un+r8K669S7Hrntxvazhh+pfto9x2Gca514F1OfOwLxJVrdH8G8hp44V/ClPftoKYaOVF6GzxIt7fIqRgQO6bh15Q82lvV9bsUS8DCctB4GVoupaHGDlbQXuZaGk9JTdJ0Qd5BaN2ufNEilgYkrAabRV2u2yHyk4pZB2yrtmzbg4rLMyX+l5TE5RKvxANUOqNLac9lw/jyta5OHpO5Uo11bNMuvszLYXg22x+o0P7N/NRnQ+/3g+mri3Hj9ga2zqr9w2mw19bn9EBdRTcHUcTrdgnr1pKx1xY2O8QAkBQ6uIPtolQcA0kGaCpZVoXS80TZr4sSrbbCNLzz6nJAMIMx51dFq7yG2ayLQZPjkzAxIWW1prTRTdjkdL+u0a5VO37hBN9Ry8PEe7V1doaXeXjoYHT+9ukRLq6dVuIO0V53X11e3zTYs0coJ1/4k/dO0sivvfNZ11RxPaxqle2gv9R4/6Gld33wkei9et+Zrll9nue257WzxmmaXX14b99rWiRVa2rWS6ktM3BXVZ0B8S8AIgHHW+CKg8HR52ssTqMo6SU/C8lpgKcccYOXZkwdx7b17o0+TjgLkof12ut9Nn21OZhAs3GcYYOTZnXMu1oPzltG4DKwPeZ2ENDhulArS6c7Ta8RqEJDdKXhpbfma/A7xhuRly+mUbTfHl61fWOse7fXLqpb5WoRWEmcbNMuvs7WCeF778aFt+ovkNpYeiFRVnwHxHCAUhuEW4nCmdVOedALvNCDttGMMrWgQIEAvZLsHcZOGgnMSR2LHZKK3k0GFPxXK12rbnHxuQa/EJp12tK3BzQ0xBHEP0ga8+pjeZlhwPArqdn+7vcO8DpE7/PqBjzs0O6uTrV2oIw8OuDoC8XZplldnb1Co7KsbFPNgie2LYOzMBsnAMzoX6lPi+mjjkEFnKE+LqM+A+KKAUihe954JA4phlzwYoYFpp8/Ng1gKvhqUBsLO9HgAdNoujkcqbmqam9NORpi9+F6lskPHZe5n6vDRdmTPIiFuGzs3oGQa3MA20ODCo2MFFh/iN25Q0pn2qPf4Sg2mqnM6RLY/kO9qO0TxIENrO8hwp81tuHDZ1G9AUr2WbdAsp87eqAPEVV01A3fVJ8SQvkE3TP+QUyeVF19VfQbEHRBNgV7ZYX2IClBjb1hDXDzbQ+ZhN3mKXEPcPtk7w5S1Ggy4Hq4FePZgInn4KLkuC+5W07Ih7nekDmhFR7NWYA9N5ZoGGxh8ZN1zrgUkszvEdMehOifdGW3ndoaGpgwdLyg7n375t36/8Zrll2XIa93OMs3zmPPamK7DetvmJV+DefMLiJcN5sLx+aCUAYT2zl2Ix6/fxJAXsCevpjEsQ69IJUCWdCY0yYS4tkFe+0nAbdJQNsg9cXM8YyagLIgXrugMZ98jPbFCD39hhU5dv07Xs5bjK/Rwby+9Gjx/ila+0KO9B3OuD15XdvhXaW/vYVo57sfL9oWO++Eq3j+44tj66uM8o/GqLYODe6kXl4mbr1OrDyfhKtG1Yl3y8tQ6zdyy9dsf14mHV09lt8s8rbZ87hStrEb1keMyfUDSjl5d1X0G5yPpA06t7lV12z13/bqb50XVZ0C8MHQV/Mq4JnX/O4k/gaEH8dR0exri7BmbaXXtjWaANRviEeAljl37aN+uBOIy7R9Pxcfx24FJfDyegpen05PBRuLBJ/medqwwwG/coOsOHGznXKgRpSBuG6bkafsBznlxO4e4Q0zZXhMoGbvUbIcAnDtMv5x4X+pdDPea5GPLsJghH63TLKPORppuL8SvkxlYSr3zBsim34jPJQDndpd3zrTLCupzdyBeBngRh/cxmuIAngbovPMzwbukjvaDDz4gLNAAdQB1oO51ABAHmLcFzHnQ1ueKADz2RHMAXveGCPsAC9QB1IF56gAgDojXFuLTAJ4H71kaw7Vr1wgLNEAdQB1oYh0AxAHxWkI8D+BZ8M4CdxMbJmwGUFAHUAeK1AFAHBCvHcRnBXgI3nmV/+rVq4QFGqAOoA60oQ4A4oB4rSA+C8B9eIfAPa2RXrlyhbBAA9QB1IGm1gFAHBCvDcTnBbgP7xC48xro5cuXCQs0QB1AHWhiHQDEAfHaQ9y/B649cA1wDe8QtKc10EuXLhEWaIA6gDrQpDoAiAPitYB4lheeBfAseGtQ+w3x/fffp7zlvffeIyzQAHUAdaBJdaD2ECf8tV6Bu3fvUmj56KOPSC937twhXn7729/Gy+3bt+nDDz+kW7duxcvNmzdJlt/85jekF/0Out7OGkTgeA2/vb6d331H2ur33lE36tA/AOKtR2T9MzgPwAXeGuACbl4LuP1G5nv2vK+n57GND26gDqAONKkOAOL1Z1yrLQwBnI/leeBZAGdwC7QF1rox6il42eb76KH75ziGp5VRB1AHmlAHAPFWI7L+mQtBPARwmUb3Ac5eN0+LC7w1nOVBN2mI+n45b/v3zGfZz7u3jnP5zx5AH+iDOlBeHQDE68+51loYAniWF87w1gCXKXMG+IEDB7BAA9QB1IFO1gFAvLWIrH/GQhAPeeH8IFsWwHm6nCGOPygABaBAFxUAxLtY6jXJc1GIay9cPHCePmeA85Q5IF6TAoUZUAAKVK4AIF655EhQFPAhHvLCNcD5NTKGOAOcH1xjgPO9NUBcFMUaCkCBrikAiHetxGuSXx/gvD8N4vohNn6AjR9Eu3jxIiBekzKFGVAAClSvACBeveZIkSj34y76oy5yL1x74TyNzk+cb25u0vnz5wFx1CgoAAU6q0AHID6mYb9PfVl2j2izs8VdbsbHy33qL48zI80773viRbxwfg9cT6NfuHCBzp49C4hnlgBOQAEo0HYFWg9xAxIBuKxzwLN9BW4HG4M1f4ixSaPdfepnDj4yrrswooHkl9eZ1xO5Gg1odGEBKmwMYxt8gE+bShcvnF8nE4jzt43ZCz99+jQgvoDiQpRQAAo0Q4EOQHxIsa8oYMsB2nYV2+bagAbLQxqkbGOID2iwu0/DjbR1fB3PMmj422MejDeGwetT3vKFEY0C6aRTnvHIjBDXr5UxxOVrbHwvnD/UwlPp586doxMnTgDiMxYFgkMBKNAeBVoPcaeoagtxC+rRBVlrq6Nja8PA1DV74UMa8QBAPHiTRzVw0VEFthniocEBBzWDgeWx46nrsHytpGvDjuysQd+mL+dloCG3NJ4Yp3/wxJ9O10+l69fK5IG2X//612Yq/dixY4B4oFxxCApAgW4o0CGIMwztvXEBT22KmMEbeeACzsQ2Abus1Rn2wpfHBraSp/T1SfjgFnvIfc9rjwIKfGNwmwFCElYgzcFt2OQcH9PnqaAnLg+1CcT1VDq/WsavlfFT6Xw//MyZM/TOO+8A4sGCxUEoAAW6oEBHIK4ebqvh/XAHdilPOoG3C2j3uIa4bBeuwCZNHuC4EHbTs7FpW/X2tLB5EPe9cJlKZ5BriMtT6Xw//Fe/+pW5H/7WW28B4oULGgGhABRomwLth3gMqGTqt16FyAMMF54Mx9j7pQTWRHb63NzjZ882GpAwQAXcIZgWzm+klaQdikuntWiIy8+Myv1wQLxwSSIgFIACHVGg5RBnANopdAFT7crVTGerV+DkifL4ATcNcTttPVgbm4fd5ClyDVZKefIz5tgbHMhAQWLRA4ztgDg/1Cae+KlTp+jNN9+snyeuBo72OQB3kCZabnmdqjsFn4Xg65z6pQeNW7YKEUABKFChAi2HuJpGFzhOed2qQu35TrIZZKQHGNo7dyFuIe2+MuZAPLoX3Y8eLovzE3w6fZNGy+578wxmAbfxxPUsgYFGAortgrg8mV5HiKc04wK4MKKhPHgYF0jWhi77rDDyWmBSFiZksIwDcQDiAVFwCAo0U4F2QzzlEUUeb+yFbHOh5XjNCUw9iEeQ1uD3Ic65sjBRHn7WswC+N6fCmTiW5Ylzjsv1KGeCeDRgYc/Ufzo96544T6frJ9N5Ol2/Xnby5El644036uOJe4Oc+WrXdIibctlKHQbE5ysaXAUFaqhAuyFeQ8GbZJKFePyWfSmmF/nQizyZLhDnj7zIr5ZpiPM74nWCuB7UZImVDqOg7Q86g6DOmr3xU7Th5LU+mV0xoXIhzvYkgz951sKPHftQAArUQwFAvB7lUEsr6gZxeUecf/REPvRSH4gXg2suxE0tUFAP1opp5/kia0sCYM+2HIi79m3SeMP/gmDQKByEAlBgmxQAxLdJ+CYkWyeIy+dW5ZfL+Jvp9fLEPVBmFLALSQ7kQ9nf9yOadt7egx8EnomIvfEpEI/D+UljHwpAgdopAIjXrkjabVDedLr/oRc9nV5/iHsft8koxq1DvMBggSGtpsTjbZmez4E4m802mmskfEZecBgKQIHtVwAQ3/4y6JQFZUP89ddfr9eDbVPAt3WIJ5/Dzaw4fG89z44pEJd4t/wAnUSENRSAAgtTABBfmLSIOKRA2RA/evRofSAeP4HvvfqlXjHzwWi9Xv3Uf4HpcjMF775maLSOXzHz74l7JZEJce+VQyecFwd2oQAUqIUCgHgtiqE7RrQb4rYcDaid6WwNdQtYmeIebqShHV+f503raW9JywnP8SZPmXN68WuJDpzd6fl4Kt1cq+3uTh1FTqFAkxQAxJtUWi2wtQsQb0ExIQtQAAo0RAFAvCEF1RYzy4T48ePHqV7T6W0pJeQDCkCBpihQe4jzhz6wtEcD/miLv/CT57zwF9l44Z8b5eXKlSvmZ0f5W+n8bvj58+fN++GnT582P0F65MgROnz4cI3uiTel2cNOKAAF2qIAII5BQqWDJB/gvA+It6U7QT6gABSoWgFAHBAHxKtudUgPCkABKFCSAoA4IA6Il9SYEA0UgAJQoGoFAHFAHBCvutUhPSgABaBASQoA4oA4IF5SY0I0UAAKQIGqFQDEAXFAvOpWh/SgABSAAiUpAIgD4oB4SY0J0UABKAAFqlYAEAfEAfGqWx3SgwJQAAqUpAAgDogD4iU1JkQDBaAAFKhagdZD/NCeHvV6ybK0/0yl0Ap+be7kKi31lmj1ZP6X2Nj2Wthb4kAHH3upuokjPSgABdqsQOcgzkDft54PzyB4UyA7RPvU4KC8eBPbZoW4P2ApI5+z2jBNO0C8zd0J8gYFoEDVCrQe4hoqZ/YvGa+8HO+WIb6PDgncjXet9uX4FtbFARoNKPYcUrMMZ2h1zyqd2UL6rF1xG5LBh9bc3wbEq27iSA8KQIE2K9ApiIunWoaHOpl4EDf7yRQ5DxicwQJDflcE1RTwz9DqrvSUvwPQ9X1mABKy3QmXAW0ZwJhbCwr25vieQwbW9raD5MG1qRcNWDgtxwa2S+KL8rUaDZb2/T3fNnAHNqdXl6j3+EHnR1Dw7fQ2dzHIGxSAAotUoAMQ96a9BTgZsPM9x+x9D+IaZpMJFYe4hWUC/EO0Gt23j+Fs4Chw9T1etiPrXBSWbZMBhOddC9wFzGY/IyxrMR3i+j6+zZvEPZmcoZVdS7Rywv0ls9ZB/MKIBv0+9eNlQKMLi2zG0+MeL2t7+tRfHk+/iIg21wYq7JiG/e3PSyHDEQgKdESB7kF8GvAKw90bHHjxFoa49tC9tC3EV8299wSEIYi73q478PBBOqGJGnAYaOuBjTdLEA8kItumQ9y1xdHB5HWFTns/R9omiBvo+aC7MKLh2uY2dSkM3jS0x8tDKoJxQHybig3JQoGCCnQA4gp6DC9+GE15mi7wVFgPqOlwnic+cWHpwIvj0rDWoFRA9dNgYPrT334YO62f54lbu+xUeTJlLxqkIO7dJtgqxHW+jSarp52p9Fb9FOnGkPr9YnAs2D63GGyTRrv7NNjCAAIQ32IR4HIosGAFAPGpsM4Cuw9xdwp9JohnDCosQA+Z++XJdHvaHh+0LugZ4tmQXzjEzRQ8p2/t8KfS2wRxnrIuAkzrrSfT2+41MmUdedBmSj6Zwg6lYabKQ9PjZlq/yKBCp+XmIRfi3m2D4caCeytEDwWgQEqBlkPcn/K2nmgeEF0ApoGZnPch7nriZso6hnPkDcu+9sSN56vvIwfuiafCeHaZ+HQcfD55Ot2AWtL2Bi2zQtwNH+kr0/FOvpSN6/toaQ8vh1JeeHsgbr3eaSCzANdgtQBNrhOgqjDaw+ft3SNKJuc5fAJ5p4Wnwjpno51Q+kl82RD38zumMSAeEhjHoMBCFegcxMsBOAMqPUBw447Abd4lX6LV9byn09245P6342FHoI6fBPdgnLbH9b45Lj2lLmm4UJZ8qfvakm78lLm2dR8d0rcDsiAeacVptvcVMx9qoXYbDpMNSolDX6e3iSgP1HnnJOpAGLZHZgeybbN2SDiJDmsoAAWqVaDlEFfeYAp6OJfMKixaCwa/HRi0F+JEoalutzlneM0OSMNhOG7x1jVY9XE3LSIqMJ1u4oqfok+m+OXpdZ0WkW+bBbl5Cj80nZ8yCAegABQoWwFAHHBXH4hZDMzZ25dZijZDPNcrNi3X86KlNTPEYwj6oORA3nUxnDmsmnaX+OK1d118XG04aavj0WY+xJPwmfflkyDYggJQYAEKAOKA+OIgLlPx6n58qyEewTb1hLp6xcx6vhq8DOLEy7bervtKmLnGuQ8eef27k2nvzL6BId13H1bjsMkrZn76bkzZEB/TMB54+O+Tu3FgDwpAgcUpAIgD4ouDeEDbdkPcNlQLajU17XnL/nmZJrdXR574moWv/WCMhn7UGRg4Jw+g5XcRFtTJx2c8qHtPmffVg3LZEFdT6Twd7w0y8u3BWSgABcpSABAPgKa6e8WLmb6us/1dgPjWGmcE8WlfeGOIA5xbkxpXQ4EWKACIA+LwxGvVkItA3HrWrgdfq0zAGCgABSpSABAHxAHxihpbsWTyIW4eIAvc4y4WN0JBASjQNgUAcUAcEG9bq0Z+oAAU6IwCgDggDoh3prkjo1AACrRNgdpD/MzkLmFpjwanb3xE/nLq+h3i5eQHvzXL8asfklmu3KJ3Lv2G3nrvBr158QM6ev4K/fLc+zQ+/Wv6f++8Sz89coxePHyEDhw40LZ2ifxAASgABQopAIhjkFDpIMkHOO8D4oXaKgJBASgABVIKAOKAOCCeahY4AAWgABRohgKAOCAOiDejrcJKKAAFoEBKAUAcEAfEU80CB6AAFIACzVAAEAfEAfFmtFVYCQWgABRIKQCIA+KAeKpZ4AAUgAJQoBkKAOKAOCDejLYKK6EAFIACKQUAcUAcEE81CxyAAlAACjRDAUAcEAfEm9FWYSUUgAJQIKVApyD+7J4+3dPr01fXa/wFtJPP0Od6A/rWyTwbD9NXp4bJu/4unTHpPEHPVjyIwcdeUm0QB6AAFIACcyvQHYgbaFUFcYasTYsHDeUPHIpDfGP/wKQvdtwj8C8B4jwo+tz+izN58q2G+IURDfp96qtlsLY5d+Ms9cKUbQMaTfvNcmPAJo1290l+9nRzbUD95fGMptk4tC4z/RY6fjt9Rr0RvEsKdATiF+lbuxKoLt4TZ8gqL7cEYLrfj58N4kHQlmATIO51FQaUQ0oQZ3/3e0sgT8XppVlg14C370H7woiGhQYY5UFcBgJENs7CugDiBUoZQbqqQCcgbr3RAX11j/VKK4f4xIUuw8+xYf0JumfPYevRpuDK1/oDkHR89+hBg5oi57wXg7hOx5/OdwdBn9v/hjMokrSNznueic7ZQYw7E/AE/bjNP4ASAi4DaGbPVXVHoTjV6ambnH5fDyymXuEFWATEiWby6AFxr0ywCwUSBdoPcQZkz077ClAcgCrgud7ulHvKudcxEJUnriE9uUvFIW7BGtt78oPWKeEAACAASURBVBn6lrmfn0Dc5GnXM7SRYQ+fnw5xC2mdzudi++25JI7D9K1oCt33xK2+egBwkb61J7GNw9/z+Gvt/RWzAHAdWBmgJtPtjidqrk3OGa/VC68HA9a7lvDZkB4v98lJJ2n77paTlvba8yHO8cfT5LtHFL554MZBxDMUOg02xYaJ49IDnxTEvbA6Lj+s0VWnFUrblQJ7UKBJCrQc4pF3GUGuWogn3nN8HzoCbWGIe/BPBhkRxPfzAEUNFgIglzzH98QF+Nrj53TkuImDwR3BmMM555LBTRDiMqMQsOUMp9MpiNvpdJlGHi8r2Bpoyn4acuONqBvJGhgoYDoDBaf38eN1TiY7ji1E5IDQjcNJywlHtLkxzoV4DOi+P7CwaSSDDTfNkD16QGPOx7MNLqRlsBPHzXoq7RIRsAUFmqlAqyGeApialk48ywRKCSS3eowhq+HqerpFIc72h+2MBicFnrTPjMOHuNJGgG8888yBhJ1R0PYZvX2I8/U67tZDXHmm/eSBsHT3oGHjQ0yFTkHcA5wJynHJgEBdG3m3MojQZ/R22lvnNMR7ddNLQTyYro6dt904+Ijx4MXbTuUxGkjIeT1YCIX14ue4Jc/j5QGN1pJbGmx/DHTfTOxDgQYqAIiHPMYtH/Mhfpc0TGeBeHyv3LEp8sTXp7+OptN1Bik+xH34SnopLz0Z4Ez1xP1reb/1EA/BNOoZDIA05AWUfN6CznirAi8+nIKW9e61V2u3dVxJT5QGdHLObql01VP1HKcFoQtgB+IcgfHiOU/h9HUaAlZ7TA084ji0Nn2Kn2DXENfbNiLzX+czsXFMQ+N1S1qclzw7VYTYhAINUaDVEHegNbEgZa8wvvcroCp97UPc9cRdjzXyqgWiGq5mW9kbuCdupqjltbFAPgpB3Dx4p9Jx4rH2JR73lHvikg/RW+0z9DsL8SCMw0DJ91JnBFEG9HT/pD1XfdxuT4G4XGDyF86PDFAciGs9eDtvilvnQV8naUeefRy/hNkYxl63zaPAXF2ITSjQcAUAcQdYiZfpDwBm24/ArKaREwhyGvr8E/Qse6gCOw1xti0CuZ2SlofG+HrZzh+cmAGDsiMexOSm0/fukWt7Fexj29ST6JIPo6sdvMh0+lf3dNgT1yDiToP3Y+91TEPlfSeeZMgTj57szoOe0ylZCKeeUNevmBlbsmYQsiG+uTZU75ozIItD3BmoRLMQmdPcjnZRfpRe7j1xzjyHGdBAe90M9N3zvOPuiIkdKFA7BToF8dlAXBbQEY/Wvf0fe8mHYTwNvjxU0BPQRtPJHqAN8HiqW4ErPiZT4OpcqJcxAwMJa9aunanzsQ3ZEE+m0q3dsSecMsDLH6cfxy+BeRDgTqfH8TkQ5/B+fG5eTAj+KI1Ow3jncotA0sQaCjRfAUB8IZ44wK3BrbdbDfHm9wfIARSAAg1TABAHxGf6bKoG8jzbgHjDegiYCwWgQK0VAMQBcUC81k0UxkEBKAAFshUAxAFxQDy7feAMFIACUKDWCgDigDggXusmCuOgABSAAtkKAOKAOCCe3T5wBgpAAShQawUAcUAcEK91E4VxUAAKQIFsBQBxQBwQz24fOAMFoAAUqLUCtYf4ZDIhLO3R4MaNG+Qv169fJ14++OADs1y9epV4uXLlCl26dInee+89unjxIp0/f57OnTtHp0+fpnfeeYeOHDlChw8fpgMHDtS6kcE4KAAFoMCiFADEMUiodJDkA5z3AfFFNW/ECwWgQNsVAMQBcUC87a0c+YMCUKC1CgDigDgg3trmjYxBASjQdgUAcUAcEG97K0f+oAAUaK0CgDgg3gqIX758mbBAA9QB1IGu1QFAHBBvBcRbO8xGxqAAFIACOQoA4oA4IJ7TQHAKCkABKFBnBQBxQBwQr3MLhW1QAApAgRwFAHFAHBDPaSA4BQWgABSoswKAOCAOiNe5hcI2KAAFoECOAoA4IA6I5zQQnIICUAAK1FkBQBwQB8Tr3EJhGxSAAlAgRwFAHBAHxHMaCE5BASgABeqsACAOiAPidW6hsA0KQAEokKMAIA6IA+I5DQSnoAAUgAJ1VgAQB8QB8Tq3UNgGBaAAFMhRABAHxAHxnAaCU1AACkCBOivQPoiv76Ner0e9Xat0ZjKhM/uX7P6eQzSZHKJ9fK63RKsnJ5XCa9KpwcIZWt3Vo33rE5qcXKWl3j46xPk323vp4I0bdEMt169fJ14++OADs1y9epV4uXLlCl26dInee+89unjxIp0/f57OnTtHp0+fpnfeeYeOHDlChw8fpgMHDtSjjV0Y0aDfp763DDdmMW+TRrv7NNs1s8SfEXZj6Nk9pHFGUOcwX7d7RJvm4DbZ7hiEHSjQLQUA8U7BtaqBS5ch7sFPwL5cCIlENBsIN9cG1C8cd7hzGy/zwMOze2NYbCABiIdFxVEoUJEC7YM4oFzrGQbtgct2uzxxD4amIY9p2B/Q6EKRVl0txM0gIPaki9jnhQHEPUGwCwWqVQAQB/Qrhb6AW6/bD3Ei9nYHa3bS2TRxZ/paAz4A8Yyw1oNOpu+TKXgeNCTHnXSd/iWQlnNedmy4+DaB9vxzIV7UDkkHaygABWZVABAHxAHxWVtNVngzdR7yxImcaW8DZRUuD4S5Yb14jV0WnC7Q9SBBG19khsACPBkIeODPsd0duGzSeEMNYrQZ2IYCUGBuBQBxQBwQn7v5eBcWhLgLN46DwSigdSGZHzYAcQeq1j4eQCQQ1jYXgHgoT5yGeONOemnb43A6WWxDAShQmgKAOCAOiJfVnELAi+JOYGxBF09Nq2lv6z1rEE4Lm4a48fhVnHE6Al0nrzot50Syw5AOxSf30XMgzpFwvs31Ej6JGVtQAAqUoAAgDogD4iU0JBNFFsS94wy2ZLrbT9wFa37YNMRJe8l+1IF9Z5o/cJ7Y9jwAT4G4RGnSyYtHAmINBaDATAoA4oA4ID5Tk8kJ7MHahIw8WQfa5pi6J+5E6ULcQNl//UuFT0PYvyeuAgc3bfjkXe8oUPyKmbUnPB1PZOyL4axt36TRsrw/7ocLGoKDUAAKzKEAIA6IA+JzNJzgJQbiyVPhedPIqWnvIAhtKtlh+XwE4b7y7lN2yP32oNXmYDztLVPnsT1uGjK1Hg9KcjxxN86sQUu2TTgDBaDAdAUAcUAcEJ/eThACCkABKFBLBWoP8VqqBqPmVuDu3bvkLx999BHxcufOHbPcvn2bePnwww/p5s2bZpDB75Lzp1j5M6z8CdazZ8/S8ePH6ejRo/X57OrcquBCKAAFoMB8CgDi8+mGq+ZUwAc47wPic4qJy6AAFOi8AoB456tAtQIA4tXqjdSgABRotwKAeLvLt3a5A8RrVyQwCApAgQYrAIg3uPCaaDog3sRSg81QAArUVQFAvK4l01K7APGWFiyyBQWgwLYosCPUqdbp2LaogkQXpkCobuHBtoXJjYihABRouQI7pAOt67rl+ncue4B454ocGYYCUGCBCuz47W9/S3VeFph3RL0NCgDi2yA6koQCUKC1Cuzgj2nUeWmt8h3NGCDe0YJHtqEAFFiIAjv4S1h1XgrnOvqhidS3nfXPIfJ3oZ1vQheOHQFZgdAPfMyoDCA+o2AIDgWgABTIUWDH+++/T3VecmxPTnkAZ5DLDzS4P8IQ/ThF8LeVk+gWvuX8aIROjX/MYkCjF4fpwYYBqP9DFlH4CzoOu5360QwewGw134B4WmgcgQJQAApsowI7zp07R3VepmuT/IpT+ucS5ZzAz9+fHvtiQmTAl+FuQBs4Hw1UnDwyVDNmFhjiTth4RmILvyZVY4hfvnyZsEAD1AHUga7VgR3Hjh2jOi9TISpeeAhmBjp6Ct3+3rH21KfGv6AAPEPgQlb/FjMRn5fZBDbB7C8L5K1RxtvO8K5DEJd4knRlUGNnKJLjMnUuP6sZgV8gLpoHvHs986Hjs/kd0RPmlsYzdFH9EIq8GYEfQFlQZUO0UAAKtFaBHUeOHKE6L9OUl2njwdqIhgwIs2jo8DHfE3cBOS2NhZxnEOqBhwAySswFNMOW8yRrDuRC37cxC+Kk090YJgMFk77WSbZVzDIokoGDcw2Ra7Nrn4X7kA4reMv9cUBcaYxNKAAFoMAMCux47bXXqM7LtLwIxOWBtnhtQJN43vHxCPTay52WxmLOs20JKF0AiiccDUYYvCo/1nYN9LSFmRBn8OrBQ3yptscFcBzEQDuyyRzU4dieJD98WttgPfHN1M+QMsgB8VhhbEABKAAFZlIAn12dSS4E3qoC4n3rNSC+VVVxPRSAAl1VABDvaslvU741vGUbEN+mwkCyUAAKNF4BQLzxRdisDAi49RoQb1YZwlooAAXqowAgXp+y6IQlGt6yDYh3ouiRSSgABRagACC+AFERZbYCAm69BsSz9cIZKAAFoECeAoB4njo4V7oCGt6yDYiXLjMihAJQoCMKAOIdKei6ZFPArdeAeF1KB3ZAASjQNAUA8aaVWMPt1fCWbUC84YUK86EAFNg2BQDxbZO+mwkLuPUaEO9mXUCuoQAU2LoCgPjWNUQMMyig4S3brYK4/q48fx1QPlFbRCP9SVzzid05Pw/s22A+2VvAACd9/TW+AtciCBSAAq4C8pnq+HPg7hct3cChvWJtEBAPaYdjC1NAwK3XbYG4+XSu90nb8dqINouq6UDUvyj9WVs/BO/LN+rH+qT+Rr4+7m876RfrQPwosA8FoED0OxLeZ6jpwoiGa1FvkPqEdUi1Ym0QEA9ph2MLU0DDW7bbAnH5Pvzc4jkQ9WOZDvHQIMKPJXffSb9YB5IbH05CgS4qwO1o2uzXViHO10cOAyDexUq2jXkWcOt1WyCeD1GBol3LD/I4P8STBVF/Ws7z9m1xSvzTCtdN35nuz0p/WpQ4DwWgQKzA1MG8gbz84qbccova79qIBjz9btp4TpsGxGO9sVGxAhrest0WiLOUdjrb/614PiPwVPfFDJzVfi5Ep3ni084nNiS/8+51ErnpV1xRkBwUaKQCXpvKyoNp++lfhHR+njrv56YB8SxlcXzRCgi49bpNELf6BYCd0SCdUXsuRKdBetp57+dtpaA5TXn4Ljd9uQBrKAAFshXYGsSdmblUnyH9ivLi+33CdHp2aeDMAhTQ8Jbt9kE8Eo6hGN8bCzdu/ZvrlAvRaZAOx+8UobHH7QDMtL5Mz+em78SEHSgABTIUcAbmGWH4IbdB3DdwoFD7DR2LIoQnnqUsji9aAQG3XrcW4uY1MZkuDzdIp8HnQnQaxKMnYsWrDhWkavih0/mDiOAVOAgFoICvgNOO/ZPRPiCeIQwO114BDW/ZbgvEx8v6HheRgWI82rYQd+55Gc9YXeM0fh/60yFO0bvlThpcI+JXzGycyT1xr7rkpu+FxS4UgAIZCkRtPW77UbDcV8z89s7XhI4lceHp9Az5cXixCgi49botEDdPp8cfduBpawVoaZDy9KkJJ156pPkUiMbxy/R3RlHJw3XyBLwLdR4MuFPq8X24KelnJIfDUAAKBBSI22vc3nR/kDwEa59JCQE7dCydEO6JpzXBkQUqoOEt222BeL5sxRpkfhw4CwWgABRwFQDEXT2wt2AFBNx6DYgvWHREDwWgQGsVAMRbW7T1zJiGt2wD4vUsK1gFBaBA/RUAxOtfRq2yUMCt192AeKuKEZmBAlCgJgoA4jUpiK6YoeEt24B4V0of+YQCUKBsBQDxshVFfLkKCLj1GhDPlQwnoQAUgAKZCgDimdLgxCIU0PCWbUB8EUojTigABbqgACDehVKuUR4F3HoNiNeogGAKFIACjVIAEG9UcTXf2Lt3D9MT0ccPBqOLxDAHxJtfrsgBFIAC26NAqyB+7fTLtPL1R+mBz3yc4q9V9T9O939+QI/tW6H1C7e2R2WkGisAiMdSYAMKQAEosGUF2gHxO+/Sc199gHbGn7dzPyspQM/8ZvSWZUQERRUAxIsqhXBQAApAgekKNB/ik6P05NLO2PP++EOP0VPPH6V3r12ja7xsHqPx+nO08vVH6NFnN6crkhfi8lEafe9r9OhnH6XnLuQF9M7Ne50XTRt2AfE2lCLyAAWgQF0UaDjEb9H61wXgO+mRHxyjhU6Y8w9EGG/f++GKaaU573XT4m3geUC8gYUGk6EAFKitAs2G+OtP0u9HU+j3L48XC3AuwnlhPO91ta028xsGiM+vHa6EAlAACvgKNBri42Xxwh+l5y77WSu4f+1devl7j9FD/yaKa+cn6aEvPkXreubd/IB7+D67+zOPXpqzXHdnk9b381T9J+N7+x//zIAe+97L9O41L1753eh+n8x9/mvHaPT1h+iTO9nGnfTJpcfoKScD/vXh/WtvP0dPfnFA9/9elNedn6QH/mxIow0tRnStMzC5RUe/94hNf+eQjoajN0dTEL/6Dj3zl39EnzC230efePh/0Hd+9j7dvn2bPvzwQ7p58yZNJhO6fv06Xb16lc78037a96cP0r/7V/fQxz72Mfqdf/Gv6cCBAzkp4hQUgAJQoL0KNBjiR+nJT0Ww+eLLlOJckTI7+xw9qoH1+QciEHK899NwI5qcnwXGOt2i12k7Qg/n/d6j9NxZHXHym9CD749o+JlIB+fanfTI2rv6opztWzT+Zt6DgTvpgW96Mx0K4sNvPhIPPNzf0E4n6UD86WfoiYDtvd599Cc/OuFB/H16Zd9/pN+9916699576Z57Ioj/zu8A4mmZcQQKQIGOKNBgiCcgsz+qrkvM/nazPJWerPWPsh+lJw1AdtJD3z6qBgHXaLx8v733/ZknXa9SgWs0y4NtudeJHX3qf+YxGr0uw5FbdO31FXrEeKh96j+0QgmSVd77O+mhb67TZjTeuHXhZdojYNz5GL0s0Wl5vO1rP3ksgrAbF03epZe/EWnR30l71tUTB3Ge7IBnz8sBb91Lh3c1xHnW4IG9L9Gvf/OReVd8cu4AfeM/9KnX61Hvvr+mVz5IPPFf/f2fRwD/XfpP33iWfnniLB0/fpyOHv5JfSAeDdqy34KI6uXuERVTKyDgIg45ZdmndHvKSpTr4YzPh2RFtcXjm2uDAnan+4XssnINcuOvT75dK7HXJQVMnez3qbMQv/Xq1yyoP/8UHfNL/uyIBsar/X168nV1Mu7sZuy4cq5jgNpBxkO0clqlFW2ynfbVOW2LgngICMefogcir/xrryrwpqMnomRGY+fX1wPPFbxLKw8FZjziPPXpoR8kw4tgEuqgA/Hdz9BF/2Mvb3+PPssQ7/Xoyy99EE2nH6bhv7Ue+O8+9hydvXiRzp6NIH70aL0gvntAg74eLKrMi2ahMlPBqtw0HYFnz3it6CCjJJiZwU+GZgXFcCEbuCg4wNqk0XKxvLrxl5TvgJk4BAWKKDBeTgbb7YB4AD635BWza9fo2A8GESiTjuLoN+V+emgqOjk23FCSSic8q/eRc11sx+e1p63SvLVOX4uA/Ej8ilwC8bAnMe28in/zOXpkCvDf/cEDVj99vzvO0wPBwYdKwdnUEA9/se01+psI4l/40TkL8TMj+i/RNPoXnz1PF+sM8f6ABrv75NQbo4D1AofLQ8p9jsJRa/E73BmE61CRtEuC2cIhbttDukyK5NGGAcSLa4WQi1Yg6ksiNjUY4pv03J9GsN2ZwDkkn2mABlRJODOS4WP88NbnH8hcnvpnFWMMrvI88dgOzxtKUg0BOXQsuYL8B9/0KX878lB4NiCrkwvpN++T+nNB/OQqPRxB/K//76WaQ3xIY64nfnmyznwscC7Rl+tzUkdtOQ5otBa92rg8jkovKf8+Dyg3eOZIX0eUHadbAUw431Y3SE5cbIduC950dWxvFKGqa1zfzOAhblNRW46vmRKXc92QRnnT6QHNvSxGu1pXd3BjdIptc/NdVOtwmjgKBWZXoCWeONG15x+NPOyd9NhPsm/+Jo0s6ehieA5G6l7zFDHjjkN3XFOu4dM51021Q3niSR6TzibsRY1pT+Rdh88rm1XHuuc1dVxtxp74p9QzAjl5UpemNqdCfHKQvuJ74griXznQAIiTO1JmEbicY2gpaPoQTcMimTazYtqyTwZcArukbufHmSoSY1sMVe90flwaZtaOpL55Gph6ptvNmEZr0ZMB5lxiP0X6ZcZl6p4KL3U4hqybCVdT91yy5+uq8xYNiuL41Tnf9o0xyVAriRtbUKBsBaTdN/qeOBHdOpo8mb3zEVo5Hr7/axqx54knx8L3ooOSzwmuPIgXvyeu7bQdDne8O7+Rvo8dvo8ezNFM98SdtObUQkN8518eopvePfHJK1+l+wzEP0V/+wt5sK1B98Qjj9gBh+7oWbcY4h7oTBFx2QqgFCyk+Jzro4M6/sAAwnr0EqdE5K+lU9CgncE+x4YobrY1Al88iPGT5X3/Wn+fw0yJy9HbSyPvXBw0oCtfJwMJNw5VLsZWrVkcIzagQAUKjJv8YJvV59bGkB6QJ7j7H6fB8ojWj28mn1x9/il69LNy/1t1ZJdfpsfkus/soZf1j6PcuUXvrj9Jw+e9Z4i5oZvBwE56LL4/XaCc8q67NaahvCqX83T6Tuc1ugTi/CrcY8+/Gz+Qdu3tleS1OeeJ9mw7TQcV5Us/6e4+nX5/KQ/5aYjz0+mP/K8jdOW2fTr98j//gP7rfdHT6Q/+HZ1Q74mfWnnYvFp27711fzpd6ljS0TsAcGChyzF5DsNMkZu3H5I4pPScuOSgA71pccpFGWtTV3UetF2yLdBS9sV1XMJEazNgCQ0GVPqO/XrmqnhcQV0kCUdzOeiukzbgpRkNQtz4Vb45GmO/vS6ZIXHjxx4UWJQCDb4nnkhy67gCVzSNbGHrNsidn3WfROcBwP054WUUHqd0a532CPjlutirikOlN6Zcx/bHr5JJvGq9c+lJOjrR0SYd9eCLj4XzsPMRGjnvluvr/e1rtB6/SuZqZnW8nx77ifcEetxpS4fuxxne1xAf/OVf0EMqn1Jmvfv+hH540v/Yy6/oxa/8+wjkNX1P3IMRd/yDtZF739gBCsMtTz8PFhRN6/p1zpSFgHdanOFySY7qNKfFpcJy3n27kkiT2wnqWLzp6WagOGNc5rZUPN0dxxxtKDv9U7LPGmZenzOdLtebNaeT/WyJExQ7UKAkBVoBcaPFnWt07OWn6LHd+oMtO+mTn32AHv36Cj33+mbsrTraba7TU/orZeanSx+lr+1P3r3W4W8dH9HXluSrah+n+79R7EMzU6+7doye+5/qy3H85bXPPkrD54+pd9jFEttZMPR4oOHEHX1xbjzHF+w211foa392P31cwPp799PA/3qdmFAGxEcX6eaxZ+gvHv6EfY3uvk/QH/3379E/Xbyd+cW2d577Nv233Z+ifykfe7nvk/V6xSyeClcemoaDA/EMKIvG5gFFD/KR15d4fNbL1Q/EGa8xB4Jx9OZevcA/OuoMCGaxz9qRGvhKYiZenZdZ7olLJHZtvWZld6TJVAinAKtfMcsHcKYnvjFUD4RaDZKyce3GHhQoRwFdb6n50+nliNK0WFyIN8n6u3fvkr989JGdTr9z5w7xwp9czfrs6qVLTXiwLSkR9hCdTt2DOIc0XqQMnHgdQ5/LWYMvitcAUWZMwk+nZ8eZ2MZbFogSF68VHKOg2XH59iX1UmZVUnlX+dTn4jScvGu7XB0du3ePaJz3dLpkWWCfYYOeFrf2J9pnQtyLM3MQIzZgDQVKUCBuL83+2EsJSjQ2iqSzbFqn4QOc91sD8e2qT4GBwXaZgnShABSoVoH2TKdXq9s2pwaI1/KLbdtSK2xdaNpgblukQqJQoIUKAOKNLFRAvLsQT8pepqwB8EY2YhgNBUpRABAvRcaqI0k68qZ14JhOr7quID0oAAXarAAg3ubSrWHeAPEaFgpMggJQoLEKAOKNLbpmGg6IN7PcYDUUgAL1VAAQr2e5tNYqQLy1RYuMQQEosA0KAOLbIHqXkwTEu1z6yDsUgAJlKwCIl60o4stVABDPlQcnoQAUgAIzKQCIzyQXAm9VAUB8qwrieigABaBAogAgnmiBrQoUAMQrEBlJQAEo0BkFAPHOFHU9MgqI16McYAUUgALtUAAQb0c5NiYXgHhjigqGQgEo0AAFAPEGFFKbTFwUxC9fvkxYoAHqAOpA1+oAIN4mQjYgL4uCeAOyDhOhABSAAqUrAIiXLikizFMAEM9TB+egABSAArMpAIjPphdCb1EBQHyLAuJyKAAFoIBSABBXYmBz8Qq0D+LchGZZytR4lnT9sGXaUVVcfh5m2a/KRqQDBapVgFsB/qBAZQoA4lttcrOAq8ywlVURL6Gq8pCVjmcOdqFAzRTgmos/KFCZAoB4FixwfLYZjar1qqyJICEoMJMCgPhMciHwVhUAxKuGD9Irb3Cw1dqP66FA+QrUHuKnT58mLO3R4NSpU+QvJ0+eJF5OnDhhlmPHjpEsb7/9Nr355pv0xhtv0JEjR+gXv/gFHT58mH72s5/RK6+8Qi+88AIdOHCg/JZROEZAsjxINlnLUIVZZH62ml7oehxrogKNgPhkMiEs7dDgxo0b5C/Xr18nXj744AOzXL16lXi5cuUKXbp0id577z26ePEinT9/ns6dO2cGde+8846BOgMdEF8kLBB3fQcpjJyyyqeJ+ILNUgNqrQR74QB4OwDO5egDnPcB8bI6YsRTHtSgpdWy1niAcdEwrtZCAOLtAXj7II6OHtDsQh2oNSI6bxzXwFr/AeKAeD2n07vQeSOPGKRk1YFaY6NTxgHiuN9e6e2K9kynZ3VuO4gu7KBBfwf11TK6kBO+tPuac6SxYe0crKWvHS/voNDxMsE22r2DhhvptDmNYX8HsW6bazuov3sHbVagE9vD5VZ2vk0eVH3gNMYV5KfMsnLj6hQna51Zbj21/oMnDk+8SZ64dNYOtC/soGEAkm6nGAbZtDAmveX5rjVxM8R320GHD5UqIJ4J6MiuKsAtGm9ZZ/lZmAAAEstJREFUywwos44paEeDp6wBjNjkr2Vg4x/fyv585VxrbHTKOEAcnjg88bmafACcUcfsw3ArHey0a7cMngiWIwaNNxiYr3MP6JIBN5O3aNbCGfTQDqokbc+uhaSZVyci7WcZqADiczXWVl8EiAPigPhcTTwNqyIQEOjKtK3AyxxX0616IOCfE++N09NT9nKc4Sjxm/MenJ2BgQIJA0LHEcpPMN4IxNpmPy6+TsetbeBz/vS1Aytlo1xXth1OfNE0Pqflax+0k6f7uewCOoc0lDzwWuczFDY+79+iiW4vSH1y7NR2BMpGruH0s/KtbQxvz9VocNECFADEAXFAfK6GlYY4d4hZoJKOUDpbgTcfN8fUPV/dyfJ59pLFW5sW1oT3oJhrlwYkb6t7tT5UOB4NMR2v3qYoHg01hpGGvOiRmX+lh4lP7S/KDj+/Ulbabs6HLmPe1/nU+ZKy0OHzzvvpc9gY4tGsgb8vNvrlEu9PgTinEUrXtzO9P1ejwUULUGAHf0CjrL+bN2+aL2g988wzxEsZcc9yT3z9K3168PtnUlA68/0Hqd/vR8uXab3x4P4lfekPRvTdMxXdLz9zkD79By/SS0HdDtG+3hKtnixmi32w7SDt7e2lg9GHX5r5nvgWIK49pcgb8jt67qw1POJO1OuUfeDLQ3XOtQxVL804Pg1xr0N3OncvXXO9ipftEHDwNj8DMBDwqnBxunoqO4pbBjbOgIDDaRsXaIeT34xy8fX2oernL5UXnW8vDT99jsuP39/37YnLRbQP6OVfE0rXz0d6vyxqIJ6tKuB44rdv36af/vSnBsAM4ddee82Jn6EsgA6dl8ASTy0gfvJperD/ID1dEDKlfljm1S9T/w+fpjNB+BWDXtieukD8DK3u6tHSfm/g9PMv0Y6/2pEsf/dSPLCSp9NPry5Rb9cKnW7sx17SEC/SGfodqHTUelpctgVqsWcbmG5PxcfAU+HibenUPYg4gORzCqhOfqbFq0DL4OJBBK85DxyPP0jxoRCnFYCOY+MC7YhtiDTygWlsVvmUsovLydfWGxT5efav99P3z4f2U+UvZSjlHdDTvyaUbshW95j09lhvtwIOxN966y3ihf/Eq5Z931ABdei8nKsFxBcG0gIQXlja9YD4mf0M4tUpg5ST9N1v7qBP/+NJA3KBOK8PPt6jpdXTDf1iWxriDmwCHTp3gn4HyscEdm4nGcXvQUMgK552Kj7utKUDz7DBScePX2xkAOtXzArEa6C3sYMGkdfPtrF3nps/sTGyY8z3l/1ZA23jAu3wYcZ2pwYfbIuyLwh6yROvte36eOCcnz6Xkx+/v58qfz9eQNxHV+v2HYj7uWNP3PfGdRg+NyvEZXCQF69OI386/Qw9/YcyTW7X/nS6mUqf1xtmCMfT8H368qsJuE28X1mPPcyJ8fijqXqz7dpl49FT+ev0ZRX3bB57BPGf8zT3Ku3g5c8P0knH4+cw0bk/WKVPj95NbOVwP3/RXheF+dLPk7wZ7987vyM1nV58Gv2lvwtD/MaJFVrq7aVXG/nt9ADEIyCz9yuQNcBUr5iFOl1zLAO8fnizr+L3z3N6DB+Z2naA7UPE7/DVeY6D86HjmRavgZAGXwTcooMKBhSnGQSn0mdRdvgQ9bUWqGr7fKiG9OZ4U++5M9zVA3R8nUlP5dNc54Xx0xMb9WwAh/FtjPcjqOuBiJ/vUB7Sx3Qvje3tVGBuiDOMX3rpJbp8+XLK/jxPvEyI8z3wvgKpviduzmlIRts+5MPT1Qy0dfqyintigJ5AOBfiAtNMT9wOPhJbosGITk/iCK4F0HKf2u4nIH6XvvvnGtz++V/Sl/72lwnUDbAlrglNzD1wdc89dE98fV8BL5x1fIm+9Fefpu+esoME7YnfuHGaVnb1aO/BJv4AShjicWfsTWkL1EPQ5Wukw+aO3SyetyfH2cvlD8lIfHxtCH5yTK6LO3EFatMxZ3mKUWevIa7TCsYbgcm3zY/DpOvboTTQ12fZmJu/Oe0IwUwgGcxvwFPOypsfD8eXyqcaBPJ5LjPOpwZ0HE8Ee6lPMuji61J6R3qYPPCtDn+2Q8DupZWVF3s81e3jwDYpkAlxngp/9tlnU5Bmz1vui2d503kQnzWfmZ649nwj0GmIC5y35Ik7AGXPObm3viWIewMCY2sgP5KH9JqhrCA7mdBLf6ugzdD1PPOToxHt0OB28ubG58TF4QIQN1Ppew4lAwEnvgmd/MdP594TF5jzlPrDq6ca+Ctm2RDP7/xwHfQprw4IxKvXdNaeHOEXpUAQ4uxdM8Cn3dNmoPODcAxt/dcWiKe9+RIh7k/xlwnx1FR4NK2uIM6gNtPw8ZR7MigoAvFDe3rUy4G4HngYoH/zu2a6X+Ata/OA2+OvAuIBz7T6jrk8uMD2arQExDV5urmdgrgAPHSv25eIw/KUOk+R6782QNx42v1k+txMry/SEw955553m4DR9Zz5uANehrjniSfXTsh45c49bjc+J645PXGd3uTUd+nTf/Ul84qawFvW8MSr6ewB1XbqDIhr8nRz24H4LABnuXg6PTSlngfx8u6J2wfD5GEzC930e+LzTqf711mv3PPEY29aHlLT0J+QfdgtuSYBmw3v3xNP9qOHzE6u0lKvF7j37EI3BfEJn1fT695gwEBcQd565Ykn7p63caUebCt8T3xC/GDbjug1M4G3XbfznjiA2U5golx1uXYTmHXMtQNxBrLc75a1vi+u74fP+554eRCfkH3YLHoK/CvrxOD1QejDOAFpBEoPcMl5AbM89f60c098MtFPxj9IT7/K76N7EJ9MjE3JE+76vBu/fkAvtmFuiHPeBL7JtHny4Jt77tOjg949dvtgnJ1uZ7hzePXgm9GMn07v0b71tI7O/XB+XzzwnriBeAufTkdHrzt6bLe3PtQRZ920yYF4WRLkeeKzppH5YFsmfNNQiaGIazIfRJtHo2LvibvloT3x1r0njvva1F5olT0gyeoJy05nUfFl2Y/jVSsAiAPsWwB7xhfbcjQViDf/i23cVOfpIENNfJ54un7NvPpvl26hcp/n2HbZz+nir44KtOrb6UW9STPFHniH3E57h+5hu95k0XS6Ea74R19YDwvxNnw7nZvzLB1qHZv/rHmYJb+zhi1bn1nTnzd82XbPG9+89k+7bl57cF1VCnAJ1voP0+ntGkCIJ67XzfwBlFo3GxgHBaBARxQAxHOmfrvhSVc7SNDwlm1AvCO9DbIJBaBA6QoA4oD4Fu6Jzz4AEHDrNSBeertGhFAACnREAUAcEAfEO9LYkU0oAAXapwAgDogD4u1r18gRFIACHVEAEAfEWwFx/togFmiAOoA60LU6AIgD4q2AeEcG3cgmFIACUMBRoBEQ59fMsLRDg1OnTpG/nDx5kng5ceKEWY4dO0ayvP322/Tmm2/SG2+8QUeOHKFf/OIXdPjwYfrZz35Gr7zyCr3wwgt04MABp1JjBwpAASjQFQVqD/GuFERX8nn37l3yl48++oh4uXPnjln4s728fPjhh+YX8vhVP36C/erVq3Tp0iW6ePEinT17lo4fP05Hjx4FxLtSeZBPKAAFUgoA4ilJcGCRCvgA531AfJGKI24oAAXarAAg3ubSrWHeAPEaFgpMggJQoLEKAOKNLbpmGg6IN7PcYDUUgAL1VAAQr2e5tNYqQLy1RYuMQQEosA0KAOLbIHqXkwTEu1z6yDsUgAJlKwCIl60o4stVABDPlQcnoQAUgAIzKQCIzyQXAm9VAUB8qwrieigABaBAogAgnmiBrQoUAMQrEBlJQAEo0BkFAPHOFHU9MgqI16McYAUUgALtUAAQb0c5NiYXnYD4hREN+n3qx8uARhdsEW2uDai/PK60vEyasS1s15CqtaDS7CIxKNApBQDxThX39me27RC3wEygbRS/MKLh2qbZrBri4+UAtDeGZoAx3JihPpiBSfnwZ/sGkTYzWIOgUAAKRAoA4qgKlSrQaogbOOaDrlKI59nD53aPyA4tClQBQLyASAgCBapXABCvXvNOp9hmiBfxKtMQH9NQTXW7Xql7zoFuBOGR8bT7FPKq8+3huJMZg3RYdd4MBtTtgeh2gOTFrCUP6laBnE8q/CaNdoutdjt9y8HNs6tHEhO2oAAUsAoA4qgJlSrQXohrQGVL6oLNAisBsAInR7ExVHD2wkZgTa7105xmj3s+F+IcdcATF3gnoLVxyr6bV47ETZOP+Om6+5s03ig8V+ALgH0o0AkFAPFOFHN9MgmIqwfbAlPaDD6BoF9qDuAC17rh08DMO+/EbQJ6A4osiCvP21ym7JoX4lU/+Ofqgj0o0CwFdpw/f740i2/evEkvvPACPfPMM2YpM+7SjEREMynAnTtPeWZ7fDNFl/otcYZ6W36KNA3CtDYabGZbpqH1OgajBXEy5aweAlOwTKdij+Tb40I6HdY9n+mJx7ZGVjDso3vtOq/2bHpgkU7Xeucmz7Pcs88SAcehQMsVcDzx27dv009/+tMYwq+99pqTfYayAJrX/nkJLPEA4qJIc9fcyQLiBcuvAFgdsHF4H4JxUlOAVyAtno537qPHcdupen0uDdM5Ia7SdPJq0p6SJ20fT77z63gAuacKdqGAq4AD8bfeeot44T/xqmXfvYxIQB06L+cAcV+15u0D4rOUmXjO3hPqma+YMSizZjk8iJING0+1K1jmWWjKzwchX6seauPrfWDack8efMv0xJ14vPyYdBItTBpeft3BwyaNltUT8wXzmJd/nIMCbVfAgbifWfa0s7xtDsvnZoW4DA7y4vXtwP4sCghIuAO2nSp70rbzT/a1Byidazxtq7xDF+LJ9TFMzL3S5Mnl+HiGybn3xM/9H9rV71Pvj39Iv/r531Cv1zPL0v4zdP36dbp69QR9f+leuvfeB+k7//gd+uzHPkb/cfhTOnDgQEZq23M4paf6uIo5p/S1cEz003B14xnSUL9TPQPg3Hg4rQSsiUJSb2TmhctaQTx6CM3Ukch+m5eReeJc6o5f/lJ/bB0cq6fTo5Tj+mPT0uHDdiYWYwsKQAGiuSHOMH7ppZfo8uXLKR3zPHFAPCVXyQekMx7QYLcLB39fviLmdpz2GumM5RzfE5fteAAQd8A6nSzP0mazGMS/QF+IAG5BvkT/+7gL8c89eA99rKYQL7lAaxtdakBSW0thGBRorwKZEOep8GeffTYFafa85b54ljedB/H2SlmXnAnExfvO3g8+rGamQPvxvVoB93DZfuVLe0dyToBv7sHyA1ra0/RkKQTx3hfoh+/epg8//JDW91lv/OGVU8oTv5c+991f0vHjx+no0aO188S9LLd2FxBvbdEiYw1SIAhx9q4Z4NPuaTPQ+UE4hrb+A8S1GlVvC7STqVCBrUDb3ZfwrjctIJawoelS/5yEkWtDOS8EcZ5Ov20h/u4P/7OZUnch/iB9Z+MsIB4SuMJjgHiFYiMpKJChQAriAvDQvW4/Dg7LU+o8Ra7/AHGtRtXbAuWCEPc8b9MxK29aQD1ckx/1SMcbe+IFsloI4r2/oX+KIB72xAHxAlIjCBSAAh1QwIH4LABnbXg6PTSlngdx3BNfdK2aEeIZ97XFm44hvhE9wcyAl6edZQCg33H2Hobyc1sM4nYKXR5s6/XS98ThifvKYh8KQIEuKuBAnIEs97tlre+L6/vh874nDogvuprNCHH91DE/tSxgju5ra4jLZzPNtLnc95bwMcgTTz2U00IQ/+Mf0j/96Avx0+n71ifpp9MxnR6SF8egABTomAIOxMvKe54nXlYaiKeZChSFuNwT50HfZCIQv0qXLl2iixcv0tmzuCfezBoAq6EAFChTAUC8TDUR11QFAPGpEiEAFIACUKCwAvh2emGpELAMBQDxMlREHFAACkABq8BCPHGICwWyFMiF+J07dOfOHfPKIt+S4ffEMZ2epSSOQwEoAAWmfLENAkGBshUAxMtWFPFBASjQZQXgiXe59Lch74D4NoiOJKEAFGitAoB4a4u2nhkDxOtZLrAKCkCBZioAiDez3BprNSDe2KKD4VAACtRQAUC8hoXSZpMA8TaXLvIGBaBA1QrUHuL8W9FY2qPBT37yE8paXnzxReLl+eefN8tzzz1HvPzDP/wD/fjHP6a1tTXzRcEf/ehHtLKyQk8//TR9+9vfxq+YVd1rID0oAAVqo0DtIV4bpWBIKQp0whNPfY8++RTtdv3yl/l8rnzz3inJMQ3j793bT/bKr905wdRO/CM58ad2hzRW57EJBaBAdQoA4tVpjZSIqO0Qt4BLoG0K/cKIhmubZnNbIM6Dit1DGu727DIWzQZx+y19D9rR9/Onwd9pAGag48XjBJhvh+2b5Vf15ksFV0GB+igAiNenLDphSashbmCWD6btgDinyWALpz0DxPPyx+eCnn5GtQbEM4TBYSgwmwL/H5SeTaHZQ0TIAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip:** if you use JupyterLab you can create a new window with the notebook output like so:   \n",
    "\n",
    "![image.png](attachment:3aff5b53-5087-4c8e-b18b-e6583f5c52cc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get top or bottom of dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AMC Concord</td>\n",
       "      <td>4099</td>\n",
       "      <td>22</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2930</td>\n",
       "      <td>186</td>\n",
       "      <td>40</td>\n",
       "      <td>121</td>\n",
       "      <td>3.58</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMC Pacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>17</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11</td>\n",
       "      <td>3350</td>\n",
       "      <td>173</td>\n",
       "      <td>40</td>\n",
       "      <td>258</td>\n",
       "      <td>2.53</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12</td>\n",
       "      <td>2640</td>\n",
       "      <td>168</td>\n",
       "      <td>35</td>\n",
       "      <td>121</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "0  AMC Concord   4099   22    3.0       2.5     11    2930     186    40   \n",
       "1    AMC Pacer   4749   17    3.0       3.0     11    3350     173    40   \n",
       "2   AMC Spirit   3799   22    NaN       3.0     12    2640     168    35   \n",
       "\n",
       "   displacement  gear_ratio   foreign  \n",
       "0           121        3.58  Domestic  \n",
       "1           258        2.53  Domestic  \n",
       "2           121        3.08  Domestic  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>VW Rabbit</td>\n",
       "      <td>4697</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>1930</td>\n",
       "      <td>155</td>\n",
       "      <td>35</td>\n",
       "      <td>89</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>VW Scirocco</td>\n",
       "      <td>6850</td>\n",
       "      <td>25</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>1990</td>\n",
       "      <td>156</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.78</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>Volvo 260</td>\n",
       "      <td>11995</td>\n",
       "      <td>17</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>14</td>\n",
       "      <td>3170</td>\n",
       "      <td>193</td>\n",
       "      <td>37</td>\n",
       "      <td>163</td>\n",
       "      <td>2.98</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "71    VW Rabbit   4697   25    4.0       3.0     15    1930     155    35   \n",
       "72  VW Scirocco   6850   25    4.0       2.0     16    1990     156    36   \n",
       "73    Volvo 260  11995   17    5.0       2.5     14    3170     193    37   \n",
       "\n",
       "    displacement  gear_ratio  foreign  \n",
       "71            89        3.78  Foreign  \n",
       "72            97        3.78  Foreign  \n",
       "73           163        2.98  Foreign  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get an X amount of random rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Buick Regal</td>\n",
       "      <td>5189</td>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16</td>\n",
       "      <td>3280</td>\n",
       "      <td>200</td>\n",
       "      <td>42</td>\n",
       "      <td>196</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Cad. Eldorado</td>\n",
       "      <td>14500</td>\n",
       "      <td>14</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>16</td>\n",
       "      <td>3900</td>\n",
       "      <td>204</td>\n",
       "      <td>43</td>\n",
       "      <td>350</td>\n",
       "      <td>2.19</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Cad. Seville</td>\n",
       "      <td>15906</td>\n",
       "      <td>21</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>13</td>\n",
       "      <td>4290</td>\n",
       "      <td>204</td>\n",
       "      <td>45</td>\n",
       "      <td>350</td>\n",
       "      <td>2.24</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Plym. Champ</td>\n",
       "      <td>4425</td>\n",
       "      <td>34</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>1800</td>\n",
       "      <td>157</td>\n",
       "      <td>37</td>\n",
       "      <td>86</td>\n",
       "      <td>2.97</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Merc. Marquis</td>\n",
       "      <td>6165</td>\n",
       "      <td>15</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>23</td>\n",
       "      <td>3720</td>\n",
       "      <td>212</td>\n",
       "      <td>44</td>\n",
       "      <td>302</td>\n",
       "      <td>2.26</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "7     Buick Regal   5189   20    3.0       2.0     16    3280     200    42   \n",
       "11  Cad. Eldorado  14500   14    2.0       3.5     16    3900     204    43   \n",
       "12   Cad. Seville  15906   21    3.0       3.0     13    4290     204    45   \n",
       "42    Plym. Champ   4425   34    5.0       2.5     11    1800     157    37   \n",
       "30  Merc. Marquis   6165   15    3.0       3.5     23    3720     212    44   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "7            196        2.93  Domestic  \n",
       "11           350        2.19  Domestic  \n",
       "12           350        2.24  Domestic  \n",
       "42            86        2.97  Domestic  \n",
       "30           302        2.26  Domestic  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = 5\n",
    "df_auto.sample(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select column(s) based on name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* the below returns a pandas `Series` object, this is different than a pandas `Dataframe` object!   \n",
    "You can tell by the way that it looks when shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    AMC Concord\n",
       "1      AMC Pacer\n",
       "2     AMC Spirit\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['make'].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the column name has no whitespace you can also use a dot followed with the column name:  \n",
    "**Warning:** you can't change / overwrite a column using the dot syntax, only do that with the \\[...\\] syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    AMC Concord\n",
       "1      AMC Pacer\n",
       "2     AMC Spirit\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.make.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If you want multiple columns you need to use double brackets:**   \n",
    "*Note:* you are passing a standard Python list of column names to the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AMC Concord</td>\n",
       "      <td>4099</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMC Pacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>4816</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Buick Electra</td>\n",
       "      <td>7827</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Buick LeSabre</td>\n",
       "      <td>5788</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Buick Opel</td>\n",
       "      <td>4453</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Buick Regal</td>\n",
       "      <td>5189</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Buick Riviera</td>\n",
       "      <td>10372</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Buick Skylark</td>\n",
       "      <td>4082</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            make  price  mpg\n",
       "0    AMC Concord   4099   22\n",
       "1      AMC Pacer   4749   17\n",
       "2     AMC Spirit   3799   22\n",
       "3  Buick Century   4816   20\n",
       "4  Buick Electra   7827   15\n",
       "5  Buick LeSabre   5788   18\n",
       "6     Buick Opel   4453   26\n",
       "7    Buick Regal   5189   20\n",
       "8  Buick Riviera  10372   16\n",
       "9  Buick Skylark   4082   19"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[['make', 'price', 'mpg']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select  row based on index value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_auto[['make', 'price', 'mpg', 'trunk', 'headroom']].set_index('make')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "price       10372.0\n",
       "mpg            16.0\n",
       "trunk          17.0\n",
       "headroom        3.5\n",
       "Name: Buick Riviera, dtype: float64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['Buick Riviera']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* notice the appearance, this returned a pandas.Series object not a pandas.Dataframe object "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select row based on index position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>trunk</th>\n",
       "      <th>headroom</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>make</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AMC Spirit</th>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>12</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Buick Century</th>\n",
       "      <td>4816</td>\n",
       "      <td>20</td>\n",
       "      <td>16</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Buick Electra</th>\n",
       "      <td>7827</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               price  mpg  trunk  headroom\n",
       "make                                      \n",
       "AMC Spirit      3799   22     12       3.0\n",
       "Buick Century   4816   20     16       4.5\n",
       "Buick Electra   7827   15     20       4.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[2:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You can also include columns based on their column (!) index position:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>make</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AMC Concord</th>\n",
       "      <td>22</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AMC Pacer</th>\n",
       "      <td>17</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AMC Spirit</th>\n",
       "      <td>22</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             mpg  trunk\n",
       "make                   \n",
       "AMC Concord   22     11\n",
       "AMC Pacer     17     11\n",
       "AMC Spirit    22     12"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0:3, 1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* In the example above the first `0:3` selects the first 3 rows, the second `1:3` selects the 2nd and 3rd column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select based on condition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases you want to filter rows based on a condition. You can do this in Pandas by putting the condition inside square brackets.  \n",
    "\n",
    "It is worth explaining the intuition behind this method as a lot of people find it confusing:  \n",
    "\n",
    "1. You request Pandas to filter a dataframe by putting a condition between square brackets: `df[ `*condition*` ]`\n",
    "2. The `condition` is a sequence of `True` or `False` values for each row (so the length of the `condition` always has to match the number of rows in the dataframe!)\n",
    "3. In Pandas you can generate a `True` or `False` value for each row by simply writing a boolean expression on the whole column. \n",
    "4. Pandas will then only show those rows where the value is `True`\n",
    "\n",
    "In more practical terms:\n",
    "\n",
    "`df_auto['price'] < 3800` will evaluate each row of `df_auto['price']` and return, for that row, whether the condition is `True` or `False`:\n",
    "\n",
    "``\n",
    "0     False\n",
    "1     False\n",
    "2      True\n",
    "3     False\n",
    "4     False\n",
    "5     False\n",
    "``\n",
    "\n",
    "By putting that condition in square brackets `df_auto[ df_auto['price'] < 3800 ]` pandas will first generate a sequence of `True` / `False` values and then only display the rows for which the value is `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMC Spirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12</td>\n",
       "      <td>2640</td>\n",
       "      <td>168</td>\n",
       "      <td>35</td>\n",
       "      <td>121</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Chev. Chevette</td>\n",
       "      <td>3299</td>\n",
       "      <td>29</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>9</td>\n",
       "      <td>2110</td>\n",
       "      <td>163</td>\n",
       "      <td>34</td>\n",
       "      <td>231</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667</td>\n",
       "      <td>24</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2750</td>\n",
       "      <td>179</td>\n",
       "      <td>40</td>\n",
       "      <td>151</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Merc. Zephyr</td>\n",
       "      <td>3291</td>\n",
       "      <td>20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>17</td>\n",
       "      <td>2830</td>\n",
       "      <td>195</td>\n",
       "      <td>43</td>\n",
       "      <td>140</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Subaru</td>\n",
       "      <td>3798</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2050</td>\n",
       "      <td>164</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.81</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Toyota Corolla</td>\n",
       "      <td>3748</td>\n",
       "      <td>31</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2200</td>\n",
       "      <td>165</td>\n",
       "      <td>35</td>\n",
       "      <td>97</td>\n",
       "      <td>3.21</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "2       AMC Spirit   3799   22    NaN       3.0     12    2640     168    35   \n",
       "13  Chev. Chevette   3299   29    3.0       2.5      9    2110     163    34   \n",
       "17     Chev. Monza   3667   24    2.0       2.0      7    2750     179    40   \n",
       "33    Merc. Zephyr   3291   20    3.0       3.5     17    2830     195    43   \n",
       "65          Subaru   3798   35    5.0       2.5     11    2050     164    36   \n",
       "67  Toyota Corolla   3748   31    5.0       3.0      9    2200     165    35   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "2            121        3.08  Domestic  \n",
       "13           231        2.93  Domestic  \n",
       "17           151        2.73  Domestic  \n",
       "33           140        3.08  Domestic  \n",
       "65            97        3.81   Foreign  \n",
       "67            97        3.21   Foreign  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[ df_auto['price'] < 3800 ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also combine multiple conditions by wrapping each condition in parentheses and chaining them with:\n",
    "\n",
    "* For an **AND** statement use: `&`\n",
    "* For an **OR** statement use: `|`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Subaru</td>\n",
       "      <td>3798</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2050</td>\n",
       "      <td>164</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.81</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Toyota Corolla</td>\n",
       "      <td>3748</td>\n",
       "      <td>31</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2200</td>\n",
       "      <td>165</td>\n",
       "      <td>35</td>\n",
       "      <td>97</td>\n",
       "      <td>3.21</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "65          Subaru   3798   35    5.0       2.5     11    2050     164    36   \n",
       "67  Toyota Corolla   3748   31    5.0       3.0      9    2200     165    35   \n",
       "\n",
       "    displacement  gear_ratio  foreign  \n",
       "65            97        3.81  Foreign  \n",
       "67            97        3.21  Foreign  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[ (df_auto['price'] < 3800) & (df_auto['foreign'] == 'Foreign') ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** this will return a new dataframe that isn't automatically assigned to a new variable.   \n",
    "If we want to keep it as a separate dataframe we have to assign it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Subaru</td>\n",
       "      <td>3798</td>\n",
       "      <td>35</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>11</td>\n",
       "      <td>2050</td>\n",
       "      <td>164</td>\n",
       "      <td>36</td>\n",
       "      <td>97</td>\n",
       "      <td>3.81</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Toyota Corolla</td>\n",
       "      <td>3748</td>\n",
       "      <td>31</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>2200</td>\n",
       "      <td>165</td>\n",
       "      <td>35</td>\n",
       "      <td>97</td>\n",
       "      <td>3.21</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "65          Subaru   3798   35    5.0       2.5     11    2050     164    36   \n",
       "67  Toyota Corolla   3748   31    5.0       3.0      9    2200     165    35   \n",
       "\n",
       "    displacement  gear_ratio  foreign  \n",
       "65            97        3.81  Foreign  \n",
       "67            97        3.21  Foreign  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_small = df_auto[(df_auto.price < 3800) & (df_auto.foreign == 'Foreign')]\n",
    "df_auto_small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sort dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229</td>\n",
       "      <td>23</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6</td>\n",
       "      <td>2370</td>\n",
       "      <td>170</td>\n",
       "      <td>35</td>\n",
       "      <td>119</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934</td>\n",
       "      <td>18</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7</td>\n",
       "      <td>3470</td>\n",
       "      <td>198</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486</td>\n",
       "      <td>26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8</td>\n",
       "      <td>2520</td>\n",
       "      <td>182</td>\n",
       "      <td>38</td>\n",
       "      <td>119</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389</td>\n",
       "      <td>28</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9</td>\n",
       "      <td>1800</td>\n",
       "      <td>147</td>\n",
       "      <td>33</td>\n",
       "      <td>98</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667</td>\n",
       "      <td>24</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2750</td>\n",
       "      <td>179</td>\n",
       "      <td>40</td>\n",
       "      <td>151</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "55      Datsun 200   6229   23    4.0       1.5      6    2370     170    35   \n",
       "47  Pont. Firebird   4934   18    1.0       1.5      7    3470     198    42   \n",
       "44   Plym. Sapporo   6486   26    NaN       1.5      8    2520     182    38   \n",
       "23     Ford Fiesta   4389   28    4.0       1.5      9    1800     147    33   \n",
       "17     Chev. Monza   3667   24    2.0       2.0      7    2750     179    40   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "55           119        3.89   Foreign  \n",
       "47           231        3.08  Domestic  \n",
       "44           119        3.54  Domestic  \n",
       "23            98        3.15  Domestic  \n",
       "17           151        2.73  Domestic  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.sort_values(by=['headroom', 'trunk'], inplace=True) ## Note: `inplace=True` will overwrite the existing dataframe after sorting\n",
    "df_auto.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">View a dataframe using `qgrid`</span><a id='qgrid'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are used to RStudio or Stata you might be looking for an interactive dataframe viewer. Python / Jupyter does not come with such a feature by default. The primary workflow to inspect parts of the data would be to write quick snippets of code to pull up the data you'd like to see (such as illustrated above). \n",
    "\n",
    "This can be cumbersome in the beginning, a neat workaround is to use a package called `qgrid` to quickly inspect your data:  \n",
    "\n",
    "This is the GitHub page for `qgrid`: https://github.com/quantopian/qgrid  \n",
    "\n",
    "The first time you run it you do have to run the following command in the terminal / command line:\n",
    "1. `jupyter labextension install @jupyter-widgets/jupyterlab-manager qgrid2`   \n",
    "\n",
    "*Note 1:* you can create a terminal window from within JupyterLab and run it from there (click the plus in the top left and click terminal)  \n",
    "*Note 2:* you might have to click on the puzzle icon in the left sidebar and click \"enable\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**First make sure you import the `show_grid` function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qgrid import show_grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You can inspect a Dataframe as follows:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41cb0c31ba7c4c2f8e4dd01ea1a9dcfa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "QgridWidget(grid_options={'fullWidthRows': True, 'syncColumnCellResize': True, 'forceFitColumns': True, 'defau…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_grid(df_auto, grid_options={'editable' : False})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Several things to note about qgrid:** \n",
    "- I strongly recommend to never edit values directly inside qrid. It is best to include `grid_options={'editable' : False}` when using it to avoid this.  \n",
    "- Saving a notebook with active `qgrids` in them dramatically increases the file-size. Generally it is best to remove the qgrid as soon as your are done. \n",
    "- Opening very big dataframes using `show_grid()` might cause stability issues.  \n",
    "- These `qgrids` will only display locally, not on GitHub. Therefore, if you see this on GitHub, you will not see the actual `qgrid`.\n",
    "- There are a bunch of options you can use with `show_grid()`, however, I strongly discourage you to modify a dataframe using the qgrid toolbars. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Dealing with datatypes</span><a id='datatypes'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to pay attention to the datatypes contained in a column. A lot of errors that you will encounter relate to wrong datatypes (e.g. because of data errors)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remark:** you can enter any Python object into a dataframe, but that doesn't mean it is good idea. My recommendation is to mainly include `int`, `float`, `date`, or `str` data types in a dataframe. In other cases you are probably better of using native Python data structures and not a dataframe.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'TestKey': 'TestVal'}</td>\n",
       "      <td>&lt;function &lt;lambda&gt; at 0x000001E2E01BE678&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     col1                                       col2\n",
       "0  {'TestKey': 'TestVal'}  <function <lambda> at 0x000001E2E01BE678>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([({'TestKey' : 'TestVal'}, lambda x: x + 2)], columns=['col1', 'col2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show current datatypes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "make             object\n",
       "price             int64\n",
       "mpg               int64\n",
       "rep78           float64\n",
       "headroom        float64\n",
       "trunk             int64\n",
       "weight            int64\n",
       "length            int64\n",
       "turn              int64\n",
       "displacement      int64\n",
       "gear_ratio      float64\n",
       "foreign          object\n",
       "dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important note about strings**\n",
    "Prior to Pandas 1.0 strings would be stored in a dataframe as an `object` datatype. However, the `object` dtype is not specific to just strings and is also assigned to any other dtype that isn't recognized by Pandas. \n",
    "\n",
    "This is why Pandas 1.0 introduced a new datatype for strings called `string`.  ([Link with details](https://pandas.pydata.org/pandas-docs/stable/user_guide/text.html)) \n",
    "\n",
    "As af June 2020, the `pandas.StringDtype` is still considered experimental, so I will not use it explicitly in the demos below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert datatypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert the datatype of a column in two ways:  \n",
    "\n",
    "1. Use the built-in Pandas functions to convert the column in one go\n",
    "2. Loop over the values and convert them individually"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*1) Use Pandas functions*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to convert a column to `string`, I recommend to use `.astype(str)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['length'] = df_auto['length'].astype(str)\n",
    "df_auto['length'].dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to convert a column to `numeric`, I recommend to use `df.to_numeric()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['length'] = pd.to_numeric(df_auto['length'])\n",
    "df_auto['length'].dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*2) Convert values individually*   \n",
    "Using the built-in Pandas fuctions is usually sufficient, however, sometimes you want to have more control over the conversion.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['length'].apply(lambda x: str(x)).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: `'O'` stands for 'object'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['length'].apply(lambda x: int(x)).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The section `dealing with dates` will discuss how to convert a column with `dates`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Honorable mention:*    \n",
    "\n",
    "Pandas 1.0 introduced a new feature (`convert_dtypes()`) that can attempt to resolve dtype conflicts for you. See the example below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229</td>\n",
       "      <td>23</td>\n",
       "      <td>4</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6</td>\n",
       "      <td>2370</td>\n",
       "      <td>170</td>\n",
       "      <td>35</td>\n",
       "      <td>119</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7</td>\n",
       "      <td>3470</td>\n",
       "      <td>198</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486</td>\n",
       "      <td>26</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8</td>\n",
       "      <td>2520</td>\n",
       "      <td>182</td>\n",
       "      <td>38</td>\n",
       "      <td>119</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389</td>\n",
       "      <td>28</td>\n",
       "      <td>4</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9</td>\n",
       "      <td>1800</td>\n",
       "      <td>147</td>\n",
       "      <td>33</td>\n",
       "      <td>98</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667</td>\n",
       "      <td>24</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2750</td>\n",
       "      <td>179</td>\n",
       "      <td>40</td>\n",
       "      <td>151</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Olds Omega</td>\n",
       "      <td>4181</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "      <td>4.5</td>\n",
       "      <td>14</td>\n",
       "      <td>3370</td>\n",
       "      <td>200</td>\n",
       "      <td>43</td>\n",
       "      <td>231</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>4816</td>\n",
       "      <td>20</td>\n",
       "      <td>3</td>\n",
       "      <td>4.5</td>\n",
       "      <td>16</td>\n",
       "      <td>3250</td>\n",
       "      <td>196</td>\n",
       "      <td>40</td>\n",
       "      <td>196</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Olds Cutlass</td>\n",
       "      <td>4733</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "      <td>4.5</td>\n",
       "      <td>16</td>\n",
       "      <td>3300</td>\n",
       "      <td>198</td>\n",
       "      <td>42</td>\n",
       "      <td>231</td>\n",
       "      <td>2.93</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dodge St. Regis</td>\n",
       "      <td>6342</td>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "      <td>4.5</td>\n",
       "      <td>21</td>\n",
       "      <td>3740</td>\n",
       "      <td>220</td>\n",
       "      <td>46</td>\n",
       "      <td>225</td>\n",
       "      <td>2.94</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Plym. Volare</td>\n",
       "      <td>4060</td>\n",
       "      <td>18</td>\n",
       "      <td>2</td>\n",
       "      <td>5.0</td>\n",
       "      <td>16</td>\n",
       "      <td>3330</td>\n",
       "      <td>201</td>\n",
       "      <td>44</td>\n",
       "      <td>225</td>\n",
       "      <td>3.23</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>74 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "55       Datsun 200   6229   23      4       1.5      6    2370     170    35   \n",
       "47   Pont. Firebird   4934   18      1       1.5      7    3470     198    42   \n",
       "44    Plym. Sapporo   6486   26   <NA>       1.5      8    2520     182    38   \n",
       "23      Ford Fiesta   4389   28      4       1.5      9    1800     147    33   \n",
       "17      Chev. Monza   3667   24      2       2.0      7    2750     179    40   \n",
       "..              ...    ...  ...    ...       ...    ...     ...     ...   ...   \n",
       "38       Olds Omega   4181   19      3       4.5     14    3370     200    43   \n",
       "3     Buick Century   4816   20      3       4.5     16    3250     196    40   \n",
       "36     Olds Cutlass   4733   19      3       4.5     16    3300     198    42   \n",
       "22  Dodge St. Regis   6342   17      2       4.5     21    3740     220    46   \n",
       "45     Plym. Volare   4060   18      2       5.0     16    3330     201    44   \n",
       "\n",
       "    displacement  gear_ratio   foreign  \n",
       "55           119        3.89   Foreign  \n",
       "47           231        3.08  Domestic  \n",
       "44           119        3.54  Domestic  \n",
       "23            98        3.15  Domestic  \n",
       "17           151        2.73  Domestic  \n",
       "..           ...         ...       ...  \n",
       "38           231        3.08  Domestic  \n",
       "3            196        2.93  Domestic  \n",
       "36           231        2.93  Domestic  \n",
       "22           225        2.94  Domestic  \n",
       "45           225        3.23  Domestic  \n",
       "\n",
       "[74 rows x 12 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.convert_dtypes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Handling missing values</span><a id='missing-values'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dealing with missing values is easy in Pandas, as long as you are careful in defining them as `np.nan` (and **not** a string value like 'np.nan')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/missing_data.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add some missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* We define a missing value as `np.nan` so we can consistently select them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto.loc['Example_Car'] = [np.nan for x in range(0,len(df_auto.columns))]\n",
    "df_auto.loc['Example_Bike'] = [np.nan for x in range(0,len(df_auto.columns))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Example_Car</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Example_Bike</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "Example_Car   NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "Example_Bike  NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "\n",
       "              displacement  gear_ratio foreign  \n",
       "Example_Car            NaN         NaN     NaN  \n",
       "Example_Bike           NaN         NaN     NaN  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.loc[['Example_Car', 'Example_Bike']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select missing or non-missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always use `pd.isnull()` or `pd.notnull()` when conditioning on missing values. This is the most reliable.  \n",
    "`df_auto.make == np.nan` will **not** work consistently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Example_Car</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Example_Bike</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make  price  mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "Example_Car   NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "Example_Bike  NaN    NaN  NaN    NaN       NaN    NaN     NaN     NaN   NaN   \n",
       "\n",
       "              displacement  gear_ratio foreign  \n",
       "Example_Car            NaN         NaN     NaN  \n",
       "Example_Bike           NaN         NaN     NaN  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[pd.isnull(df_auto.make)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Plym. Sapporo</td>\n",
       "      <td>6486.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2520.0</td>\n",
       "      <td>182.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "44   Plym. Sapporo  6486.0  26.0    NaN       1.5    8.0  2520.0   182.0   \n",
       "23     Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0   \n",
       "17     Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  \n",
       "55  35.0         119.0        3.89   Foreign  \n",
       "47  42.0         231.0        3.08  Domestic  \n",
       "44  38.0         119.0        3.54  Domestic  \n",
       "23  33.0          98.0        3.15  Domestic  \n",
       "17  40.0         151.0        2.73  Domestic  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto[pd.notnull(df_auto.make)].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fill missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fill missing values with something we can use `.fillna()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Example_Car</th>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Example_Bike</th>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 make    price      mpg    rep78 headroom    trunk   weight  \\\n",
       "Example_Car   Missing  Missing  Missing  Missing  Missing  Missing  Missing   \n",
       "Example_Bike  Missing  Missing  Missing  Missing  Missing  Missing  Missing   \n",
       "\n",
       "               length     turn displacement gear_ratio  foreign  \n",
       "Example_Car   Missing  Missing      Missing    Missing  Missing  \n",
       "Example_Bike  Missing  Missing      Missing    Missing  Missing  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_auto.fillna('Missing')\n",
    "df.loc[['Example_Car', 'Example_Bike']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop rows with missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To drop missing values we can use `.dropna()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45              Plym. Volare\n",
       "Example_Car              NaN\n",
       "Example_Bike             NaN\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['make'].tail(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36       Olds Cutlass\n",
       "22    Dodge St. Regis\n",
       "45       Plym. Volare\n",
       "Name: make, dtype: object"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto = df_auto.dropna(axis=0)\n",
    "df_auto['make'].tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note 1:* `axis=0` indicates that we want to drop rows and not columns.  \n",
    "*Note 2:* the `subset=[...]` parameter can be used in `dropna()` to only look for missing values in certain columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Work with data in the dataframe</span><a id='work-with-data'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine columns (and output it to a new column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Remember:* You can select a column using:\n",
    "1. `df_auto['price']`\n",
    "2. `df_auto.price` --> but this one only works if there are no spaces in the column name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>trunk</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>6229.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1038.166667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>704.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>4389.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>487.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>3667.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>523.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>4172.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>596.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     price  trunk  price_trunk_ratio\n",
       "55  6229.0    6.0        1038.166667\n",
       "47  4934.0    7.0         704.857143\n",
       "23  4389.0    9.0         487.666667\n",
       "17  3667.0    7.0         523.857143\n",
       "51  4172.0    7.0         596.000000"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['price_trunk_ratio'] = df_auto.price / df_auto.trunk\n",
    "df_auto[['price', 'trunk', 'price_trunk_ratio']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a new column by iterating over the dataframe per row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to iterate over rows.  \n",
    "They mainly different in their trade-off between ease-of-use, readability, and performance.  \n",
    "\n",
    "I will show the three main possibilities.\n",
    "\n",
    "For the sake of demonstration, let's say our goal is to achieve the following:    \n",
    "> If the car is a foreign brand, multiple the price by 1.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 1: use `.apply()` with `lambda`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* `lambda` is a so-called anonymous function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>foreign</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   foreign  new_price\n",
       "55      Datsun 200  6229.0   Foreign     9343.5\n",
       "47  Pont. Firebird  4934.0  Domestic     4934.0\n",
       "23     Ford Fiesta  4389.0  Domestic     4389.0\n",
       "17     Chev. Monza  3667.0  Domestic     3667.0\n",
       "51   Pont. Sunbird  4172.0  Domestic     4172.0"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logic = lambda x: x.price * 1.5 if x.foreign == 'Foreign' else x.price\n",
    "df_auto['new_price'] = df_auto.apply(logic, axis=1)\n",
    "df_auto[['make', 'price', 'foreign', 'new_price']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* make sure to include the `axis = 1` argument, this tells Pandas to iterate over the rows and not the columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 2: use `.apply()` with a function**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above we used an anonymous `lambda` function.  \n",
    "For more complex processing it is possible to use a defined function and call it in `.apply()`  \n",
    "\n",
    "**Personal note:** This is often my preferred method as it is the most flexible and a lot easier to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def new_price_function(row):\n",
    "    if row.foreign == 'Foreign':\n",
    "        return row.price * 1.5\n",
    "    else:\n",
    "        return row.price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>foreign</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   foreign  new_price\n",
       "55      Datsun 200  6229.0   Foreign     9343.5\n",
       "47  Pont. Firebird  4934.0  Domestic     4934.0\n",
       "23     Ford Fiesta  4389.0  Domestic     4389.0\n",
       "17     Chev. Monza  3667.0  Domestic     3667.0\n",
       "51   Pont. Sunbird  4172.0  Domestic     4172.0"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['new_price'] = df_auto.apply(new_price_function, axis=1)\n",
    "df_auto[['make', 'price', 'foreign', 'new_price']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* make sure to include the `axis = 1` argument, this tells Pandas to iterate over the rows and not the columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 3: loop over the dataframe row by row**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* Using apply is generally considered best practices over using `iterrows()`. I'd only use `iterrows()` if you have to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>foreign</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   foreign  new_price\n",
       "55      Datsun 200  6229.0   Foreign     9343.5\n",
       "47  Pont. Firebird  4934.0  Domestic     4934.0\n",
       "23     Ford Fiesta  4389.0  Domestic     4389.0\n",
       "17     Chev. Monza  3667.0  Domestic     3667.0\n",
       "51   Pont. Sunbird  4172.0  Domestic     4172.0"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_list = []\n",
    "for index, row in df_auto.iterrows():\n",
    "    if row.foreign == 'Foreign':\n",
    "        new_list.append(row.price * 1.5)\n",
    "    else:\n",
    "        new_list.append(row.price)\n",
    "df_auto['new_price'] = new_list\n",
    "df_auto[['make', 'price', 'foreign', 'new_price']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Combining dataframes</span><a id='combining-dataframes'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can combine dataframes in three ways:\n",
    "\n",
    "1. Merge\n",
    "2. Join\n",
    "3. Append"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will demonstrate that using the following two datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto_p1 = df_auto[['make', 'price', 'mpg']]\n",
    "df_auto_p2 = df_auto[['make', 'headroom', 'trunk']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg\n",
       "55      Datsun 200  6229.0  23.0\n",
       "47  Pont. Firebird  4934.0  18.0\n",
       "23     Ford Fiesta  4389.0  28.0"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_p1.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make  headroom  trunk\n",
       "55      Datsun 200       1.5    6.0\n",
       "47  Pont. Firebird       1.5    7.0\n",
       "23     Ford Fiesta       1.5    9.0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_p2.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Merge datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html  \n",
    "The `.merge()` function is one of my personal favorites, it is really easy to use.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make   price   mpg  headroom  trunk\n",
       "0      Datsun 200  6229.0  23.0       1.5    6.0\n",
       "1  Pont. Firebird  4934.0  18.0       1.5    7.0\n",
       "2     Ford Fiesta  4389.0  28.0       1.5    9.0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_auto = pd.merge(df_auto_p1, df_auto_p2, how='left', on='make')\n",
    "merged_auto.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Join datasets on index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if both dataframes share the same index you can also join on the index. This can be faster compared to `.merge`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto_p1.set_index('make', inplace=True)\n",
    "df_auto_p2.set_index('make', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make   price   mpg  headroom  trunk\n",
       "0      Datsun 200  6229.0  23.0       1.5    6.0\n",
       "1  Pont. Firebird  4934.0  18.0       1.5    7.0\n",
       "2     Ford Fiesta  4389.0  28.0       1.5    9.0"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joined_auto = df_auto_p1.join(df_auto_p2)\n",
    "joined_auto.reset_index().head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Append data to the dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See http://pandas.pydata.org/pandas-docs/stable/merging.html#concatenating-objects  \n",
    "\n",
    "*Note:* There is also a shortcut function called `.append()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_auto_i1 = df_auto.iloc[0:3]\n",
    "df_auto_i2 = df_auto.iloc[3:6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>1038.166667</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>704.857143</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>487.666667</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "23     Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "55  35.0         119.0        3.89   Foreign        1038.166667     9343.5  \n",
       "47  42.0         231.0        3.08  Domestic         704.857143     4934.0  \n",
       "23  33.0          98.0        3.15  Domestic         487.666667     4389.0  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_i1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>523.857143</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>596.000000</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Dodge Colt</td>\n",
       "      <td>3984.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2120.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>498.000000</td>\n",
       "      <td>3984.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make   price   mpg  rep78  headroom  trunk  weight  length  turn  \\\n",
       "17    Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0  40.0   \n",
       "51  Pont. Sunbird  4172.0  24.0    2.0       2.0    7.0  2690.0   179.0  41.0   \n",
       "19     Dodge Colt  3984.0  30.0    5.0       2.0    8.0  2120.0   163.0  35.0   \n",
       "\n",
       "    displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "17         151.0        2.73  Domestic         523.857143     3667.0  \n",
       "51         151.0        2.73  Domestic         596.000000     4172.0  \n",
       "19          98.0        3.54  Domestic         498.000000     3984.0  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_i2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the higher level function `concat()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>1038.166667</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>704.857143</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>487.666667</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>523.857143</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>596.000000</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Dodge Colt</td>\n",
       "      <td>3984.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2120.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>498.000000</td>\n",
       "      <td>3984.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "23     Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0   \n",
       "17     Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0   \n",
       "51   Pont. Sunbird  4172.0  24.0    2.0       2.0    7.0  2690.0   179.0   \n",
       "19      Dodge Colt  3984.0  30.0    5.0       2.0    8.0  2120.0   163.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "55  35.0         119.0        3.89   Foreign        1038.166667     9343.5  \n",
       "47  42.0         231.0        3.08  Domestic         704.857143     4934.0  \n",
       "23  33.0          98.0        3.15  Domestic         487.666667     4389.0  \n",
       "17  40.0         151.0        2.73  Domestic         523.857143     3667.0  \n",
       "51  41.0         151.0        2.73  Domestic         596.000000     4172.0  \n",
       "19  35.0          98.0        3.54  Domestic         498.000000     3984.0  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([df_auto_i1, df_auto_i2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the shortcut fuction `append()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>rep78</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>turn</th>\n",
       "      <th>displacement</th>\n",
       "      <th>gear_ratio</th>\n",
       "      <th>foreign</th>\n",
       "      <th>price_trunk_ratio</th>\n",
       "      <th>new_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>3.89</td>\n",
       "      <td>Foreign</td>\n",
       "      <td>1038.166667</td>\n",
       "      <td>9343.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>231.0</td>\n",
       "      <td>3.08</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>704.857143</td>\n",
       "      <td>4934.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.15</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>487.666667</td>\n",
       "      <td>4389.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>523.857143</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>2.73</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>596.000000</td>\n",
       "      <td>4172.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Dodge Colt</td>\n",
       "      <td>3984.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2120.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>Domestic</td>\n",
       "      <td>498.000000</td>\n",
       "      <td>3984.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price   mpg  rep78  headroom  trunk  weight  length  \\\n",
       "55      Datsun 200  6229.0  23.0    4.0       1.5    6.0  2370.0   170.0   \n",
       "47  Pont. Firebird  4934.0  18.0    1.0       1.5    7.0  3470.0   198.0   \n",
       "23     Ford Fiesta  4389.0  28.0    4.0       1.5    9.0  1800.0   147.0   \n",
       "17     Chev. Monza  3667.0  24.0    2.0       2.0    7.0  2750.0   179.0   \n",
       "51   Pont. Sunbird  4172.0  24.0    2.0       2.0    7.0  2690.0   179.0   \n",
       "19      Dodge Colt  3984.0  30.0    5.0       2.0    8.0  2120.0   163.0   \n",
       "\n",
       "    turn  displacement  gear_ratio   foreign  price_trunk_ratio  new_price  \n",
       "55  35.0         119.0        3.89   Foreign        1038.166667     9343.5  \n",
       "47  42.0         231.0        3.08  Domestic         704.857143     4934.0  \n",
       "23  33.0          98.0        3.15  Domestic         487.666667     4389.0  \n",
       "17  40.0         151.0        2.73  Domestic         523.857143     3667.0  \n",
       "51  41.0         151.0        2.73  Domestic         596.000000     4172.0  \n",
       "19  35.0          98.0        3.54  Domestic         498.000000     3984.0  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto_i1.append(df_auto_i2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Group-by operations</span><a id='groupby'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often you want to perform an operation within a group, in Pandas you achieve this by using `.groupby()`.\n",
    "\n",
    "Pandas `.groupby()` is a process involving one or more of the following steps (paraphrasing from the docs):  \n",
    "1. **Splitting** the data into groups based on some criteria\n",
    "2. **Applying** a function to each group independently\n",
    "3. **Combining** the results into a data structure\n",
    "\n",
    "For the full documentation see: http://pandas.pydata.org/pandas-docs/stable/groupby.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the dataframe by creating a group object:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 1 is to create a `group` object that specifies the groups that we want to create.  \n",
    "After creating a `group` object we can apply operations to it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_list = ['price', 'mpg', 'headroom', 'trunk', 'weight', 'length']\n",
    "grouped = df_auto[col_list + ['foreign']].groupby(['foreign'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* it is also possible to groupby on multiple column (e.g. `df_auto.groupby(['foreign', 'trunk'])`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying example 1) Compute mean summary statistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>6179.250000</td>\n",
       "      <td>19.541667</td>\n",
       "      <td>3.187500</td>\n",
       "      <td>15.083333</td>\n",
       "      <td>3368.333333</td>\n",
       "      <td>197.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6070.142857</td>\n",
       "      <td>25.285714</td>\n",
       "      <td>2.571429</td>\n",
       "      <td>11.285714</td>\n",
       "      <td>2263.333333</td>\n",
       "      <td>167.428571</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                price        mpg  headroom      trunk       weight      length\n",
       "foreign                                                                       \n",
       "Domestic  6179.250000  19.541667  3.187500  15.083333  3368.333333  197.416667\n",
       "Foreign   6070.142857  25.285714  2.571429  11.285714  2263.333333  167.428571"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying example 2) Retrieve particular group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "      <th>foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>4389.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>3667.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>4172.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3984.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2120.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>Domestic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     price   mpg  headroom  trunk  weight  length   foreign\n",
       "47  4934.0  18.0       1.5    7.0  3470.0   198.0  Domestic\n",
       "23  4389.0  28.0       1.5    9.0  1800.0   147.0  Domestic\n",
       "17  3667.0  24.0       2.0    7.0  2750.0   179.0  Domestic\n",
       "51  4172.0  24.0       2.0    7.0  2690.0   179.0  Domestic\n",
       "19  3984.0  30.0       2.0    8.0  2120.0   163.0  Domestic"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.get_group('Domestic').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying example 3) Iterate over the groups in the `group` object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By iterating over each group you get a lot of flexibility as you can do anything you want with each group.   \n",
    "\n",
    "It is worth noting that each group is a dataframe object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Domestic\n",
      "     price   mpg  headroom  trunk  weight  length   foreign\n",
      "47  4934.0  18.0       1.5    7.0  3470.0   198.0  Domestic\n",
      "23  4389.0  28.0       1.5    9.0  1800.0   147.0  Domestic\n",
      "17  3667.0  24.0       2.0    7.0  2750.0   179.0  Domestic\n",
      "51  4172.0  24.0       2.0    7.0  2690.0   179.0  Domestic\n",
      "19  3984.0  30.0       2.0    8.0  2120.0   163.0  Domestic\n",
      "Foreign\n",
      "     price   mpg  headroom  trunk  weight  length  foreign\n",
      "55  6229.0  23.0       1.5    6.0  2370.0   170.0  Foreign\n",
      "56  4589.0  35.0       2.0    8.0  2020.0   165.0  Foreign\n",
      "68  5719.0  18.0       2.0   11.0  2670.0   175.0  Foreign\n",
      "72  6850.0  25.0       2.0   16.0  1990.0   156.0  Foreign\n",
      "61  4499.0  28.0       2.5    5.0  1760.0   149.0  Foreign\n"
     ]
    }
   ],
   "source": [
    "for name, group in grouped:\n",
    "    print(name)\n",
    "    print(group.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### It is also possible to use the `.apply()` function on `group` objects:   \n",
    "\n",
    "Using a `lambda df: ....` with `.apply()` is a nice way to iterature over subsets of the data.\n",
    "\n",
    "For example, let's say we want to get the cheapest car within each \"trunk\" size category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trunk</th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trunk</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5.0</th>\n",
       "      <td>5.0</td>\n",
       "      <td>Honda Civic</td>\n",
       "      <td>4499.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6.0</th>\n",
       "      <td>6.0</td>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7.0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8.0</th>\n",
       "      <td>8.0</td>\n",
       "      <td>Dodge Colt</td>\n",
       "      <td>3984.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9.0</th>\n",
       "      <td>9.0</td>\n",
       "      <td>Chev. Chevette</td>\n",
       "      <td>3299.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       trunk            make   price\n",
       "trunk                               \n",
       "5.0      5.0     Honda Civic  4499.0\n",
       "6.0      6.0      Datsun 200  6229.0\n",
       "7.0      7.0     Chev. Monza  3667.0\n",
       "8.0      8.0      Dodge Colt  3984.0\n",
       "9.0      9.0  Chev. Chevette  3299.0"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto.groupby('trunk').apply(lambda df: df.sort_values('price').iloc[0]).head()[['trunk', 'make', 'price']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create new dataframe with one row for each group"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to aggregate each group to one row in a new dataframe you have many options, below a couple of examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) `grouped.sum()` and `grouped.mean()`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>296604.0</td>\n",
       "      <td>938.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>724.0</td>\n",
       "      <td>161680.0</td>\n",
       "      <td>9476.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>127473.0</td>\n",
       "      <td>531.0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>237.0</td>\n",
       "      <td>47530.0</td>\n",
       "      <td>3516.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             price    mpg  headroom  trunk    weight  length\n",
       "foreign                                                     \n",
       "Domestic  296604.0  938.0     153.0  724.0  161680.0  9476.0\n",
       "Foreign   127473.0  531.0      54.0  237.0   47530.0  3516.0"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>6179.250000</td>\n",
       "      <td>19.541667</td>\n",
       "      <td>3.187500</td>\n",
       "      <td>15.083333</td>\n",
       "      <td>3368.333333</td>\n",
       "      <td>197.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6070.142857</td>\n",
       "      <td>25.285714</td>\n",
       "      <td>2.571429</td>\n",
       "      <td>11.285714</td>\n",
       "      <td>2263.333333</td>\n",
       "      <td>167.428571</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                price        mpg  headroom      trunk       weight      length\n",
       "foreign                                                                       \n",
       "Domestic  6179.250000  19.541667  3.187500  15.083333  3368.333333  197.416667\n",
       "Foreign   6070.142857  25.285714  2.571429  11.285714  2263.333333  167.428571"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) `grouped.count()` and `grouped.size()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          price  mpg  headroom  trunk  weight  length\n",
       "foreign                                              \n",
       "Domestic     48   48        48     48      48      48\n",
       "Foreign      21   21        21     21      21      21"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "foreign\n",
       "Domestic    48\n",
       "Foreign     21\n",
       "dtype: int64"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) `grouped.first()` and `grouped.last()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>198.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6229.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>170.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           price   mpg  headroom  trunk  weight  length\n",
       "foreign                                                \n",
       "Domestic  4934.0  18.0       1.5    7.0  3470.0   198.0\n",
       "Foreign   6229.0  23.0       1.5    6.0  2370.0   170.0"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.first()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>mpg</th>\n",
       "      <th>headroom</th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>4060.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>3330.0</td>\n",
       "      <td>201.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>3995.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1980.0</td>\n",
       "      <td>154.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           price   mpg  headroom  trunk  weight  length\n",
       "foreign                                                \n",
       "Domestic  4060.0  18.0       5.0   16.0  3330.0   201.0\n",
       "Foreign   3995.0  30.0       3.5   11.0  1980.0   154.0"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.last()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4) You can also use the `.agg()` function to specify which operations to perform for each column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th colspan=\"2\" halign=\"left\">mpg</th>\n",
       "      <th colspan=\"2\" halign=\"left\">trunk</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>mean</th>\n",
       "      <th>median</th>\n",
       "      <th>mean</th>\n",
       "      <th>&lt;lambda_0&gt;</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foreign</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Domestic</th>\n",
       "      <td>4934.0</td>\n",
       "      <td>19.541667</td>\n",
       "      <td>19.0</td>\n",
       "      <td>15.083333</td>\n",
       "      <td>1508.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Foreign</th>\n",
       "      <td>6229.0</td>\n",
       "      <td>25.285714</td>\n",
       "      <td>25.0</td>\n",
       "      <td>11.285714</td>\n",
       "      <td>1128.571429</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           price        mpg             trunk             \n",
       "           first       mean median       mean   <lambda_0>\n",
       "foreign                                                   \n",
       "Domestic  4934.0  19.541667   19.0  15.083333  1508.333333\n",
       "Foreign   6229.0  25.285714   25.0  11.285714  1128.571429"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.agg({'price' : 'first', \n",
    "             'mpg' : ['mean', 'median'], \n",
    "             'trunk' : ['mean', (lambda x: 100 * np.mean(x))]}, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add a column to the original dataframe with a value from the groupby operation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequently it is useful to generate a new column in the existing dataframe based on a groupby operation.  \n",
    "This operation is generally called a `transform` operation. \n",
    "\n",
    "For example, let's say we want to group the data by trunk size and we want to create a new column which has the weight as a ratio of the group mean. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 1: using `transform`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>weight_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>1.168350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>0.828539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>0.925926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>0.905724</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    trunk  weight  weight_ratio\n",
       "55    6.0  2370.0      1.000000\n",
       "47    7.0  3470.0      1.168350\n",
       "23    9.0  1800.0      0.828539\n",
       "17    7.0  2750.0      0.925926\n",
       "51    7.0  2690.0      0.905724"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['weight_ratio'] = df_auto.groupby('trunk')['weight'].transform(lambda x: x / x.mean())\n",
    "df_auto[['trunk', 'weight', 'weight_ratio']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 2: manually using `.merge()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `transform` method can sometimes become hard to use, in those cases you might have to resort to just doing it manually using a `.merge` operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: this new DF only has 18 rows!\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight_group_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.0</td>\n",
       "      <td>1760.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   trunk  weight_group_mean\n",
       "0    5.0             1760.0\n",
       "1    6.0             2370.0"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_per_group = df_auto.groupby('trunk').apply(lambda df: df['weight'].mean()).reset_index()\n",
    "mean_per_group = mean_per_group.rename(columns={0 : 'weight_group_mean'})\n",
    "print('Note: this new DF only has {} rows!'.format(len(mean_per_group)))\n",
    "mean_per_group.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trunk</th>\n",
       "      <th>weight</th>\n",
       "      <th>weight_group_mean</th>\n",
       "      <th>weight_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>2370.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.0</td>\n",
       "      <td>3470.0</td>\n",
       "      <td>2970.0</td>\n",
       "      <td>1.168350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>1800.0</td>\n",
       "      <td>2172.5</td>\n",
       "      <td>0.828539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2750.0</td>\n",
       "      <td>2970.0</td>\n",
       "      <td>0.925926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2690.0</td>\n",
       "      <td>2970.0</td>\n",
       "      <td>0.905724</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   trunk  weight  weight_group_mean  weight_ratio\n",
       "0    6.0  2370.0             2370.0      1.000000\n",
       "1    7.0  3470.0             2970.0      1.168350\n",
       "2    9.0  1800.0             2172.5      0.828539\n",
       "3    7.0  2750.0             2970.0      0.925926\n",
       "4    7.0  2690.0             2970.0      0.905724"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp_df = pd.merge(df_auto, mean_per_group, on='trunk', how='left')[['trunk', 'weight', 'weight_group_mean']]\n",
    "tmp_df['weight_ratio'] = tmp_df['weight'] / tmp_df['weight_group_mean']\n",
    "tmp_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And a lot of other groupby operations!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many-many more things you can do with Pandas `.groupby`, too much to show here.  \n",
    "Feel free to check out the comprehensive documentation:  \n",
    "https://pandas.pydata.org/pandas-docs/stable/groupby.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Reshaping and Pivot Tables</span><a id='reshaping-pivot'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas includes a variety of tools that allow you to reshape your DataFrame.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create some sample data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuples = [('bar', 'one',   1, 10),\n",
    "          ('bar', 'two',   2, 11),\n",
    "          ('bar', 'three', 3, 12),\n",
    "          ('baz', 'one',   4, 13),\n",
    "          ('baz', 'two',   5, 14),\n",
    "          ('baz', 'three', 6, 15),\n",
    "          ('foo', 'one',   7, 16),\n",
    "          ('foo', 'two',   8, 17),\n",
    "          ('foo', 'three', 9, 18)\n",
    "         ]\n",
    "df = pd.DataFrame(tuples)\n",
    "df.columns = ['first', 'second', 'A', 'B']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>second</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bar</td>\n",
       "      <td>one</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bar</td>\n",
       "      <td>two</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bar</td>\n",
       "      <td>three</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>baz</td>\n",
       "      <td>one</td>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>baz</td>\n",
       "      <td>two</td>\n",
       "      <td>5</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>baz</td>\n",
       "      <td>three</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>foo</td>\n",
       "      <td>one</td>\n",
       "      <td>7</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>foo</td>\n",
       "      <td>two</td>\n",
       "      <td>8</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>foo</td>\n",
       "      <td>three</td>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first second  A   B\n",
       "0   bar    one  1  10\n",
       "1   bar    two  2  11\n",
       "2   bar  three  3  12\n",
       "3   baz    one  4  13\n",
       "4   baz    two  5  14\n",
       "5   baz  three  6  15\n",
       "6   foo    one  7  16\n",
       "7   foo    two  8  17\n",
       "8   foo  three  9  18"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1) Create a pivot table (i.e. long to wide)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `pivot()` function:  \n",
    "http://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-by-pivoting-dataframe-objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "second  one  three  two\n",
       "first                  \n",
       "bar       1      3    2\n",
       "baz       4      6    5\n",
       "foo       7      9    8"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.pivot(index='first', columns='second', values='A')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `pd.pivot_table()` function:  \n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.pivot_table.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>13</td>\n",
       "      <td>15</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>18</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A             B          \n",
       "second one three two one three two\n",
       "first                             \n",
       "bar      1     3   2  10    12  11\n",
       "baz      4     6   5  13    15  14\n",
       "foo      7     9   8  16    18  17"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.pivot_table(df, values=['A', 'B'], index='first', columns='second')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note 1:* the above illustrates that Pandas essentially has two indexes: the usual 'row index' but also a 'column index'  \n",
    "\n",
    "Pandas also has an \"unpivot\" function called `pandas.melt` (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   A  B  C\n",
       "0  1  2  3\n",
       "1  4  5  6\n",
       "2  7  8  9"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wide_df = pd.DataFrame([(1, 2, 3), (4, 5, 6), (7, 8 ,9)], columns = ['A', 'B', 'C'])\n",
    "wide_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>variable</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>B</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>B</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "      <td>B</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>C</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>C</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7</td>\n",
       "      <td>C</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   A variable  value\n",
       "0  1        B      2\n",
       "1  4        B      5\n",
       "2  7        B      8\n",
       "3  1        C      3\n",
       "4  4        C      6\n",
       "5  7        C      9"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.melt(wide_df, id_vars = ['A'], value_vars=['B', 'C'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Stack and Unstack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Stack` and `Unstack` are higher level operators to reshape a dataframe based on a multi-level index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the documentation:  \n",
    ">stack: “pivot” a level of the (possibly hierarchical) column labels, returning a DataFrame with an index with a new inner-most level of row labels.  \n",
    "unstack: inverse operation from stack: “pivot” a level of the (possibly hierarchical) row index to the column axis, producing a reshaped DataFrame with a new inner-most level of column labels.\n",
    "\n",
    "http://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-by-stacking-and-unstacking\n",
    "\n",
    "In other words:  \n",
    "**Stack** --> move the data \"down\"  \n",
    "**Unstack** --> move the data \"up\"  \n",
    "\n",
    "These operations are primarily useful when moving complex data from wide to long or long to wide format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>13</td>\n",
       "      <td>15</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>18</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A             B          \n",
       "second one three two one three two\n",
       "first                             \n",
       "bar      1     3   2  10    12  11\n",
       "baz      4     6   5  13    15  14\n",
       "foo      7     9   8  16    18  17"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivot_df = pd.pivot_table(df, values=['A', 'B'], index='first', columns='second')\n",
    "pivot_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th>second</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">bar</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">baz</th>\n",
       "      <th>one</th>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>5</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">foo</th>\n",
       "      <th>one</th>\n",
       "      <td>7</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>8</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              A   B\n",
       "first second       \n",
       "bar   one     1  10\n",
       "      three   3  12\n",
       "      two     2  11\n",
       "baz   one     4  13\n",
       "      three   6  15\n",
       "      two     5  14\n",
       "foo   one     7  16\n",
       "      three   9  18\n",
       "      two     8  17"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivot_df.stack(level=['second'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note* We could also just use `pivot_df.stack()` as it will by default choose the 'last' level of the index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unstack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_index(['first', 'second'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th>second</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">bar</th>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">baz</th>\n",
       "      <th>one</th>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>5</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">foo</th>\n",
       "      <th>one</th>\n",
       "      <td>7</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>8</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              A   B\n",
       "first second       \n",
       "bar   one     1  10\n",
       "      two     2  11\n",
       "      three   3  12\n",
       "baz   one     4  13\n",
       "      two     5  14\n",
       "      three   6  15\n",
       "foo   one     7  16\n",
       "      two     8  17\n",
       "      three   9  18"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th>bar</th>\n",
       "      <th>baz</th>\n",
       "      <th>foo</th>\n",
       "      <th>bar</th>\n",
       "      <th>baz</th>\n",
       "      <th>foo</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>one</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>13</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>three</th>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "      <td>15</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>two</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>11</td>\n",
       "      <td>14</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A           B        \n",
       "first  bar baz foo bar baz foo\n",
       "second                        \n",
       "one      1   4   7  10  13  16\n",
       "three    3   6   9  12  15  18\n",
       "two      2   5   8  11  14  17"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.unstack(level=['first'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">A</th>\n",
       "      <th colspan=\"3\" halign=\"left\">B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "      <th>one</th>\n",
       "      <th>three</th>\n",
       "      <th>two</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>first</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>baz</th>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>13</td>\n",
       "      <td>15</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>18</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         A             B          \n",
       "second one three two one three two\n",
       "first                             \n",
       "bar      1     3   2  10    12  11\n",
       "baz      4     6   5  13    15  14\n",
       "foo      7     9   8  16    18  17"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.unstack(level=['second'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Dealing with dates</span><a id='dates'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas has a lot of functionality to deal with timeseries data  \n",
    "http://pandas.pydata.org/pandas-docs/stable/timeseries.html\n",
    "\n",
    "A nice overview from the documentation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Class           | My interpretation         |        Example          | How to create:                               |\n",
    "|-----------------|---------------------------|------|----------------------------------------------|\n",
    "| `Timestamp`     | Specifiek date and time   | `2017-01-01 03:00:00`           | `to_datetime`, `date_range`                  |\n",
    "| `Timedelta`     | An absolute time duration | `5 hours`          | `to_timedelta`, `timedelta_range`            |\n",
    "| `Period`        | Regular intervals of time | `daily` |`Period`, `period_range`                   |\n",
    "| `DateOffset`    | Similar to `timedelta` but follows calendar rules | `same time next day`  | `DateOffset`                |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make an example DF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ad = df_auto.copy()[['make', 'price']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Create a `date` column with the right dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we load in the data and all the dates are imported as strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ad['date'] = ['01-01-2018', '30-05-2020', '10-10-2019', '05-04-2013', '05-07-2015']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['date'].dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert the dates to a date dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "make             object\n",
       "price           float64\n",
       "date     datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['date'] = pd.to_datetime(df_ad['date'], dayfirst=True)\n",
    "df_ad.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55   2018-01-01\n",
       "47   2020-05-30\n",
       "23   2019-10-10\n",
       "17   2013-04-05\n",
       "51   2015-07-05\n",
       "Name: date, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Select observations that fall in a certain range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `pd.to_datetime` to create a datetime object that we can use in a condition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2011-02-01 00:00:00')"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Timestamp('2011-02-01')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* it is usually a good idea to explicitly include the format, to avoid unexpected behavior"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2011-02-01 00:00:00')"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_datetime('01-02-2011', format='%d-%m-%Y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select all observations with a date later than Jan 1st 2016:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2018-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2020-05-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2019-10-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date\n",
       "55      Datsun 200  6229.0 2018-01-01\n",
       "47  Pont. Firebird  4934.0 2020-05-30\n",
       "23     Ford Fiesta  4389.0 2019-10-10"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad[df_ad.date > pd.to_datetime('01-01-2016', format='%d-%m-%Y')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use `isin()` with a `date_range` object to get all observations in a range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2018-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>2015-07-05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             make   price       date\n",
       "55     Datsun 200  6229.0 2018-01-01\n",
       "51  Pont. Sunbird  4172.0 2015-07-05"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_date = pd.to_datetime('01-01-2015', format='%d-%m-%Y')\n",
    "end_date = pd.to_datetime('31-12-2018', format='%d-%m-%Y')\n",
    "df_ad[df_ad.date.isin(pd.date_range(start_date, end_date, freq='D'))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Date ranges:\n",
    "You can use `date_range` to create an array of datetime objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03', '2011-01-04',\n",
       "               '2011-01-05', '2011-01-06', '2011-01-07', '2011-01-08',\n",
       "               '2011-01-09', '2011-01-10', '2011-01-11', '2011-01-12',\n",
       "               '2011-01-13', '2011-01-14', '2011-01-15', '2011-01-16',\n",
       "               '2011-01-17', '2011-01-18', '2011-01-19', '2011-01-20',\n",
       "               '2011-01-21', '2011-01-22', '2011-01-23', '2011-01-24',\n",
       "               '2011-01-25', '2011-01-26', '2011-01-27', '2011-01-28',\n",
       "               '2011-01-29', '2011-01-30', '2011-01-31', '2011-02-01',\n",
       "               '2011-02-02', '2011-02-03', '2011-02-04', '2011-02-05',\n",
       "               '2011-02-06', '2011-02-07', '2011-02-08', '2011-02-09',\n",
       "               '2011-02-10', '2011-02-11', '2011-02-12', '2011-02-13',\n",
       "               '2011-02-14', '2011-02-15', '2011-02-16', '2011-02-17',\n",
       "               '2011-02-18', '2011-02-19', '2011-02-20', '2011-02-21',\n",
       "               '2011-02-22', '2011-02-23', '2011-02-24', '2011-02-25',\n",
       "               '2011-02-26', '2011-02-27', '2011-02-28', '2011-03-01',\n",
       "               '2011-03-02', '2011-03-03', '2011-03-04', '2011-03-05',\n",
       "               '2011-03-06', '2011-03-07', '2011-03-08', '2011-03-09',\n",
       "               '2011-03-10'],\n",
       "              dtype='datetime64[ns]', freq='D')"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.date_range('1/1/2011', periods=len(df_auto.index), freq='D')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['2015-01-01', '2015-01-02', '2015-01-03', '2015-01-04',\n",
       "               '2015-01-05', '2015-01-06', '2015-01-07', '2015-01-08',\n",
       "               '2015-01-09', '2015-01-10',\n",
       "               ...\n",
       "               '2018-12-22', '2018-12-23', '2018-12-24', '2018-12-25',\n",
       "               '2018-12-26', '2018-12-27', '2018-12-28', '2018-12-29',\n",
       "               '2018-12-30', '2018-12-31'],\n",
       "              dtype='datetime64[ns]', length=1461, freq='D')"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_date = pd.to_datetime('01-01-2015', format='%d-%m-%Y')\n",
    "end_date = pd.to_datetime('31-12-2018', format='%d-%m-%Y')\n",
    "pd.date_range(start_date, end_date, freq='D')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Select components of the dates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access special date operations by using the `.dt.` accessor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See: https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#basics-dt-accessors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2020-05-30</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2019-10-10</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>2015-07-05</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date  day\n",
       "55      Datsun 200  6229.0 2018-01-01    1\n",
       "47  Pont. Firebird  4934.0 2020-05-30   30\n",
       "23     Ford Fiesta  4389.0 2019-10-10   10\n",
       "17     Chev. Monza  3667.0 2013-04-05    5\n",
       "51   Pont. Sunbird  4172.0 2015-07-05    5"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['day'] =  df_ad['date'].dt.day\n",
    "df_ad.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2020-05-30</td>\n",
       "      <td>30</td>\n",
       "      <td>2020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2019-10-10</td>\n",
       "      <td>10</td>\n",
       "      <td>2019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>2015-07-05</td>\n",
       "      <td>5</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date  day  year\n",
       "55      Datsun 200  6229.0 2018-01-01    1  2018\n",
       "47  Pont. Firebird  4934.0 2020-05-30   30  2020\n",
       "23     Ford Fiesta  4389.0 2019-10-10   10  2019\n",
       "17     Chev. Monza  3667.0 2013-04-05    5  2013\n",
       "51   Pont. Sunbird  4172.0 2015-07-05    5  2015"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['year'] =  df_ad['date'].dt.year\n",
    "df_ad.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Manipulate (off-set) the date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects\n",
    "\n",
    "Add 1 year to each date:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>year</th>\n",
       "      <th>new_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>2018</td>\n",
       "      <td>2019-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2020-05-30</td>\n",
       "      <td>30</td>\n",
       "      <td>2020</td>\n",
       "      <td>2021-05-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2019-10-10</td>\n",
       "      <td>10</td>\n",
       "      <td>2019</td>\n",
       "      <td>2020-10-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>2014-04-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>2015-07-05</td>\n",
       "      <td>5</td>\n",
       "      <td>2015</td>\n",
       "      <td>2016-07-05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date  day  year   new_date\n",
       "55      Datsun 200  6229.0 2018-01-01    1  2018 2019-01-01\n",
       "47  Pont. Firebird  4934.0 2020-05-30   30  2020 2021-05-30\n",
       "23     Ford Fiesta  4389.0 2019-10-10   10  2019 2020-10-10\n",
       "17     Chev. Monza  3667.0 2013-04-05    5  2013 2014-04-05\n",
       "51   Pont. Sunbird  4172.0 2015-07-05    5  2015 2016-07-05"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['new_date'] = df_ad.date.apply(lambda x: x + pd.DateOffset(years=1))\n",
    "df_ad.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add 2 business days to each date:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>year</th>\n",
       "      <th>new_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>6229.0</td>\n",
       "      <td>2018-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018-01-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>4934.0</td>\n",
       "      <td>2020-05-30</td>\n",
       "      <td>30</td>\n",
       "      <td>2020</td>\n",
       "      <td>2020-06-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>4389.0</td>\n",
       "      <td>2019-10-10</td>\n",
       "      <td>10</td>\n",
       "      <td>2019</td>\n",
       "      <td>2019-10-14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>3667.0</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>2013-04-09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>4172.0</td>\n",
       "      <td>2015-07-05</td>\n",
       "      <td>5</td>\n",
       "      <td>2015</td>\n",
       "      <td>2015-07-07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              make   price       date  day  year   new_date\n",
       "55      Datsun 200  6229.0 2018-01-01    1  2018 2018-01-03\n",
       "47  Pont. Firebird  4934.0 2020-05-30   30  2020 2020-06-02\n",
       "23     Ford Fiesta  4389.0 2019-10-10   10  2019 2019-10-14\n",
       "17     Chev. Monza  3667.0 2013-04-05    5  2013 2013-04-09\n",
       "51   Pont. Sunbird  4172.0 2015-07-05    5  2015 2015-07-07"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ad['new_date'] = df_ad.date.apply(lambda x: x + 2* pd.offsets.BDay())\n",
    "df_ad.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Dealing with strings</span><a id='strings'></a> [(to top)](#toc)  \n",
    "Pandas has a variety of built-in operations that you can use to modify string values:  \n",
    "https://pandas.pydata.org/pandas-docs/stable/reference/series.html#api-series-str\n",
    "\n",
    "You can access the functions by using the `.str.` accessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example:** Get the length (i.e. number of characters) for each make:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>make_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Olds Omega</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Olds Cutlass</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dodge St. Regis</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Plym. Volare</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>69 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               make  make_len\n",
       "55       Datsun 200        10\n",
       "47   Pont. Firebird        14\n",
       "23      Ford Fiesta        11\n",
       "17      Chev. Monza        11\n",
       "51    Pont. Sunbird        13\n",
       "..              ...       ...\n",
       "38       Olds Omega        10\n",
       "3     Buick Century        13\n",
       "36     Olds Cutlass        12\n",
       "22  Dodge St. Regis        15\n",
       "45     Plym. Volare        12\n",
       "\n",
       "[69 rows x 2 columns]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['make_len'] = df_auto['make'].str.len()\n",
    "df_auto[['make', 'make_len']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example:** Convert the make to all uppercase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>make_upper</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>DATSUN 200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>PONT. FIREBIRD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>FORD FIESTA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>CHEV. MONZA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>PONT. SUNBIRD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Olds Omega</td>\n",
       "      <td>OLDS OMEGA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>BUICK CENTURY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Olds Cutlass</td>\n",
       "      <td>OLDS CUTLASS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dodge St. Regis</td>\n",
       "      <td>DODGE ST. REGIS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Plym. Volare</td>\n",
       "      <td>PLYM. VOLARE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>69 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               make       make_upper\n",
       "55       Datsun 200       DATSUN 200\n",
       "47   Pont. Firebird   PONT. FIREBIRD\n",
       "23      Ford Fiesta      FORD FIESTA\n",
       "17      Chev. Monza      CHEV. MONZA\n",
       "51    Pont. Sunbird    PONT. SUNBIRD\n",
       "..              ...              ...\n",
       "38       Olds Omega       OLDS OMEGA\n",
       "3     Buick Century    BUICK CENTURY\n",
       "36     Olds Cutlass     OLDS CUTLASS\n",
       "22  Dodge St. Regis  DODGE ST. REGIS\n",
       "45     Plym. Volare     PLYM. VOLARE\n",
       "\n",
       "[69 rows x 2 columns]"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['make_upper'] = df_auto['make'].str.upper()\n",
    "df_auto[['make', 'make_upper']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example:** Replace all whitespace characters with `-`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>make</th>\n",
       "      <th>make_no_ws</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Datsun 200</td>\n",
       "      <td>Datsun-200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Pont. Firebird</td>\n",
       "      <td>Pont.-Firebird</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Ford Fiesta</td>\n",
       "      <td>Ford-Fiesta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Chev. Monza</td>\n",
       "      <td>Chev.-Monza</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Pont. Sunbird</td>\n",
       "      <td>Pont.-Sunbird</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Olds Omega</td>\n",
       "      <td>Olds-Omega</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Buick Century</td>\n",
       "      <td>Buick-Century</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Olds Cutlass</td>\n",
       "      <td>Olds-Cutlass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dodge St. Regis</td>\n",
       "      <td>Dodge-St.-Regis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Plym. Volare</td>\n",
       "      <td>Plym.-Volare</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>69 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               make       make_no_ws\n",
       "55       Datsun 200       Datsun-200\n",
       "47   Pont. Firebird   Pont.-Firebird\n",
       "23      Ford Fiesta      Ford-Fiesta\n",
       "17      Chev. Monza      Chev.-Monza\n",
       "51    Pont. Sunbird    Pont.-Sunbird\n",
       "..              ...              ...\n",
       "38       Olds Omega       Olds-Omega\n",
       "3     Buick Century    Buick-Century\n",
       "36     Olds Cutlass     Olds-Cutlass\n",
       "22  Dodge St. Regis  Dodge-St.-Regis\n",
       "45     Plym. Volare     Plym.-Volare\n",
       "\n",
       "[69 rows x 2 columns]"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_auto['make_no_ws'] = df_auto['make'].str.replace(' ', '-')\n",
    "df_auto[['make', 'make_no_ws']]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}