Pandas Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Pandas\n",
    "\n",
    "[Pandas](https://pandas.pydata.org) is designed to make data pre-processing and data analysis fast and easy in Python. At the very basic level, Pandas objects can be thought of as enhanced versions of NumPy structured arrays in which the rows and columns are identified with labels rather than simple integer indices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To use Pandas, you need to import the `pandas` module, using for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T12:57:09.909716Z",
     "start_time": "2020-06-24T12:57:02.830486Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:15:18.240575Z",
     "start_time": "2020-06-24T13:15:18.229186Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# This line imports the NumPy package\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Pandas has two main data structures, **Series** and **DataFrame**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Pandas `Series` object\n",
    "\n",
    "\n",
    "A `Series` is a single dimension array-like object containing a *sequence of values*, with an array of *data labels*, called its **index**.\n",
    "\n",
    "A Series can be created easily from a Python list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:06:29.883282Z",
     "start_time": "2020-06-24T14:06:29.873731Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    4\n",
      "1    8\n",
      "2    1\n",
      "3    3\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "ts = pd.Series([4, 8, 1, 3])\n",
    "print(ts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As we see in the output, the `Series` wraps both a sequence of values and a sequence of indices, which we can access with the `values` and `index` attributes. The values are simply a familiar NumPy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:06:30.975916Z",
     "start_time": "2020-06-24T14:06:30.966676Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 8, 1, 3])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ts.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `index` is an array-like object of type `pd.Index`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=4, step=1)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ts.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The labels in the index can be used to select values in the `Series` through the index operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:13:05.940761Z",
     "start_time": "2020-06-24T13:13:05.922266Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "print(ts[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2    1\n",
      "3    3\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(ts[2:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The essential difference between a NumPy 1D array and a Pandas `Series` is the presence of the index: while the NumPy array has an implicitly defined integer index used to access the values, the Pandas `Series` has an explicitly defined index associated with the values.\n",
    "\n",
    "This explicit index definition gives the `Series` object additional capabilities. For example, the index need not be an integer, but can consist of values of any desired type. For example, if we wish, we can use strings as an index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:10:32.009532Z",
     "start_time": "2020-06-24T13:10:31.999642Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first     4\n",
      "second    8\n",
      "third     1\n",
      "fourth    3\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "ts = pd.Series([4, 8, 1, 3], index=['first', 'second', 'third', 'fourth'])\n",
    "print(ts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And the item access works as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "print(ts['fourth'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:13:05.940761Z",
     "start_time": "2020-06-24T13:13:05.922266Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first     4\n",
      "second    8\n",
      "third     1\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(ts['first':'third'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:13:05.940761Z",
     "start_time": "2020-06-24T13:13:05.922266Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "second    8\n",
      "fourth    3\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(ts[['second', 'fourth']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can think about a `Series` as a kind of fixed-length, ordered Python's `dict`, mapping index values to data values. In fact, it is possible to create a `Series` directlty from a Python's `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:18:31.486140Z",
     "start_time": "2020-06-24T13:18:31.465527Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pisa       80\n",
      "London    300\n",
      "Paris       1\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "my_dict = {'Pisa': 80, 'London': 300, 'Paris': 1}\n",
    "ts = pd.Series(my_dict)\n",
    "print(ts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Constructing Series\n",
    "\n",
    "`Series` objects can be constructed in many different ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1    4\n",
       "2    6\n",
       "dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series([2, 4, 6])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100    5\n",
       "200    5\n",
       "300    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(5, index=[100, 200, 300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    a\n",
       "1    b\n",
       "3    c\n",
       "dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series({2:'a', 1:'b', 3:'c'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    c\n",
       "2    a\n",
       "dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series({2:'a', 1:'b', 3:'c'}, index=[3, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using Series\n",
    "\n",
    "Using NumPy functions or NumPy-like operations, such as boolean indexing, universal functions, and so on, will preserve the indexes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:15:28.579927Z",
     "start_time": "2020-06-24T13:15:28.567266Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pisa       80\n",
      "London    300\n",
      "dtype: int64\n",
      "Pisa       5.540622e+34\n",
      "London    1.942426e+130\n",
      "Paris      2.718282e+00\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(ts[ts > 3])\n",
    "print(np.exp(ts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Arithmetic operations on `Series` are automatically aligned on the index labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:29:55.852232Z",
     "start_time": "2020-06-24T13:29:55.833699Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first      8.0\n",
      "fourth     NaN\n",
      "pisa       NaN\n",
      "second    16.0\n",
      "third      NaN\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "ts1 = pd.Series([4, 8, 1, 3], index=['first', 'second', 'third', 'fourth'])\n",
    "ts2 = pd.Series([4, 8, 1], index=['first', 'second', 'pisa'])\n",
    "ts_sum = ts1 + ts2\n",
    "print(ts_sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here two index values are correctly computed (corresponding to the label `first` and `second`). The two other index labels `third` and `fourth` in `ts1` are missing in `ts2`, as well as the `pisa` index label in `ts2`. Hence, for each of these index label, a `NaN` value (*not a number*) appears, which Pandas considers as a **missing value**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `pd.isnull` and `pd.notnull` functions detects missing data, as well as the corresponding instance methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:31:20.930884Z",
     "start_time": "2020-06-24T13:31:20.911850Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first     False\n",
      "fourth     True\n",
      "pisa       True\n",
      "second    False\n",
      "third      True\n",
      "dtype: bool\n",
      "first      True\n",
      "fourth    False\n",
      "pisa      False\n",
      "second     True\n",
      "third     False\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print(pd.isnull(ts_sum))\n",
    "print(ts_sum.notnull())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# DataFrame\n",
    "\n",
    "A `DataFrame` is a rectangular table of data. It contains an ordered list of columns. Every column can be of a different value type. A `DataFrame` has both a *row index* and a *column index*. It can be thought as a dict of `Series` (one per column) all sharing the same index labels.\n",
    "\n",
    "If a Series is an analog of a one-dimensional array with flexible indices, a DataFrame is an analog of a two-dimensional array with both flexible row indices and flexible column names. Just as you might think of a two-dimensional array as an ordered sequence of aligned one-dimensional columns, you can think of a DataFrame as a sequence of aligned Series objects. Here, by \"aligned\" we mean that they share the same index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There are many way to construct a `DataFrame`, but the most common is using a dictionary of equally-sized Python's lists (or NumPy's arrays):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:46:58.794613Z",
     "start_time": "2020-06-24T13:46:58.779817Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            Brand  Price\n",
      "0     Honda Civic  22000\n",
      "1  Toyota Corolla  25000\n",
      "2      Ford Focus  27000\n",
      "3         Audi A4  35000\n"
     ]
    }
   ],
   "source": [
    "cars = {'Brand': ['Honda Civic', 'Toyota Corolla', 'Ford Focus', 'Audi A4'],\n",
    "        'Price': [22000, 25000, 27000, 35000]}\n",
    "\n",
    "df = pd.DataFrame(cars)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Given a two-dimensional array of data, such as a NumPy two-dimensional array, we can create a `DataFrame` with any specified column and index names. If omitted, an integer index will be used for each:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        foo       bar\n",
      "a  0.455037  0.286076\n",
      "b  0.600862  0.805786\n",
      "c  0.178555  0.432043\n"
     ]
    }
   ],
   "source": [
    "df = pd.DataFrame(np.random.rand(3, 2),\n",
    "                  columns=['foo', 'bar'],\n",
    "                  index=['a', 'b', 'c'])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To pretty-print a `DataFrame` in a Jupyter notebooks, it is enough to write its name (or using the `head()` instance method for very long `DataFrame`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>foo</th>\n",
       "      <th>bar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.455037</td>\n",
       "      <td>0.286076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>0.600862</td>\n",
       "      <td>0.805786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>0.178555</td>\n",
       "      <td>0.432043</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        foo       bar\n",
       "a  0.455037  0.286076\n",
       "b  0.600862  0.805786\n",
       "c  0.178555  0.432043"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:48:19.586597Z",
     "start_time": "2020-06-24T13:48:19.564159Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>foo</th>\n",
       "      <th>bar</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.455037</td>\n",
       "      <td>0.286076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>0.600862</td>\n",
       "      <td>0.805786</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        foo       bar\n",
       "a  0.455037  0.286076\n",
       "b  0.600862  0.805786"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It is possible to change the order of the colums at `DataFrame` construction time. If you provide a column's name not included in the dictionary, a column with missing values will appear:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T13:50:34.030261Z",
     "start_time": "2020-06-24T13:50:34.008485Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Color</th>\n",
       "      <th>Price</th>\n",
       "      <th>Brand</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>22000</td>\n",
       "      <td>Honda Civic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>25000</td>\n",
       "      <td>Toyota Corolla</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>27000</td>\n",
       "      <td>Ford Focus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>35000</td>\n",
       "      <td>Audi A4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Color  Price           Brand\n",
       "0   NaN  22000     Honda Civic\n",
       "1   NaN  25000  Toyota Corolla\n",
       "2   NaN  27000      Ford Focus\n",
       "3   NaN  35000         Audi A4"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(cars, columns=['Color', 'Price', 'Brand'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If working with a large table, it might be useful to sometimes have a list of all the columns' names. This is given by the `keys()` methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:19:07.147448Z",
     "start_time": "2020-06-24T14:19:07.137275Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Color', 'Price', 'Brand'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(df.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another common way to create a DataFrame is to use a *nested dict of dicts*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:27:29.650782Z",
     "start_time": "2020-06-24T14:27:29.643762Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "pop = {'Nevada': {2001: 2.4, 2002: 2.9},\n",
    "       'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If this nested dict is passed to the DataFrame, the outer dict keys are interpreted as column labels, and the inner keys are interpreted as row labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:28:37.554310Z",
     "start_time": "2020-06-24T14:28:37.535827Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Nevada</th>\n",
       "      <th>Ohio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2001</th>\n",
       "      <td>2.4</td>\n",
       "      <td>1.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002</th>\n",
       "      <td>2.9</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Nevada  Ohio\n",
       "2001     2.4   1.7\n",
       "2002     2.9   3.6\n",
       "2000     NaN   1.5"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(pop)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Accessing a DataFrame\n",
    "\n",
    "We now create a more complex DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:10:32.187393Z",
     "start_time": "2020-06-24T14:10:32.162017Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city\n",
       "Mahler         1860   1911           Kaliste\n",
       "Beethoven      1770   1827              Bonn\n",
       "Puccini        1858   1924           Lucques\n",
       "Shostakovich   1906   1975  Saint-Petersburg"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict_of_list = {'birth': [1860, 1770, 1858, 1906], \n",
    "                'death':[1911, 1827, 1924, 1975], \n",
    "                'city':['Kaliste', 'Bonn', 'Lucques', 'Saint-Petersburg']}\n",
    "composers_df = pd.DataFrame(dict_of_list, index=['Mahler', 'Beethoven', 'Puccini', 'Shostakovich'])\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are multiple ways of accessing values or series of values in a Dataframe. Unlike in Series, a simple bracket gives access to a column and not an index, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:11:08.259631Z",
     "start_time": "2020-06-24T14:11:08.241496Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Mahler                   Kaliste\n",
       "Beethoven                   Bonn\n",
       "Puccini                  Lucques\n",
       "Shostakovich    Saint-Petersburg\n",
       "Name: city, dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df['city']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "returns a Series. Alternatively one can also use the attributes syntax and access columns by using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:11:37.321286Z",
     "start_time": "2020-06-24T14:11:37.309806Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Mahler                   Kaliste\n",
       "Beethoven                   Bonn\n",
       "Puccini                  Lucques\n",
       "Shostakovich    Saint-Petersburg\n",
       "Name: city, dtype: object"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df.city"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The attributes syntax has some limitations, so in case something does not work as expected, revert to the brackets notation.\n",
    "\n",
    "When specifiying multiple columns, a DataFrame is returned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:12:06.558313Z",
     "start_time": "2020-06-24T14:12:06.540600Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>birth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>Kaliste</td>\n",
       "      <td>1860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>Bonn</td>\n",
       "      <td>1770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>Lucques</td>\n",
       "      <td>1858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>1906</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          city  birth\n",
       "Mahler                 Kaliste   1860\n",
       "Beethoven                 Bonn   1770\n",
       "Puccini                Lucques   1858\n",
       "Shostakovich  Saint-Petersburg   1906"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df[['city', 'birth']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "One of the important differences with a regular Numpy array is that **with Pandas' DataFrame regular indexing doesn't work**. Instead one has to use either the `iloc` or the `loc` attributes.\n",
    "\n",
    "**Remember that `loc` and `iloc` are attributes, not methods, hence they use brackets `[]` and not parenthesis `()`.**\n",
    "\n",
    "The `loc` attribute allows to recover elements by using the index labels, while the `iloc` attribute can be used to recover the regular indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:16:36.990555Z",
     "start_time": "2020-06-24T14:16:36.979449Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1911\n",
      "1911\n"
     ]
    }
   ],
   "source": [
    "print(composers_df.iloc[0,1])\n",
    "print(composers_df.loc['Mahler', 'death'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Adding columns\n",
    "\n",
    "It is very simple to add a column to a Dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:25:29.267510Z",
     "start_time": "2020-06-24T14:25:29.253143Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>???</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>???</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>???</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>???</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city country\n",
       "Mahler         1860   1911           Kaliste     ???\n",
       "Beethoven      1770   1827              Bonn     ???\n",
       "Puccini        1858   1924           Lucques     ???\n",
       "Shostakovich   1906   1975  Saint-Petersburg     ???"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df['country'] = '???'\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Alternatively, an existing list can be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:25:30.513553Z",
     "start_time": "2020-06-24T14:25:30.500562Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>country</th>\n",
       "      <th>country2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>???</td>\n",
       "      <td>Austria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>???</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>???</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>???</td>\n",
       "      <td>Russia</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city country country2\n",
       "Mahler         1860   1911           Kaliste     ???  Austria\n",
       "Beethoven      1770   1827              Bonn     ???  Germany\n",
       "Puccini        1858   1924           Lucques     ???    Italy\n",
       "Shostakovich   1906   1975  Saint-Petersburg     ???   Russia"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df['country2'] = ['Austria','Germany','Italy','Russia']\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Deleting columns\n",
    "\n",
    "The `del`keyword is used to delete columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:25:32.996744Z",
     "start_time": "2020-06-24T14:25:32.983364Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>country2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>Austria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>Russia</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city country2\n",
       "Mahler         1860   1911           Kaliste  Austria\n",
       "Beethoven      1770   1827              Bonn  Germany\n",
       "Puccini        1858   1924           Lucques    Italy\n",
       "Shostakovich   1906   1975  Saint-Petersburg   Russia"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del composers_df['country']\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Handling Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Python missing data: `None`\n",
    "\n",
    "`None`is a Python singleton object that is often used for missing data in Python code. Because it is a Python *object*, `None` cannot be used in any arbitrary NumPy/Pandas array, but only in arrays with data type '`object'` (i.e., arrays of Python objects)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Python missing numbers: `NaN`\n",
    "\n",
    "`NaN` (acronym for *Not a Number*) is a special floating-point value recognized by all systems that use the standard IEEE floating-point representation.\n",
    "\n",
    "`NaN` is a bit like a data virus–it infects any other object it touches. Regardless of the operation, the result of arithmetic with `NaN` will be another `NaN`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 + np.nan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `None` and `NaN` in Pandas\n",
    "\n",
    "Pandas is built to handle the two of them nearly interchangeably, converting between them where appropriate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    NaN\n",
       "2    2.0\n",
       "3    NaN\n",
       "dtype: float64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series([1, np.nan, 2, None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Working with Missing Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There are several useful methods for detecting, removing, and replacing null values in Pandas data structures. \n",
    "\n",
    "They are:\n",
    "\n",
    "* `isnull()`: Generate a boolean mask indicating missing values\n",
    "* `notnull()`: Opposite of `isnull()`\n",
    "* `dropna()`: Return a filtered version of the data\n",
    "* `fillna()`: Return a copy of the data with missing values filled or imputed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "data = pd.Series([1, np.nan, 'hello', None])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2    False\n",
       "3     True\n",
       "dtype: bool"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "2    hello\n",
       "dtype: object"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data.notnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "0  1.0  NaN  2\n",
       "1  2.0  3.0  5\n",
       "2  NaN  4.0  6"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame([[1,      np.nan, 2],\n",
    "                   [2,      3,      5],\n",
    "                   [np.nan, 4,      6]])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "1  2.0  3.0  5"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.dropna()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Importing Excel files as DataFrames\n",
    "\n",
    "Another very common way of \"creating\" a Pandas Dataframe is by importing a table from another format like CSV or Excel. In order to import Excel files, you need to install the `openpyxl` package:\n",
    "\n",
    "```shell\n",
    "pip install openpyxl\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mDEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621\u001b[0m\u001b[33m\n",
      "\u001b[0mRequirement already satisfied: openpyxl in /usr/local/lib/python3.9/site-packages (3.0.7)\n",
      "Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.9/site-packages (from openpyxl) (1.0.1)\n",
      "\u001b[33mDEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: You are using pip version 22.0.3; however, version 22.0.4 is available.\n",
      "You should consider upgrading via the '/usr/local/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install openpyxl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "An Excel table is provided in the [composers.xlsx](data/composers.xlsx) file and can be read with the `pd.read_excel` function. There are many more readers for other types of data (csv, json, html etc.) but we focus here on Excel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:46:22.696917Z",
     "start_time": "2020-06-24T14:46:22.651062Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>composer</th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mahler</td>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Beethoven</td>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Puccini</td>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shostakovich</td>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       composer  birth  death              city\n",
       "0        Mahler   1860   1911           Kaliste\n",
       "1     Beethoven   1770   1827              Bonn\n",
       "2       Puccini   1858   1924           Lucques\n",
       "3  Shostakovich   1906   1975  Saint-Petersburg"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df = pd.read_excel('data/composers.xlsx')\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The reader automatically recognized the heaers of the file. However it created a new index. If needed we can specify which column to use as header:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:47:02.153798Z",
     "start_time": "2020-06-24T14:47:02.086758Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city\n",
       "composer                                    \n",
       "Mahler         1860   1911           Kaliste\n",
       "Beethoven      1770   1827              Bonn\n",
       "Puccini        1858   1924           Lucques\n",
       "Shostakovich   1906   1975  Saint-Petersburg"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df = pd.read_excel('data/composers.xlsx', index_col = 'composer')\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we open the file in Excel, we see that it is composed of more than one sheet. Clearly, when not specifying anything, the reader only reads the first sheet. However we can specify a sheet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:51:26.183305Z",
     "start_time": "2020-06-24T14:51:26.141838Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sibelius</th>\n",
       "      <td>unknown</td>\n",
       "      <td>unknown</td>\n",
       "      <td>unknown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Haydn</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Rohrau</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                birth    death              city\n",
       "composer                                        \n",
       "Mahler           1860     1911           Kaliste\n",
       "Beethoven        1770     1827              Bonn\n",
       "Puccini          1858     1924           Lucques\n",
       "Shostakovich     1906     1975  Saint-Petersburg\n",
       "Sibelius      unknown  unknown           unknown\n",
       "Haydn             NaN      NaN            Rohrau"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df = pd.read_excel('data/composers.xlsx', index_col = 'composer', sheet_name='Sheet2')\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As you can see above, some information is missing. Some missing values are marked as \"`unknown`\" while other are `NaN`. `NaN` is the standard symbol for unknown/missing values and is understood by Pandas while \"`unknown`\" is just seen as text. \n",
    "This is impractical as now we have columns with a mix of numbers and text which will make later computations difficult. What we would like to do is to replace all \"irrelevant\" values with the standard `NaN` symbol that says \"*no information*\".\n",
    "For this we can use the `na_values` argument to specify what should be a `NaN`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T14:58:59.168497Z",
     "start_time": "2020-06-24T14:58:59.139749Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860.0</td>\n",
       "      <td>1911.0</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770.0</td>\n",
       "      <td>1827.0</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858.0</td>\n",
       "      <td>1924.0</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906.0</td>\n",
       "      <td>1975.0</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sibelius</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Haydn</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Rohrau</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               birth   death              city\n",
       "composer                                      \n",
       "Mahler        1860.0  1911.0           Kaliste\n",
       "Beethoven     1770.0  1827.0              Bonn\n",
       "Puccini       1858.0  1924.0           Lucques\n",
       "Shostakovich  1906.0  1975.0  Saint-Petersburg\n",
       "Sibelius         NaN     NaN               NaN\n",
       "Haydn            NaN     NaN            Rohrau"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df = pd.read_excel('data/composers.xlsx', index_col = 'composer', sheet_name='Sheet2', \n",
    "                             na_values=['unknown'])\n",
    "composers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Plotting DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We will learn more about plotting later, but let's see here some possibilities offered by Pandas. Pandas builds on top of Matplotlib but exploits the knowledge included in Dataframes to improve the default output. \n",
    "\n",
    "We can pass Series to Matplotlib which manages to understand them. Here's a default scatter plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:01:47.227488Z",
     "start_time": "2020-06-24T15:01:47.005823Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAZFklEQVR4nO3df5BV533f8fdHy8pZTWMvGtapWKFCMogMGsZgbSTNqG4kpRVImQ6UNA5K2lBLUzWu5Ik1Li0kja3YmRFjEnviieyOUlNFdQMiMaG4cUzIyLGmGf1aDPqBBNZakgMrVeBBuLW1woC+/eM+C5fL/XH27v15zuc1s6Oz33P27nm07Pc++zzf5zmKCMzMrBgu6fYNmJlZ5zjpm5kViJO+mVmBOOmbmRWIk76ZWYHM6fYN1DNv3rxYuHBht2/DzKyv7Nu37/sRMVLtXE8n/YULFzI+Pt7t2zAz6yuSvlfrnId3zMwKxEnfzKxAnPTNzArESd/MrECc9M3MCqSnq3fMzIpm1/5Jtuw5zOsnp5g/PMSGlUtYs2K0Za/vpG9m1iN27Z9k087nmTp9FoDJk1Ns2vk8QMsSv4d3zMx6xJY9h88l/GlTp8+yZc/hln0PJ30zsx7x+smpGcWb4aRvZtYj5g8PzSjeDCd9M7MesWHlEoYGBy6IDQ0OsGHlkpZ9j4ZJX9ICSd+U9KKkg5J+M8Uvl7RX0svpv3NTXJK+IGlC0nOSPlj2WuvT9S9LWt+yVpiZ5cCaFaM8sHYZo8NDCBgdHuKBtctaWr2jRs/IlXQFcEVEfFvSTwL7gDXAvwFORMRmSRuBuRHxnyTdDnwMuB24HvjDiLhe0uXAODAGRHqdayPirVrfe2xsLLzhmpnZzEjaFxFj1c417OlHxBsR8e10/P+Al4BRYDXwJ+myP6H0RkCKPxIlTwLD6Y1jJbA3Ik6kRL8XWNV8s8zMbKZmVKcvaSGwAngK+KmIeCOd+j/AT6XjUeBI2ZcdTbFa8crvcTdwN8BVV101k9szM+tZ7V50lVXmpC/pHwBfBT4eEf9X0rlzERGS6o8TZRQRDwEPQWl4pxWvaWbWaeVJfviyQX74zhlOv1tKaeWLroCOvhlkSvqSBikl/P8RETtT+E1JV0TEG2n45liKTwILyr78yhSbBG6qiP9t87duZtabKlfWvvX26YuumTp9lvt3H+TUmXfbugK3UpbqHQFfBl6KiM+VndoNTFfgrAf+Z1n811MVzw3AD9Iw0B7gVklzU6XPrSlmZpYr1VbWVnNy6nTbV+BWytLTvxH418Dzkg6k2G8Bm4Edku4Cvgd8OJ37OqXKnQngbeAjABFxQtJngGfSdZ+OiBOtaISZWS+YHtKZnOUK2lauwK3UMOlHxP8GVOP0L1S5PoB7arzWVmDrTG7QzKwfVA7pNDI0OMBPDF5SdeinlStwK3lFrplZC2Qd0gGYe9kgD6xdxqf++TVtX4FbyVsrm5m1wEyGZN45/S5wfrK256p3zMysfq39/OGhzGP505O1a1aMnvvoFA/vmJllMD1mP3lyiuB8eeWu/ZNA9c3SBgdqTYe2d7K2Hid9M7MMGj3gpNpmaVv+5QcY7cB2yTPh4R0zswyyPOCk1lBNZVVPuydr63FP38wsg2YfcNKJ7ZJnwj19Myu8LJuhbVi5pOkee6cna+tx0jezQqtcVFVr/5ta5ZUAN25+rOu7Z2blpG9mhVZvgrYyeVf22LO+YfQSj+mbWaFlmaCtpVFFTy9y0jezQnvf0OCM4uVm84bRLU76ZlZoqrF+qla8XLMVPd3kpG9mhXayyi6X9eLlqq3C7WYNfhZO+mZWaLPprfdaDX4Wrt4xs0Jrpv6+Vx5y3gwnfTMrtJlub9yPZZrlnPTNrDBq9dBnsmJ2JnX9vchJ38wKoVU99H4s0yzniVwzK4RaPfRP7Hj23J74WfRjmWY5J30zK4RaPfGzERc8DKWRfizTLOekb2aFUK8nPpOtE/qxTLOcx/TNrBCqlWaWmzw5lbkUs5e2Sp4pJ30zy5V6FToAH3/0QNWvE/R1KWZWHt4xs9xo9PDyesk7oO92zGyGk76Z5UaWrY5rPai8ln4pxczKSd/MciNLDX216pt6+qUUM6uGSV/SVknHJL1QFvuApCckPS/pa5LeW3Zuk6QJSYclrSyLr0qxCUkbW98UM+t3u/ZPcuPmx1i08S+5cfNjM6qfBxi+rPoe+OXx6eqbLPqpFDOrLD39h4FVFbH/CmyMiGXAXwAbACQtBdYB16Sv+aKkAUkDwIPAbcBS4I50rZkZ0Hg8PouIbPE1K0YZqLNhfj+WYmbVsHonIh6XtLAifDXweDreC+wBfgdYDWyPiFPAq5ImgOvSdRMR8QqApO3p2hdn3QIzy4VGe9pUq8qZ/rrp2Mmp6nvg/6BK/I7rF/CVJ//+ovi/uuEqfm9Ntr8E+lGzJZsHKSXtXcAvAwtSfBR4suy6oykGcKQifn2T39vMcqjWePzkySlWfPqv+eE7Zzj9bpyLbfjzZyG4ICZKVTiVqo3LTyf2bU8d4WwEAxJ3XL8g1wkfmk/6dwJfkPQ7wG7gx626IUl3A3cDXHXVVa16WTPrcfOHh5iskfjfqvIUq9NnL07vARcl/nrj8r+3Zlnuk3ylpqp3IuJQRNwaEdcC24DvplOTnO/1A1yZYrXi1V77oYgYi4ixkZGRZm7PzPrQTKtqagno2y0SOqGpnr6k90fEMUmXAP8Z+C/p1G7gTyV9DpgPLAaepvTmu1jSIkrJfh3wq7O9eTPLj/KHmdTq8WcxPDTI3228pVW3lTtZSja3AU8ASyQdlXQXpeqb7wCHgNeB/wYQEQeBHZQmaL8B3BMRZyPiDHAvpQnfl4Ad6Vozs5Y6OXW6qXLPolDUqnHqAWNjYzE+Pt7t2zCzDqh8yMlsDQ0OFHZoR9K+iBirds4rcs2sJ1Qr2ZyNPO6b0wpO+mbWE2qVbE5PyNY618xrFpmTvpn1hHqPIaz1tKpfu+Gquhuo5W3fnFbwfvpm1hWVK2xv/tkRvrpv8oIhnuka+/LKnmoPOKk2H5DHfXNawRO5ZtZxtZL0L107yjcPHW/45Kpar5nlqVdFUG8i1z19M+u4WvvsfPPQ8aZr7Pv5EYad5DF9M+u4LPveW3s46ZtZx9WbtLX2ctI3s46rVo0zOCB+dOpM0w9QsWw8pm9mHVdZjTN82SA/fOfMuf3wpx+gUn6ttYZ7+mbWFWtWjPJ3G2/h1c2/yGWXzjm3L/40r6htD/f0zawj6pVUemK3c9zTN7O2a/T8W0/sdo6Tvpm1Xb3n30L1iV2vqG0PD++YWds1Gr5ptM2CtY6Tvpm1Xa3n3wZw4+bHziV4J/n28/COmbVdveffVo7vW3s56ZtZ261ZMcoDa5fV3AbZ5Zmd4+EdM8tsNjtZTg/fLNr4l1Tb29flmZ3hnr6ZZdKo7DIrl2d2l5O+mWXSqOwyK5dndpeHd8wsk1atmnV5Znc56ZtZJrXKLpsZlnF5Zvd4eMfMMvGwTD64p29mmXhYJh+c9M1yqF0PCfewTP9z0jfLmenSyulKmywPJGnXm4T1Ho/pm+XMTEsrW1V/b/2hYdKXtFXSMUkvlMWWS3pS0gFJ45KuS3FJ+oKkCUnPSfpg2desl/Ry+ljfnuaYWbUKG6heWrlr/ySf2PFsS+rvrT9k6ek/DKyqiH0W+N2IWA58Mn0OcBuwOH3cDXwJQNLlwKeA64HrgE9JmjvLezezCrv2T6Ia5ypLK6d7+Gej2qYI3hYhrxom/Yh4HDhRGQbem47fB7yejlcDj0TJk8CwpCuAlcDeiDgREW8Be7n4jcTMZmnLnsNV97URXFRaWW0YqJy3RcinZsf0Pw5skXQE+H1gU4qPAkfKrjuaYrXiF5F0dxoyGj9+/HiTt2dWTLV658HFk7iNevI/OnXG4/o51GzS/yhwX0QsAO4DvtyqG4qIhyJiLCLGRkZGWvWyZoVQq3debUvjRj35k1OnPaGbQ80m/fXAznT8Z5TG6QEmgQVl112ZYrXiZtZCM1k1W+/BJtM8oZs/zSb914GfT8e3AC+n493Ar6cqnhuAH0TEG8Ae4FZJc9ME7q0pZmYtVP6wElHq4T+wdlnVmvvKa2vxhG6+NFycJWkbcBMwT9JRSlU4/xb4Q0lzgHcoVeoAfB24HZgA3gY+AhARJyR9BngmXffpiKicHDazFpjJqtnya2/c/FjLNlSz3tUw6UfEHTVOXVvl2gDuqfE6W4GtM7o7M+uYDSuXXLCSF7yhWh55GwYzA7yhWlE46ZvZOd5QLf+8946ZWYE46ZuZFYiHd8w6wFsXW69w0jdrs2b2tzdrFw/vmLVZrf3tf/drB7t0R1ZkTvpmbVZrRetbb5/2vjbWcU76Zm1Wb0Xr/bvd27fOctI3a7N6K1pPTp3u4J2YOembtZ0na62XOOmbdcAlNbaxrBU3axcnfbMOeLf6Y2hrxs3axUnfrAOqPbmqXtysXbw4y6wFGq24nc22xV7Na63kpG82S1lW3Da7bbFX81qrOembzVKtFbdb9hy+IDE3s21x1tc2y8pj+mazVGvFbSueLdvO17ZictI3m6VaK25b8WzZdr62FZOTvtksbVi5hKHBgQtirXq2bDtf24rJY/pms9TOZ8v6ubXWaoro3dUhY2NjMT4+3u3bMDPrK5L2RcRYtXMe3jEzKxAnfTOzAnHSNzMrECd9M7MCcfWOGd7fxoqjYU9f0lZJxyS9UBZ7VNKB9PGapANl5zZJmpB0WNLKsviqFJuQtLHlLTFr0vT+NpMnpwjO72/j59daHmUZ3nkYWFUeiIhfiYjlEbEc+CqwE0DSUmAdcE36mi9KGpA0ADwI3AYsBe5I15p1Xb39bczypuHwTkQ8LmlhtXOSBHwYuCWFVgPbI+IU8KqkCeC6dG4iIl5JX7c9Xfvi7G7fbPZmsr+Nh4Gs3812IvdDwJsR8XL6fBQ4Unb+aIrVil9E0t2SxiWNHz9+fJa3Z9ZY1v1tPAxkeTDbpH8HsK0VNzItIh6KiLGIGBsZGWnlS5tVlXV/Gw8DWR40Xb0jaQ6wFri2LDwJLCj7/MoUo07crKuy7m/jbY4tD2ZTsvlPgUMRcbQsthv4U0mfA+YDi4GnAQGLJS2ilOzXAb86i+9t1lJZHnAyf3iIySoJ3tscWz/JUrK5DXgCWCLpqKS70ql1VAztRMRBYAelCdpvAPdExNmIOAPcC+wBXgJ2pGvN+oa3ObY88C6bZjPg6h3rB/V22fSKXLMZaOY5t2a9xHvvmJkViJO+mVmBOOmbmRWIk76ZWYE46ZuZFYirdyz3XGZpdp6TvuXa9CZp03vmTG+SBjjxWyF5eMdyzZukmV3ISd9yzZukmV3ISd9yLete+WZF4aRvueZN0swu5Ilca5teqJrJule+WVE46VtbzLRqpp1vEN4kzew8D+9YW8ykasbPnjXrHCd9a4uZVM24rNKsc5z0rS1mUjXjskqzznHSt7aYSdWMyyrNOsdJ31pq1/5Jbtz8GPc9eoCfGLyE4aFBBIwOD/HA2mVVJ1RdVmnWOa7esZaprNh56+3TDA0O8PlfWV63esZllWad46RvNc20jLLehGyjBO6ySrPOcNK3qprZndITsma9z2P6VlUzZZSekDXrfU76VlUzvfZqE7IAPzp1xgutzHqEk75V1Uyvfc2KUR5Yu4y5lw1eED85ddorbM16hJO+VdVsGeWaFaNcdunFU0VeYWvWGzyRa1XNpozSE7pmvctJ32pqtoxy/vAQk1USvCd0zbqv4fCOpK2Sjkl6oSL+MUmHJB2U9Nmy+CZJE5IOS1pZFl+VYhOSNra2GdZLvMLWrHdl6ek/DPwR8Mh0QNLNwGrgAxFxStL7U3wpsA64BpgP/I2kq9OXPQj8M+Ao8Iyk3RHxYqsaYr3DK2zNelfDpB8Rj0taWBH+KLA5Ik6la46l+Gpge4q/KmkCuC6dm4iIVwAkbU/XOul3SeVq25t/doRvHjresiTtFbZmvanZ6p2rgQ9JekrStyT9XIqPAkfKrjuaYrXiF5F0t6RxSePHjx9v8vasnmoPLfnKk3/vh5iYFUCzSX8OcDlwA7AB2CFJrbihiHgoIsYiYmxkZKQVL2kVqq22reQSS7N8arZ65yiwMyICeFrSu8A8YBJYUHbdlSlGnbh1WNbSSZdYmuVPsz39XcDNAGmi9lLg+8BuYJ2k90haBCwGngaeARZLWiTpUkqTvbtnee/WpKylky6xNMufLCWb24AngCWSjkq6C9gK/HQq49wOrI+Sg8AOShO03wDuiYizEXEGuBfYA7wE7EjXWhfU2iOnnEsszfJJpRGa3jQ2Nhbj4+Pdvo1canf1jpl1j6R9ETFW7ZxX5BaUSyrNiskbrpmZFYiTvplZgTjpm5kViJO+mVmBOOmbmRWIk76ZWYE46ZuZFYiTvplZgTjpm5kViJO+mVmBOOmbmRWIk76ZWYE46ZuZFYiTvplZgTjpm5kViJO+mVmBOOmbmRWIk76ZWYE46ZuZFYiTvplZgTjpm5kViJO+mVmBOOmbmRWIk76ZWYE46ZuZFYiTvplZgTRM+pK2Sjom6YWy2P2SJiUdSB+3l53bJGlC0mFJK8viq1JsQtLG1jfFzMwaydLTfxhYVSX++YhYnj6+DiBpKbAOuCZ9zRclDUgaAB4EbgOWAneka83MrIPmNLogIh6XtDDj660GtkfEKeBVSRPAdencRES8AiBpe7r2xZnfspmZNWs2Y/r3SnouDf/MTbFR4EjZNUdTrFb8IpLuljQuafz48eOzuD0zM6vUbNL/EvAzwHLgDeAPWnVDEfFQRIxFxNjIyEirXtbMzMgwvFNNRLw5fSzpj4H/lT6dBBaUXXplilEnbmZmHdJUT1/SFWWf/gtgurJnN7BO0nskLQIWA08DzwCLJS2SdCmlyd7dzd+2mZk1o2FPX9I24CZgnqSjwKeAmyQtBwJ4Dfh3ABFxUNIOShO0Z4B7IuJsep17gT3AALA1Ig62ujFmZlafIqLb91DT2NhYjI+Pd/s2zMz6iqR9ETFW7ZxX5JqZFUhTE7m9btf+SbbsOczrJ6eYPzzEhpVLWLOiaoWomVmh5C7p79o/yaadzzN1+iwAkyen2LTzeQAnfjMrvNwN72zZc/hcwp82dfosW/Yc7tIdmZn1jtwl/ddPTs0obmZWJLlL+vOHh2YUNzMrktwl/Q0rlzA0OHBBbGhwgA0rl3TpjszMekfuJnKnJ2tdvWNmdrHcJX0oJX4neTOzi+VueMfMzGpz0jczKxAnfTOzAnHSNzMrECd9M7MC6emtlSUdB743y5eZB3y/BbfT69zO/ChCG8HtbKd/FBFVnzfb00m/FSSN19pXOk/czvwoQhvB7ewWD++YmRWIk76ZWYEUIek/1O0b6BC3Mz+K0EZwO7si92P6ZmZ2XhF6+mZmljjpm5kVSF8mfUlbJR2T9EKVc5+QFJLmpc8l6QuSJiQ9J+mDZdeul/Ry+ljfyTZkUa2dku6XNCnpQPq4vezcptTOw5JWlsVXpdiEpI2dbkc9tX6Wkj4m6ZCkg5I+WxbvuzZCzZ/lo2U/x9ckHSg7l6d2Lpf0ZGrnuKTrUrwvfzdrtPEDkp6Q9Lykr0l6b9m53vpZRkTffQD/BPgg8EJFfAGwh9KCrnkpdjvwV4CAG4CnUvxy4JX037npeG6329aoncD9wH+ocu1S4FngPcAi4LvAQPr4LvDTwKXpmqXdbluDNt4M/A3wnvT5+/u5jfX+zZad/wPgk3lsJ/DXwG3p+Hbgb8uO++53s0YbnwF+Ph3fCXymV3+WfdnTj4jHgRNVTn0e+I9A+ez0auCRKHkSGJZ0BbAS2BsRJyLiLWAvsKrNtz4jddpZzWpge0SciohXgQnguvQxERGvRMSPge3p2p5Qo40fBTZHxKl0zbEU78s2Qv2fpSQBHwa2pVDe2hnAdM/3fcDr6bgvfzdrtPFq4PF0vBf4pXTccz/Lvkz61UhaDUxGxLMVp0aBI2WfH02xWvF+cG/6c3irpLkplqd2Xg18SNJTkr4l6edSPE9tLPch4M2IeDl9nrd2fhzYIukI8PvAphTPUzsPcj5p/zKlUQfowTbmIulLugz4LeCT3b6XDvgS8DPAcuANSsMCeTOH0p/2NwAbgB2pN5xXd3C+l59HHwXui4gFwH3Al7t8P+1wJ/DvJe0DfhL4cZfvp6ZcJH1KSXAR8Kyk14ArgW9L+ofAJOffdUnnJuvEe1pEvBkRZyPiXeCPKf2ZCPlq51FgZ/qz/2ngXUqbVuWpjQBImgOsBR4tC+etneuBnen4z8jhv9mIOBQRt0bEtZTewL+bTvVeG7s9KTKLyZSF1J4Ue43zE7m/yIWTRU/H+cmiVylNFM1Nx5d3u12N2glcUXZ8H6XxQoBruHDC6BVKk0Vz0vEizk8YXdPtdjVo428An07HV1P6M1j93MZa/2YpjVV/qyKWq3YCLwE3peNfAPal47793azSxulig0uAR4A7e/Vn2fX/eU3+D99GaWjjNKVe4V0V51/jfNIX8CCld97ngbGy6+6kNLEyAXyk2+3K0k7gv6d2PAfs5sI3gd9O7TxMqpZI8duB76Rzv93tdmVo46XAV4AXgG8Dt/RzG+v9mwUeBn6jyvW5aSfwj4F9KbE9BVybru3L380abfzN9HP5DrCZtNtBL/4svQ2DmVmB5GVM38zMMnDSNzMrECd9M7MCcdI3MysQJ30zswJx0jczKxAnfTOzAvn/1ArNOBFD/JAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "composers_df = pd.read_excel('data/composers.xlsx', index_col = 'composer', sheet_name='Sheet5')\n",
    "plt.plot(composers_df.birth, composers_df.death, 'o')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different types of plots are accessible when using the `plot` function of DataFrame instances via the `kind` option. The variables to plot are column names passed as keywords instead of whole series like in Matplotlib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:03:52.577393Z",
     "start_time": "2020-06-24T15:03:52.310044Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAeR0lEQVR4nO3de5zddX3n8dd7LpkEgyEkkYVcmmgS2+BCxJHLplrAVRBX0jXqolZYsbK1wUdllZs3UGpXYrWrK9JGZClbS6SGSryyKAqVh1wmNMSEi46AzQwslxCBYDLXz/5xfpOcOXPO/M7MnN+5zfv5eMyDcz6/3/nN98sk88n3rojAzMxsPC21LoCZmdU/JwszM0vlZGFmZqmcLMzMLJWThZmZpWqrdQGyMH/+/Fi6dGmti2Fm1lC2bt36TEQsKHatKZPF0qVL6erqqnUxzMwaiqTflLrmbigzM0vlZGFmZqmcLMzMLJWThZmZpXKyMDOzVE4WZmZNYvfePu7f9Vt27+2r+LObcuqsmdl0c/O2Xi7evJ32lhYGhofZsO4Yzly9sGLPd8vCzKzB7d7bx8Wbt7N/YJgX+gbZPzDMRZu3V7SF4WRhZtbgevbso71l9K/z9pYWevbsq9j3cLIwM2twi+bOYmB4eFRsYHiYRXNnVex7OFmYmTW4ebM72LDuGGa2t3BoRxsz21vYsO4Y5s3uqNj3yGyAW9Ji4HrgCCCAjRHxJUmHA98ElgKPAe+MiD2SBHwJOAP4HfBfI+K+5FnnAJ9IHv2XEfH3WZXbzKwRnbl6IWuWz6dnzz4WzZ1V0UQB2bYsBoGPRMQq4ERgvaRVwCXAjyNiBfDj5D3Am4EVydd5wNUASXK5DDgBOB64TNLcDMttZtaQ5s3u4NjFh1U8UUCGySIinhhpGUTEC8CDwEJgLTDSMvh74I+T12uB6yPnLuAwSUcCpwG3RsSzEbEHuBU4Patym5nZWFUZs5C0FHg1cDdwREQ8kVz6f+S6qSCXSHblfawniZWKF36P8yR1Sep6+umnK1sBM7MaynKxXbkyX5QnaTawGfhwRDyfG5rIiYiQFJX4PhGxEdgI0NnZWZFnmpnVwu69fQfGHn7W/UzJxXb592XR9ZQv02QhqZ1covhGRNyUhJ+UdGREPJF0Mz2VxHuBxXkfX5TEeoGTC+I/zbLcZma1kr8Su39omKHhYQaHYT+5qbEXbd7OmuXzx00iWcisGyqZ3fR14MGI+GLepS3AOcnrc4Cb8+JnK+dE4Lmku+oW4E2S5iYD229KYmZmTaVwJXbfYC5R5GtvaWHn489nvmK7UJYtizXAe4FfSNqWxD4GfA64UdL7gd8A70yufZ/ctNluclNn3wcQEc9KugK4N7nvMxHxbIblNjOrut17+/jJQ0/R1qJx78stvgvaW1oOtDbg4IrtrLqjMksWEfEzoFSt31Dk/gDWl3jWtcC1lSudmVn9GOl6apV4sX+o6D0vmdHKUAQb1h3D0UfNyXzFdiHvOmtmVkP5XU+lzGpv5cLTVvLWYxceaDlsWHcMFxWMWWQ5yO1kYWZWQyObAOZ3KRXaNzDE//jBQxz+ko4Dg9hZr9gu5L2hzMyqoNRaiWKbALa3ihmto3vx+wZjzCB2liu2CzlZmJll7OZtvay58jb+5Jq7WXPlbWzZ1nvgWrFNAL/wjmO55pzXckh766jnVHrb8YlwN5SZWYbyxyQK10qMtAiKdSnt3tvHMKPXF2c9iD0etyzMzDJU7sFEhV1K1dh2fCLcsjAzy9BUDiaq9iD2eNyyMDObgrRN/qbaQqjmIPZ43LIwM5uk/H2cxtufqVQLoZobAU6Vk4WZ2SSUM3Cdb97sjlHxchNNvXA3lJnZJJQ7cF1M4YaB1dgIcKqcLMzMJmHR3FnsGxgcFds3MFjWwPVUEk2tOFmYmU1S/mFuxd6XMpUZUrXiZGFmNgk9e/Yxs230CuuZba1ltQ7qbQ1FOTzAbWY2CVNtHdTTGopyuGVhZjYJk20d5K/LqJc1FOVwy8LMbJIm2jpotOmy+dyyMDMrQ6mV2uW2Dhpxumw+tyzMzFJUokVQ7JCjrM/NriS3LMzMxlGsRfDRb22n+8kXJvScRpwum8/JwsxsHMUW0PUPDnPGl/9l1CFGaRpxumw+d0OZmY2jWIsAoH8oxt0LqphGmy6bzy0LM7NxjLQIZrSN/XUZw3FgEV7aVuX5z2uU6bL53LIwM0uU2jL8zNULOWrOTN7+d3eNur9vKHjJjNaGnhJbLicLMzPSZzy1t7Uys72F/QMHu6TaWsS9jz3Lp7/7QNlblTcqd0OZ2bRXzhqIYrOWBoeDS/95B0PDMSpe7zvIToaThZlNe+VsGX5g7KJ17M6yA0Ojk0UjTYktV2bJQtK1kp6StCMvdqykn0v6haTvSHpp3rVLJXVLeljSaXnx05NYt6RLsiqvmTW2cgeYi1k0dxb7B4dGxfYPDo35hX/m6oV87Iw/KPqM9lY15JTYcmU5ZnEd8BXg+rzYNcBHI+J2SecCFwKflLQKOAs4GjgK+JGklclnrgLeCPQA90raEhEPZFhuM2swlRhgjohx34/4w+Xzi8Zv+NMTaG9rbbgpseXKrGUREXcAzxaEVwJ3JK9vBdYlr9cCmyKiLyIeBbqB45Ov7oh4JCL6gU3JvWZmQHnjDcVaHfmxnj37mNU++t/Os9rbio47LD/iUM4+acmo2NknLaFz2byGnBJbrmrPhtpJ7pf9t4F3AIuT+EIgf05aTxID2FUQPyHbIppZIym251KrxE8eeopTfv9l/Kz7mTGtjoBRsU/+p1UT2orjM2v/PWefuJRtu37L6sWHsfyIQ7OsYl2odrI4F/iypE8CW4D+Sj1Y0nnAeQBLlixJudvMmkWxFdYv9g9x+Xd28vFv72BoeJjBYQ4kkwu/tR0I+gbjQOyK7z7AJ9+yiiu+98CopDJeK2H5EYdOiyQxoqrJIiIeAt4EkIxJvCW51MvBVgbAoiTGOPHCZ28ENgJ0dnYW72w0s6YzMkvpos3baZV4sT83UL23b6jo/a0tghBw8Hp7SwuvWjiHOy8+tSG34qiGqk6dlfSy5L8twCeAv00ubQHOktQhaRmwArgHuBdYIWmZpBnkBsG3VLPMZlb/zly9kDsvPpVPn3k0sztax713aDjoK5j5tG9g8ECCaOZxh6nIcursDcDPgVdK6pH0fuBdkn4JPAQ8DvxvgIjYCdwIPAD8EFgfEUMRMQicD9wCPAjcmNxrZjbGy17awcDQ6C6ploJlEWtXH0VLYRDY+fhzDXMQUS2o1PSwRtbZ2RldXV21LoaZVUn+1Nn9g0NEBLPa2+gfGmI4Ri+am9HWQnvLwe6qEYfMaGU4oin3dSqXpK0R0Vnsmldwm1lDK5w6OzAUtLa0cNV7juNrZ3cys210t1R7q8a0PgB+1z/UcEedVpOThZk1tGJbdcxobWHOrHaOPmrOmJlSQ8PBZW89mpntLRzSPnZ8oxn3daoEJwsza2jjHVda6nS695z4e9x58an87XtfQ0fBORXNuK9TJXiLcjNrOIXnToxMnS22RqLU6XTzZnfw+pUL+PzbS3/WDnKyMLOGUmofqPGOK503u6NkAmjko06rycnCzBpG/mB2sYOGJvuLfiqfnS48ZmFmDaOccycsG04WZtYwxhvMtmw5WZhZwyg1uwmY9MFHVh6PWZhZQykckP5Z9zOsufK2KR18ZOncsjCzhjOy4R+QevCRVYaThZnVvVLna3vAu3rcDWVmdW2887U94F09blmYWd1KO1+71IC310xUnlsWZla3ip2vPdLNlLadh1WWk4WZ1a1i3Uz9Q0M8t6+f3Xv7Ru3z5CSRLXdDmVndKuxmamuB4YD13/hX1lx5G1u29da6iNOGk4WZ1bWR87Wves9xtLa0MDAUniZbA04WZlYVpaa/lmPe7A7mzGpnRqunydaKxyzMLHPjTX8tl6fJ1pZbFmaWqbTpr+XyNNnacsvCzDJVzvTXcnmabO04WZhZpirdfeRpsrXhbigzy5S7j5qDWxZmljl3HzU+JwszG2X33r5Mfqm7+6ixOVmY2QGTmeKaVXKx+uJkYWbA6CmuIzOXLtq8nTXL55dMApVYP2GNIbMBbknXSnpK0o682GpJd0naJqlL0vFJXJK+LKlb0nZJx+V95hxJv0q+zsmqvGbTXc+efbRKo2LjrZDufvIFLvyWT6mbLrKcDXUdcHpBbAPw6YhYDXwqeQ/wZmBF8nUecDWApMOBy4ATgOOByyTNzbDMZtPWjt7neLF/aFSs1BTXm7f1csaX/4X+wdFTYr39RvPKLFlExB3As4Vh4KXJ6znA48nrtcD1kXMXcJikI4HTgFsj4tmI2APcytgEZGZTtHtvH1d874Ex8U++ZdWYLqiR7qr+oRhzv7ffaF7VHrP4MHCLpL8ml6j+QxJfCOzKu68niZWKjyHpPHKtEpYsWVLRQps1u2KrrF/S0cqrFs4p616AGa1i/cnLMy+r1Ua1F+V9ELggIhYDFwBfr9SDI2JjRHRGROeCBQsq9VizaaHYKuuh4SjaSih2b24z2GDjHY/4nIkmVe1kcQ5wU/L6n8iNQwD0Aovz7luUxErFzayCJrLKuvDejrYWBPQP4YHuJlbtbqjHgT8CfgqcCvwqiW8Bzpe0idxg9nMR8YSkW4C/yhvUfhNwaXWLbDY9TGSVdf69z+0bYP037uOFvsED1ye7UaDVr8yShaQbgJOB+ZJ6yM1q+gDwJUltwH6SMQbg+8AZQDfwO+B9ABHxrKQrgHuT+z4TEYWD5mZWIRNZZT1y7+69fT5nYhpQxNgZDY2us7Mzurq6al0Ms2ljy7ZeLvLivIYnaWtEdBa75hXcZjZl3iiw+TlZmFlFeKPA5ubzLMzMLFVZyULS25K9mZ6T9LykFyQ9n3XhzMysPpTbDbUBeGtEPJhlYcxsLG8BbvWg3GTxpBOFWfV5C3CrF+MmC0lvS152Sfom8G3gwLLMiLip2OfMbOqKnS9x4bfGP1/CLCtpLYu35r3+HbkV1COCg1t3mFmFFduwr29wmH+8+9/40BtW1LBkNh2Nmywi4n0AktZExJ351yStybJgZtPdormz6B8aHhP/nz/6Je8+YYlbF1ZV5U6d/V9lxsysQubN7uDcNUvHxIcCdj7uyYhWXWljFieRO3NigaT/nnfppUBrlgUzMzjpFfO4+vZHilxpvm16rL6ljVnMAGYn9x2aF38eeHtWhTKznKPmFN+Mr1TcLCtpYxa3A7dLui4iflOlMplZ4sX+IWa2t7B/4ODYxcz2ljFnZZtlrdx1Fr+T9HngaGDmSDAiTs2kVGYGUHKbb2//bdVW7gD3N4CHgGXAp4HHOHjGhJlNwe69fdy/67dFT5abyAl2E3222USU27KYFxFfl/QXeV1TThZmU1TOCu3Jbv/t1d9WSeW2LAaS/z4h6S2SXg0cnlGZzKaF/BXaaWdXz5vdwbGLD5tQi6LcZ5uVo9xk8ZeS5gAfAT4KXANckFmpzKaBkRXa+UbOrq7nZ9v0VFY3VER8N3n5HHBKdsUxmz4WzZ2V2dnVWT7bpqdyz7NYKenHknYk74+R9Ilsi2bW3KY6eF2rZ9v0pIj0laCSbgcuBP4uIl6dxHZExKsyLt+kdHZ2RldXV62LYVaWLM+r8FkYNhGStkZEZ7Fr5c6GOiQi7pGUHxuccsnMLNOzq30utlVKuQPcz0h6BcmGNJLeDjyRWanMzKyulNuyWA9sBH5fUi/wKPCezEplZmZ1JW3X2fydZr8P/IRca+RFYB3wxeyKZmZm9SKtZTGy0+wrgdcCNwMC3gvck2G5zMysjqTtOvtpAEl3AMdFxAvJ+8uB72VeOrMG4BlHNh2UO8B9BNCf974/iZUk6VpJT42szUhi35S0Lfl6TNK2vGuXSuqW9LCk0/LipyexbkmXlFles6q4eVsva668jT+55m7WXHkbW7b11rpIZpkod4D7euAeSf+cvP9j4LqUz1wHfCX5LAAR8V9GXkv6ArkV4UhaBZxFbgv0o4AfSVqZ3HoV8EagB7hX0paIeKDMcptlJn//pf3kVktftHk7a5bPdwvDmk652318VtIPgNclofdFxL+mfOYOSUuLXVNuwcY7gZHzMNYCmyKiD3hUUjdwfHKtOyIeST63KbnXycJqbmT/pZFEAQf3XyqWLNxdZY2s3JYFEXEfcF+Fvu/rgCcj4lfJ+4XAXXnXe5IYwK6C+AnFHijpPOA8gCVLllSomGalTWT/JW8Xbo2u3DGLSnsXcEMlHxgRGyOiMyI6FyxYUMlHmxVV7v5L3i7cmkHZLYtKkdQGvA14TV64F1ic935REmOcuFnNlXMw0US7q8zqUdWTBfAfgYcioicvtgX4R0lfJDfAvYLcOg4BKyQtI5ckzgLeXeXymo0rbf8lbxduzSCzbihJNwA/B14pqUfS+5NLZ1HQBRURO4EbyQ1c/xBYHxFDETEInA/cAjwI3Jjca9YwvF24NYOytihvNN6i3OqRZ0NZvavEFuVmNkXeLtwaWa1mQ5mZWQNxsjAzs1ROFmZmlsrJwszMUjlZmJlZKicLs3Hs3tvH/bt+6605bNrz1FmzErz5n9lBblmYFeHN/8xGc7IwK2Jk8798I5v/mU1HThZmRXjzP7PRnCzMivDmf2ajeYDb6lI9bLpXzlkVZtOFk4XVnYnOQsoysXjzP7McJwurK/mzkEZOlrto83bWLJ9f9Je2p7eaVYfHLKyuTGQWkqe3mlWPk4XVlYnMQvL0VrPqcbKwujKRWUie3mpWPR6zsLoxMlC9Zvl87rz41NRB65HEclHBmIUHpM0qz8nC6sJkB6o9vdWsOtwNZZmYyG6tUx2onje7g2MXH+ZEYZYhtyys4ibaShgZqB6ZKgsHB6qdAMzqg1sWVlGTaSV4oNqs/jlZWEVNZjpr4QyojrYW1p+8POuimtkEOFlYRU22lXDm6oXcefGpfOD1LweCjXc8wporb2PLtt4MS2tm5XKysIqa6m6tX/1pN32D4RXZZnXGA9xWcZOdzuqBbrP65WRhmZjMbq0e6DarX5l1Q0m6VtJTknYUxD8k6SFJOyVtyItfKqlb0sOSTsuLn57EuiVdklV5rfZ84JBZ/cqyZXEd8BXg+pGApFOAtcCxEdEn6WVJfBVwFnA0cBTwI0krk49dBbwR6AHulbQlIh7IsNxWQ16RbVafMksWEXGHpKUF4Q8Cn4uIvuSep5L4WmBTEn9UUjdwfHKtOyIeAZC0KbnXyaKGCg8bqvThQz5wyKz+VHvMYiXwOkmfBfYDH42Ie4GFwF159/UkMYBdBfETij1Y0nnAeQBLliypcLFtROHq7He+ZhE3bu3x4UNmTa7aU2fbgMOBE4ELgRslqRIPjoiNEdEZEZ0LFiyoxCOtQLHV2dff9W8+fMhsGqh2sugBboqce4BhYD7QCyzOu29REisVtxootjq7kA8fMmtO1U4W3wZOAUgGsGcAzwBbgLMkdUhaBqwA7gHuBVZIWiZpBrlB8C1VLrMlik1tLeSprmbNKcupszcAPwdeKalH0vuBa4GXJ9NpNwHnJK2MncCN5Aaufwisj4ihiBgEzgduAR4EbkzutRooNrX17JOWeKqr2TSgiKh1GSqus7Mzurq6al2MppX1bCgzqw1JWyOis9g1r+C2CSuc2uqprmbNzxsJmplZKicLMzNL5WRhZmapnCzMzCyVk4WZmaVysjAzs1ROFmZmlsrJwszMUjlZmJlZKicLMzNL5WRhZmapnCzMzCyVk4WZmaVysjAzs1ROFmZmlsrJwszMUjlZmJlZKicLMzNL5WRhZmapnCzMzCyVk4WZmaVysjAzs1ROFmZmlsrJwszMUjlZmJlZKicLMzNLlVmykHStpKck7ciLXS6pV9K25OuMvGuXSuqW9LCk0/LipyexbkmXZFVeMzMrLcuWxXXA6UXifxMRq5Ov7wNIWgWcBRydfOarkloltQJXAW8GVgHvSu41M7MqasvqwRFxh6SlZd6+FtgUEX3Ao5K6geOTa90R8QiApE3JvQ9UurxmZlZaLcYszpe0PemmmpvEFgK78u7pSWKl4mNIOk9Sl6Sup59+Ootym5lNW9VOFlcDrwBWA08AX6jUgyNiY0R0RkTnggULKvVYMzMjw26oYiLiyZHXkr4GfDd52wsszrt1URJjnLiZmVVJVVsWko7Me/ufgZGZUluAsyR1SFoGrADuAe4FVkhaJmkGuUHwLdUss5mZZdiykHQDcDIwX1IPcBlwsqTVQACPAf8NICJ2SrqR3MD1ILA+IoaS55wP3AK0AtdGxM6symxmZsUpImpdhorr7OyMrq6uWhfDzKyhSNoaEZ3FrnkFt5mZpXKyKLB7bx/37/otu/f21booZmZ1o6qzoerdzdt6uXjzdtpbWhgYHmbDumM4c3XRZR1mZtOKWxaJ3Xv7uHjzdvYPDPNC3yD7B4a5aPN2tzDMzHCyOKBnzz7aW0b/72hvaaFnz74alcjMrH44WSQWzZ3FwPDwqNjA8DCL5s6qUYnMzOqHk0Vi3uwONqw7hpntLRza0cbM9hY2rDuGebM7al00M7Oa8wB3njNXL2TN8vn07NnHormznCjMzBJOFgXmze5wkjAzK+BuKDMzS+VkYWZmqZwszMwslZOFmZmlcrIwM7NUTblFuaSngd9M8THzgWcqUJx653o2j+lQR3A9s/R7EVH0XOqmTBaVIKmr1L7uzcT1bB7ToY7getaKu6HMzCyVk4WZmaVysihtY60LUCWuZ/OYDnUE17MmPGZhZmap3LIwM7NUThZmZpZqWiULSddKekrSjiLXPiIpJM1P3kvSlyV1S9ou6bi8e8+R9Kvk65xq1qEcxeop6XJJvZK2JV9n5F27NKnnw5JOy4ufnsS6JV1S7XqMp9TPUtKHJD0kaaekDXnxhqsjlPxZfjPv5/iYpG1515qpnqsl3ZXUs0vS8Um8If9ulqjjsZJ+LukXkr4j6aV51+rrZxkR0+YLeD1wHLCjIL4YuIXcQr75SewM4AeAgBOBu5P44cAjyX/nJq/n1rpuafUELgc+WuTeVcD9QAewDPg10Jp8/Rp4OTAjuWdVreuWUsdTgB8BHcn7lzVyHcf7M5t3/QvAp5qxnsD/Bd6cvD4D+Gne64b7u1mijvcCf5S8Phe4ol5/ltOqZRERdwDPFrn0N8BFQP5o/1rg+si5CzhM0pHAacCtEfFsROwBbgVOz7joEzJOPYtZC2yKiL6IeBToBo5Pvroj4pGI6Ac2JffWhRJ1/CDwuYjoS+55Kok3ZB1h/J+lJAHvBG5IQs1WzwBG/qU9B3g8ed2QfzdL1HElcEfy+lZgXfK67n6W0ypZFCNpLdAbEfcXXFoI7Mp735PESsUbwflJs/1aSXOTWDPVcyXwOkl3S7pd0muTeDPVMd/rgCcj4lfJ+2ar54eBz0vaBfw1cGkSb6Z67uTgL/t3kOvlgDqs47ROFpIOAT4GfKrWZamCq4FXAKuBJ8h1XzSbNnJdECcCFwI3Jv/6blbv4mCrohl9ELggIhYDFwBfr3F5snAu8OeStgKHAv01Lk9J0zpZkPvluQy4X9JjwCLgPkn/DujlYJYnudY7TryuRcSTETEUEcPA18g1Z6G56tkD3JR0T9wDDJPbjK2Z6giApDbgbcA388LNVs9zgJuS1/9EE/6ZjYiHIuJNEfEacon/18ml+qtjrQd9qv0FLKX0YOFjHBzgfgujB9HuSeKHA4+SG0Cbm7w+vNb1SqsncGTe6wvI9YcCHM3ogbRHyA2itSWvl3FwIO3oWtcrpY5/Bnwmeb2SXHNdjVzHUn9myfXF314Qa6p6Ag8CJyev3wBsTV437N/NInUcmYTRAlwPnFuvP8ua/8+r8g/qBnJdMAPk/hX6/oLrj3EwWQi4ilym/wXQmXffueQGnLqB99W6XuXUE/g/ST22A1sYnTw+ntTzYZLZJ0n8DOCXybWP17peZdRxBvAPwA7gPuDURq7jeH9mgeuAPytyf9PUE/hDYGvyC/Fu4DXJvQ35d7NEHf8i+bn8Evgcya4a9fiz9HYfZmaWarqPWZiZWRmcLMzMLJWThZmZpXKyMDOzVE4WZmaWysnCrAIkLS2xm/E1klaV+MyHk10ERt7vzbKMZlPhZGGWoYj404h4oDAuqZXc3keHjPmQWR1ysjCrnDZJ35D0oKRvSTpE0k8ldUKu5SDpC5LuJ7fg6ijgJ5J+MvIASZ+VdH9yjsMRNaqH2RhOFmaV80rgqxHxB8DzwJ8XXH8JubMXjo2Iz5DbcvuUiDgl7/pdEXEsuW2rP1ClcpulcrIwq5xdEXFn8vofyG1XkW8I2DzO5/uB7yavt5LbR8isLjhZmFVO4d45he/3R8TQOJ8fiIP77wyR2zTOrC44WZhVzhJJJyWv3w38LOX+F8idYWBW95wszCrnYWC9pAfJbZF9dcr9G4Ef5g9wm9Ur7zprZmap3LIwM7NUThZmZpbKycLMzFI5WZiZWSonCzMzS+VkYWZmqZwszMws1f8HjjhbWWtXK4kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "composers_df.plot(x = 'birth', y = 'death', kind = 'scatter')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:04:46.851228Z",
     "start_time": "2020-06-24T15:04:46.568575Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEbCAYAAADnH5IjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvWUlEQVR4nO3de3hddZ3v8fcnaXqjF0KBOkAVsCI3oWrnQayjBXQQ5OLDZWBEHZlxGGbgKMNIUUbmVJQjdIbLHJkZQDzj8aggtxFBkOEWkLsgpQItNy30Agg1vaQNbZp8zx9r7XZlZSfZabKy904+r+fZT7LW+q7fXr/uNN/81u+yFBGYmZkNtYZqX4CZmY1MTjBmZlYIJxgzMyuEE4yZmRXCCcbMzArhBGNmZoVwgjEbRpJC0sxejp0i6b8HWN5cScuH5uoGRtIXJD04gPilkj5e0LUUVrZtOycYG1aSPiPpCUltkl6TdIekj1T7umpBRPwoIv60r5i+EtRoIen7kr5V7euw/jnB2LCRdDZwOfC/gOnAO4F/B46t4mUVQtKYWi7PbDg4wdiwkDQVuAA4IyJujoj1EdEREbdGxDlpzDhJl0tamb4ulzQuPTZX0nJJ8yT9Pm39fFrSkZJekPQHSedl3m++pBsl/UTSOkm/lnRg5vg+klokrZb0rKRjMseOlPRcet4KSV/JHDtK0sL0vIclHZA5tlTSuZIWAev7SApHSvqtpLck/bOkhvT8brec0tbKGZJeBF6U9EB66Om0BXhSJvYfMv8up/bxOZwqaXFat99K+pvMsdK/cdmyJE2T9DNJayU9Dry7t/dJ4z8n6RVJqyT9Y+5Yg6SvSno5PX69pB0yx2+Q9LqkNZIekLRfuv804BRgXvpvcGum2FmSFqXn/ETS+L6uz4ZBRPjlV+Ev4JPAZmBMHzEXAI8COwM7AQ8D30yPzU3P/yegCfhr4E3gx8BkYD+gHdgjjZ8PdAAnpPFfAX6Xft8EvAScB4wFDgXWAe9Nz30N+JP0+2bgA+n37wd+DxwENAJ/ASwFxqXHlwILgRnAhF7qGMB9wA4kLbgXgC+mx74APJiLvSuNnZDZNzMTU/p3uSCt15HABqC5l/f/FEliEPCxNPYDlZQFXAdcD2wH7A+syF5v7n32BdqAjwLjgEvTsj+eHv9y+lnvlh6/Crg2c/5fpp/rOJJW78LMse8D38q931LgcWCX9N9rMXB6tX/uR/ur6hfg1+h4kfzV+Xo/MS8DR2a2DweWpt/PJUkgjen25PSX7UGZ+CeBT6ffzwcezRxrIE0c6et1oCFz/Fpgfvr9q8DfAFNy1/cfpAkvs+954GPp90uBv+ynjgF8MrP9d8A96fdfoGeCObTM+fkE004mcZMkwQ9V+Ln8FPhyf2WRJNQOYO/Msf/VR4L5J+C6zPZ2wKZMglkMHJY5/kdp+T3+AAG2T+s9Nd3uLcF8NrO9ALiy2j/3o/3lW2Q2XFYBO/bTl7AL8Epm+5V035YyIqIz/b49/fpG5ng7MCmzvaz0TUR0AcvT8nYBlqX7su+1a/r98SR/vb8i6X5JB6f73wX8Q3p7bLWk1SStlew1LqN/2Zh8HfuK7c2qiNic2d5A93+HLSQdIenR9JbiapJ67lhBWTsBY8pce292ofu//3qSn4GSdwH/lfl3XAx0AtMlNUq6KL19tpYkeZC7znJeL3PdVkVOMDZcHgE2Ap/uI2YlyS+eknem+7bVjNI3aT/Hbml5K4EZpb6PzHutAIiIX0XEsSS36n5KclsIkl+YF0bE9pnXxIi4NlNOJcuTz8h8318dh2y587Q/6ybgX4DpEbE9cDvJ7bL+vElyiyt/7b15je7//hOBaZnjy4Ajcv+W4yNiBfAZkoEfHwemAruXikm/egn4OuEEY8MiItaQ3Db5t7RzfqKkpvQv6gVp2LXA1yXtJGnHNP6Hg3jbD0o6Lm01nUWS4B4FHiP5C3deeg1zgaOB6ySNVTIfZWpEdABrgVJL57vA6ZIOUmI7SZ+SNHmA13WOpGZJM0j6In4ygHPfAPYc4PuVjCXp03gT2CzpCKDPYdElacvxZmB++tntS9IH1ZsbgaMkfUTSWJJ+nezvmyuBCyW9CyD9zEujCSeTfFargIkkt+KyBvNvYMPICcaGTURcApwNfJ3kl9wy4EySVgLAt4AngEXAb4Bfp/u21S3ASUAr8DnguEhGrm0iSShHAG+RDJX+fEQsSc/7HLA0vT1zOkn/ERHxBMnggivSMl8i6TfZlut6kmRAwM+B7w3g3PnA/01vLf3ZQN40ItYBXyJpkbWStBR+NoAiziS57fQ6ST/If/bxXs8CZ5AMwngtfb/shNB/Td/7vyWtI0n8B6XHfkBy+20F8Fx6LOt7wL7pv8FPB3D9NsyUdoiZjSiS5pN0hn+22tdiNlq5BWNmZoVwgjEzs0L4FpmZmRXCLRgzMyuEF9BL7bjjjrH77rsPqoz169ez3XbbDc0F1TDXc+QYDXWE0VHPatXxySeffCsidip3zAkmtfvuu/PEE08MqoyWlhbmzp07NBdUw1zPkWM01BFGRz2rVUdJva7o4FtkZmZWCCcYMzMrhBOMmZkVwgnGzMwK4QRjZmaFcIIxMxvFVrVt5Ollq1nVtnHIy/YwZTOzUeqWhSs496ZFNDU00NHVxYLjD+CYWbv2f2KF3IIxMxuFVrVt5NybFvF2RxfrNm7m7Y4u5t20aEhbMk4wZmaj0PLWdpoauqeApoYGlre293LGwBWWYCSdKOlnklZIapP0pKQ/LxP315JelPR2GnNYmZhdJf2XpHWS3pJ0RfoI1gGXZWZmsFvzBDq6urrt6+jqYrfmCUP2HkW2YM4G2oC/B44B7gN+LOl/lALShHMlyRPsjgCeBW6TtH8mpgm4k+RZ7SeTPGL2RODq7JtVUpaZmSWmTRrHguMPYHxTA5PHjWF8UwMLjj+AaZPGDdl7FNnJf3REvJXZvlfSLiSJ5zvpvvnA/42IbwJIuh94P/BVoPQkwhOAfUieTvi7NK6D5Pnp34iIFwdQlpmZpY6ZtStzZu7I8tZ2dmueMKTJBQpsweSSS8lTwC4AkvYE9iJ5PnjpnC7gBpIWSMkRwK9KySX1U2AT8MkBlmVmZhnTJo3jwBnbD3lygeHv5D8YeCH9fu/065JczGJgB0k7ZeK6xUTEJuDlTBmVlmVmZsNk2ObBpB3unwb+Mt3VnH5dnQttzRx/M/2ajynFNWdiKykrf02nAacBTJ8+nZaWlr6q0K+2trZBl1EPXM+RYzTUEUZHPfN17OwKNnV2MbaxgcYGVeWahiXBSNod+DFwS0R8fzjesxIRcTXpYIHZs2fHYJ+lMBqeOQGu50gyGuoII7+eq9o28vgjD/K+2QczbdK4PidQrmrbWFifS17ht8gk7QDcAbwCnJI5VGpdTM2d0pw73lomphTXmovtrywzsxHlloUrmHPxvfzuzfXMufhefvTYK71OoCzFfvaax5hz8b38bOGKQq+t0ASTzlW5DRgLHBURGzKHS/0le+dO2xv4Q0S8mYnrFiNpLLBnpoxKyzIzGzGys/E7I3i7o4v5P3u2xy/2poYGnl25tvCZ+3lFTrQcQzKK6z3AJyPi99njEfFbkg7/EzPnNKTbd2RC7wD+WNK7MvuOAcYBvxhgWWZmI8azK9fQoO79Kx2dwYaOnhMoIQqfuZ9XZB/MvwNHkkyMnCZpWubYUxGxkWTuyg8lLQUeAv6CJCF9JhN7I/CPwM2Szie5DXYZ8OPMHBgqLMvMbES4ZeEK5t34NBs3R68x241tpDOCBccfwH67TC185n5ekQnmT9Ov/1rm2B7A0oi4VtIk4FzgfJLZ90dFxDOlwIjokPRJ4AqSeS4bgeuAc7IFVlKWmdlIULo11ldymdDUyDmH78XRB+66pTN/wfEHMC/X+V9kR39hCSYidq8w7rvAd/uJWU4yxHnQZZmZ1bvSQpVv09VrTHtHJ9++Ywk7bDduywiyomfu53k1ZTOzGtXbw8DKLVQpibGN3ftjNm6OHh35Rc7cz3OCMTOrQX0NKS63UOWM5glc8xd/zMSmxm7lFN2R3xc/0dLMrMZkhx+XboPNu2kRc2buuKXlkb/d9ZsnHmG/XabQRfd+maI78vviFoyZWY2p9GFg+dtdw7EE/0C4BWNmVmMG8zCw4e7I74tbMGZmw6y3zvuSwbZEhrMjvy9uwZiZDaO+FqLM6q0lMpyLVQ6WE4yZ2TCppPM+a9qkcd32V5qcaoVvkZmZDZNKO+/LySan4VqscrCcYMzMhsluzRNo79jcbV97x+aKOu8Hk5yqxQnGzGwYKbf6cX67N4MZWVYtTjBmZsNkeWs748d0n2k/fkxjRa2QWpvjUgl38puZDZPBtkJqaY5LJdyCMTMbJtvaCsnOm6mVOS6VcAvGzGwYDbQVUm9Dk7PcgjEzK0hvM/YrbYXU49DkLLdgzMwKMBQtj3IPFisNTa6HW2RuwZiZDbFyLY+v3LiIl95YN6By6nFocpYTjJnZECs3KXLT5i6O/N+/7PbgsP7U49DkLN8iMzMbYuVaHgCbOqPPtcfKqbehyVluwZiZDbFSy2PsmJ6/YqMrWN7a3u+S/fny6mVocpZbMGZmg9Db8vnHzNqVXaaO54SrHu0Wv7EzePS3q7js7hfqcujxQDjBmJlto/5GijWNaWR8UwNvd2RHgcEld73Aps2VLdlfzwq9RSZppqSrJC2S1CmppUzM9pL+j6Q/SGqTdIekmWXi9pV0j6QNklZKukBSYy5Gks6TtExSu6QHJM0qroZmNlpVMkel3Givji7o6opu+2p9VeRtVXQfzH7AkcDzwAu9xPwEOBz4MvAZYBpwj6QppQBJzcDdQADHAhcA/wB8I1fWV4HzgYuBo4E24G5J7xii+piZAclIsTEN3VdCzieKaZPGcfYn9upx7uZcgqmnoccDUfQtslsj4hYASTcCO2YPSjoY+FPg4xFxT7rvMeB3wGnAv6ShpwMTgOMiYi1wV5qA5ktaEBFrJY0nSTDfjogr0rIeAZYCZwJfL7SmZlZ3BvP44WdWrKFtY2e3feUSxQ4Tx5Y9v6lRjB/TuOXW2ki7PQYFJ5iI6DlOr7tZQAfQkjnnDUlPA59ia4I5ArgzTS4l15G0VD4G3Ap8GJgCXJ8pa72kW9PznWDMbIvBzLRf1baRb/78uR77zz9q3x6JYtaM7cuWce0XD6JpTGPdDT0eiGoPUx4PdEZEZ27/JmCfzPbewJJsQES8CmxIj5ViOoEXc2UtzsSYmVXUf1JuGHFp37Mr1/aYSLnd2Eb232Vqj/eaOX0ynz/4nd32ff7gdzJ7j2l1OfR4IKo9iuwlYLyk90XEbwAkTQD2ByZn4pqB1WXOb02PlWLayiSrVmCipLERsWkoL97M6lO5Nb4aJe5b8nsO2XtnHnzprR6tm4At+zZ1dtGZm0jZGdFrP8oFx76Pz39odxYuW82sGdszc/rksnEjjSKi/6iheKO0DyYi5mb2jSVpmbwOnAqsBS4CTgE2R8T4NK4DOCciLs+VuRz4QUScJ+kf05jtczFfBL4LjMsnGEmnkfT1MH369A9ed911g6pjW1sbkyZNGlQZ9cD1HDlGQx2hZz07u4Ilr6+jK/f7r1EigIgge6QhfaxxNl4SAkQy+mi35glMndBUWB36U63P8pBDDnkyImaXO1bVFkxEbJJ0MnAtW2+BPQj8ADg0E9oK9Gx7Jq2W1kzMJEmNuVZMM7ChXOslIq4GrgaYPXt2zJ07dxC1gZaWFgZbRj1wPUeO0VBHKF/P9QtXMO+mRTRKrN+Uv/HR3cSxjRCwoWNr3ORxY/i3U97P1Alja6IfpRY/y2rfIiMiHk/nvexF0mp5WdJtQHb66xJy/SiSZgAT2ZqYlgCNwEySYdElPfpvzMxKa3zdt+T3zL/12R4jwrI6u4LNnd1vibV3bGa/XaZWPbHUsmp38gMQiefT5PIe4OPA9zIhdwCHS8reuDwJaAfuT7cfJrnFdmIpQNJEkvkwdxR5/WZWv3aeMo6OXPLITW/h2Fm70JDfCTy7ck3dPPyrGgptwaS/4I9MN3cFpkg6Id2+PSI2SDqfpIXxFvA+komS10XEXZmirgS+BNws6WJgT2A+cGlp6HJEvC3pIuB8Sa1pmWeTJNHvFFhNM6tD2WHKXQFjGmBC0xg2dXbSFdDVubW/5acLVzK2sYGOzq2tnM1dcPoPf01XxIhdS2ywir5FtjNwQ25faXsPkkmQ04DLSSZhLiOZ+3JJ9oSIaJV0GHAFyZyX1cBlJEkm6yKShPK1tNwngE9ExBtDUBczGyGyw5RLI8nGjWng3075ABCc8aOn6OjcvCW+qVF0bO45rW9D2nczUtcSG6yiJ1ouJRlk0VfMWcBZFZT1HN07/svFBHBh+jIzK6vcMOWxjQ1MndBU9lkunV3B/zx6P7758+doQN06+6G+HmM8nGqiD8bMbDj19Sji3p4iecqH3sVD5x7KlZ/7IONyz3kZqWuJDVbVR5GZmQ2H/LpjC44/gHm5yZSlFkhvT5GcNmkcH91rJ/75hN7Pta2cYMxsxFvT3sGci+/tse5YX48injZpXK9Jo54fYzycnGDMbEQrtVze7mgs+4CvbU0Ogzl3tHAfjJmNaMtb23uMNBqpD/iqNU4wZjai7dY8gfyKi+6UHx5OMGY2ok2bNI7dmif0GBUG9FiO34aW+2DMbMSbOqGJh8796JZO+Qdfeqtsp78NLbdgzGxUmDZpHAemT5fs72FjNjScYMxsRCr3RErYOos/y53+xfAtMjMbcbILWXZ0dXHxh7f+qutrFr8NLbdgzGxEyS5kWboFtry1fUtLprelYDynZei5BWNmI0q5hSyV7u9vKRgbWk4wZjailLsF1gVsN7ax2z7PxC+eb5GZ2YiSvQU2rjGZwy/gqCse5GcLV1T34kYZJxgzG3GOmbUrt535EUJJgumK8HDkKnCCMbOa1dtQ40qs39TJuEYPR64m98GYWU3KDzUe6Gx7D0euPrdgzKzmlBtqPNDbW9m+mEbJw5GrwC0YM6s55YYab8tz70vDkR9/5EEeOuYjTi7DzC0YM6s5Q3l7a9qkcUxoanRyqQInGDOrOZ5tPzL4FpmZ1STPtq9/TjBmNmil594PdSLwbPv6VugtMkkzJV0laZGkTkktZWL+SNJ/SlohqU3SU5JOKRO3r6R7JG2QtFLSBZIaczGSdJ6kZZLaJT0gaVZxNTSzWxauYM7F9/LZax5jzsX3VjRbfjDzW6x+FN2C2Q84EngUaMoflNQA/AyYBswDXgdOAH4oqT0ibk7jmoG7geeAY4F3A5eQJMivZ4r8KnA+cA6wBDgbuFvS/hHxehEVNBvNssOJSyO+5t20iDkzd+y15THY+S1WP4ru5L81ImZExInAs2WO7wXMBr4cET+KiHsi4m+Bp4CTMnGnAxOA4yLiroi4EvgGcLakKQCSxpMkmG9HxBURcTdwIhDAmUVV0Gw0W97aTmO6HEtJb7PlV7Vt5IEX3mTejX6a5GhRaIKJiK5+QkqtmjW5/atJ1qcrOQK4MyLWZvZdR5J0PpZufxiYAlyfef/1wK3p+WY2xJ5ZsYb1mzq77Ss3nLh0G+30//ckGzd3/7Xg5VtGrmoPU34GeAy4QNJ7JE2R9AVgDnBlJm5vklteW0TEq8CG9FgpphN4MfceizMxZjZEVrVt5Js/f67H/vM/tW+322PZ22gbOjp7xHv5lpGrqqPIIiIkHQHcAryQ7u4ATo2IezOhzSStmrzW9Fgppi0i8j/BrcBESWMjYlP2gKTTgNMApk+fTktLyyBqA21tbYMuox64niPHYOrY3tHJl/bpoDNiy74GiR3WvURLy+/6jCvFBrDL1CZ+88Qj23QNlfJnWR1VTTBpJ/8PSDr5TwJ+TzIo4HuSVkXEL4p8/4i4GrgaYPbs2TF37txBldfS0sJgy6gHrufIMZg6rmrbyN9ffC9vd2y95TW+qaHHkizl4sY0gARjGxvpjE4WHP+eQjv6/VlWR7VvkR2Vvj4dEddHREtEzAP+C1iQiWsFppY5vzk9VoqZlB+6nMZsyLdezGxwKp1tn48bNyb5tdPRmSyp747+kavaEy33Jvnln+83eQo4JrO9hFw/iqQZwES29s0sARqBmcDzuffo1n9jZkOj0tn22bg17R2c8aNfs27j5i3Ht2UhS6t91W7BvELSP/Le3P4PAksz23cAh0uanNl3EtAO3J9uPwysJRmaDICkicDR6flmVoBpk8Zx4Izt+00Opbj9dpni57SMEkXP5J8o6QRJJwC7AjuVttNf/rcDrwI/lfTnkj4u6TLgz4B/yxR1JbARuDmNOQ2YD1xaGrocEW8DFwHnSTpD0mHADWkdv1NkPc2scl7IcvQo+hbZziS/5LNK23tExNI0EXybZGb+FOBlkomVV5dOiIjWNO4Kknktq4HLSJJM1kUkCeVrJAMHngA+ERFvDF2VzGywvJDl6FBogomIpXSfMFku5iUyt7X6iHsOOLSfmAAuTF9mVsO8kOXIV+0+GDMzG6EqSjCSjpP0oqQ1ktZKWidpbf9nmpnZaFXpLbIFwNERsbjIizGzwSvq2SxmA1VpgnnDycWs9nkpfKslfSYYScel3z4h6SfAT0mGCwNQel6LmVVfuWezfOXGRez7R1OYOX1yP2ebDb3++mCOTl9TSFYu/tPMvqOKvTQzG4jlre00NXT/L71pcxdH/OsDFT1l0myo9dmCiYhTASTNiYiHssckzSnywsxsYHZrntBjhjxARxecff3CPp8yaVaESocpl5sJ79nxZjWkNEO+qbHn1LPNXfDsSg/8tOHVXx/MwSRPitxJ0tmZQ1NIFpY0sxpyzKxdaZA489qnyhyNMvvMitPfKLKxwKQ0LttLuBY4oaiLMrNtt/c7ynfo7zLVi0na8OqvD+Z+4H5J34+IV4bpmsxsENZv6mR8U0OPB4Gt39TzccVmRap0HswGSf8M7AeML+2MiD7XBjOz4dfbsvdeDt+GW6Wd/D8ieWjXHsA3SJ7V8quCrsnM+rGqbSNPL1td9imQg10Ov6+yzQai0hbMtIj4nqQvZ26bOcGYVUEls/W3dTl8rwRgQ6nSFkxH+vU1SZ+S9H5gh4Kuycx6kZ2tv27j5j6fZ1/pkya3pWyzSlSaYL4laSrwD8BXgGuAvy/sqsysrHKz9UvPs6/lsm10qugWWUTcln67BjikuMsxs76Um60/VM+zL7JsG50qfR7MXpLukfRMun2ApK8Xe2lmllfk8+yLLNtGp0o7+b8LnANcBRARiyT9GPhWURdmZuUV+Tz7Isu20afSBDMxIh6Xuq1xtLmA6zGzChT5PPsiy7bRpdJO/rckvZt0MSNJJwCvFXZVZmZW9yptwZwBXA3sLWkF8DvglMKuyszM6l5/qylnV1C+HbiPpNWzHjgeuLS4SzMzs3rW3y2yyelrNvC3QDOwPXA68IH+Cpc0U9JVkhZJ6pTUkjs+V1L08rozF7tvOpJtg6SVki6Q1JiLkaTzJC2T1C7pAUmz+rtOMzMbev2tpvwNAEkPAB+IiHXp9nzg5xWUvx9wJPAo0FTm+K+Bg3P73gn8BLijtENSM3A38BxwLPBu4BKSBJkdLv1V4HySEW9LgLOBuyXtHxGvV3C9ZsNiVdtGj9SyEa/SPpjpwKbM9qZ0X39ujYhbACTdCOyYPRgRa0mSzxaS/gToAq7P7D4dmAAcl55zl6QpwHxJCyJiraTxJAnm2xFxRVrWIyQLc55J90RkVjVe78tGi0pHkf0AeFzS/LT18hjw/f5OioieDwjv358D90fEysy+I4A70+RSch1J0vlYuv1hkidtbklMEbEeuDU936zqvN6XjSYVJZiIuBA4FWhNX6dGxLeH+mIk7QW8H7g2d2hvklte2Wt6FdiQHivFdAIv5s5dnIkxq6qBrvflpfOtnlV6i4yI+DVJn0mRTiZZufmm3P5mYHWZ+Nb0WCmmLSLyj+1rBSZKGhsR2dt8SDoNOA1g+vTptLS0DOri29raBl1GPXA9t11nV3D6ezfSFbFlX4M6eevFp2h5udtEZta0d7C8tR2RTEDbrXkCUyeU68rcdv4sR45arGPFCWaYnAz8d0T8YTjeLCKuJpnfw+zZs2Pu3LmDKq+lpYXBllEPXM/BWb9wBfNyfTCH5fpgVrVtZM7F9/J2x9aBkuObNvPQuR8d0kEB/ixHjlqsY80kGEkHAvsAF5Y53ApMLbO/OT1WipkkqTHXimkGNuRbL2bVUsl6X6VbaW+ztRuzdCvNo86sXtRMgiFpvbQDt5Q5toRcP4qkGcBEtvbNLAEagZnA85nQHv03ZtXW33pfXjrfRoJKR5ENh5NJhjW3lTl2B3C4pMmZfSeRJKT70+2HgbXAiaUASROBo8nMqTGrB14630aCQlsw6S/4I9PNXYEp6UKZALdHxIY07kPA7vT+lMwrgS8BN0u6GNgTmA9cWhq6HBFvS7oIOF9SK1snWjYA3xniqpkVzkvnW70r+hbZzsANuX2l7T1IJkFC0npZQy8tjYholXQYcAXJvJbVwGUkSSbrIpKE8jVgGvAE8ImIeGMQdTCrGi+db/Ws0AQTEUsBVRB3FnBWPzHPAYf2ExMkgwTKDRQwM7NhVEt9MGZmNoI4wZiZWSGcYMzMrBBOMGZmVggnGLMh5gUqzRK1NJPfrO75WS9mW7kFYzZE/KwXs+6cYMyGyECf9WI20jnBmA0RL1Bp1p0TjNkQ8QKVZt25k99GjFVtG6u+MKQXqDTbygnGRoSBjt4qMhl5gUqzhBOM1b3s6K3SEyDn3bSIOTN3LPuL3kOJzYaH+2Cs7g1k9JaHEpsNHycYq3sDGb3locRmw8cJxureQEZveSix2fBxH4zVtVJn/ZyZO/LQuYf223FfSkbzcn0w7pQ3G3pOMFa3trWz3kOJzYaHb5FZzRjIKsSD7ayfNmkcB87Y3snFrEBuwVhNGGhrpNRZXxqWDFs76500zGqDWzBWddvSGnFnvVntc4KxqtuWocP5kWPjxjRwxtyZRV+qmQ2AE4xV3ba2Ro6ZtSsPnXsof/3RPYHg6gd+y5yL7+VnC1cUeLVmViknGKu6wa5C/O8tL7Fxc3hmvlmNKTTBSJop6SpJiyR1SmrpJe59km6TtEbSOkmPS/pgLmZfSfdI2iBppaQLJDXmYiTpPEnLJLVLekDSrOJqaEOl1Br54RcP4qFzD614bTDPzDerXUWPItsPOBJ4FGgqF5AmgF8CtwAnpbv/GJiQiWkG7gaeA44F3g1cQpIgv54p7qvA+cA5wBLgbOBuSftHxOtDVSkrxrasQuzOfrPaVXSCuTUibgGQdCOwY5mYK9O4z2b2/SIXczpJwjkuItYCd0maAsyXtCAi1koaT5Jgvh0RV6Tv+QiwFDiT7onIRgjPzDerXYUmmIjo6uu4pH2Bg4C/76eoI4A70+RSch1wMfAx4Fbgw8AU4PrM+6+XdGt6vhPMCOWZ+Wa1qdqd/AelX5slPS1ps6SXJf1VLm5vklteW0TEq8CG9FgpphN4MXfu4kyMVUl+lv5AZu1XwjPzzWpPtWfyvyP9+gNgAfAr4ATgGkmvRcTt6fFmYHWZ81vTY6WYtojoLBMzUdLYiNiUPSDpNOA0gOnTp9PS0jKoyrS1tQ26jHow0Hquae9geWs7AgJonthE64aOLdu7NU9g6oSyXXRVNRo+z9FQRxgd9azFOlY7wSj9ek1ELEi/v0/SPsDXgNvLnzY0IuJq4GqA2bNnx9y5cwdVXktLC4Mtox4MpJ6r2jYy5+J7ebsjO+CvC9i6Pb5pMw+d+9Gaa32Mhs9zNNQRRkc9a7GO1b5F1pp+vS+3/15g31zc1DLnN2fKaAUm5YcupzEb8q0XGx7lhhHneVix2chU7QSzOP2q3H4B2QECS8j1o0iaAUxka9/MEpI/i/PrhfTov7HhU24YcZ6HFZuNTNVOMA+TtDwOze0/DHg6s30HcLikyZl9JwHtwP2ZstYCJ5YCJE0Ejk7PtyooN0v/8we/c5tn7ZtZ/Si0Dyb9BX9kurkrMEXSCen27RGxQdIFwAJJq0k6+Y8HPkoy/LjkSuBLwM2SLgb2BOYDl5aGLkfE25IuAs6X1MrWiZYNwHeKq6X1p9ww4i8ftpeHFZuNcEV38u8M3JDbV9reA1gaEZdLagD+B0nSeB44ISJ+WTohIlolHQZcQTLnZTVwWRqfdRFJQvkaMA14AvhERLwxdFWybZGfpb8ts/bNrL4UPdFyKT37V8rFXQpc2k/Mc/S8lZaPCeDC9GVmZlVU7T4YMzMboZxgzMysEE4wZmZWCCcYMzMrhBOMmZkVwgnGzMwK4QRjZmaFcIIxM7NCOMGYmVkhnGDMzKwQTjBmZlYIJxgzMyuEE4yZmRXCCcbMzArhBGNmZoVwgjEzs0I4wZiZWSGcYMzMrBBOMGZmVggnGDMzK4QTjJmZFcIJxszMCuEEY2ZmhSg0wUiaKekqSYskdUpqKROzVFLkXq+XidtX0j2SNkhaKekCSY25GEk6T9IySe2SHpA0q7gamplZb8YUXP5+wJHAo0BTH3E/Br6T2d6UPSipGbgbeA44Fng3cAlJgvx6JvSrwPnAOcAS4Gzgbkn7R0SPpGVmZsUpOsHcGhG3AEi6Edixl7jXIuLRPso5HZgAHBcRa4G7JE0B5ktaEBFrJY0nSTDfjogr0vd8BFgKnEn3RGRmZgUr9BZZRHQNUVFHAHemyaXkOpKk87F0+8PAFOD6zPuvB25Nzzczs2FUK538fyVpk6Q1km6U9K7c8b1JbnltERGvAhvSY6WYTuDF3LmLMzFmZjZMir5FVolbSPpolgP7AP8T+KWk90XEmjSmGVhd5tzW9Fgppi0iOsvETJQ0NiLyfTunAacBTJ8+nZaWlkFVpK2tbdBl1APXc+QYDXWE0VHPWqxj1RNMRHw5s/lLSQ8DC4FTgcsLfu+rgasBZs+eHXPnzh1UeS0tLQy2jHrgeo4co6GOMDrqWYt1rJVbZFtExDPA88AHMrtbgallwpvTY6WYSfmhy2nMhnzrxczMilVzCSYV6atkCbl+FEkzgIls7ZtZAjQCM3Nl9ei/MTOz4tVcgpG0P0lSeDKz+w7gcEmTM/tOAtqB+9Pth4G1wImZsiYCR6fnm5nZMCq0Dyb9BX9kurkrMEXSCen27cAhwGeB24CVJInl68CrwPczRV0JfAm4WdLFwJ7AfODS0tDliHhb0kXA+ZJa2TrRsoHukzjNzGwYFN3JvzNwQ25faXsPYFkaczmwPbAK+AVwXnbOS0S0SjoMuIJkXstq4DKSJJN1EUlC+RowDXgC+EREvDFE9TEzswoVmmAiYimgfsIOq7Cs54BD+4kJ4ML0NWxWtW1keWs7nV3Rf7CZ2ShR9WHK9e6WhSs496ZFNDU0cPp7N7J+4QqOmbVrtS/LzKzqaq6Tv56satvIuTct4u2OLtZt3ExXBPNuWsSqto3VvjQzs6pzghmE5a3tNDV0/ydsamhgeWt7la7IzKx2OMEMwm7NE+jo6r6eZ0dXF7s1T6jSFZmZ1Q4nmEGYNmkcC44/gPFNDUweN4YGiQXHH8C0SeOqfWlmZlXnTv5BOmbWrsyZuSPLW9t568WnOMwd/GZmgFswQ2LapHEcOGN7Ghv6G5FtZjZ6OMGYmVkhnGDMzKwQTjBmZlYIJxgzMyuEE4yZmRVCyfqQJulN4JVBFrMj8NYQXE6tcz1HjtFQRxgd9axWHd8VETuVO+AEM4QkPRERs6t9HUVzPUeO0VBHGB31rMU6+haZmZkVwgnGzMwK4QQztK6u9gUME9dz5BgNdYTRUc+aq6P7YMzMrBBuwZiZWSGcYMzMrBBOMClJMyVdJWmRpE5JLf3EXyYpJP1LmWP7SrpH0gZJKyVdIKkxFyNJ50laJqld0gOSZg1trcped7/1lLQ0rVv29XqZuJqsZ6WfpaT3SbpN0hpJ6yQ9LumDuZiarGP6vn3WU9LcMp9j6XXnSKlnGvNHkv5T0gpJbZKeknRKmbiarGeFddxe0v+R9Ie0jndImlkmrnbqGBF+Jf1QxwLLgBuAxUBLH7H7AmuBNcC/5I41AyuBu4FPAKcD64Fv5eK+BrQDZwIfB24nmST1jmrXE1gK/Aj4UOb1gXqpZ4V1nAWsA34IfDJ9nQ98pB7qWEk9gSm5z/BDwJ8BAZw1gurZAPwK+C1wCnAY8B9pPY+rh3pW+DN7J7AC+BxwDPA4yeTwKbVax8J+KOrtBTRkvr+x3AecOX4P8E2SX8T5BPM1oDX3oc8DNpT2AeNJktM/ZWK2A97M/yBUo57l6lUmpmbrWWEdHwV+XK91HOjPbCbuHKAT2GWk1BPYmySZHJ3b/2vgJ/VQzwrqeHBax8My+6an1/6VWq2jb5GlIqKrkjhJJ5D8QF/US8gRwJ0RsTaz7zpgAvCxdPvDJH9dXp95//XAren5ham0nhWo2Xr2V0dJ+wIHAd/pp6iarWP6PtvyWf45cH9ErMzsq/d6NqVf1+T2rwayTwGs2XpWUMdZQAfQkjnnDeBp4FOZuJqqoxPMAEiaAFwCfDX9QMrZG1iS3RERr5L8BbF3JqYTeDF37uJMTLX9laRNaf/EjZLelTtez/U8KP3aLOlpSZslvSzpr3Jx9VzHHiTtBbwfuDZ3qN7r+QzwGHCBpPdImiLpC8Ac4MpMXD3XczzQGRGduf2bgH0y2zVVRyeYgfka8BrJffveNJP85ZTXmh4rxbSV+WFpBSZKGjvI6xysW4C/I7mXfQ5J8/yXkqZmYuq5nu9Iv/6ApK/pE8AvgGskHZmJq+c6lnMyyV/BN+X213U9I7nHcwTJ77MXSFoyVwN/GRH3ZkLruZ4vAeMlva+0I/2Dd39gh0xcTdVxzFAVNNJJ2gP4CnBI+gM9YkXElzObv5T0MLAQOBW4vBrXNMRKt02uiYgF6ff3SdqH5I+I26tzWYU7GfjviPhDtS9kKElqIPljYRpwEvB74Ejge5JWRcQvqnl9Q+RO4HfAVZJOJRlkdBEwFdhczQvri1swlbsIuAN4Ph0uuD3Jv9+4dLv0S6uV5EPPa06PlWIm5YcOpjEbImLTkF/9IETEM8DzwAcyu+u5nqXruy+3/16SEYLZuHqtYzeSDiS5lZK/PQb1X8+j0tenI+L6iGiJiHnAfwELMnF1W8/0fU8m6dhfQjJSbE+SxJqdQlBTdXSCqdx7geNIPpzSawbJML9WYNc0bgm5+5iSZgAT2XpvdAnQCOTHsPe4f1pDIn2V1HM9F6dfldsvINvZWs91zDuZZFjqLWWO1Xs99yb5xZjvU3gKeHdmu67rGRGPk1zX3sDMiPgTYGeSEZElNVVHJ5jKfRE4JPd6g2QkxiEkQ/wgaeUcLmly5tyTSP5z359uP0zSxD2xFCBpInB0en5NkbQ/yQ/fk5nd9VzPh0n+KDg0t/8wklE5JfVcx7yTgVsjoq3MsXqv5yskfQfvze3/IMmQ+5J6ryeReD4iXpb0HpI5LN/LhNRWHYsa111vL5IMf0L6egR4NrM9sZdzllJ+ouVrwF0kH/5pQBvlJzptAM4g+cX2c5KJTtOrWU+SIY/XkkxYOwT4W5LJXb+l54SumqxnJZ8lcBbJCJzzSDr5ryRpvfxJPdRxID+zJBMsg+QWUrly6rqewGSSJLOYZBj2x4HL0jr/XT3Us8Kf2fNJksIhwJdI/qj9fi1/loX9UNTbC9idrbeB8q/dezlnKWUmJJLcx7+X5K+G10gmZTbmYgT8I7A8jfsl8P5q1xM4gGQi6ZskI45eB75PZmJerdez0s8SOJuk43QT8Bsys75rvY4DrOflJCOLxvVRVl3Xk+R2zw0kfRNtJC3RvyFdMb7W61lhHS8n+WNvI8mosnOBMbX8WXq5fjMzK4T7YMzMrBBOMGZmVggnGDMzK4QTjJmZFcIJxszMCuEEY2ZmhXCCMasSSbtLeqbM/mvSZ9aUO+esdNZ1abvczHyzmuAEY1ZjIuKLEfFcfn+6OOFZJLO+zWqeE4xZdY2R9CNJi9MHu02U1CJpNiQtFEmXSHqaZOb1LiSPFtiyErSkC9MHpz0qaXqV6mHWgxOMWXW9F/j3iNiHZAHCv8sd3w54LCIOjIgLSJZCOSQiDskcfzQiDgQeAP56mK7brF9OMGbVtSwiHkq//yHwkdzxTno+gTJrE3Bb+v2TJGtamdUEJxiz6sovBpjffjt6Pto2qyO2LijYiZ9SazXECcasut4p6eD0+88AD/YTv45keXqzmucEY1ZdzwNnSFpM8iyP/+gn/mrgF9lOfrNa5eX6zcysEG7BmJlZIZxgzMysEE4wZmZWCCcYMzMrhBOMmZkVwgnGzMwK4QRjZmaF+P9OzIxleaviVAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "composers_df.plot(x = 'birth', y = 'death', kind = 'scatter',\n",
    "                  title = 'Composer birth and death', grid = True, fontsize = 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Some additional plotting options are available in the `plot` module. For example histograms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:05:37.947613Z",
     "start_time": "2020-06-24T15:05:37.641676Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXC0lEQVR4nO3dfZRV9X3v8fdXHqRcND6A1ogUzIpWkgbFMVIFI2obavAhFaOu0KUhdq7aeI0PTTXepax740quscXWmzRFY9XEaHyqCanplaQQdYkPgBAfEI1KzChRgvdGoyKi3/vH2eA4mYEzMOfsObPfr7Vmsc9v73N+39+cmQ/7/PaevSMzkSRVx3ZlFyBJai6DX5IqxuCXpIox+CWpYgx+SaqYwWUXUI+RI0fm2LFjyy5DklrKkiVLfpOZo7q2t0Twjx07lsWLF5ddhiS1lIj4ZXftTvVIUsUY/JJUMQa/JFVMS8zxd+ftt9+mo6ODdevWlV1KKYYNG8bo0aMZMmRI2aVIajEtG/wdHR3ssMMOjB07logou5ymykzWrl1LR0cH48aNK7scSS2mZad61q1bx6677lq50AeICHbdddfKftqRtG0aFvwRcW1EvBwRj3VpPzsinoyIxyPi8m3sY9uKbGFVHrukbdPIPf7rgGmdGyJiKnAcMCEzPwJc0cD+JUndaNgcf2beExFjuzSfCXwtM98qtnm5r/qbM/+pvnopAM79s322uM2qVauYPn06jz32vg81nH766Zx33nmMHz/+955z5ZVX0t7ezvDhwwEYMWIEv/vd7/qmaEmqQ7MP7u4DTImIy4B1wAWZ+XB3G0ZEO9AOMGbMmOZV2AeuueaabtvfeecdrrzySmbOnLkp+KVW09c7Wb1Rzw6ZtqzZB3cHA7sAk4C/BW6JHiarM3NuZrZlZtuoUb93qYl+Y8OGDXz2s59lv/32Y8aMGbzxxhscfvjhmy4xMWLECM4//3wmTJjAZZddxosvvsjUqVOZOnXqpte4+OKLmTBhApMmTeKll14qayiSKqLZwd8B3JE1DwHvAiObXEOfWrlyJWeddRYrVqxgxx135Jvf/Ob71r/++uscfPDBLF++nEsuuYQPfvCDLFiwgAULFmxaP2nSJJYvX85hhx3G1VdfXcYwJFVIs4P/TmAqQETsAwwFftPkGvrUXnvtxaGHHgrAzJkzue+++963ftCgQZxwwgk9Pn/o0KFMnz4dgAMPPJBVq1Y1rFZJggbO8UfETcDhwMiI6AAuBa4Fri1O8VwPnJotfrf3rjNVXR8PGzaMQYMG9fj8IUOGbHrOoEGD2LBhQ98XKUmdNGyPPzNPycw9MnNIZo7OzG9n5vrMnJmZH83MiZn5n43qv1mef/55Fi1aBMD3vvc9Jk+evNntd9hhB1577bVmlCZJ3WrZSzZ0VdbR/n333ZdvfOMbzJo1i/Hjx3PmmWcyb968Hrdvb29n2rRpm+b6JanZohVmWtra2rLrjVhWrFjBfvvtV1JF/YPfA5XB0zlbR0Qsycy2ru0te60eSdLWMfglqWIMfkmqGINfkirG4Jekihkwp3NKqoAFXy2n36kXldNvgwyc4O/rH4hevtGzZ89mxIgRXHDBBb163sKFCxk6dCiHHHIIAKeddhrTp09nxowZvXodSaqXUz0lW7hwIffff3/ZZUiqEIN/G1x22WXss88+TJ48mZUrVwLwzDPPMG3aNA488ECmTJnCk08+CcC8efM4+OCDOeCAAzjqqKN46aWXWLVqFd/61reYM2cO+++/P/feey8A99xzD4cccgh77703t912W2njkzQwGfxbacmSJdx8880sW7aMu+66i4cfrt1Ppr29nauuuoolS5ZwxRVXcNZZZwEwefJkHnjgAR555BFOPvlkLr/8csaOHcsZZ5zBueeey7Jly5gyZQoAq1ev5r777uNHP/oRF154YWljlDQwDZw5/ia79957+fSnP73pTlrHHnss69at4/777+fEE0/ctN1bb70FQEdHByeddBKrV69m/fr1jBs3rsfXPv7449luu+0YP368N2aR1OcM/j707rvvstNOO7Fs2bLfW3f22Wdz3nnnceyxx7Jw4UJmz57d4+tsv/32m5Zb4VpKklqLUz1b6bDDDuPOO+/kzTff5LXXXmPevHkMHz6ccePGceuttwK10F6+fDkAv/3tb9lzzz0BuP766ze9jpdpltRsA2ePv8nn2U6cOJGTTjqJCRMmsNtuu3HQQQcBcOONN3LmmWfyla98hbfffpuTTz6ZCRMmMHv2bE488UR23nlnjjjiCJ577jkAjjnmGGbMmMEPfvADrrrqqqaOQVI1NeyyzBFxLTAdeDkzP9pl3fnAFcCozNzirRe9LHP3/B6oDKVelnnw7eV03KJ/wFXGZZmvA6Z1U8hewJ8Dzzewb0lSDxp568V7gFe6WTUH+BLgUUtJKkFTD+5GxHHAC5m5vC9er8pnvFR57JK2TdOCPyKGA18GLqlz+/aIWBwRi9esWfN764cNG8batWsrGYCZydq1axk2bFjZpUhqQc08q+dDwDhgeUQAjAaWRsTHM/PXXTfOzLnAXKgd3O26fvTo0XR0dNDdfwpVMGzYMEaPHl12GZJaUNOCPzMfBXbb+DgiVgFt9ZzV050hQ4Zs9q9fJUnda9hUT0TcBCwC9o2Ijoj4fKP6kiTVr2F7/Jl5yhbWj21U35KknnnJBkmqGINfkirG4JekijH4JaliDH5JqhiDX5IqxuCXpIox+CWpYgx+SaoYg1+SKsbgl6SKMfglqWIMfkmqGINfkirG4JekijH4JaliDH5JqphG3nrx2oh4OSIe69T29Yh4MiJ+HhH/FhE7Nap/SVL3GrnHfx0wrUvbfOCjmfkx4Cngogb2L0nqRsOCPzPvAV7p0nZ3Zm4oHj4AjG5U/5Kk7pU5xz8L+HFPKyOiPSIWR8TiNWvWNLEsSRrYSgn+iLgY2ADc2NM2mTk3M9sys23UqFHNK06SBrjBze4wIk4DpgNHZmY2u39JqrqmBn9ETAO+BHwiM99oZt+SpJpGns55E7AI2DciOiLi88D/BnYA5kfEsoj4VqP6lyR1r2F7/Jl5SjfN325Uf5Kk+viXu5JUMQa/JFWMwS9JFWPwS1LFGPySVDEGvyRVjMEvSRVj8EtSxRj8klQxBr8kVYzBL0kVY/BLUsUY/JJUMQa/JFWMwS9JFWPwS1LFGPySVDGNvPXitRHxckQ81qltl4iYHxFPF//u3Kj+JUndqyv4I+JPtuK1rwOmdWm7EPhpZn4Y+GnxWJLURPXu8X8zIh6KiLMi4gP1PCEz7wFe6dJ8HHB9sXw9cHyd/UuS+khdN1vPzCkR8WFgFrAkIh4C/jUz5/eyv90zc3Wx/Gtg9542jIh2oB1gzJgxvexGGvjmzH+q7BLUouqe48/Mp4H/Dvwd8AngnyLiyYj4y63pODMTyM2sn5uZbZnZNmrUqK3pQpLUjXrn+D8WEXOAFcARwDGZuV+xPKcX/b0UEXsUr7kH8HIv65UkbaN69/ivApYCEzLzbzJzKUBmvkjtU0C9fgicWiyfCvygF8+VJPWBuub4gU8Bb2bmOwARsR0wLDPfyMzvdPeEiLgJOBwYGREdwKXA14BbIuLzwC+Bz2xj/ZKkXqo3+H8CHAX8rng8HLgbOKSnJ2TmKT2sOrLu6iRJfa7eqZ5hmbkx9CmWhzemJElSI9Ub/K9HxMSNDyLiQODNxpQkSWqkeqd6vgjcGhEvAgH8IXBSo4qSJDVOvX/A9XBE/DGwb9G0MjPfblxZkqRGqXePH+AgYGzxnIkRQWbe0JCqJEkNU1fwR8R3gA8By4B3iuYEDH5JajH17vG3AeOLyyxIklpYvcH/GLUDuqu3tKEkDTgLvlpe31Mv6vOXrDf4RwJPFFflfGtjY2Ye2+cVSZIaqt7gn93IIiRJzVPv6Zw/i4g/Aj6cmT+JiOHAoMaWJklqhHovy/zXwG3AvxRNewJ3NqgmSVID1XvJhr8BDgVehU03ZdmtUUVJkhqn3uB/KzPXb3wQEYPZzN2zJEn9V73B/7OI+DLwBxHxZ8CtwLzGlSVJapR6g/9CYA3wKPBfgbvo3Z23JEn9RL1n9bwLXF18SZJaWL3X6nmObub0M3Pvrek0Is4FTi9e81Hgc5m5bmteS5LUO725Vs9Gw4ATgV22psOI2BP4b9Su/fNmRNwCnAxctzWvJ0nqnbrm+DNzbaevFzLzSmo3YN9ag6kdKB5M7RaOL27Da0mSeqHeqZ6JnR5uR+0TQG+u5b9JZr4QEVcAz1O7fePdmXl3N322A+0AY8aM2ZquJDXApOfnltb3opL6/dO9dy2p58aoN7z/vtPyBmAV8Jmt6TAidgaOA8YB/4/aLR1nZuZ3O2+XmXOBuQBtbW3+zYAk9ZF6z+qZ2od9HgU8l5lrACLiDuAQ4LubfZYkqU/UO9Vz3ubWZ+Y/9KLP54FJxYXe3gSOBBb34vmSpG3Qm7N6DgJ+WDw+BngIeLq3HWbmgxFxG7CU2rTRIxRTOpKkxqs3+EcDEzPzNYCImA38e2bO3JpOM/NS4NKtea4kadvUe8mG3YH1nR6vL9okSS2m3j3+G4CHIuLfisfHA9c3pCJJUkPVe1bPZRHxY2BK0fS5zHykcWVJkhql3qkeqP2F7auZ+Y9AR0SMa1BNkqQGqvfWi5cCfwdcVDQNwfPuJakl1bvH/2ngWOB1gMx8EdihUUVJkhqn3uBfn5lJcWnmiPgvjStJktRI9Z7Vc0tE/AuwU0T8NTALb8qirhZ8tby+p1605W0kAXUEf0QE8H3gj4FXgX2BSzJzfoNrkyQ1wBaDPzMzIu7KzD8BDHtJanH1zvEvjYiDGlqJJKkp6p3jPxiYGRGrqJ3ZE9Q+DHysUYVJkhpjs8EfEWMy83ngk02qR5LUYFva47+T2lU5fxkRt2fmCU2oSZLUQFua449Oy3s3shBJUnNsKfizh2VJUova0lTPhIh4ldqe/x8Uy/Dewd0dG1qdJKnPbTb4M3NQIzqNiJ2Aa4CPUvskMSszFzWiL0nS+9V7Omdf+0fgPzJzRkQMpXbJZ0lSEzQ9+CPiA8BhwGkAmbme99/WUZLUQGXs8Y8D1gD/GhETgCXAOZn5eueNIqIdaAcYM2ZM04tsZXPmP1VKv+eW9fmR8sY86fm5pfQLMKm0ntXqenMHrr4yGJgI/HNmHkDtL4Ev7LpRZs7NzLbMbBs1alSza5SkAauM4O8AOjLzweLxbdT+I5AkNUHTgz8zfw38KiL2LZqOBJ5odh2SVFVlzcqeDdxYnNHzLPC5kuqQpMopJfgzcxnQVkbfklR1ZczxS5JKZPBLUsUY/JJUMQa/JFWMwS9JFWPwS1LFGPySVDEGvyRVjMEvSRVj8EtSxRj8klQxBr8kVYzBL0kVY/BLUsUY/JJUMQa/JFWMwS9JFVNa8EfEoIh4JCJ+VFYNklRFZe7xnwOsKLF/SaqkUoI/IkYDnwKuKaN/Saqysvb4rwS+BLzb0wYR0R4RiyNi8Zo1a5pWmCQNdE0P/oiYDrycmUs2t11mzs3MtsxsGzVqVJOqk6SBr4w9/kOBYyNiFXAzcEREfLeEOiSpkpoe/Jl5UWaOzsyxwMnAf2bmzGbXIUlV5Xn8klQxg8vsPDMXAgvLrEGSqsY9fkmqGINfkirG4JekijH4JaliDH5JqhiDX5IqxuCXpIop9Tx+DSyLnl1bWt+TmFta31KrcY9fkirG4JekijH4JaliDH5JqhiDX5IqxuCXpIox+CWpYgx+SaoYg1+SKqbpwR8Re0XEgoh4IiIej4hzml2DJFVZGZds2ACcn5lLI2IHYElEzM/MJ0qoRZIqp+l7/Jm5OjOXFsuvASuAPZtdhyRVValz/BExFjgAeLCbde0RsTgiFq9Zs6bptUnSQFVa8EfECOB24IuZ+WrX9Zk5NzPbMrNt1KhRzS9QkgaoUoI/IoZQC/0bM/OOMmqQpKoq46yeAL4NrMjMf2h2/5JUdWXs8R8K/BVwREQsK76OLqEOSaqkpp/OmZn3AdHsfiVJNf7lriRVjMEvSRVj8EtSxRj8klQxBr8kVYzBL0kVY/BLUsWUcVnmppoz/6nS+j538O0l9XxCSf1KA9OiZ9eW1vefTu3713SPX5IqxuCXpIox+CWpYgx+SaoYg1+SKsbgl6SKMfglqWIMfkmqGINfkiqmrJutT4uIlRHxi4i4sIwaJKmqyrjZ+iDgG8BfAOOBUyJifLPrkKSqKmOP/+PALzLz2cxcD9wMHFdCHZJUSZGZze0wYgYwLTNPLx7/FXBwZn6hy3btQHvxcF9g5TZ2PRL4zTa+RtkcQ//gGPqHgTAGaOw4/igzR3Vt7LdX58zMucDcvnq9iFicmW199XplcAz9g2PoHwbCGKCccZQx1fMCsFenx6OLNklSE5QR/A8DH46IcRExFDgZ+GEJdUhSJTV9qiczN0TEF4D/AwwCrs3Mx5vQdZ9NG5XIMfQPjqF/GAhjgBLG0fSDu5KkcvmXu5JUMQa/JFVMywZ/RFwbES9HxGPdrDs/IjIiRhaPIyL+qbhExM8jYmKnbU+NiKeLr1PLHkNEzI6IFyJiWfF1dKd1FxVjWBkRn+zUXtolMHp6HyLi7Ih4MiIej4jL+/MYiv67ey++3+l9WBURyzqt63fj6GEM+0fEA8UYFkfEx4v2VvqdmBARiyLi0YiYFxE7dlrXH9+HvSJiQUQ8Ufz8n1O07xIR84vv6/yI2Llob/57kZkt+QUcBkwEHuvSvhe1A8e/BEYWbUcDPwYCmAQ8WLTvAjxb/LtzsbxzmWMAZgMXdLPteGA5sD0wDniG2sHxQcXy3sDQYpvxJY9hKvATYPvi8W79eQyb+3nqtP7vgUv68zh6eC/uBv6i0+/Bwhb8nXgY+ESxPAv4n/38fdgDmFgs7wA8VdR6OXBh0X4h8L/Kei9ado8/M+8BXulm1RzgS0Dno9bHATdkzQPAThGxB/BJYH5mvpKZ/xeYD0xrcOmbbGYM3TkOuDkz38rM54BfULv8RamXwOhhDGcCX8vMt4ptXi7a++UYihp7fC8iIoDPADcVTf1yHD2MIYGNe8gfAF4sllvpd2If4J5ieT5wQrHcX9+H1Zm5tFh+DVgB7FnUcH2x2fXA8Z3G0dT3omWDvzsRcRzwQmYu77JqT+BXnR53FG09tZftC8VHvms3fhyktcawDzAlIh6MiJ9FxEFFeyuNobMpwEuZ+XTxuJXG8UXg6xHxK+AK4KKivZXG8DjvBfeJvPcHoP1+DBExFjgAeBDYPTNXF6t+DexeLDd9HAMm+CNiOPBl4JKya9lG/wx8CNgfWE1tiqHVDKb28XQS8LfALcVec6s6hff29lvNmcC5mbkXcC7w7ZLr2RqzgLMiYgm1qZP1JddTl4gYAdwOfDEzX+28LmtzOaWdSz9ggp9aWI4DlkfEKmqXglgaEX9Iz5eJ6HeXj8jMlzLzncx8F7ia2sdWaKExUNszuaP46PoQ8C61C1G10hgAiIjBwF8C3+/U3ErjOBW4o1i+lRb8ecrMJzPzzzPzQGr/AT9TrOq3Y4iIIdRC/8bM3Pj9f6mYwqH4d+MUaPPH0awDHo34AsbS88G4Vbx3cPdTvP/gyUP53sGT56gdONm5WN6lzDEAe3RaPpfaHCbAR3j/gaxnqR3EGlwsj+O9A1kfKXkMZwD/o1jeh9rH1ejPY+jp54nanOrPurT123F0816sAA4vlo8ElrTg78TGkwO2A24AZvXn96H4nt4AXNml/eu8/+Du5WW9F017Mxvwzb2J2lTI29T2MD/fZf0q3gv+oHbzl2eAR4G2TtvNonZQ6BfA58oeA/CdosafU7uGUef/CC4uxrCS4kyNov1oamcOPANc3A/GMBT4LvAYsBQ4oj+PYXM/T8B1wBndbN/vxtHDezEZWFKE34PAgS34O3FO8T19CvgaxRUH+vH7MJnaNM7PgWXF19HArsBPgaepnfW2S1nvhZdskKSKGUhz/JKkOhj8klQxBr8kVYzBL0kVY/BLUsUY/JJUMQa/JFXM/wfsO2eLwgvdJQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "composers_df.plot.hist(alpha = 0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here you see again the gain from using Pandas: without specifying anything, Pandas made a histogram of the two columns containing numbers, labelled the axis and even added a legend to the plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# DataFrame Operations\n",
    "\n",
    "One of the great advantages of using Pandas to handle tabular data is how simple it is to extract valuable information from them. Here we are going to see various types of operations that are available for this.\n",
    "\n",
    "## Matrix operations\n",
    "\n",
    "The strength of Numpy is its natural way of handling matrix operations, and Pandas reuses a lot of these features. For example one can use simple mathematical operations to opereate at the cell level:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:09:22.426144Z",
     "start_time": "2020-06-24T15:09:22.385407Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>composer</th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mahler</td>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Beethoven</td>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Puccini</td>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shostakovich</td>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       composer  birth  death              city\n",
       "0        Mahler   1860   1911           Kaliste\n",
       "1     Beethoven   1770   1827              Bonn\n",
       "2       Puccini   1858   1924           Lucques\n",
       "3  Shostakovich   1906   1975  Saint-Petersburg"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_excel('data/composers.xlsx')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:09:39.792603Z",
     "start_time": "2020-06-24T15:09:39.775993Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3720\n",
       "1    3540\n",
       "2    3716\n",
       "3    3812\n",
       "Name: birth, dtype: int64"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['birth'] * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:10:00.912276Z",
     "start_time": "2020-06-24T15:10:00.899426Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    7.528332\n",
       "1    7.478735\n",
       "2    7.527256\n",
       "3    7.552762\n",
       "Name: birth, dtype: float64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.log(df['birth'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can directly use an operation's output to create a new column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:19:14.832808Z",
     "start_time": "2020-06-24T15:19:14.807838Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>composer</th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mahler</td>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Beethoven</td>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Puccini</td>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shostakovich</td>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>69</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       composer  birth  death              city  age\n",
       "0        Mahler   1860   1911           Kaliste   51\n",
       "1     Beethoven   1770   1827              Bonn   57\n",
       "2       Puccini   1858   1924           Lucques   66\n",
       "3  Shostakovich   1906   1975  Saint-Petersburg   69"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['age'] = df['death'] - df['birth']\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here we applied functions only to series. Indeed, since our Dataframe contains e.g. strings, no operation can be done on it. If however we have a homogenous Dataframe, this is possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:10:52.552872Z",
     "start_time": "2020-06-24T15:10:52.534293Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3720</td>\n",
       "      <td>3822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3540</td>\n",
       "      <td>3654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3716</td>\n",
       "      <td>3848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3812</td>\n",
       "      <td>3950</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   birth  death\n",
       "0   3720   3822\n",
       "1   3540   3654\n",
       "2   3716   3848\n",
       "3   3812   3950"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['birth', 'death']] * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Column operations\n",
    "\n",
    "There are other types of functions whose purpose is to summarize the data. For example the mean or standard deviation. Pandas by default applies such functions column-wise and returns a series containing e.g. the mean of each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:11:31.318490Z",
     "start_time": "2020-06-24T15:11:31.299730Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "birth    1848.50\n",
       "death    1909.25\n",
       "age        60.75\n",
       "dtype: float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that columns for which a mean does not make sense, like the city are discarded. \n",
    "\n",
    "Sometimes one needs to apply to a column a very specific function that is not provided by default. In that case we can use one of the different `apply` methods of Pandas.\n",
    "\n",
    "The simplest case is to apply a function to a column, or Series of a DataFrame. Let's say for example that we want to define the the age >60 as 'old' and <60 as 'young'. We can define the following general function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:16:11.585908Z",
     "start_time": "2020-06-24T15:16:11.579767Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'young'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "define_age = lambda x: 'old' if x > 60 else 'young'\n",
    "define_age(33)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'young'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def define_age2(x):\n",
    "    return 'old' if x > 60 else 'young'\n",
    "define_age2(33)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can now apply this function on an entire Series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:19:19.576840Z",
     "start_time": "2020-06-24T15:19:19.558825Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    young\n",
       "1    young\n",
       "2      old\n",
       "3      old\n",
       "Name: age, dtype: object"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['age'].apply(define_age)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can also apply a function to an entire DataFrame. For example we can ask how many composers have birth and death dates within the XIXth century:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:21:10.779069Z",
     "start_time": "2020-06-24T15:21:10.759181Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "birth    2\n",
       "death    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nineteen_century_count = lambda x: np.sum( (x >= 1800) & (x < 1900) )\n",
    "df[['birth','death']].apply(nineteen_century_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Boolean Indexing\n",
    "\n",
    "Just like with Numpy, it is possible to subselect parts of a Dataframe using boolean indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:22:52.386996Z",
     "start_time": "2020-06-24T15:22:52.375984Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     True\n",
      "1    False\n",
      "2    False\n",
      "3     True\n",
      "Name: birth, dtype: bool\n"
     ]
    }
   ],
   "source": [
    "mask = df['birth'] > 1859\n",
    "print(mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Just like in Numpy we can use this logical Series as an index to select elements in the Dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-24T15:23:15.607297Z",
     "start_time": "2020-06-24T15:23:15.591321Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>composer</th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mahler</td>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shostakovich</td>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>69</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       composer  birth  death              city  age\n",
       "0        Mahler   1860   1911           Kaliste   51\n",
       "3  Shostakovich   1906   1975  Saint-Petersburg   69"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Merging\n",
    "\n",
    "Often information is comming from different sources and it is necessary to combine it into one object. We are going to see the different ways in which information contained within separate Dataframes can be combined in a meaningful way.\n",
    "\n",
    "## Concatenation\n",
    "\n",
    "The simplest way we can combine two Dataframes is simply to \"paste\" them together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:10:31.191085Z",
     "start_time": "2020-06-25T09:10:31.070211Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city\n",
       "composer                                    \n",
       "Mahler         1860   1911           Kaliste\n",
       "Beethoven      1770   1827              Bonn\n",
       "Puccini        1858   1924           Lucques\n",
       "Shostakovich   1906   1975  Saint-Petersburg"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1 = pd.read_excel('data/composers.xlsx', index_col='composer',sheet_name='Sheet1')\n",
    "composers1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:10:50.431994Z",
     "start_time": "2020-06-25T09:10:50.378518Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Verdi</th>\n",
       "      <td>1813</td>\n",
       "      <td>1901</td>\n",
       "      <td>Roncole</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dvorak</th>\n",
       "      <td>1841</td>\n",
       "      <td>1904</td>\n",
       "      <td>Nelahozeves</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Schumann</th>\n",
       "      <td>1810</td>\n",
       "      <td>1856</td>\n",
       "      <td>Zwickau</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stravinsky</th>\n",
       "      <td>1882</td>\n",
       "      <td>1971</td>\n",
       "      <td>Oranienbaum</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            birth  death         city\n",
       "composer                             \n",
       "Verdi        1813   1901      Roncole\n",
       "Dvorak       1841   1904  Nelahozeves\n",
       "Schumann     1810   1856      Zwickau\n",
       "Stravinsky   1882   1971  Oranienbaum\n",
       "Mahler       1860   1911      Kaliste"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers2 = pd.read_excel('data/composers.xlsx', index_col='composer',sheet_name='Sheet3')\n",
    "composers2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To be concatenated, Dataframes need to be provided as a list to the `pd.concat` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:11:09.559547Z",
     "start_time": "2020-06-25T09:11:09.536473Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Verdi</th>\n",
       "      <td>1813</td>\n",
       "      <td>1901</td>\n",
       "      <td>Roncole</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dvorak</th>\n",
       "      <td>1841</td>\n",
       "      <td>1904</td>\n",
       "      <td>Nelahozeves</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Schumann</th>\n",
       "      <td>1810</td>\n",
       "      <td>1856</td>\n",
       "      <td>Zwickau</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stravinsky</th>\n",
       "      <td>1882</td>\n",
       "      <td>1971</td>\n",
       "      <td>Oranienbaum</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city\n",
       "composer                                    \n",
       "Mahler         1860   1911           Kaliste\n",
       "Beethoven      1770   1827              Bonn\n",
       "Puccini        1858   1924           Lucques\n",
       "Shostakovich   1906   1975  Saint-Petersburg\n",
       "Verdi          1813   1901           Roncole\n",
       "Dvorak         1841   1904       Nelahozeves\n",
       "Schumann       1810   1856           Zwickau\n",
       "Stravinsky     1882   1971       Oranienbaum\n",
       "Mahler         1860   1911           Kaliste"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_composers = pd.concat([composers1,composers2])\n",
    "all_composers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One potential problem is that two tables contain duplicated information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:12:01.291916Z",
     "start_time": "2020-06-25T09:12:01.254007Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          birth  death     city\n",
       "composer                       \n",
       "Mahler     1860   1911  Kaliste\n",
       "Mahler     1860   1911  Kaliste"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_composers.loc['Mahler']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It is very easy to get rid of it using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:12:18.690762Z",
     "start_time": "2020-06-25T09:12:18.656734Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Verdi</th>\n",
       "      <td>1813</td>\n",
       "      <td>1901</td>\n",
       "      <td>Roncole</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dvorak</th>\n",
       "      <td>1841</td>\n",
       "      <td>1904</td>\n",
       "      <td>Nelahozeves</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Schumann</th>\n",
       "      <td>1810</td>\n",
       "      <td>1856</td>\n",
       "      <td>Zwickau</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stravinsky</th>\n",
       "      <td>1882</td>\n",
       "      <td>1971</td>\n",
       "      <td>Oranienbaum</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city\n",
       "composer                                    \n",
       "Mahler         1860   1911           Kaliste\n",
       "Beethoven      1770   1827              Bonn\n",
       "Puccini        1858   1924           Lucques\n",
       "Shostakovich   1906   1975  Saint-Petersburg\n",
       "Verdi          1813   1901           Roncole\n",
       "Dvorak         1841   1904       Nelahozeves\n",
       "Schumann       1810   1856           Zwickau\n",
       "Stravinsky     1882   1971       Oranienbaum"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_composers.drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Joining\n",
    "\n",
    "An other classical case is that of two list with similar index but containing different information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:42:42.326225Z",
     "start_time": "2020-06-25T09:42:42.235023Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city\n",
       "composer                                    \n",
       "Mahler         1860   1911           Kaliste\n",
       "Beethoven      1770   1827              Bonn\n",
       "Puccini        1858   1924           Lucques\n",
       "Shostakovich   1906   1975  Saint-Petersburg"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1 = pd.read_excel('data/composers.xlsx', index_col='composer',sheet_name='Sheet1')\n",
    "composers1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:42:56.133368Z",
     "start_time": "2020-06-25T09:42:56.067953Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brahms</th>\n",
       "      <td>Johannes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           first name\n",
       "composer             \n",
       "Mahler         Gustav\n",
       "Beethoven  Ludwig van\n",
       "Puccini       Giacomo\n",
       "Brahms       Johannes"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers2 = pd.read_excel('data/composers.xlsx', index_col='composer',sheet_name='Sheet4')\n",
    "composers2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we use simple concatenation, this doesn't help us much. We just end up with a large matrix with lots of `NaN`'s:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:44:18.040530Z",
     "start_time": "2020-06-25T09:44:17.960321Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860.0</td>\n",
       "      <td>1911.0</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770.0</td>\n",
       "      <td>1827.0</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858.0</td>\n",
       "      <td>1924.0</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906.0</td>\n",
       "      <td>1975.0</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brahms</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Johannes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               birth   death              city  first name\n",
       "composer                                                  \n",
       "Mahler        1860.0  1911.0           Kaliste         NaN\n",
       "Beethoven     1770.0  1827.0              Bonn         NaN\n",
       "Puccini       1858.0  1924.0           Lucques         NaN\n",
       "Shostakovich  1906.0  1975.0  Saint-Petersburg         NaN\n",
       "Mahler           NaN     NaN               NaN      Gustav\n",
       "Beethoven        NaN     NaN               NaN  Ludwig van\n",
       "Puccini          NaN     NaN               NaN     Giacomo\n",
       "Brahms           NaN     NaN               NaN    Johannes"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([composers1, composers2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The better way of doing this is to **join** the tables. This is a classical database concept avaialble in Pandas. `join()` operates on two tables: the first one is the \"left\" table which uses `join()` as a method. The other table is the \"right\" one.\n",
    "Let's try the default join settings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:45:06.835088Z",
     "start_time": "2020-06-25T09:45:06.757433Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city  first name\n",
       "composer                                                \n",
       "Mahler         1860   1911           Kaliste      Gustav\n",
       "Beethoven      1770   1827              Bonn  Ludwig van\n",
       "Puccini        1858   1924           Lucques     Giacomo\n",
       "Shostakovich   1906   1975  Saint-Petersburg         NaN"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1.join(composers2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We see that Pandas was smart enough to notice that the two tables had a index name and used it to combine the tables. We also see that one element from the second table (Brahms) is missing. The reason for this is the way indices not present in both tables are handled. There are four ways of doing this with two tables called here the \"left\" and \"right\" table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Join left\n",
    "\n",
    "The two Dataframes that should be merged have a common index, but not necessarily the same items. For example here Shostakovich is missing in the second table, while Brahms is missing in the first one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:51:15.654892Z",
     "start_time": "2020-06-25T09:51:15.578862Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When using the \"left\" join, we use the first Dataframe as basis and only use the indices that appear there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:48:10.588614Z",
     "start_time": "2020-06-25T09:48:10.541269Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth  death              city  first name\n",
       "composer                                                \n",
       "Mahler         1860   1911           Kaliste      Gustav\n",
       "Beethoven      1770   1827              Bonn  Ludwig van\n",
       "Puccini        1858   1924           Lucques     Giacomo\n",
       "Shostakovich   1906   1975  Saint-Petersburg         NaN"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1.join(composers2, how = 'left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Join right\n",
    "\n",
    "When using the \"right\" join, we use the second Dataframe as basis and only use the indices that appear there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:49:12.876448Z",
     "start_time": "2020-06-25T09:49:12.836603Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860.0</td>\n",
       "      <td>1911.0</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770.0</td>\n",
       "      <td>1827.0</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858.0</td>\n",
       "      <td>1924.0</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brahms</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Johannes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            birth   death     city  first name\n",
       "composer                                      \n",
       "Mahler     1860.0  1911.0  Kaliste      Gustav\n",
       "Beethoven  1770.0  1827.0     Bonn  Ludwig van\n",
       "Puccini    1858.0  1924.0  Lucques     Giacomo\n",
       "Brahms        NaN     NaN      NaN    Johannes"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1.join(composers2, how = 'right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Inner join\n",
    "\n",
    "When using the \"inner\" join, we return only the items that appear in both Dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:49:25.070369Z",
     "start_time": "2020-06-25T09:49:25.034314Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           birth  death     city  first name\n",
       "composer                                    \n",
       "Mahler      1860   1911  Kaliste      Gustav\n",
       "Beethoven   1770   1827     Bonn  Ludwig van\n",
       "Puccini     1858   1924  Lucques     Giacomo"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1.join(composers2, how = 'inner')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Outer join\n",
    "\n",
    "When using the \"inner\" join, we return all the items that appaer in both Dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:49:35.747756Z",
     "start_time": "2020-06-25T09:49:35.701517Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770.0</td>\n",
       "      <td>1827.0</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brahms</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Johannes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860.0</td>\n",
       "      <td>1911.0</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858.0</td>\n",
       "      <td>1924.0</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906.0</td>\n",
       "      <td>1975.0</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               birth   death              city  first name\n",
       "composer                                                  \n",
       "Beethoven     1770.0  1827.0              Bonn  Ludwig van\n",
       "Brahms           NaN     NaN               NaN    Johannes\n",
       "Mahler        1860.0  1911.0           Kaliste      Gustav\n",
       "Puccini       1858.0  1924.0           Lucques     Giacomo\n",
       "Shostakovich  1906.0  1975.0  Saint-Petersburg         NaN"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1.join(composers2, how = 'outer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Merging\n",
    "\n",
    "Sometimes tables don't have the same indices but similar contents that we want to merge. For example let's imagine whe have the two Dataframes below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T10:02:35.530321Z",
     "start_time": "2020-06-25T10:02:35.378339Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>composer</th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mahler</td>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Beethoven</td>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Puccini</td>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shostakovich</td>\n",
       "      <td>1906</td>\n",
       "      <td>1975</td>\n",
       "      <td>Saint-Petersburg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       composer  birth  death              city\n",
       "0        Mahler   1860   1911           Kaliste\n",
       "1     Beethoven   1770   1827              Bonn\n",
       "2       Puccini   1858   1924           Lucques\n",
       "3  Shostakovich   1906   1975  Saint-Petersburg"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers1 = pd.read_excel('data/composers.xlsx', sheet_name='Sheet1')\n",
    "composers1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T10:02:47.472207Z",
     "start_time": "2020-06-25T10:02:47.394103Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>last name</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Puccini</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Beethoven</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brahms</td>\n",
       "      <td>Johannes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mahler</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   last name  first name\n",
       "0    Puccini     Giacomo\n",
       "1  Beethoven  Ludwig van\n",
       "2     Brahms    Johannes\n",
       "3     Mahler      Gustav"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers2 = pd.read_excel('data/composers.xlsx', sheet_name='Sheet6')\n",
    "composers2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The indices don't match and are not the composer name. In addition the columns containing the composer names have different labels. Here we can use `merge()` and specify which columns we want to use for merging, and what type of merging we need (inner, left etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T10:03:56.561292Z",
     "start_time": "2020-06-25T10:03:56.508980Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>composer</th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>city</th>\n",
       "      <th>last name</th>\n",
       "      <th>first name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mahler</td>\n",
       "      <td>1860</td>\n",
       "      <td>1911</td>\n",
       "      <td>Kaliste</td>\n",
       "      <td>Mahler</td>\n",
       "      <td>Gustav</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Beethoven</td>\n",
       "      <td>1770</td>\n",
       "      <td>1827</td>\n",
       "      <td>Bonn</td>\n",
       "      <td>Beethoven</td>\n",
       "      <td>Ludwig van</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Puccini</td>\n",
       "      <td>1858</td>\n",
       "      <td>1924</td>\n",
       "      <td>Lucques</td>\n",
       "      <td>Puccini</td>\n",
       "      <td>Giacomo</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    composer  birth  death     city  last name  first name\n",
       "0     Mahler   1860   1911  Kaliste     Mahler      Gustav\n",
       "1  Beethoven   1770   1827     Bonn  Beethoven  Ludwig van\n",
       "2    Puccini   1858   1924  Lucques    Puccini     Giacomo"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(composers1, composers2, left_on='composer', right_on='last name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Again we can use another variety of join than the default inner join."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Splitting\n",
    "\n",
    "Often Pandas tables mix regular variables (e.g. the size of cells in microscopy images) with categorical variables (e.g. the type of cell to which they belong). In that case, it is quite usual to split the data using the category to do computations. Pandas allows to do this very easily."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Grouping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:56:13.601372Z",
     "start_time": "2020-06-25T08:56:13.518195Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>period</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mahler</th>\n",
       "      <td>1860</td>\n",
       "      <td>1911.0</td>\n",
       "      <td>post-romantic</td>\n",
       "      <td>Austria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beethoven</th>\n",
       "      <td>1770</td>\n",
       "      <td>1827.0</td>\n",
       "      <td>romantic</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Puccini</th>\n",
       "      <td>1858</td>\n",
       "      <td>1924.0</td>\n",
       "      <td>post-romantic</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shostakovich</th>\n",
       "      <td>1906</td>\n",
       "      <td>1975.0</td>\n",
       "      <td>modern</td>\n",
       "      <td>Russia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Verdi</th>\n",
       "      <td>1813</td>\n",
       "      <td>1901.0</td>\n",
       "      <td>romantic</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth   death         period  country\n",
       "composer                                           \n",
       "Mahler         1860  1911.0  post-romantic  Austria\n",
       "Beethoven      1770  1827.0       romantic  Germany\n",
       "Puccini        1858  1924.0  post-romantic    Italy\n",
       "Shostakovich   1906  1975.0         modern   Russia\n",
       "Verdi          1813  1901.0       romantic    Italy"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What if we want now to count how many composers we have in each category? In classical computing we would maybe do a for loop to count occurrences. Pandas simplifies this with the `groupby()` function, which actually groups elements by a certain criteria, e.g. a categorical variable like the period:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:57:05.723610Z",
     "start_time": "2020-06-25T08:57:05.693209Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.generic.DataFrameGroupBy object at 0x117068390>"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composer_grouped = composers_df.groupby('period')\n",
    "composer_grouped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The output is a bit cryptic. What we actually have is a new object called *group* which has a lot of handy properties. First let's see what the groups actually are. As for the Dataframe, let's look at a summary of the object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:57:44.681494Z",
     "start_time": "2020-06-25T08:57:44.486278Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">birth</th>\n",
       "      <th colspan=\"8\" halign=\"left\">death</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>period</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>baroque</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1663.300000</td>\n",
       "      <td>36.009412</td>\n",
       "      <td>1587.0</td>\n",
       "      <td>1647.0</td>\n",
       "      <td>1676.5</td>\n",
       "      <td>1685.0</td>\n",
       "      <td>1710.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1720.200000</td>\n",
       "      <td>43.460838</td>\n",
       "      <td>1640.0</td>\n",
       "      <td>1697.25</td>\n",
       "      <td>1736.0</td>\n",
       "      <td>1755.25</td>\n",
       "      <td>1764.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>classic</th>\n",
       "      <td>5.0</td>\n",
       "      <td>1744.400000</td>\n",
       "      <td>12.054045</td>\n",
       "      <td>1731.0</td>\n",
       "      <td>1732.0</td>\n",
       "      <td>1749.0</td>\n",
       "      <td>1754.0</td>\n",
       "      <td>1756.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1801.200000</td>\n",
       "      <td>6.942622</td>\n",
       "      <td>1791.0</td>\n",
       "      <td>1799.00</td>\n",
       "      <td>1801.0</td>\n",
       "      <td>1806.00</td>\n",
       "      <td>1809.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>modern</th>\n",
       "      <td>13.0</td>\n",
       "      <td>1905.692308</td>\n",
       "      <td>28.595992</td>\n",
       "      <td>1854.0</td>\n",
       "      <td>1891.0</td>\n",
       "      <td>1902.0</td>\n",
       "      <td>1918.0</td>\n",
       "      <td>1971.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1974.090909</td>\n",
       "      <td>26.139834</td>\n",
       "      <td>1928.0</td>\n",
       "      <td>1962.00</td>\n",
       "      <td>1982.0</td>\n",
       "      <td>1990.00</td>\n",
       "      <td>2016.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>post-romantic</th>\n",
       "      <td>5.0</td>\n",
       "      <td>1854.200000</td>\n",
       "      <td>17.123084</td>\n",
       "      <td>1824.0</td>\n",
       "      <td>1858.0</td>\n",
       "      <td>1860.0</td>\n",
       "      <td>1864.0</td>\n",
       "      <td>1865.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1927.400000</td>\n",
       "      <td>25.540164</td>\n",
       "      <td>1896.0</td>\n",
       "      <td>1911.00</td>\n",
       "      <td>1924.0</td>\n",
       "      <td>1949.00</td>\n",
       "      <td>1957.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>renaissance</th>\n",
       "      <td>7.0</td>\n",
       "      <td>1527.142857</td>\n",
       "      <td>59.881629</td>\n",
       "      <td>1397.0</td>\n",
       "      <td>1528.5</td>\n",
       "      <td>1540.0</td>\n",
       "      <td>1564.5</td>\n",
       "      <td>1567.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1595.285714</td>\n",
       "      <td>56.295986</td>\n",
       "      <td>1474.0</td>\n",
       "      <td>1594.00</td>\n",
       "      <td>1613.0</td>\n",
       "      <td>1624.50</td>\n",
       "      <td>1643.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>romantic</th>\n",
       "      <td>17.0</td>\n",
       "      <td>1824.823529</td>\n",
       "      <td>25.468695</td>\n",
       "      <td>1770.0</td>\n",
       "      <td>1810.0</td>\n",
       "      <td>1824.0</td>\n",
       "      <td>1841.0</td>\n",
       "      <td>1867.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1883.588235</td>\n",
       "      <td>28.026904</td>\n",
       "      <td>1827.0</td>\n",
       "      <td>1869.00</td>\n",
       "      <td>1887.0</td>\n",
       "      <td>1904.00</td>\n",
       "      <td>1919.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              birth                                                          \\\n",
       "              count         mean        std     min     25%     50%     75%   \n",
       "period                                                                        \n",
       "baroque        10.0  1663.300000  36.009412  1587.0  1647.0  1676.5  1685.0   \n",
       "classic         5.0  1744.400000  12.054045  1731.0  1732.0  1749.0  1754.0   \n",
       "modern         13.0  1905.692308  28.595992  1854.0  1891.0  1902.0  1918.0   \n",
       "post-romantic   5.0  1854.200000  17.123084  1824.0  1858.0  1860.0  1864.0   \n",
       "renaissance     7.0  1527.142857  59.881629  1397.0  1528.5  1540.0  1564.5   \n",
       "romantic       17.0  1824.823529  25.468695  1770.0  1810.0  1824.0  1841.0   \n",
       "\n",
       "                      death                                                   \\\n",
       "                  max count         mean        std     min      25%     50%   \n",
       "period                                                                         \n",
       "baroque        1710.0  10.0  1720.200000  43.460838  1640.0  1697.25  1736.0   \n",
       "classic        1756.0   5.0  1801.200000   6.942622  1791.0  1799.00  1801.0   \n",
       "modern         1971.0  11.0  1974.090909  26.139834  1928.0  1962.00  1982.0   \n",
       "post-romantic  1865.0   5.0  1927.400000  25.540164  1896.0  1911.00  1924.0   \n",
       "renaissance    1567.0   7.0  1595.285714  56.295986  1474.0  1594.00  1613.0   \n",
       "romantic       1867.0  17.0  1883.588235  28.026904  1827.0  1869.00  1887.0   \n",
       "\n",
       "                                \n",
       "                   75%     max  \n",
       "period                          \n",
       "baroque        1755.25  1764.0  \n",
       "classic        1806.00  1809.0  \n",
       "modern         1990.00  2016.0  \n",
       "post-romantic  1949.00  1957.0  \n",
       "renaissance    1624.50  1643.0  \n",
       "romantic       1904.00  1919.0  "
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composer_grouped.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have a dataframe with a statistical summary of the the contents. The \"names\" of the groups are here the indices of the Dataframe. These names are simply all the different categories that were present in the column we used for grouping. Now we can recover a single group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:58:36.933456Z",
     "start_time": "2020-06-25T08:58:36.907796Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>period</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Haendel</th>\n",
       "      <td>1685</td>\n",
       "      <td>1759.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Purcell</th>\n",
       "      <td>1659</td>\n",
       "      <td>1695.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Charpentier</th>\n",
       "      <td>1643</td>\n",
       "      <td>1704.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>France</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Couperin</th>\n",
       "      <td>1626</td>\n",
       "      <td>1661.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>France</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rameau</th>\n",
       "      <td>1683</td>\n",
       "      <td>1764.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>France</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Caldara</th>\n",
       "      <td>1670</td>\n",
       "      <td>1736.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pergolesi</th>\n",
       "      <td>1710</td>\n",
       "      <td>1736.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Scarlatti</th>\n",
       "      <td>1685</td>\n",
       "      <td>1757.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Caccini</th>\n",
       "      <td>1587</td>\n",
       "      <td>1640.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Italy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bach</th>\n",
       "      <td>1685</td>\n",
       "      <td>1750.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             birth   death   period  country\n",
       "composer                                    \n",
       "Haendel       1685  1759.0  baroque  Germany\n",
       "Purcell       1659  1695.0  baroque  England\n",
       "Charpentier   1643  1704.0  baroque   France\n",
       "Couperin      1626  1661.0  baroque   France\n",
       "Rameau        1683  1764.0  baroque   France\n",
       "Caldara       1670  1736.0  baroque    Italy\n",
       "Pergolesi     1710  1736.0  baroque    Italy\n",
       "Scarlatti     1685  1757.0  baroque    Italy\n",
       "Caccini       1587  1640.0  baroque    Italy\n",
       "Bach          1685  1750.0  baroque  Germany"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composer_grouped.get_group('baroque')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If one has multiple categorical variables, one can also do a grouping on several levels. For example here we want to classify composers both by period and country. For this we just give two column names to the `groupby()` function:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T08:59:41.133661Z",
     "start_time": "2020-06-25T08:59:41.051149Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth</th>\n",
       "      <th>death</th>\n",
       "      <th>period</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>composer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Haendel</th>\n",
       "      <td>1685</td>\n",
       "      <td>1759.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bach</th>\n",
       "      <td>1685</td>\n",
       "      <td>1750.0</td>\n",
       "      <td>baroque</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          birth   death   period  country\n",
       "composer                                 \n",
       "Haendel    1685  1759.0  baroque  Germany\n",
       "Bach       1685  1750.0  baroque  Germany"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composer_grouped = composers_df.groupby(['period','country'])\n",
    "composer_grouped.get_group(('baroque','Germany'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The main advantage of this Group object is that it allows us to do very quickly both computations and plotting without having to loop through different categories. Indeed Pandas makes all the work for us: it applies functions on each group and then reassembles the results into a Dataframe (or Series depending on output).\n",
    "For example we can apply most functions we used for Dataframes (mean, sum etc.) on groups as well and Pandas seamlessly does the work for us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Unstacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's have a look again at one of our grouped Dataframe on which we applied some summary function like a mean on the age column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:08:04.942179Z",
     "start_time": "2020-06-25T09:08:04.852600Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "country  period       \n",
       "Austria  classic          56.000000\n",
       "         modern           50.000000\n",
       "         post-romantic    61.500000\n",
       "Belgium  renaissance      69.500000\n",
       "Czechia  classic          68.000000\n",
       "         modern           74.000000\n",
       "         romantic         61.500000\n",
       "England  baroque          36.000000\n",
       "         modern           81.000000\n",
       "         renaissance      73.000000\n",
       "Finland  post-romantic    92.000000\n",
       "France   baroque          59.000000\n",
       "         modern           87.500000\n",
       "         romantic         70.333333\n",
       "Germany  baroque          69.500000\n",
       "         modern           87.000000\n",
       "         post-romantic    85.000000\n",
       "         romantic         59.250000\n",
       "Italy    baroque          54.250000\n",
       "         classic          52.000000\n",
       "         post-romantic    66.000000\n",
       "         renaissance      64.000000\n",
       "         romantic         58.500000\n",
       "RUssia   modern           62.000000\n",
       "Russia   modern           79.000000\n",
       "         romantic         48.000000\n",
       "Spain    classic          52.000000\n",
       "         romantic         49.000000\n",
       "USA      modern           81.000000\n",
       "Name: age, dtype: float64"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composers_df['age'] = composers_df['death'] - composers_df['birth']\n",
    "composers_df.groupby(['country','period']).age.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we have two level of indices, with the main one being the country which contains all periods. Often for plotting we however need to have the information in another format. In particular we would like each of these values to be one observation in a regular table. For example we could have a country vs period table where all elements are the mean age. To do that we need to **unstack** our multi-level Dataframe:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-25T09:08:39.090317Z",
     "start_time": "2020-06-25T09:08:38.994496Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>period</th>\n",
       "      <th>baroque</th>\n",
       "      <th>classic</th>\n",
       "      <th>modern</th>\n",
       "      <th>post-romantic</th>\n",
       "      <th>renaissance</th>\n",
       "      <th>romantic</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Austria</th>\n",
       "      <td>NaN</td>\n",
       "      <td>56.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>61.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Belgium</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>69.5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Czechia</th>\n",
       "      <td>NaN</td>\n",
       "      <td>68.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>61.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>England</th>\n",
       "      <td>36.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>81.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>73.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Finland</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>France</th>\n",
       "      <td>59.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>87.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>70.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Germany</th>\n",
       "      <td>69.50</td>\n",
       "      <td>NaN</td>\n",
       "      <td>87.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>59.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Italy</th>\n",
       "      <td>54.25</td>\n",
       "      <td>52.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>58.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RUssia</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>62.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Russia</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>79.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spain</th>\n",
       "      <td>NaN</td>\n",
       "      <td>52.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>USA</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>81.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "period   baroque  classic  modern  post-romantic  renaissance   romantic\n",
       "country                                                                 \n",
       "Austria      NaN     56.0    50.0           61.5          NaN        NaN\n",
       "Belgium      NaN      NaN     NaN            NaN         69.5        NaN\n",
       "Czechia      NaN     68.0    74.0            NaN          NaN  61.500000\n",
       "England    36.00      NaN    81.0            NaN         73.0        NaN\n",
       "Finland      NaN      NaN     NaN           92.0          NaN        NaN\n",
       "France     59.00      NaN    87.5            NaN          NaN  70.333333\n",
       "Germany    69.50      NaN    87.0           85.0          NaN  59.250000\n",
       "Italy      54.25     52.0     NaN           66.0         64.0  58.500000\n",
       "RUssia       NaN      NaN    62.0            NaN          NaN        NaN\n",
       "Russia       NaN      NaN    79.0            NaN          NaN  48.000000\n",
       "Spain        NaN     52.0     NaN            NaN          NaN  49.000000\n",
       "USA          NaN      NaN    81.0            NaN          NaN        NaN"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "composer_unstacked = composers_df.groupby(['country','period']).age.mean().unstack()\n",
    "composer_unstacked"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Pivot Tables in Pandas\n",
    "\n",
    "A pivot table is an operation similar to `groupby` commonly seen in spreadsheets and other programs that operate on tabular data.\n",
    "\n",
    "The *pivot table* takes simple column-wise data as input, and groups the entries into a two-dimensional table that provides a multidimensional summarization of the data. \n",
    "\n",
    "The difference between pivot tables and `groupby` can sometimes cause confusion; it helps me to think of pivot tables as essentially a multidimensional version of `groupby` aggregation. \n",
    "\n",
    "That is, you split-apply-combine, but both the split and the combine happen across not a one-dimensional index, but across a two-dimensional grid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We will use the database of passengers on the Titanic, available through the Seaborn library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived  pclass     sex   age  sibsp  parch     fare embarked  class  \\\n",
       "0         0       3    male  22.0      1      0   7.2500        S  Third   \n",
       "1         1       1  female  38.0      1      0  71.2833        C  First   \n",
       "2         1       3  female  26.0      0      0   7.9250        S  Third   \n",
       "3         1       1  female  35.0      1      0  53.1000        S  First   \n",
       "4         0       3    male  35.0      0      0   8.0500        S  Third   \n",
       "\n",
       "     who  adult_male deck  embark_town alive  alone  \n",
       "0    man        True  NaN  Southampton    no  False  \n",
       "1  woman       False    C    Cherbourg   yes  False  \n",
       "2  woman       False  NaN  Southampton   yes   True  \n",
       "3  woman       False    C  Southampton   yes  False  \n",
       "4    man        True  NaN  Southampton    no   True  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "titanic = sns.load_dataset('titanic')\n",
    "\n",
    "titanic.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pivot tables by hand\n",
    "\n",
    "Let's look at survival rate by gender:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.742038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.188908</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        survived\n",
       "sex             \n",
       "female  0.742038\n",
       "male    0.188908"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.groupby('sex')[['survived']].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We might look at survival by both sex and, say, class. Using the vocabulary of `groupby`, we might proceed using something like this: \n",
    "1. we group by class and gender,\n",
    "2. select survival, \n",
    "3. apply a mean aggregate, \n",
    "4. combine the resulting groups, \n",
    "5. and then unstack the hierarchical index \n",
    "\n",
    "to reveal the hidden multidimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>class</th>\n",
       "      <th>First</th>\n",
       "      <th>Second</th>\n",
       "      <th>Third</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.968085</td>\n",
       "      <td>0.921053</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.368852</td>\n",
       "      <td>0.157407</td>\n",
       "      <td>0.135447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "class      First    Second     Third\n",
       "sex                                 \n",
       "female  0.968085  0.921053  0.500000\n",
       "male    0.368852  0.157407  0.135447"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.groupby(['sex', 'class'])['survived'].aggregate('mean').unstack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This two-dimensional `groupby` is common enough that Pandas includes a convenience routine, `pivot_table`, which succinctly handles this type of multi-dimensional aggregation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>class</th>\n",
       "      <th>First</th>\n",
       "      <th>Second</th>\n",
       "      <th>Third</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.968085</td>\n",
       "      <td>0.921053</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.368852</td>\n",
       "      <td>0.157407</td>\n",
       "      <td>0.135447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "class      First    Second     Third\n",
       "sex                                 \n",
       "female  0.968085  0.921053  0.500000\n",
       "male    0.368852  0.157407  0.135447"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.pivot_table('survived', index='sex', columns='class')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}