strata_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyData @ Strata\n",
    "## T.J. Alumbaugh\n",
    "### tj.alumbaugh@continuum.io\n",
    "### 2016-03-29"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>\n",
    "This tutorial, and Python in general, run more smoothly under Python 3.x.\n",
    "\n",
    "Whether you're running on Python 2 or Python 3, please install [Python-Future](http://python-future.org/futurize.html):\n",
    "```bash\n",
    "conda install future\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import (absolute_import, division,\n",
    "                        print_function, unicode_literals)\n",
    "from future import standard_library\n",
    "standard_library.install_aliases()\n",
    "from future.builtins import (\n",
    "         bytes, dict, int, list, object, range, str,\n",
    "         ascii, chr, hex, input, next, oct, open,\n",
    "         pow, round, super, filter, map, zip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# If your world of data is like this:\n",
    "![messy](img/messy_garage_3.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas is like this:\n",
    "![knife](img/knife.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is Pandas?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "* The python data analysis library\n",
    "  > *\"Python has long been great for data munging and preparation, but less so for data analysis and modeling. pandas helps fill this gap, enabling you to carry out your entire data analysis workflow in Python\" -- pandas.pydata.org*\n",
    "* Wes McKinney, original author:\n",
    "    * started work on Pandas in 2008 while at AQR Capital Management\n",
    "    * conveinced AQR to open-source pandas.\n",
    "    * provides a 10 minute tour of Pandas: [video](https://vimeo.com/59324550) and [notebook](http://nbviewer.ipython.org/urls/gist.github.com/wesm/4757075/raw/a72d3450ad4924d0e74fb57c9f62d1d895ea4574/PandasTour.ipynb)\n",
    "* More details from PyData.org:\n",
    "    * *\"high-level building block for doing practical, real world data analysis in Python.\"*\n",
    "    * *\"pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\"*\n",
    "    * *\"provides fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive.\"*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ``pandas`` is widely used for doing practical data analysis\n",
    "* ``pandas`` fills a gap of convenient tools for data analysis in the python ecosystem\n",
    "* ``pandas`` is consistent, built around its own data structures\n",
    "* ``pandas`` is fast, built on numpy, but goes beyond.\n",
    "* ``pandas`` is tightly integrated with the rest of the scientific Python ecosystem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical Use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``pandas`` is built for doing practical data analysis using Python\n",
    "* Used extensively in production in financial applications.\n",
    "* Well suited for tabular data (e.g. column data, spread-sheets, databases)\n",
    "* Can replace or integrate with Excel and SQL queries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fills the Data Analysis Gap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``pandas`` fills a gap in the Python scientific eco-system:\n",
    "* Convenient methods for **reading/writing** a large variety of file formats, e.g. Excel, HDF, SQL, CSV, etc.\n",
    "* Meaningful **labels** for quantitative and categorical data\n",
    "* Easy handling **missing data** and **NaN** (much harder to do with ``numpy`` arrays alone).\n",
    "* SQL-like methods for manipulating tabular data, such as pivot, groupby, merge/join.\n",
    "* Convenient methods for routine mathematical and **summary statistics**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas keeps you from doing this:\n",
    "![homemadeknife](img/homemadeknife.JPG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Overview - What Are We Going To Do Today?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### - Pandas Data Structures: How We Work With Data\n",
    "\n",
    "### - Pandas I/O: How We Get Our Data into Pandas\n",
    "\n",
    "### - Slicing/Dicing/Mixing/Matching Til Your Fingers Bleed!\n",
    "\n",
    "### -  Example: The Turning Tuesday Hypothesis: Beat the Stock Market With This One Simple Trick...\n",
    "\n",
    "### - Where To Go From Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Download Anaconda\n",
    "\n",
    "Go to [www.continuum.io](www.continuum.io) to get Anaconda!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Get Some Data\n",
    "\n",
    "```\n",
    "git clone https://www.github.com/talumbau/strata-data\n",
    "```\n",
    "\n",
    "Or go to [https://www.github.com/talumbau/strata_data](https://www.github.com/talumbau/strata_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Navigate to where you cloned the repo/downloaded the data:\n",
    "```\n",
    "   cd ~/dev/strata_data/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 4: Start the jupyter notebook:\n",
    "```\n",
    "    jupyter notebook\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 5: Start the strata_data notebook:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 6: wait a while...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cheatsheet: view this notebook online and follow the links!\n",
    "### [https://anaconda.org/talumbau/strata_pandas/notebook](https://anaconda.org/talumbau/strata_pandas/notebook) to view and download this notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas provides two primary data structures:\n",
    "* `Series` is built for 1-dimensional series data\n",
    "* `DataFrame` is built for 2-dimensional collections of tabular data\n",
    "\n",
    "These data structures differ and are related as follows:\n",
    "* **Series**: 1-dimensional array of **homogeneous** data\n",
    "* **DataFrame**: 2-dimensional table of **heterogeneous** data, composed of multiple **Series**\n",
    "\n",
    "Each of these also contains an integrated **`index`**:\n",
    "* **Series** objects include a second array called the **index** that can contain homogeneous values of any type like integers, strings or datetime objects. \n",
    "* **DataFrame** objects include a column **index**. The **index** objects allow for very sophisticated selection operations using square brackets (`[]`) along with some specialized selection functions.\n",
    "\n",
    "*side note: Pandas has another data structure called Panels which is not commonly used*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fast Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``pandas`` is fast, built on numpy, but goes beyond:\n",
    "* `Series` and `DataFrames` are built upon `numpy.ndarray`\n",
    "* ``pandas`` supports the same `ufunc` operations as in ``numpy``, and the same fast vectorized computations.\n",
    "* goes beyond ``numpy`` by providing elementwise string and datetime operations on indexing.\n",
    "* ``pandas`` uses and supports additional C extensions written in Cython."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integration with Ecosystem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``pandas`` is tightly integrated with the rest of the scientific Python ecosystem\n",
    "* built on Numpy arrays and ufuncs\n",
    "* pandas data structures can be passed into numpy, matplotlib, and bokeh methods\n",
    "* has built-in visualization using Matplotlib\n",
    "* a dependency for Python statistics library statsmodel\n",
    "* a [NumFOCUS](http://www.numfocus.org) suported project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``pandas`` is an enormous library. We will concentrate on only a few elementary data analysis tasks using ``pandas``, but more extensive descriptions of the motivations and functionality of ``pandas`` can be found in the resources below: \n",
    "\n",
    "* [pandas.pydata.org](http://pandas.pydata.org/)\n",
    "  * [pandas documentation](http://pandas.pydata.org/pandas-docs/stable/)\n",
    "  * [10 minutes to Pandas](http://pandas.pydata.org/pandas-docs/stable/10min.html)\n",
    "  * [Cookbook at PyData](http://pandas.pydata.org/pandas-docs/stable/cookbook.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to install pandas and its depedencies is by installing the Anaconda distribution.\n",
    "* http://pandas.pydata.org/pandas-docs/stable/install.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have not already installed Anaconda, do so, and then use the following conda command to install pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "Fetching package metadata: ........\n",
      "Solving package specifications: ..................\n",
      "Package plan for installation in environment /Users/talumbau/miniconda3/envs/coursebuild:\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    pytz-2016.2                |           py34_0         175 KB\n",
      "    pip-8.1.1                  |           py34_1         1.6 MB\n",
      "    python-dateutil-2.5.1      |           py34_0         237 KB\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:         2.0 MB\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    pip:             8.1.1-py34_0  --> 8.1.1-py34_1 \n",
      "    python-dateutil: 2.5.0-py34_0  --> 2.5.1-py34_0 \n",
      "    pytz:            2016.1-py34_0 --> 2016.2-py34_0\n",
      "\n",
      "Fetching packages ...\n",
      "pytz-2016.2-py 100% |################################| Time: 0:00:00 439.89 kB/s\n",
      "pip-8.1.1-py34 100% |################################| Time: 0:00:01   1.03 MB/s\n",
      "python-dateuti 100% |################################| Time: 0:00:00 316.28 kB/s\n",
      "Extracting packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Unlinking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Linking packages ...\n",
      "[      COMPLETE      ]|###################################################| 100%\n",
      "Using Anaconda Cloud api site https://api.anaconda.org\n",
      "Fetching package metadata: ........\n",
      "Solving package specifications: .....................\n",
      "# All requested packages already installed.\n",
      "# packages in environment at /Users/talumbau/miniconda3/envs/coursebuild:\n",
      "#\n",
      "pandas-datareader         0.2.0                    py34_0  \n"
     ]
    }
   ],
   "source": [
    "## Uncomment the following line to use conda to install pandas\n",
    "\n",
    "!conda install -y pandas\n",
    "!conda install -y pandas-datareader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your ``pandas`` install by trying to import it: \n",
    "* the `import` should not generate an error\n",
    "* the version should be > 0.15.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "format": "row"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.18.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas\n",
    "pandas.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Should you have an older version, use the following conda to upgrade pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## To upgrade pandas, uncomment and run the following:\n",
    "\n",
    "# !conda install pandas=0.16.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a moment to read the docstring or use tab-complete to see what's in the library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-7-5341b825b163>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-7-5341b825b163>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    pandas.   ## press <tab>\u001b[0m\n\u001b[0m                            ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "## To explore pandas contents, uncomment the following:\n",
    "\n",
    "pandas.  ## press <tab>\n",
    "# pandas?   ## press <enter>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Demonstration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# community convention, import as \"pd\"\n",
    "import pandas as pd\n",
    "\n",
    "# to imbed plots, prevents pop-up windows for plots\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Pandas makes it easy to read data from files and even from web sites.\n",
    "* Here we use the Pandas Datareader module to read Google stock data from 2010 to the end of 2014."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-01-04</th>\n",
       "      <td>313.16</td>\n",
       "      <td>314.44</td>\n",
       "      <td>311.81</td>\n",
       "      <td>313.06</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>313.28</td>\n",
       "      <td>313.61</td>\n",
       "      <td>310.46</td>\n",
       "      <td>311.68</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-06</th>\n",
       "      <td>312.62</td>\n",
       "      <td>312.62</td>\n",
       "      <td>302.88</td>\n",
       "      <td>303.83</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-07</th>\n",
       "      <td>304.40</td>\n",
       "      <td>304.70</td>\n",
       "      <td>296.03</td>\n",
       "      <td>296.75</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-08</th>\n",
       "      <td>295.70</td>\n",
       "      <td>301.32</td>\n",
       "      <td>294.26</td>\n",
       "      <td>300.71</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-11</th>\n",
       "      <td>301.93</td>\n",
       "      <td>301.93</td>\n",
       "      <td>296.72</td>\n",
       "      <td>300.25</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-12</th>\n",
       "      <td>298.74</td>\n",
       "      <td>298.78</td>\n",
       "      <td>293.71</td>\n",
       "      <td>294.94</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-13</th>\n",
       "      <td>287.96</td>\n",
       "      <td>293.90</td>\n",
       "      <td>286.66</td>\n",
       "      <td>293.25</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-14</th>\n",
       "      <td>291.66</td>\n",
       "      <td>296.80</td>\n",
       "      <td>291.11</td>\n",
       "      <td>294.63</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-15</th>\n",
       "      <td>295.50</td>\n",
       "      <td>296.48</td>\n",
       "      <td>288.73</td>\n",
       "      <td>289.71</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-19</th>\n",
       "      <td>290.45</td>\n",
       "      <td>294.91</td>\n",
       "      <td>287.86</td>\n",
       "      <td>293.52</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-20</th>\n",
       "      <td>292.70</td>\n",
       "      <td>292.70</td>\n",
       "      <td>287.36</td>\n",
       "      <td>289.91</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-21</th>\n",
       "      <td>291.43</td>\n",
       "      <td>293.12</td>\n",
       "      <td>285.84</td>\n",
       "      <td>291.20</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-22</th>\n",
       "      <td>281.97</td>\n",
       "      <td>285.01</td>\n",
       "      <td>267.16</td>\n",
       "      <td>274.73</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-25</th>\n",
       "      <td>273.02</td>\n",
       "      <td>274.66</td>\n",
       "      <td>267.49</td>\n",
       "      <td>269.73</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-26</th>\n",
       "      <td>268.72</td>\n",
       "      <td>274.53</td>\n",
       "      <td>267.88</td>\n",
       "      <td>270.94</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-27</th>\n",
       "      <td>270.36</td>\n",
       "      <td>273.55</td>\n",
       "      <td>267.39</td>\n",
       "      <td>270.78</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-28</th>\n",
       "      <td>271.97</td>\n",
       "      <td>273.23</td>\n",
       "      <td>265.03</td>\n",
       "      <td>266.88</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-29</th>\n",
       "      <td>268.98</td>\n",
       "      <td>270.22</td>\n",
       "      <td>262.54</td>\n",
       "      <td>264.71</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-01</th>\n",
       "      <td>267.03</td>\n",
       "      <td>267.64</td>\n",
       "      <td>264.88</td>\n",
       "      <td>266.24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-02</th>\n",
       "      <td>267.21</td>\n",
       "      <td>267.21</td>\n",
       "      <td>263.54</td>\n",
       "      <td>265.29</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-03</th>\n",
       "      <td>264.07</td>\n",
       "      <td>270.78</td>\n",
       "      <td>263.85</td>\n",
       "      <td>270.14</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-04</th>\n",
       "      <td>268.23</td>\n",
       "      <td>268.73</td>\n",
       "      <td>262.52</td>\n",
       "      <td>263.13</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-05</th>\n",
       "      <td>263.94</td>\n",
       "      <td>266.48</td>\n",
       "      <td>260.97</td>\n",
       "      <td>265.38</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-08</th>\n",
       "      <td>265.98</td>\n",
       "      <td>270.73</td>\n",
       "      <td>265.50</td>\n",
       "      <td>266.47</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-09</th>\n",
       "      <td>269.50</td>\n",
       "      <td>270.49</td>\n",
       "      <td>267.27</td>\n",
       "      <td>267.95</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-10</th>\n",
       "      <td>266.77</td>\n",
       "      <td>268.63</td>\n",
       "      <td>263.58</td>\n",
       "      <td>266.96</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-11</th>\n",
       "      <td>265.86</td>\n",
       "      <td>269.97</td>\n",
       "      <td>264.49</td>\n",
       "      <td>267.93</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-12</th>\n",
       "      <td>266.22</td>\n",
       "      <td>268.31</td>\n",
       "      <td>264.98</td>\n",
       "      <td>266.29</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-16</th>\n",
       "      <td>268.30</td>\n",
       "      <td>271.79</td>\n",
       "      <td>266.88</td>\n",
       "      <td>270.38</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-18</th>\n",
       "      <td>537.50</td>\n",
       "      <td>541.94</td>\n",
       "      <td>534.17</td>\n",
       "      <td>535.03</td>\n",
       "      <td>1957664.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-19</th>\n",
       "      <td>535.00</td>\n",
       "      <td>538.24</td>\n",
       "      <td>530.08</td>\n",
       "      <td>536.99</td>\n",
       "      <td>1388440.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-20</th>\n",
       "      <td>531.25</td>\n",
       "      <td>535.11</td>\n",
       "      <td>531.08</td>\n",
       "      <td>534.83</td>\n",
       "      <td>1559131.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-21</th>\n",
       "      <td>541.61</td>\n",
       "      <td>542.14</td>\n",
       "      <td>536.56</td>\n",
       "      <td>537.50</td>\n",
       "      <td>2218249.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-24</th>\n",
       "      <td>537.65</td>\n",
       "      <td>542.70</td>\n",
       "      <td>535.62</td>\n",
       "      <td>539.27</td>\n",
       "      <td>1701682.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-25</th>\n",
       "      <td>539.00</td>\n",
       "      <td>543.98</td>\n",
       "      <td>538.60</td>\n",
       "      <td>541.08</td>\n",
       "      <td>1784967.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-26</th>\n",
       "      <td>540.88</td>\n",
       "      <td>541.55</td>\n",
       "      <td>537.04</td>\n",
       "      <td>540.37</td>\n",
       "      <td>1519503.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-11-28</th>\n",
       "      <td>540.62</td>\n",
       "      <td>542.00</td>\n",
       "      <td>536.60</td>\n",
       "      <td>541.83</td>\n",
       "      <td>1145231.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-01</th>\n",
       "      <td>538.90</td>\n",
       "      <td>541.41</td>\n",
       "      <td>531.86</td>\n",
       "      <td>533.80</td>\n",
       "      <td>2109599.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-02</th>\n",
       "      <td>533.51</td>\n",
       "      <td>535.50</td>\n",
       "      <td>529.80</td>\n",
       "      <td>533.75</td>\n",
       "      <td>1522481.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-03</th>\n",
       "      <td>531.44</td>\n",
       "      <td>536.00</td>\n",
       "      <td>529.26</td>\n",
       "      <td>531.32</td>\n",
       "      <td>1279288.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-04</th>\n",
       "      <td>531.16</td>\n",
       "      <td>537.34</td>\n",
       "      <td>528.59</td>\n",
       "      <td>537.31</td>\n",
       "      <td>1392208.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-05</th>\n",
       "      <td>531.00</td>\n",
       "      <td>532.89</td>\n",
       "      <td>524.28</td>\n",
       "      <td>525.26</td>\n",
       "      <td>2558649.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-08</th>\n",
       "      <td>527.13</td>\n",
       "      <td>531.00</td>\n",
       "      <td>523.79</td>\n",
       "      <td>526.98</td>\n",
       "      <td>2327127.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-09</th>\n",
       "      <td>522.14</td>\n",
       "      <td>534.19</td>\n",
       "      <td>520.50</td>\n",
       "      <td>533.37</td>\n",
       "      <td>1871268.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-10</th>\n",
       "      <td>533.08</td>\n",
       "      <td>536.33</td>\n",
       "      <td>525.56</td>\n",
       "      <td>526.06</td>\n",
       "      <td>1716835.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-11</th>\n",
       "      <td>527.80</td>\n",
       "      <td>533.92</td>\n",
       "      <td>527.10</td>\n",
       "      <td>528.34</td>\n",
       "      <td>1610964.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-12</th>\n",
       "      <td>523.51</td>\n",
       "      <td>528.50</td>\n",
       "      <td>518.66</td>\n",
       "      <td>518.66</td>\n",
       "      <td>1989117.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-15</th>\n",
       "      <td>522.74</td>\n",
       "      <td>523.10</td>\n",
       "      <td>513.27</td>\n",
       "      <td>513.80</td>\n",
       "      <td>2812786.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-16</th>\n",
       "      <td>511.56</td>\n",
       "      <td>513.05</td>\n",
       "      <td>489.00</td>\n",
       "      <td>495.39</td>\n",
       "      <td>3953371.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-17</th>\n",
       "      <td>497.00</td>\n",
       "      <td>507.00</td>\n",
       "      <td>496.81</td>\n",
       "      <td>504.89</td>\n",
       "      <td>2875281.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-18</th>\n",
       "      <td>512.95</td>\n",
       "      <td>513.87</td>\n",
       "      <td>504.70</td>\n",
       "      <td>511.10</td>\n",
       "      <td>2918730.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-19</th>\n",
       "      <td>511.51</td>\n",
       "      <td>517.72</td>\n",
       "      <td>506.91</td>\n",
       "      <td>516.35</td>\n",
       "      <td>3680148.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-22</th>\n",
       "      <td>516.08</td>\n",
       "      <td>526.46</td>\n",
       "      <td>516.08</td>\n",
       "      <td>524.87</td>\n",
       "      <td>2723599.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-23</th>\n",
       "      <td>527.00</td>\n",
       "      <td>534.56</td>\n",
       "      <td>526.29</td>\n",
       "      <td>530.59</td>\n",
       "      <td>2191567.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-24</th>\n",
       "      <td>530.51</td>\n",
       "      <td>531.76</td>\n",
       "      <td>527.02</td>\n",
       "      <td>528.77</td>\n",
       "      <td>704035.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-26</th>\n",
       "      <td>528.77</td>\n",
       "      <td>534.25</td>\n",
       "      <td>527.31</td>\n",
       "      <td>534.03</td>\n",
       "      <td>1037727.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-29</th>\n",
       "      <td>532.19</td>\n",
       "      <td>535.48</td>\n",
       "      <td>530.01</td>\n",
       "      <td>530.33</td>\n",
       "      <td>2276104.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-30</th>\n",
       "      <td>528.09</td>\n",
       "      <td>531.15</td>\n",
       "      <td>527.13</td>\n",
       "      <td>530.42</td>\n",
       "      <td>873923.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-31</th>\n",
       "      <td>531.25</td>\n",
       "      <td>532.60</td>\n",
       "      <td>525.80</td>\n",
       "      <td>526.40</td>\n",
       "      <td>1371819.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1257 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              Open    High     Low   Close     Volume\n",
       "Date                                                 \n",
       "2010-01-04  313.16  314.44  311.81  313.06        NaN\n",
       "2010-01-05  313.28  313.61  310.46  311.68        NaN\n",
       "2010-01-06  312.62  312.62  302.88  303.83        NaN\n",
       "2010-01-07  304.40  304.70  296.03  296.75        NaN\n",
       "2010-01-08  295.70  301.32  294.26  300.71        NaN\n",
       "2010-01-11  301.93  301.93  296.72  300.25        NaN\n",
       "2010-01-12  298.74  298.78  293.71  294.94        NaN\n",
       "2010-01-13  287.96  293.90  286.66  293.25        NaN\n",
       "2010-01-14  291.66  296.80  291.11  294.63        NaN\n",
       "2010-01-15  295.50  296.48  288.73  289.71        NaN\n",
       "2010-01-19  290.45  294.91  287.86  293.52        NaN\n",
       "2010-01-20  292.70  292.70  287.36  289.91        NaN\n",
       "2010-01-21  291.43  293.12  285.84  291.20        NaN\n",
       "2010-01-22  281.97  285.01  267.16  274.73        NaN\n",
       "2010-01-25  273.02  274.66  267.49  269.73        NaN\n",
       "2010-01-26  268.72  274.53  267.88  270.94        NaN\n",
       "2010-01-27  270.36  273.55  267.39  270.78        NaN\n",
       "2010-01-28  271.97  273.23  265.03  266.88        NaN\n",
       "2010-01-29  268.98  270.22  262.54  264.71        NaN\n",
       "2010-02-01  267.03  267.64  264.88  266.24        NaN\n",
       "2010-02-02  267.21  267.21  263.54  265.29        NaN\n",
       "2010-02-03  264.07  270.78  263.85  270.14        NaN\n",
       "2010-02-04  268.23  268.73  262.52  263.13        NaN\n",
       "2010-02-05  263.94  266.48  260.97  265.38        NaN\n",
       "2010-02-08  265.98  270.73  265.50  266.47        NaN\n",
       "2010-02-09  269.50  270.49  267.27  267.95        NaN\n",
       "2010-02-10  266.77  268.63  263.58  266.96        NaN\n",
       "2010-02-11  265.86  269.97  264.49  267.93        NaN\n",
       "2010-02-12  266.22  268.31  264.98  266.29        NaN\n",
       "2010-02-16  268.30  271.79  266.88  270.38        NaN\n",
       "...            ...     ...     ...     ...        ...\n",
       "2014-11-18  537.50  541.94  534.17  535.03  1957664.0\n",
       "2014-11-19  535.00  538.24  530.08  536.99  1388440.0\n",
       "2014-11-20  531.25  535.11  531.08  534.83  1559131.0\n",
       "2014-11-21  541.61  542.14  536.56  537.50  2218249.0\n",
       "2014-11-24  537.65  542.70  535.62  539.27  1701682.0\n",
       "2014-11-25  539.00  543.98  538.60  541.08  1784967.0\n",
       "2014-11-26  540.88  541.55  537.04  540.37  1519503.0\n",
       "2014-11-28  540.62  542.00  536.60  541.83  1145231.0\n",
       "2014-12-01  538.90  541.41  531.86  533.80  2109599.0\n",
       "2014-12-02  533.51  535.50  529.80  533.75  1522481.0\n",
       "2014-12-03  531.44  536.00  529.26  531.32  1279288.0\n",
       "2014-12-04  531.16  537.34  528.59  537.31  1392208.0\n",
       "2014-12-05  531.00  532.89  524.28  525.26  2558649.0\n",
       "2014-12-08  527.13  531.00  523.79  526.98  2327127.0\n",
       "2014-12-09  522.14  534.19  520.50  533.37  1871268.0\n",
       "2014-12-10  533.08  536.33  525.56  526.06  1716835.0\n",
       "2014-12-11  527.80  533.92  527.10  528.34  1610964.0\n",
       "2014-12-12  523.51  528.50  518.66  518.66  1989117.0\n",
       "2014-12-15  522.74  523.10  513.27  513.80  2812786.0\n",
       "2014-12-16  511.56  513.05  489.00  495.39  3953371.0\n",
       "2014-12-17  497.00  507.00  496.81  504.89  2875281.0\n",
       "2014-12-18  512.95  513.87  504.70  511.10  2918730.0\n",
       "2014-12-19  511.51  517.72  506.91  516.35  3680148.0\n",
       "2014-12-22  516.08  526.46  516.08  524.87  2723599.0\n",
       "2014-12-23  527.00  534.56  526.29  530.59  2191567.0\n",
       "2014-12-24  530.51  531.76  527.02  528.77   704035.0\n",
       "2014-12-26  528.77  534.25  527.31  534.03  1037727.0\n",
       "2014-12-29  532.19  535.48  530.01  530.33  2276104.0\n",
       "2014-12-30  528.09  531.15  527.13  530.42   873923.0\n",
       "2014-12-31  531.25  532.60  525.80  526.40  1371819.0\n",
       "\n",
       "[1257 rows x 5 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reading data from a file\n",
    "\n",
    "google = pd.read_csv('data/goog.csv', index_col='Date', parse_dates=True)\n",
    "google"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Reading data from the Web\n",
    "from pandas_datareader import data as pd_data\n",
    "\n",
    "start  = pd.Timestamp('2010-1-1')\n",
    "end    = pd.Timestamp('2014-12-31')\n",
    "google = pd_data.DataReader(\"GOOG\", 'google', start, end)\n",
    "\n",
    "## Note: this is how we generated the data file read above!\n",
    "# google.to_csv('data/goog.csv', index=True, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Containers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The `google` object is a Pandas DataFrame container\n",
    "* standard library python methods like `type()` work on it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(google)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the available attributes (\"fields\") and methods of the DataFrame by using tab-completion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-14-90252ee06138>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-14-90252ee06138>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    google.   ## uncomment, place cursor after dot, press <tab>\u001b[0m\n\u001b[0m                                                               ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "## To explore the DataFrame contents, uncomment the following:\n",
    "\n",
    "google.   ## uncomment, place cursor after dot, press <tab>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Inspection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the method `DataFrame.head()` to inspect the first few rows of data:\n",
    "* great way to inspect smaller data sets\n",
    "* useful for verifying you've read the right file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-01-04</th>\n",
       "      <td>313.16</td>\n",
       "      <td>314.44</td>\n",
       "      <td>311.81</td>\n",
       "      <td>313.06</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>313.28</td>\n",
       "      <td>313.61</td>\n",
       "      <td>310.46</td>\n",
       "      <td>311.68</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-06</th>\n",
       "      <td>312.62</td>\n",
       "      <td>312.62</td>\n",
       "      <td>302.88</td>\n",
       "      <td>303.83</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-07</th>\n",
       "      <td>304.40</td>\n",
       "      <td>304.70</td>\n",
       "      <td>296.03</td>\n",
       "      <td>296.75</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-08</th>\n",
       "      <td>295.70</td>\n",
       "      <td>301.32</td>\n",
       "      <td>294.26</td>\n",
       "      <td>300.71</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-11</th>\n",
       "      <td>301.93</td>\n",
       "      <td>301.93</td>\n",
       "      <td>296.72</td>\n",
       "      <td>300.25</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-12</th>\n",
       "      <td>298.74</td>\n",
       "      <td>298.78</td>\n",
       "      <td>293.71</td>\n",
       "      <td>294.94</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-13</th>\n",
       "      <td>287.96</td>\n",
       "      <td>293.90</td>\n",
       "      <td>286.66</td>\n",
       "      <td>293.25</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-14</th>\n",
       "      <td>291.66</td>\n",
       "      <td>296.80</td>\n",
       "      <td>291.11</td>\n",
       "      <td>294.63</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-15</th>\n",
       "      <td>295.50</td>\n",
       "      <td>296.48</td>\n",
       "      <td>288.73</td>\n",
       "      <td>289.71</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Open    High     Low   Close  Volume\n",
       "Date                                              \n",
       "2010-01-04  313.16  314.44  311.81  313.06     NaN\n",
       "2010-01-05  313.28  313.61  310.46  311.68     NaN\n",
       "2010-01-06  312.62  312.62  302.88  303.83     NaN\n",
       "2010-01-07  304.40  304.70  296.03  296.75     NaN\n",
       "2010-01-08  295.70  301.32  294.26  300.71     NaN\n",
       "2010-01-11  301.93  301.93  296.72  300.25     NaN\n",
       "2010-01-12  298.74  298.78  293.71  294.94     NaN\n",
       "2010-01-13  287.96  293.90  286.66  293.25     NaN\n",
       "2010-01-14  291.66  296.80  291.11  294.63     NaN\n",
       "2010-01-15  295.50  296.48  288.73  289.71     NaN"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the first few recods of the DataFrame\n",
    "google.head(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use the Pandas `DataFrame.info()` method to see a bit more detail, including:\n",
    "* the name of each column\n",
    "* the Numpy dtype of each column\n",
    "* how many real values are present\n",
    "* amount of memory used\n",
    "\n",
    "*side note: Pandas has excellent support for not-a-number (NaN) entries in DataFrames and Series.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 1257 entries, 2010-01-04 to 2014-12-31\n",
      "Data columns (total 5 columns):\n",
      "Open      1257 non-null float64\n",
      "High      1257 non-null float64\n",
      "Low       1257 non-null float64\n",
      "Close     1257 non-null float64\n",
      "Volume    194 non-null float64\n",
      "dtypes: float64(5)\n",
      "memory usage: 58.9 KB\n"
     ]
    }
   ],
   "source": [
    "# What data do we have\n",
    "# Weekends and holidays are not included\n",
    "google.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Pandas DataFrames have a special Index, goes beyond integer indexing\n",
    "* Pandas DataFrames are indexed by column label, then (row) Index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2010-01-04    313.16\n",
       "2010-01-05    313.28\n",
       "2010-01-06    312.62\n",
       "2010-01-07    304.40\n",
       "2010-01-08    295.70\n",
       "2010-01-11    301.93\n",
       "2010-01-12    298.74\n",
       "2010-01-13    287.96\n",
       "2010-01-14    291.66\n",
       "2010-01-15    295.50\n",
       "2010-01-19    290.45\n",
       "2010-01-20    292.70\n",
       "2010-01-21    291.43\n",
       "2010-01-22    281.97\n",
       "2010-01-25    273.02\n",
       "2010-01-26    268.72\n",
       "2010-01-27    270.36\n",
       "2010-01-28    271.97\n",
       "2010-01-29    268.98\n",
       "2010-02-01    267.03\n",
       "2010-02-02    267.21\n",
       "2010-02-03    264.07\n",
       "2010-02-04    268.23\n",
       "2010-02-05    263.94\n",
       "2010-02-08    265.98\n",
       "2010-02-09    269.50\n",
       "2010-02-10    266.77\n",
       "2010-02-11    265.86\n",
       "2010-02-12    266.22\n",
       "2010-02-16    268.30\n",
       "               ...  \n",
       "2014-11-18    537.50\n",
       "2014-11-19    535.00\n",
       "2014-11-20    531.25\n",
       "2014-11-21    541.61\n",
       "2014-11-24    537.65\n",
       "2014-11-25    539.00\n",
       "2014-11-26    540.88\n",
       "2014-11-28    540.62\n",
       "2014-12-01    538.90\n",
       "2014-12-02    533.51\n",
       "2014-12-03    531.44\n",
       "2014-12-04    531.16\n",
       "2014-12-05    531.00\n",
       "2014-12-08    527.13\n",
       "2014-12-09    522.14\n",
       "2014-12-10    533.08\n",
       "2014-12-11    527.80\n",
       "2014-12-12    523.51\n",
       "2014-12-15    522.74\n",
       "2014-12-16    511.56\n",
       "2014-12-17    497.00\n",
       "2014-12-18    512.95\n",
       "2014-12-19    511.51\n",
       "2014-12-22    516.08\n",
       "2014-12-23    527.00\n",
       "2014-12-24    530.51\n",
       "2014-12-26    528.77\n",
       "2014-12-29    532.19\n",
       "2014-12-30    528.09\n",
       "2014-12-31    531.25\n",
       "Name: Open, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Access an entire column using []\n",
    "google['Open']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2010-01-04    313.16\n",
       "2010-01-05    313.28\n",
       "2010-01-06    312.62\n",
       "2010-01-07    304.40\n",
       "2010-01-08    295.70\n",
       "Name: Open, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the first 5 rows or the 'Open' column, using integer indexing\n",
    "\n",
    "google['Open'].iloc[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2010-01-04    313.16\n",
       "2010-01-05    313.28\n",
       "2010-01-06    312.62\n",
       "2010-01-07    304.40\n",
       "2010-01-08    295.70\n",
       "Name: Open, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the first 5 rows or the 'Open' column, using date indexing\n",
    "\n",
    "google.loc['2010-01-04':'2010-01-08','Open']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-01-08</th>\n",
       "      <td>295.70</td>\n",
       "      <td>301.32</td>\n",
       "      <td>294.26</td>\n",
       "      <td>300.71</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-13</th>\n",
       "      <td>287.96</td>\n",
       "      <td>293.90</td>\n",
       "      <td>286.66</td>\n",
       "      <td>293.25</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-14</th>\n",
       "      <td>291.66</td>\n",
       "      <td>296.80</td>\n",
       "      <td>291.11</td>\n",
       "      <td>294.63</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-19</th>\n",
       "      <td>290.45</td>\n",
       "      <td>294.91</td>\n",
       "      <td>287.86</td>\n",
       "      <td>293.52</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-26</th>\n",
       "      <td>268.72</td>\n",
       "      <td>274.53</td>\n",
       "      <td>267.88</td>\n",
       "      <td>270.94</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Open    High     Low   Close  Volume\n",
       "Date                                              \n",
       "2010-01-08  295.70  301.32  294.26  300.71     NaN\n",
       "2010-01-13  287.96  293.90  286.66  293.25     NaN\n",
       "2010-01-14  291.66  296.80  291.11  294.63     NaN\n",
       "2010-01-19  290.45  294.91  287.86  293.52     NaN\n",
       "2010-01-26  268.72  274.53  267.88  270.94     NaN"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filtering by conditional\n",
    "google_up = google[ google['Close'] > google['Open'] ]\n",
    "google_up.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2014-03-27</th>\n",
       "      <td>564.86</td>\n",
       "      <td>565.40</td>\n",
       "      <td>550.50</td>\n",
       "      <td>556.58</td>\n",
       "      <td>3826676.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-03-28</th>\n",
       "      <td>558.94</td>\n",
       "      <td>566.03</td>\n",
       "      <td>558.33</td>\n",
       "      <td>559.51</td>\n",
       "      <td>2258918.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-03-31</th>\n",
       "      <td>564.63</td>\n",
       "      <td>566.93</td>\n",
       "      <td>555.87</td>\n",
       "      <td>556.70</td>\n",
       "      <td>1942277.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-04-01</th>\n",
       "      <td>559.57</td>\n",
       "      <td>568.18</td>\n",
       "      <td>558.44</td>\n",
       "      <td>566.88</td>\n",
       "      <td>2182626.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-04-02</th>\n",
       "      <td>282.43</td>\n",
       "      <td>571.83</td>\n",
       "      <td>561.44</td>\n",
       "      <td>566.98</td>\n",
       "      <td>2088804.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Open    High     Low   Close     Volume\n",
       "Date                                                 \n",
       "2014-03-27  564.86  565.40  550.50  556.58  3826676.0\n",
       "2014-03-28  558.94  566.03  558.33  559.51  2258918.0\n",
       "2014-03-31  564.63  566.93  555.87  556.70  1942277.0\n",
       "2014-04-01  559.57  568.18  558.44  566.88  2182626.0\n",
       "2014-04-02  282.43  571.83  561.44  566.98  2088804.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filtering out missing data\n",
    "\n",
    "google_filtered = google[pd.isnull(google['Volume']) == False ]\n",
    "google_filtered.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Pandas we can very easily perform many types of statistical operations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1257.000000</td>\n",
       "      <td>1257.000000</td>\n",
       "      <td>1257.000000</td>\n",
       "      <td>1257.000000</td>\n",
       "      <td>1.940000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>375.275593</td>\n",
       "      <td>378.450247</td>\n",
       "      <td>372.132474</td>\n",
       "      <td>375.327064</td>\n",
       "      <td>1.937264e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>115.684354</td>\n",
       "      <td>116.288827</td>\n",
       "      <td>114.935742</td>\n",
       "      <td>115.664301</td>\n",
       "      <td>9.842775e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>218.940000</td>\n",
       "      <td>220.920000</td>\n",
       "      <td>216.600000</td>\n",
       "      <td>217.820000</td>\n",
       "      <td>7.040350e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>285.790000</td>\n",
       "      <td>288.760000</td>\n",
       "      <td>283.060000</td>\n",
       "      <td>285.450000</td>\n",
       "      <td>1.338451e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>318.330000</td>\n",
       "      <td>320.800000</td>\n",
       "      <td>315.180000</td>\n",
       "      <td>317.260000</td>\n",
       "      <td>1.684634e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>452.540000</td>\n",
       "      <td>456.020000</td>\n",
       "      <td>449.740000</td>\n",
       "      <td>452.830000</td>\n",
       "      <td>2.164369e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>612.790000</td>\n",
       "      <td>613.830000</td>\n",
       "      <td>608.690000</td>\n",
       "      <td>609.470000</td>\n",
       "      <td>6.795393e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Open         High          Low        Close        Volume\n",
       "count  1257.000000  1257.000000  1257.000000  1257.000000  1.940000e+02\n",
       "mean    375.275593   378.450247   372.132474   375.327064  1.937264e+06\n",
       "std     115.684354   116.288827   114.935742   115.664301  9.842775e+05\n",
       "min     218.940000   220.920000   216.600000   217.820000  7.040350e+05\n",
       "25%     285.790000   288.760000   283.060000   285.450000  1.338451e+06\n",
       "50%     318.330000   320.800000   315.180000   317.260000  1.684634e+06\n",
       "75%     452.540000   456.020000   449.740000   452.830000  2.164369e+06\n",
       "max     612.790000   613.830000   608.690000   609.470000  6.795393e+06"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Computations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One simple example motivated by finance is to compute the \"volatility\" -- the rolling standard deviation of the log-return over the business days of the year.\n",
    "* First we use the `pandas` method `.pct_change()` to compute the *return* per day based on percent change.\n",
    "* Next we use `numpy` to compute the log of the return.\n",
    "* Finally we use `pandas` method `.rolling_std()` to compute the volatility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2010-01-04         NaN\n",
       "2010-01-05   -0.004408\n",
       "2010-01-06   -0.025186\n",
       "2010-01-07   -0.023303\n",
       "2010-01-08    0.013345\n",
       "Name: Return, dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the returns from daily percent change\n",
    "# Append a new column\n",
    "google['Return'] = google['Close'].pct_change()\n",
    "google['Return'].iloc[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2010-01-04         NaN\n",
       "2010-01-05   -0.004418\n",
       "2010-01-06   -0.025509\n",
       "2010-01-07   -0.023578\n",
       "2010-01-08    0.013256\n",
       "Name: LogReturn, dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the log() of the return using numpy\n",
    "# Append another new column\n",
    "import numpy as np\n",
    "google['LogReturn'] = np.log(1+google['Return'])\n",
    "google['LogReturn'].iloc[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pd' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-d15b49574519>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# Append yet another new column\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mwindow_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m252\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mgoogle\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Volatility'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrolling_std\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgoogle\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'LogReturn'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwindow\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mwindow_size\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwindow_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mgoogle\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Volatility'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mwindow_size\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mwindow_size\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'pd' is not defined"
     ]
    }
   ],
   "source": [
    "# Compute volatility as rolling standard deviation of the log() of the returns\n",
    "# Append yet another new column\n",
    "window_size = 252\n",
    "google['Volatility'] = pd.rolling_std(google['LogReturn'], window=window_size) * np.sqrt(window_size)\n",
    "google['Volatility'].iloc[window_size-5:window_size+5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, Pandas is coupled to Matplotlib:\n",
    "* Columns from a single DataFrame can be plotted using `DataFrame.plot()`.\n",
    "* Here we will plot the newly created columns of data, `LogReturn` and `Volatility`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 1257 entries, 2010-01-04 to 2014-12-31\n",
      "Data columns (total 8 columns):\n",
      "Open          1257 non-null float64\n",
      "High          1257 non-null float64\n",
      "Low           1257 non-null float64\n",
      "Close         1257 non-null float64\n",
      "Volume        194 non-null float64\n",
      "Return        1256 non-null float64\n",
      "LogReturn     1256 non-null float64\n",
      "Volatility    1005 non-null float64\n",
      "dtypes: float64(8)\n",
      "memory usage: 88.4 KB\n"
     ]
    }
   ],
   "source": [
    "google.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x1159d8d30>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x11545f2e8>], dtype=object)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA20AAAFzCAYAAACtseEIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmcjfX7x/HXByUlouyEUNFCsiS/Mm3aQ4ukfVHf0EaF\nNrR8S4tW+iZtKolKkRLKVEoKibJnzS5Ldobr98d1phnMMGfmzMyZmffz8TiPOec+9/I598zhvu7r\n87k+wcwQERERERGR+FQotxsgIiIiIiIi6VPQJiIiIiIiEscUtImIiIiIiMQxBW0iIiIiIiJxTEGb\niIiIiIhIHFPQJiIiIiIiEseiDtpCCCVDCENCCDNCCH+EEBqHELqHEP4KIUyOPM5LtX63EMKcyPrN\nY9t8ERERERGR/C1EO09bCOFt4FszeyuEUAQ4BLgb2GBmvfdYtzYwEGgIVAbGALVMk8OJiIiIiIhk\nSFSZthBCCeA0M3sLwMySzGx98ttpbNICGBRZbwEwB2iUhfaKiIiIiIgUKNF2j6wOrA4hvBXpBtkv\nhHBw5L2OIYQpIYT+IYSSkWWVgMWptl8SWSYiIiIiIiIZEG3QVgSoD/Qxs/rAZqAr0Bc4yszqAcuB\n52LaShERERERkQKqSJTr/wUsNrOJkdcfAV3MbFWqdV4HhkeeLwGqpHqvcmTZXkIIGucmIiIiIiIF\nmpntNewsqkybma0AFocQjo4sOguYHkIon2q1S4HfI8+HAW1CCAeGEKoDNYGf97F/PbLw6N69e57Y\nZ156FPTPr3MZn4/kc6lzGvtzqofOaTx+3oJ2LvPy76ogPuL5fMZz29J7pCfaTBvAncD7IYQDgHnA\njcDLIYR6wC5gAXBbJAibHkIYDEwHdgDtbV+tkbiTkJCQ200QkT3oeykSf/S9FIk/+el7GXXQZma/\n4SX8U7tuH+s/CTwZ7XEkPuSnP3aR/ELfS5H4o++lSPzJT9/LqCfXlviVn/4w44XOaezoXMaezmns\n6FzGns5p7OhcxpbOZ2zpfOaMqCfXzi4hBPWcFBERERGRAiuEgKVRiCQzY9pyVLVq1Vi4cGFuNyPP\nqFq1KgsWLMjtZoiIiIiISIzEfaYtEm3mQovyJp0vEREREZG8Kb1Mm8a0iYiIiIiIxDEFbSIiIiIi\nInEs6qAthFAyhDAkhDAjhPBHCKFxCKFUCGFUCGFWCOGrEELJVOt3CyHMiazfPLbNFxERERERyd8y\nk2l7EfjCzGoDdYGZQFdgjJkdA3wDdAMIIdQBWgO1gfOBviGEvfpo5lc9e/bk2muvze1miIiIiEgO\n+v57GDs25fWUKbBtW+61R/K+qIK2EEIJ4DQzewvAzJLMbD3QAngnsto7QMvI80uAQZH1FgBzgEax\naHg8GThwIA0bNuTQQw+lUqVKXHjhhfzwww+ADyYUERERkYJh5Uo46yy49lp4803o0wdOOgluvhn+\n/DO3Wyd5VbSZturA6hDCWyGEySGEfiGEg4FyZrYCwMyWA2Uj61cCFqfafklkWb7Ru3dvOnXqxEMP\nPcTKlStZtGgRHTp0YPjw4bndNBERERHJZjt3wtNPe2C2fj1cdRV06ACjRsH990OvXjB8OJQuDSef\nDP36gRmMGQO6ty8ZFW3QVgSoD/Qxs/rAJrxr5J415gtEzfl//vmH7t2707dvX1q0aEGxYsUoXLgw\nF1xwAU899dRe6w8bNozjjz+e0qVLc+aZZzJz5sx/3+vVqxeVK1emRIkS1K5dm7GRnLqZ8dRTT1Gz\nZk3KlClDmzZtWLduXY59RhERERFJ25o1cPrp8OWXsGoVHHYYVK0Kzz4LderAI4/A2rVw7rnw0ktQ\nvjzcdhssXw4zZvg+1q/P3c8geUO0k2v/BSw2s4mR1x/jQduKEEI5M1sRQigPrIy8vwSokmr7ypFl\naerRo8e/zxMSEkhISIiyeTlr/PjxbNu2jZYtW+533dmzZ9O2bVuGDRtGs2bN6N27NxdffDEzZsxg\n3rx59OnTh0mTJlGuXDkWLVrEzp07AXjppZcYNmwY33//PUcccQR33nkn7du3Z+DAgdn98URERERk\nH/r1gyOPhPffh/nzPetWq1ZKBu2OO+CSS+CAA/x1//5w2mnw++++Png27rXXcqf9OeXPP+Goo7I/\ns7hrF/z1l/9O8orExEQSExP3u17Uk2uHEL4F2pnZ7BBCd+DgyFtrzKxXCKELUMrMukYKkbwPNMa7\nRY4GaqU1i3ZmJ9eO1S8/M/NRDxw4kHvvvZelS5em+X7Pnj35888/GTBgAI8//ji///47gwYNihzP\nqFKlCgMHDqRSpUo0bdqU999/n2bNmlGkSEosXadOHfr06cMZZ5wBwLJly6hatSpbt26lUKG9E6Wa\nXFtEREQk+/32G9SrB4MHwxVXZHy7O+6A6tVh4EC45RZ44glYvHj/2+U18+fD5ZfDPff4+L4PP4TW\nrbP3mEOHQrt2MG8elCiRvcfKLrGcXPtO4P0QwhS8euR/gV7AOSGEWcBZwFMAZjYdGAxMB74A2qcZ\nmWWBWWwemXH44YezevVqdu3atd91ly5dStWqVf99HUKgSpUqLFmyhBo1avDCCy/Qo0cPypUrR9u2\nbVm+fDkACxcupFWrVpQuXZrSpUtTp04dDjjgAFasWJG5RouIiIhIVP7+27NjqSWXLzjrrOj2dcIJ\nnqHbuNEDjC1b8m/QNmUKXH89nHOOj/t79FGYPDk2+1+3DmbN2n3Z+PGweTO8+GJsjhFPog7azOw3\nM2toZvXM7FIzW29ma8zsbDM7xsyam9m6VOs/aWY1zay2mY2KbfNzV5MmTShatCiffvrpftetWLEi\nCxcu3G3Z4sWLqVTJ67K0adOG77///t91unTpAsCRRx7Jl19+yZo1a1izZg1r165l06ZNVKhQIcaf\nRkRERETSMnQodOrkz5cvh2bNfFn//l5gJBonnODBRpcuULgwXHQRfPxx7Nuc29asgYsv9kzb66/D\nIYfA1197MZZ58/a97ezZewfJqU2fDg0awI037r78m2/g5ZfhhRegVCkPiPOLzGTaJKJEiRL07NmT\nDh068Nlnn7FlyxaSkpIYOXLkv0FXstatWzNixAjGjh1LUlISzz77LAcddBCnnnoqs2fPZuzYsWzf\nvp0DDzyQYsWK/dv18bbbbuOBBx5g0aJFAKxatYphw4bl+GcVERERKajWrvWskRlMmOCvy5f3IiTR\nOuEE7yZ49dX++ppr4L33YtvenDJ+vGcaX3klpaDK/PkemPXqBWXKeFGWqlXh229TPueIEfveb5s2\nfp5++SXt96+/3gO28eNh7lxftnChP66/HkqW9Exc5PI5X1DQlkWdOnWid+/ePP7445QtW5YjjzyS\nPn360KpVq93WO/roo3nvvffo2LEjZcqUYcSIEQwfPpwiRYqwbds2unbtSpkyZahYsSKrVq3iySef\nBOCuu+6iRYsWNG/enJIlS3Lqqafy888/58ZHFREREcn3Vq3ysWqdO6csW7vWly9dCtOmwfnne+BR\nq1b0+y9e3Md3HXigvz7jDN9vcjXJvOKff+DUU/1cjR4Nl13mXRNbtfIs2MSJnu1KrUoVeOABD6jS\ns2qVB37PP++ZurZtvXtqsj/+8PPVtau/rlXLA8ZPPoEWLaBIETjuOH8vPwVtURciyS6ZLUQiu9P5\nEhEREYnekiXe5XHAAKhbF95+G7Ztg0KF4Pbb4X//83Fs777rwcQ118Tu2PffD888A0lJ3mUynp10\nknd1vPRS+Ogj+PFHWLbMg7dOnfz1p5/6ebvjDp/qILVXXvEAtU+ftPffuTNs3ervb9wId97phQff\neMODs1atoGFDz+TNnQu33gp33eW/n1tv9fc3bPD2rV/vbQDP3g0e7IFjZjKkOSWWhUhERERERPKN\nDRu8FP8dd3jA1q+fBybJGaG1a6FyZe8iOW2ad92Lpf/+17v05YU6c1OmeAarTx9o2tSXlS8PK1d6\n9uvCC1Oqu69cuff25cv7uMDNm3f/vHPnwvHHQ+/e8OCDvqx4cejeHb74Alav9q6YNWvCww/7+zVr\neobv3nvhhx+gSRNffuihcMwx3r3ytdfguuvg7LO9fa+95lMD5DXRztMmIiIiIpLnmXkXvgYNvKLh\n/Pnw889Qv74HHWXKeFe90qU9eDvzTPjuO1/v2GNj25YiRbyb3+LFULGiL1u40MeCxRMzOOggLwRy\nyCEpy1NPwVW7tv/8+muoUWPvfVSs6J/z5ps9ozZ0qC+fONHP93vvpZwD8DnXtm2DSpW8qMmTT+5+\nvPbtfQ64L7/0gDDZ8897li8Ez/p99x189ZVPdL5xI3z2WdbPR05Spk1ERERECpwhQ6BRI59vbfFi\n7z7XsGFK98TkoA0803bmmT52q3VrKFo09u2pUiWl9P/IkVCtWsoE3Nll587o1l+zxoO24sU9GEod\nPP32mxdpSc6+nXlm2kFn3bqerRwzBsaN8yqaRx/t3U2vvz6lQEuyEDyzdsEFewdsye+ff/7e3TAP\nOMCDtRA8KL7sspT2fPttdJ87HihoExEREZEC59df/ectt3g3uipVdn8/ddC2cqUX3WjVyseeZYcj\nj/TsWnKbwDN827f7RNyx9scffszNmzO+zf33e9CWlhNP9CC40H6ii0MO8Szbhx96l8enn/YukDt3\nerfGtNxzj2fk9gzYMmrVKh/PFplpa7csYV4RddAWQlgQQvgthPBrCOHnyLLuIYS/QgiTI4/zUq3f\nLYQwJ4QwI4TQPJaNFxERERHJqH/+8SAIfAxV375evfGll/YO2sqW9SqFH30ECxb4mLZPPvHl2eHY\nYz2Q3LrVg5lTTvFCGiNH7p19ioXnnvPj9OqVck7Auyk2brx35cUZM/xcxGJOuVde8Uzc8uVw992e\n5Wzc2DOd2eGwwzyYLF7ci5bkxZp9UVePDCHMA042s7WplnUHNphZ7z3WrQ0MBBoClYExQK20ykSq\nemRs6HyJiIiIuE2bfLzayy/7OKm//vLxUUOHetGL997zMW3jxvmYsnLlUrYdNMgnz/76a3+d3ZdX\n33/vVQ1r1vTXxx4L7dp54Hj77V48IwQv3pG6nZmxbJmXxR8xAv7zHz8Hb7zhQVyDBp4xW7rUz0uy\nyy/35fffn7Vj57akJChWzIPjfVXq3LXLs4ddu8a2Uuj+xLJ6ZEhnu7QSli2AQWaWZGYLgDlAo2gO\nVrVqVUIIemTwUTXeRqyKiIiI5JJ77vFAqEQJD94+/ti7IJ5xhmfMTjrJ1/u//9s7EGrVCqZOzbm2\n1qsHl1wCDz3kpetLlPBM259/+vvJ3RjLl4c5c7J2rJdf9uxdkyZedfHHH30qg44dfezXgAF+vrZs\n8fX79PHXHTtm7bjxoEgRz7ylnvstLQsXeibwkUegZ8+cadu+ZKZ6pAGjQwg7gX5m9npkeccQwrXA\nRKCzma0HKgHjU227JLIswxYsWJCJJoqIiIhIQTZ+PHz+uV98V6niAchnn3kRkXbtvHDGvjItRYt6\nYYxnn/UMU3Y79NDdKxq2b+/dOf/6y1+vXZsynuyffzJ/nF274PXX/fyAdxl8/nkPTqZN82DmoIOg\nQgX//O+95yX1p0+Hgw/O/HHjSblyHpDtq6vr1KneXfPtt6F6dZ+GoEgRmDXLC6dkdnxdZmUmaGtq\nZstCCGXw4G0G0Bd41MwshPA48BxwS7Q77tGjx7/PExISSEhIyETzRERERKSge+opePRRL7YBHnBc\ndZV3czzzTM+27E/79n6h/uST2dvWtJQo4cFZ8lxm69alBG3r12d+v3PmeCGO5G6Y4BnHX37xLpnJ\ngdnAgZ75e/VVL+BRvXrmjxlvkueKO/HEtN//8UevEtqvnwd4lSt7sFakiJ+jjz7yjGQsJCYmkpiY\nuN/1og7azGxZ5OeqEMJQoJGZperxyuvA8MjzJUDqYZ2VI8vSlDpoExERERHJDDMvP//yy3u/F0LG\nAjbwQCU3AjbwybbXr/egrUQJz7QdcIC/t7+uffvy009e5CS15IxThQopyxo3hjffhIsu8sqZ+Um5\ncj6ub08zZ8IRR8CoUd619vrrfXn9+j6+b8AAD2Q7dvTpAxo0yHpb9kxU9UynL2ZUY9pCCAeHEIpH\nnh8CNAd+DyGkmsqOS4HfI8+HAW1CCAeGEKoDNYGfozmmiIiIiMi+zJnj1RCTvfWWZ0X2rAiZl9Ss\n6YVClizx7M7atSnB2po1md9vWkFbCL78hRd2X37hhZ6dPOeczB8vHjVp4pNx76l2bT83Y8d6oJas\nTRsv2FK0qBexefppuO46L2qS7LfffPntt0c//11GRFuIpBwwLoTwK/ATMNzMRgFPhxCmhhCmAM2A\newDMbDowGJgOfAG0T7NEpIiIiIhIJiQl+Ri1e+/16ocAH3wAL76Y8+OOYunyy6FZM9iwwStdLlmS\nErRlJdM2YYJn0fbUuHHa3QUHDoT81hnummt8KoV+/XYPgIsV88IvFSvuHqi2aOGVRN9916cOuOYa\nX6dvX/j9d+jUCRISfMLwCRP8psHGjTB7duzaHHXJ/+ySXsl/EREREZG07NwJl17q84wdfLAHOQsX\nQu/ePmYpq6Xxc9vOnT6+avx4/zyVK0PnznD++T5Z9Icf+kTV+5J8eR1CygTWK1d6AZKC7MYbvcjI\niy/CnXf6eSpZ0ueny0j32d9/hxNO8OctW3rBmho1YNIkuOACz/JOmuT73bgx/fO9bZsXeUmuZBrL\nkv8iIiIiIjlu40YYPjyl+9mkSV4gYtgwLxry+OM+7qhXr7wfsIFXtzztNJ9DbvhweOYZzxD9/bcX\nU7nllpRzYeZzy+2ZA3ngAT8f4JOEly2rgA08UANYvNh/Ll/uYwYzOt7x+OO9S+SIEZ6Bq1HDl598\nsneRnDTJX69c6YHy1q1772PFCp/ioUEDePhhLzaTHgVtIiIiIpIndO/uGZL69T2IadsWzjrLL7bP\nOMMvpIcNy/sTQO+pSRPPIn74IZx7LnzyiWeEIKUL3jPP+Piz++/3qQ7Ag9y+fT3QA+/muOd4toLq\npJP8vEyY4K+//x6aNo1uHyee6Fm1PYPgHj08E5q8X0gJElNr2tQzdOPGefGTfVWkVPdIEREREckT\natTwoGz2bOja1UvXDx68e/n6gmLRIujWDQ4/3IuwHHigj72qWNGzPwsXwv/+B0OGwM8/+xxwN94I\nM2bkn/nWsmrTppRKkvff7xnNTp1it/+yZT3DtmGDv96xwwvkgHfpPeIIGD3axxNu3erVSpcvT7t7\npII2EREREYl7M2d6V8GVK3181q5d/iiSmVmH84m5c72b5NdfQ/PmMH++BwPFi8PmzZ5NeuEFL1E/\nc6Z347vmmtxudXxp0AD69PHy/h98kDK2LBYmTvRArGRJv7EwZkzKDYa+ff0GRHIWFDxwK1Ys7aCt\nAP+Zi4iIiEhuWbnSs0NpjSFKLt5w6KEpr2+/3cdnJVeELFTIHwVZ9epeMXPCBM8SgZ/TMmV8Auik\nJA/qWrSA0qW9O6nsrkoV+PRTz7rVrRvbfaeex+3YYz3jWbOmH+vxx1O6sSZLnjw9LQX8T11ERERE\ncsqKFT4eC+C886BUKe/Gt6fHH/cJpU87zQuLFCrkwUla44IKssKFfVzV7bf7pM/Jqlb1yogXX+xB\n7lNPwQ8/KMhNS5UqXm20U6fsPT/33ecTdk+Z4lMCnHrq7nPB7U/UmbYQwgJgPbAL2GFmjUIIpYAP\ngarAAqC1ma2PrN8NuAlIAu6KzOsmIiIiIgXMBx/4heuKFT4nVtWqHnQce6xfNLdu7UHGV195FmLn\nTs8SgXcjK1w4d9sfj77+2jM0qc9N/fre5e+GG3KtWXlGlSo+NrJdu+w9zlln+c2ILl28cM7110e3\nfWa6R+4CEsxsbaplXYExZvZ0CKEL0A3oGkKoA7QGagOVgTEhhFoavCYiIiJS8CSXNB80yEutJyZ6\nUYyPP4Y2bbxrX8uWXmikfn2oUAH++18vtlGpUq42PW4dcsjeyy6/3IO2hIQcb06ec955HrjlxDQI\njRp5Vm/pUs+ERiPqQiQhhPlAAzP7O9WymUAzM1sRQigPJJrZsSGEroCZWa/Iel8CPcxsQhr7VSwn\nIiIikg/99BMULQovveRj1X74wS+Sp09PKSRywQXw5Zc+DmjKFC+oEfYqxyCSdy1d6jcf6tWDX39N\ne530JtfOTKbNgNEhhJ3Aa2bWHyhnZisAzGx5CKFsZN1KwPhU2y6JLBMRERGRAqJ/f+8aWbq0VzAc\nMmTvdf7v/7w4ya23erU9BWyS3xxxhP9s0iT6bTMTtDU1s2UhhDLAqBDCLDyQS00pMxEREREBPGt2\nww1eDfL009Ne54EH/CGSXx14oP+sUCH6baMO2sxsWeTnqhDCp0AjYEUIoVyq7pErI6svAaqk2rxy\nZFmaevTo8e/zhIQEEtQRV0RERCTP27bNuz+2aZPbLRHJfeXKpTxPTEwkMTFxv9tENaYthHAwUMjM\nNoYQDgFGAT2Bs4A1ZtYrUoiklJklFyJ5H2iMd4scDaRZiERj2kRERETyp1at4Lrr/KdIQfbVV9Cs\nWfpzssVqTFs5YGgIwSLbvm9mo0IIE4HBIYSbgIV4xUjMbHoIYTAwHdgBtFdkJiIiIlKwbN+e0jVM\npCA799zMbRdV0GZm84F6aSxfA5ydzjZPAk9mqnUiIiIikudt26agTSQrNC+6iIiIiGSr7du95L+I\nZI6CNhERERHJVuoeKZI1CtpEREREJFupe6RI1ihoExEREZFspUybSNYoaBMRERGRbKUxbSJZo6BN\nRERERLKVMm0iWZOpoC2EUCiE8GsIYVjkdfcQwl8hhMmRx3mp1u0WQpgTQpgRQmgeq4aLiIiIFEQ7\nd8L06TB3bm63JOM0pk0ka6KdXDvZXcAfQIlUy3qbWe/UK4UQauMTbdcGKgNjQgi1NMG2iIiISPS2\nb4dTToENG2DtWvjyS2jYMLdbtX/qHimSNVFn2kIIlYELgP57vpXG6i2AQWaWZGYLgDlAo2iPKSIi\nIhLPRo+Gww/P/uNMnuyZttmz4bHHoG1bMIONG7P/2Fmh7pEiWZOZ7pHPA/cBe2bLOoYQpoQQ+ocQ\nSkaWVQIWp1pnSWSZiIiISL4xaxasWQOlS0PdujBuXOyPsWoVvPkmNGkCIcB//uPLX30VypSBBQtS\n1t20CZ55Bt56y7NysWTmj2ioe6RI1kQVtIUQLgRWmNkUds+s9QWOMrN6wHLgudg1UURERCS+JQdG\n774LF18MAwZEt/2HH3qQlZ6tW+G882DlSujY0ZeFAE2bwp13QuPGkJCQsn7fvvDZZ/DGG9CzZ3Rt\n2Zf//Q8KFYLLL8/4NuvWQVISHHBA7NohUtBEO6atKXBJCOECoBhwaAhhgJldl2qd14HhkedLgCqp\n3qscWZamHj16/Ps8ISGBhNT/+oiIiIjEqaVL4YUX4MILoUQJ6NJl/9uYeeAF8OCD8Oef0LmzB0V7\nrtehA9SsCYMGpWwDHsglJXmwWKaMt6N8efjhB7j1Vh/vduKJcOaZcP75u2+bUffdB4ceCo88AlOn\n+rJPP4Xff4fjj9//9qef7j8zc2yR/C4xMZHExMT9rhcyWxMkhNAM6Gxml4QQypvZ8sjye4CGZtY2\nhFAHeB9ojHeLHA2kWYgkhKD6JCIiIpIntWoFV1/tGai1a6FqVVi2DA45JO31e/TwAOiTTzyLdvjh\ncNBB8PPPUKOGr5OUBE8+CatXw9dfw/jxHjyl59xzPYjbudMDrSlTvKvmLbfA++97Vu7FF+G44zL+\nuV59Fe65BypUgHLlYMIE+Oor+OMP6NcPhg/3YHLWLHj6ac/w7VlwpHRpPye6zBPZvxACZrbXLY5Y\nBW0DgHrALmABcJuZrYis1w24GdgB3GVmo9LZn4I2ERERyVN++skDlvvu84CmenVf3qqVZ7bKl4cV\nK6Bdu5Rt3nvPs1Z//w3z5sFDD/nzQw/1cV+vvgqbN8Nll3n2bfVqD+Zq1tx3W375Bdq3h4kT4amn\n4P77U7JbO3b4fp95xtt78MH73tfatZ79+/prGDIEqlWD337zsXp33OFdHS+4AEqVgoEDPUP4yiuQ\nmAjNmqXsZ8sWPwcTJ0KtWtGeXZGCJ+ZBW6wpaBMREZG8ZPp0z1r93/95Numzz1Le+/hjuOkm7yoZ\nggdd5cr584YNPSv18MPejbFaNc+6mXk5/7vugpEjPbB65x0P4EqUSLcZe1m+POVYe0pI8GDuggv8\n9eDBHmSVK7f7evfdB99/759pz/eSzZ7tn//gg73QyNln+7r9+/uxJ070z1ismGcURWT/FLSJiIiI\nxNBNN3l1xrQqI27d6kVAbrgBihf3ZSec4Nmo88/3SpCTJsEXX3j3w7JlfZ05c+DUUz1ImzEj9hUX\nn3rKx7317u1FRe64w7s+du4MnTp5sGXm3Tu//DJjXSlXr/bPe8ghcM45UKWKZ+nOPx+6d/fztL/M\nnog4BW0iIiIiMfLTT156/+uvvcjHvsya5dm05PFhjRp5V8r0jBsH//yTkg2LpV9/hUsvhQYNYNo0\nDyyLFfMA64cfvAvmH3/42LhFi6IvHrJtm4/rmzzZu4i+8krsP4NIfpZe0BZt9UgRERGRAm3HDmjZ\n0p/Xq7f/9Y85xn+2a+ddBu+5Z9/r/9//Za19+1K3rhc9WbfOC5UcdJAvb9zYu2XedBM8+qi3MTPV\nHosWhWuugc8/330KAhHJGmXaRERERKLwyiswbJhXUcyLZeyTL7dSt/2JJ7wgCvg8cC+9lPnPtmED\nfPCBB4BFlB4QiYq6R4qIiEi+tXlzzo2bOv54eP117x6ZX+zY4V02f/sN/vMfKFw4t1skUjApaBMR\nEZF8KwR47jkfkzVvHtx9d/YcZ8sWrxS5bt3e85GJiGSVxrSJiIhIvrRihf/s2RNOPNFL0Z96qhf8\niLWpU+HooxWwiUjOKpSZjUIIhUIIk0MIwyKvS4UQRoUQZoUQvgohlEy1brcQwpwQwowQQvNYNVxE\nREQEvKBGs2awfr3PLfbKK3DJJbBkSWyP8+OPPo9a6smjRURyQqaCNuAuYHqq112BMWZ2DPAN0A0g\nhFAHaA0v4Nk+AAAgAElEQVTUBs4H+oaQF4fsioiISDwy87nHWrdOWXbFFXDVVb58xw7o1w+2b/d1\n//knZb1lyzJ2jKlTYdcueOABr4jYqVNMP4KIyH5FHbSFECoDFwD9Uy1uAbwTef4OECmEyyXAIDNL\nMrMFwBwgGzoriIiISH63aRMsWLD7ssGDfXzZbbftvvzee2HsWJ8c+rbb4KijfNLnkiU9eFu2DCpW\nhJkz933M6dO9TH7btrB8OYwe7XOuiYjkpMxk2p4H7gNSVw0pZ2YrAMxsOVA2srwSsDjVeksiy0RE\nREQybMcOn6z56qth8WIPvBYtgs6d4eWX9652WKkS/PIL3HGHZ9fGjoUhQ6B8ec+cffqpr/fuu2kf\nb+dOSEyE7t19oukPP/QxcyphLyK5Iap/ekIIFwIrzGxKCCFhH6tmqgxkjx49/n2ekJBAgmZlFBER\nKfDM4Pbb/fmPP8KRR8KDD3qJ+muuSX8y6mLFPGgDOPRQqFXL17/0Uu8u+eijMGKEz1GW2sqVPk6u\nTRvvDvnWWx70XXFFtn1EESmgEhMTSUxM3O96UZX8DyH8F7gGSAKKAYcCQ4EGQIKZrQghlAfGmlnt\nEEJXwMysV2T7kUB3M5uQxr5V8l9ERET+Zeal/D/4wAOrn36Cli3huuvgmWfg999h40Y45JCM73Pe\nPPjoI+jQwbNzZcrAX395N8nOneGMM6BXL8/stWwJQ4dm3+cTEdlTzOdpCyE0Azqb2SUhhKeBv82s\nVwihC1DKzLpGCpG8DzTGu0WOBmqlFZ0paBMREZFk770H99zjBUaGDvWujU2bpry/ZQssXQo1amTt\nOGef7Rm12bO9+uT06TBunL93663w2mtZ27+ISDSye562p4DBIYSbgIV4xUjMbHoIYTBeaXIH0F6R\nmYiIiKTlp5+8UEjt2t5tsUMHH1v2xBO7B2zgXR+zGrCBZ9UuvNDnevvuO2jQwAPE66+HUqWyvn8R\nkVjIdKYt1pRpExER2b/Jkz2Qadgwt1sSe1dc4dUhX3vNi39MneqFQ7Lb7Nnw7bdwyy3eHRP8pzJt\nIpLTsjvTJiIiItlo82bPAN15pwdtn3wCzZvndqtia9YsL6t/8sle1TEnAjaAo4/2R2qNGsHpp+fM\n8UVE9iezk2uLiIhIDti+3eccq18fbrjBx2CNHJkyb1iyMWNg7dpca2amJCXBpEmwdauPJZsxwzNb\nRxzhny83TZjg0wuIiMQDdY8UERGJU4sXw803e/e9Bx+ECy6Aww7zaomXXOJB28UXw0MPQaFC0Ldv\nSmn8vOCTT+Cyy3wcW6FC0LGjl+EXESmo1D1SREQkj7n9di+GMXUqlCix+3tNmsADD/gE0mvW+LJ4\nLZwxbZpnzypU2H15YqIXArn+ep+0+vDDc6V5IiJxT90jRURE4tCIETBnjo/t2jNgA580evp0mDgR\ntm2D6tVhw4acb2d6hg/3wiJvvgn16nnhlK1bd1/njz/gpJOgXDkFbCIi+6KgTUREJM6sWQN33w3P\nPQcHHpj2OsWLe2n8k0/2bpGXXBJfQdvHH3uXx/794dVXU9qZ2urVULZs7rRPRCQviap7ZAihKPAd\ncGBk24/MrGcIoTvQDlgZWfUBMxsZ2aYbcBOQBNxlZqNi1XgREZH86MYboW5dnz8sow49FDZuzL42\n7Wn1au+2Wby4V1pMbd48r3Q5bpxn0sDnWTv+eC8+cv/9vmzVKu82KSIi+xZ1IZIQwsFmtjmEUBj4\nAbgTOB/YYGa991i3NjAQaAhUBsYAtdKqOKJCJCIiIl4Bslo1L0KSVrfI9Dz9tAdBzzyTbU37186d\nUKeOt3HLFvjnHw8akyXPd9avX8q8ZwDPPw+PPALr1/vyokV924MOyv42i4jkBekVIom6e6SZbY48\nLYpn25Ijrb12DrQABplZkpktAOYAjdJYT0REpEAzg9dfh9tug3POiS5gAw+acqp75Mcf+xi0TZt8\nbF3duvDSS7BokS/7+GOvAhn2uDK45x4f39anj7e1aFEFbCIiGRF10BZCKBRC+BVYDow2s18ib3UM\nIUwJIfQPIZSMLKsELE61+ZLIMhERkQJvzhw49lh46y347DPPRJ10kldUjFbx4jnTPXL2bA/UHnzQ\ng7KXXvIgbMoUOO00uOoqOO+8vStFJnv5ZQ/o7rhDXSNFRDIq6pL/ZrYLOCmEUAIYGkKoA/QFHjUz\nCyE8DjwH3BLtvnv06PHv84SEBBISEqLdhYiISJ7Rv78HaU8/7ZmnDh2gW7fM7SvWmbb33oNvvoGb\nboL/+z9fNm0aNG8OZ5zhc8YlO/98f1x1lU878MIL6e+3Xj0f7zZliu9HRKQgS0xMJDExcb/rZWly\n7RDCw8Cm1GPZQghVgeFmdmIIoStgZtYr8t5IoLuZTUhjXxrTJiIi+ZqZZ8OSx3+dey7ceSesWwfX\nXAMjR/qyzPjpJ2jfHiZP3vu95HncSpfO2L527YKaNeHyy+Gdd7yAyFtv+Xi0EiU8uyYiIrEXkzFt\nIYQjkrs+hhCKAecAM0MI5VOtdinwe+T5MKBNCOHAEEJ1oCbwc2Y+gIiISF43eLAHPa1b+/xl06Z5\nQY+rroKvvoIzz8z8vuvVg1mzvJBJ6nug27bBMcd4cZNu3bzwx/6MGuUZs1694MMPPeN2wQXw6afe\nrVFERHJWtN0jKwDvhBAK4QHfh2b2RQhhQAihHrALWADcBmBm00MIg4HpwA6gvdJpIiJSEHzyCaxc\nCZUq+Xi1Ro1g0iR/r0EDz6qVLQtVq/p8Zs2bZ+14Bx0E11/v+9u0CW6/HY48Ej74ABo39rnSbr0V\nnn3Wx5Tty4gR0KaNj1lLSPBM3Vdf+bxwtWplrZ0iIhK9LHWPjCV1jxQRic6mTXDAAelPviy5w8wz\nanfe6ZmzQYOgWTMv4LF+vWfXjjoq+46/ahX89ptXcNy1y4Ov00+HwoVh2DCf4HrkyN23WbPG51Q7\n6yw4+GCoX9+DvFNOyb52iojI3tLrHqmgTUQkD1q/Hg47DDp39syJxI/vvvMg7b33oG1bnzetY0dY\nssQLcDzwQO61belS7475yCPwxRc+Nq1OHWjVyrs+HnCAZ9YmToQVK/y1iIjkHAVtIiL5SMeOnhnZ\nuhVmzNh7PqzctmSJdwssiG680cv4d+mS2y1JWwjelbJtW0hKgh49oGFDmD7ds7aPP+7r5MQk3SIi\nsjsFbSIi+cQvv8All8Dvv3u5+C+/hOOOy+1Wpfj8c7j4Ym/fnu3asQMGDvSxV/nRvHk+dm327IxX\nasxpIcAVV8Brr0GNGt4lslo1BWkiIvEgvaAt6nnaREQk9/zzD9x8s8/rdfjhXpL9o4/iI2i7/36Y\nOxeGDvXX3br5WKn27T2rA54dvOEGuO66+MsOZtayZf65jzjCs1Tt28dvwAbe3hIlfOzavff6JNnz\n5uV2q0REZF+iKvkvIm7bNs8UzJ+f2y2RgmLDBg+KKlXyiY6vucaXJwdte7r6ap+3K6dMneqZmsMO\n8/F2ixd7INO8uQcGO3b4etOn+89Vq3KubbG2axds3+7Pp0zxcvq33+4ZtsWL4e67c7d9+1O+vAds\n4OPrtm+H6tVzt00iIrJvCtpEMqFTJy/n/eqrGd/GzC/25s+H1auzr22SvyxZ4t0fa9f2QOfHH6FP\nn5Qs1SmneDZr5kyfo8vMfw4c6MUkcsKwYVC3LvTu7XOPlSgBlSt7gPbII3DCCXDhhT7G6/nnfZvF\nizN3rO3bUwKm3PLmm1CyJJx6qndPvfJK7wq6YYMXIYnnLFtaVGxERCT+RTu5dtEQwoQQwq8hhGkh\nhO6R5aVCCKNCCLNCCF8lT8Adea9bCGFOCGFGCCGLs9BIQbdqld+1T0jw7lW5Efy8/z6MHu0Xz2+8\n4RfM+/Puu3D00X4n/qijfAyJhnCmb+3a3G5Bztm1C95+24Obu+/2AhZt2viExrfe6n8rF1zgxSLe\nessDoNTdCgsVgssugxYtfNuGDX0OsCJFYtf9cPt2+PvvtN/78kto184nX77nnrTX+egj7xJ54IHw\n55/QpAksWpS5ttx0E1SokLltY+WPP/zfoUcf9WD0yitztz0iIpL/RRW0mdk24AwzOwmoB5wfQmgE\ndAXGmNkxwDdAN4AQQh2gNVAbOB/oG0J+GcUg2WHHDr8IBNi50wOke+7xC8KtW70q27Jl3qWnWDG/\nWMwJM2d6gPjjj3DHHTBggF88163rRSFefBEefhg2btx9u9WroXt3uO02vzt/+eUe6O3c6Zk62d07\n78C113qmYt263G5NbKxYse8g9MorPWOb/HfUuXPK3ForV3qVv6+/hltuSX8fN9zgXd6GDYOHHvKy\n7p07+/i3rFq40Of4OuIIn98r9c2GN9+E1q09KDvjjPT3ceihPqbtscdg82YvgvH++3uvt327f8/3\nZfVqv1Gyv/Wyg5nPjTd3Lpx8Mpx9thddOfvsnG+LiIgULFEXIjGzzZGnRSPbG9ACaBZZ/g6QiAdy\nlwCDzCwJWBBCmAM0AiZkrdmSX73+upcyf/xxv4A//HDvuvPCC1C0qGfY3nrL79ifdppfSG7Y4BeF\n2WX4cL8o3rrV7/C3a5cy4WyDBt4lbNQof12tGpx/vl/cHnCAZ9i2b/cL6dNO8wd4xbaLL/YL2BUr\n4Ntv/XMddphnSAqirVv9d//ccz7x7zPPwBNP5HarMufFFz2YmTwZLrrIM0s//rj3erNnww8/pIyN\nvOgizyBfeaVn1665JmN/D/Xr+99QaiVL+tiy9CQlwV9/wZgx3sXvm2/grrv877VDB99nw4b+PTzk\nEJ+s+eqrfdv27WHLFi+I0qNHyt91RhQr5hnEp57yjPWZZ/qkz2be3XDuXB8ft32738QpUcLPS82a\nPsdZcsGMgQP9e/e///n3JrtNnOjj1jZs8L/VmjWz/5giIiL/MrOoHnh27lfgH+DJyLK1e6yzJvLz\nZaBtquX9gUvT2a+tXm22a5dJAbVunVm5cmYnn2xWtKjZ2LEpfw+rV5tt3773Nk2amH3zTdaP/ccf\nZo8+ajZ8uNn8+SnLv/nGrHBhs+++M9uxw2znzt23mzfP7IknzJ591uy558w6dDC7/36zevXMOnY0\n++GH9P+m//tfswYNzG680axmTTMwu/rqrH+WvOrzz80aN/bnV1zh56NdO7P77svddkVrwABv+8kn\nm9WoYTZ4sFnJkmbLl++97ttvm7Vtmz3teOUVs9tvT//9Nm28nWBWurTZ0UebVa9uVqlSyvmvXNn/\nRn//3bcZOdKXX3ml2bvvmjVsmPn2PfaY7+vmm/31t9+a1alj9swz3o4yZcwuv9zsootS2tm1q1mF\nCmZnneX/RiQv37Il8+3IqIsv9u9s+fL+d6r/q0REJDt4eJZGrJTWwow8gBLA18BxyUFaqvf+tkwE\nbWB2yCFm48blwBmJAf2nHVsvv+wXg0lJ/siIF180O+YYsz//NFu5MvO/k/r1zVq1Mjv3XP8brFLF\ngzEwa948Y/v4/nuzYsXMDj7YbPz46I5/991+rMMPT/viPr/atcts2DCzl17yz9+liy/fts0Dmgce\n8OWnn5677UzLqlVmixZ5gPPVV/5Z+vb13+GDD3q7+/XzdS+7zOx//9t7H3fe6QF/dnj33fRvAgwa\n5EHarFl+42HJErP16/2cDx3q68yebbZmze7b7dhh9tZbHtgddZTZmDGZb9+2bWY//ugB7RdfmJ1z\njtnTT/t7/fubPfmkn8OEBF931CizTp3Mhgwxu+sus9atU4K2zz/PfDtS27Ej7ZtDZh5IzpzpAWJG\n/30SERGJVnpBW5Ym1w4hPAxsBm4BEsxsRQihPDDWzGqHELpGDtwrsv5IoLuZ7dU9MoRg1at3p3p1\n79LTvXsCF1+ckOm25YR69XwcR3I1NMmahx/2LoWPPBLddr17+/iYyZO9+9Jxx3l3ytdf9+6Uqc2Y\nkdKtato0L2qwejVMmuTj1goX9kIJAwd617xRo3w8T0YtW+bdp2rViq4IxLZt3lXypZe8KMU11/jc\nSQcdtP/jff45XHUVFC+e8ePFi7VroUwZKFXKfw+JidCs2e7rJJ/H3CzcsnEjjB3r3/eNG+Hjj738\n/lFH+fjENWv8b69HD//9tWnjv7uZM70c/Pff++/0jz92/z2ddpoXs9jXeLDMGjbMvwPDh+/9Xo0a\nPi6zadPM7Xv1au+2eOCBWWsjwCuv+PetcWPo1Stln7t2ebXJbt32rsa4ebOvt2OHd6P97DMvyHL1\n1Zkvt79rl3dJPfFEnwMvtY0boWxZHyNYULsvi4hI9khMTCQxMfHf1z179kxzcu1os2tHACUjz4sB\n3wEXAL2ALpHlXYCnIs/r4F0pDwSqA3PBA8U09m3btpnNnet3TitUMHvvPbMzzvAMSrwZMcLbeeON\nud2S/OPuu816945+ux07Uu6433BDyvMyZTwrkGzFCrOyZT2b8803Zh9+6JmI554z++efvfe7fn2m\nP0qWDBzod/WHDdv7vU2bzG65xWzDBn996aVmxx/vXbb2zIrkBYsWmVWsuO91/vrLrFSpnGlPej7/\nPOXvKrnbYJcu/nzAALOPP/bn112X/j6uu84zRMmuvda3ya7fW2Kin9v77zdbuzZl+ezZ/u9rfukp\nsGCBd6kcNMi7V2e2q+Srr5rVquX/btxwg9nUqSnvjR5t1rRpbNorIiKyL6STaYv2nmEF4J0QQiF8\nbNuHZvZFCOEnYHAI4SZgIV4xEjObHkIYDEwHdgDtI41J04EH+h3g+fOhTh2/g7p4sd/hzI677Hff\n7YPd+/aNftshQ3zwfiHNdBczGzZkLltUpIgXLgnBq0pWrw5HHul37s84wzOi9epB//5wzjleACFZ\n69bp77dEiejbEgtXXeUV+wYP9gIMyZmmnTvhvvtSCrEsW+bFEWbP9rLwDz3khVmOP97n9Fq3zidh\njmebNu3/d16xomfkPvkELr00Z9q1p+XLvULgsmWeLXv0UX89daq3KblKY4sW6e+jd2/PAlev7lm5\nd9+FqlU9y5gdGjXy78WwYZ4B3LABqlTxrFa7drGbDiC3Va3q5ffBi6Z8+KEXThk0yDOCb7/tBYKS\nVavmFVzPOstfz53rhVcmTfIM6vr1Pil5u3ZeyfaAA/xvb88MsIiISE7KUvfIWAoh7BbPzZjh1bnW\nroVy5bzSWeHCsTve0qX+n/cRR/hF8Lnn7nv9+fO9yt8pp/h/6ied5BfXM2f6f/SSdVdeCa1aedey\nWLnsMg/evv0W5szx0uQnnhi7/WeXtWu9a+cJJ3g1xXLlvLvksGH+9/ryyzB+vFf0K1bMy8G/8YYH\nDbNn+/enWrWUqoTxatIkvziePHnf67Vq5Z/r7ru9tP3FF+dM+5L9978emD31lP9u0gq0hg7187+v\nGzkTJ3og/uqr3q36u++yr83JfvvNb1pUq+YBZpMm/jM/3nB6/30PwJKrZjZt6pUex43z7qqzZvlc\ndscf7/9uH320f5dGj/ZAvF49327HDv9e7dzpP4880ru4limTe59NREQKhhBCmt0j4/a/7dq1/Q5n\n2bJe9j29iV2jtWGDl9j+7ju/KO7Tx+cM6tLF79AmW7vWMxUdO/qd8VNP9QvHgw7y8UoXXOBZnII0\nCXB2y47S/VWq+Bi5SpV8DFteCNjAg4Lx4z1jUKOGB5yvvQZffeXTDLzzjgcxxYr5+mec4Vm4Tz/1\nMUznneeBXrzbuNE/4/588IFPxnzrrf5YsiT725ba8uUeLEL6mbFWrfYfCDVo4OPgbr/dx7jlhBNO\n8Hadf77fALj88vwZsIEH8//5j4+NfPhhD7SOOsrnVhw3Dpo393kff//dx6o+8wzceaePLUwO2MD/\n73nxRc9Q/vMPTJ+ugE1ERHJXnhhSXa6cZ7nKls36vl580QO15cv9wqllSw/ANm70AG3VKg8Sb77Z\nu92dcgqMGOFB5NSpfiHQubN3lZk8WUFbLGW2e+S+VKniWdpHH/V53vKSgw/2QK1fP88IPPmkdxVM\ny9VXp8yhVaOGZ3Fatsy5tmZWRrpHgt8sadrUg9bly6FyZb+gfvjhnAlAVqzwDFUsfPihz/t3+eWx\n2d/+FCrkhW7ipFNFtipRIqX7c3J3xjfe8Pnm+vf3giWXXOJz0C1a5N2p27RJu5t0hw45124REZH9\nyRNBW/nyftF0wglZ39cnn/hYhxo1/EIwBH8NcOGFfrE7ZoyPWdvzoqpuXe+elqxUKQ/yhg71bM6Q\nId71RjJn48bYZ9qOP95/j8lZkryod28fw9a+fca3OewwzxTHu02bMpZpA8+SfPWVP09M9BsrZ52V\n/eP2zODnnz1AjIUqVeDrr2Ozr4wqyBUPDz3U/17GjfOADTzLaQbXXuu/DxERkXgXt2PaUrvrLh9Q\n/u67Gb+rvn27d3HxffvP+fN9fNOyZemPj/vzTx9Lt23b/stZb9nid/83b/ZsQcWKPk4i+biScbt2\neUGBb77x7qeSNVu3QsmS/nPPghObN3uwtGd3L7OcKU4xYACceaZny95+20vpv/PO/rf7/feUGze7\ndvk4vkaN4Lbbojv+mjU+PnDLFs/G3Hjjvtf/5RcP/BcsyD/FO0RERCQ+5bkxbak9+STMm+eB1p13\n+rLly/1OO/jFZlKSX8gNHOgZtKJFvVvlLbek7OfBB73owb4KmtSo4fvJyPxDxYp5F8mZM/1O7saN\nPreXZMy0aR4cf/ONB79//eWBhmRdchZ569bdl2/d6t0oq1f3OeHAiy3ce2/OVJo0g+uv927K4ONL\nM5ppO+44LySTHFwed5xXctzTvHk+TnXAgLT307Ond2tu2hQ6dfKM5M6dnjFPS48ePuZVAZuIiIjk\nljyRaQO/yBo71u+Kv/OOF12YO9fHKwwZAiNHenfF8uW9C0zjxj4APSHBuzYVLpwyjiG7SrkvXeqD\n2T/7zIPIU0/1QE6BiEsuOhGCZy2qV/ff0WGH+YX8xRf7MomN8uX9pkLqcXAdOniBk759vdLhP//4\n2LnSpf379OST0LWrf0+Skvz3E0uDBnnV1datvYvaJZf4mNIRI6Lf19SpXvV1/vyUSci3bPFxqNWr\ne2Zuzhy/GXDFFZ7Vq1jRs3VffOFB35ln+ufduNErjS5atHt3uTlzPLhbuDCl6IuIiIhIdkkv0xbV\nSIcQQmVgAFAO2AX0M7OXQwjdgXbAysiqD5jZyMg23YCbgCTgLjMblZkPUKqUl6meNQsee8wDsLlz\nfZxPr17w7LOeuWnZMuUCDvyu+umnewnnxx/P3rm3KlaE//3PgzXw8W3ly3uwWZBt2uQXzaNHe/XG\na6/1wO3oo/0CeeTIgj3mJrucfrpXxbvjDu8KWbiwl0SfO9enupgwwedBPOoor645d64X2zj8cK/Q\nCLt3E5482Yv3bNjgGa86daJrz4ABnrEaMcILQFx2mS/P7DxlJ57o286Z44HYrl0egB1zjBf7qFPH\nx7AuWOCf9cor/d+Iiy9OaXudOn5+1qyBChU845i6iuwrr3i2XgGbiIiI5KaoMm0hhPJAeTObEkIo\nDkwCWgBXAhvMrPce69cGBgINgcrAGKBWWim1/WXa0jJ1ql/07+uib+dOv1hs0iTnioRs2eIBytKl\nHij+9ZcHLrfc4hfN+bXcdnrGjPGL9BEjfLLasWO9oMvdd8euuIPszcwzShdd5Bmp4sU9kEvuYpyW\nTz7x8VsdOvjf77Jl3g35p588g9ytm5ffnznTb060aeNB+P7mUPz0Uw+ORo/27+GuXZ7xa9zYt81s\n18Pmzb2L49ln+w2Sv//28a+HH+6B5SOP+Jx1jzzi2bSuXXcfy/fEEz4p+Zw5vrxCBf/MJUt6cFqt\nGkyZomIVIiIikjPSy7RlqXtkCOFT4GXg/4CNZvbcHu93BczMekVefwn0MLMJaewr6qAt3iUledez\nDRvg88/9Yvjnn+HkkwtO4Pbdd17soUMHz1okS0ryC/VYTpguaVu5MrrpMl57zUuk16sHr7/ugfZh\nh3mGOzl4WbPGA+8XXvDArWvXfe/zmmt8LrnkcaixcuONPhavfn2fh+yDD/w4qaXulrunzZs9SKtR\nw1+fdZZP/dGypQe6hx0GgwfHts0iIiIi6Yl50BZCqAYkAscDnYEbgPXARKCzma0PIbwMjDezgZFt\n+gNfmNknaewv3wVt4BeKxxzjk+oOH+5jiGrV8nnfXnst/VL0v/3mUwzklEmTvD3PPOOFHE46Kev7\n3LnTP0PLlt7d7sgjs75PiS+//eaBzk03edXU9ArxHHOMjz2N9eTmDz3kxz32WN//Rx9lbX/vvQdv\nvgkvv+w3GxYsiP3cgSIiIiLpiWn1yEjXyI/wMWobgb7AUWZWD1gOPLev7QuSfv18HN6wYd5N8N57\nfQxeUlL6F5jz53uWY8qU3ZevWOFj+F5/PfYT5b74ou+3enXPBP7wQ9b3OWiQjyF87DEFbPlV3br+\ndz1zJvz3vylzHqb2559eoTHaMXAZcdRR3rVx9uzYTBVx5ZXe3lNO8e6cCthEREQkHkSdaQshFAE+\nB740sxfTeL8qMNzMTkyje+RIoHt63SO7d+/+7+uEhAQSEhKialu8KlTIg6zUp3rsWLjhBr/YTS5y\nkFzK/I03fPxbs2ZeDj+5K2XnzvDrr148onNn+M9/9n/sXbu8OMpll/kUCMk2b/Zj16/vx61b1/d3\nyineHWzBAu9qlizaMUc7dvhF+muveYU+yf8+/xyee84njh471jNw4Fm4UqX8vVibNs3Hs4Xg8zie\nc07W9/noo9C9u2eKC0o3ZhEREckdiYmJJCYm/vu6Z8+esekeGUIYAKw2s06plpU3s+WR5/cADc2s\nbQihDvA+0BioBIwmhoVI8ooZM3wMUNOmuy+vW9e7Yp18Mowb55X+FizwYg8JCX4ReuaZXtnvt9/g\nxx+9SuaRR/q+3njDK+Hty733+sXykCE+pm7OHC9GsW6dz9k1bJh3aZswwQO5YsW8q2TyOLTk7Z5+\nOs1JZvIAACAASURBVLrPPGiQB4up/gYln1u1yiemf+klvyHxyy/eLbJiRS9qcuihsT/mzp0+9qxb\nt5QgMavMfBxcdrRXREREZF9iMqYthNAU+A6YBljk8QDQFqiHTwOwALjNzFZEtukG3AzsYB8l//Nz\n0JaeK6/0ubKuusrHknXp4hXs1qzx4MrMq16uXAnXXecV+BYs8KzFhAleFXDiRKhaNf1jHHOM77Nt\nW8+qNWniWYTrr/cL3Rkz4K67fP+py+5PmwbPP+/737TJu4xlxJAhXjJ99mzPBl5/fZZOkeQxrVvD\n9On+NxmCj+UcPx6++iq3WyYiIiIS/7KlemQsFcSgrXt3D77mzfMg7d13PbhasiSlSt/993tAl9y9\nMnU3xXvv9a6SN9/spdP3nMR7/HifuLhjR+/mNXOmd31ctcrn6cqInTt94uU//9z3NuvWeXZu2zYP\n1I44wn8mz/ElBcvWrZ51K1XKq09WrpzbLRIRERGJfwra4tCSJXDPPR50HXecjzk74IDd11m3zic1\nTmtc2Lp1np377jvvwvjYYynv/fijZ/HefhsWLvQ57aZMgaee8q6X0TjrLJ9T7aKL9h7bllwWvUQJ\nzxAOHAhFi0a3f8mfZs/26qjZOaG9iIiISH6ioC0fGzzYi4YMHZqy7KqrfIxc+/ZepfK663wuqp9+\n8jmrovH66z459ubNHgwecYSPsxsyxOfn+s9/4J13fEqD6tVj+9lERERERAoKBW352PTpnlWbM8df\nb93q3dEmT/aiJUuWeODVpUtKpcpo7NzpwV6/fjBggC+79lqfKLxvX1WHFBERERGJBQVt+VhSElSo\n4NX6qlXzwOrDD2HEiNgeZ8QI7yJ55ZU+f9UXX/iYusKFY3scEREREZGCSEFbPnf77V4w5IknoF07\nn5y7Q4fsPeaWLZnL3ImIiIiIyN4UtOVzixd7BcmKFb2b5Ndfe7l1ERERERHJGxS0FQB//w3z53t3\nycaN9670KCIiIiIi8Su9oK1QlDupHEL4JoTwRwhhWgjhzsjyUiGEUSGEWSGEr0IIJVNt0y2EMCeE\nMCOE0DzrH0XSM21aIg0awCmnKGCLlcTExNxuQr6hcxl7Oqexo3MZezqnsaNzGVs6n7Gl85kzogra\ngCSgk5kdBzQBOoQQjgW6AmPM7BjgG6AbQAihDtAaqA2cD/QNQeFEdtGXJvZ0TmNH5zL2dE5jR+cy\n9nROY0fnMrZ0PmNL5zNnRBW0mdlyM5sSeb4RmAFUBloA70RWewdoGXl+CTDIzJLMbAEwB2gUg3ZL\nDtEXUST+6HspEn/0vRSJP/npexltpu1fIYRqQD3gJ6Ccma0AD+yAspHVKgGLU222JLJM8oj89Mcu\nkl/oeykSf/S9FIk/+el7malCJCGE4kAi8JiZfRZCWGNmpVO9/7eZHR5CeBkYb2YDI8v7A1+Y2Sdp\n7FNVSEREREREpEBLqxBJkWh3EkIoAnwEvGtmn0UWrwghlDOzFSGE8sDKyPIlQJVUm1eOLMtQ40RE\nRERERAq6zHSPfBOYbmYvplo2DLgh8vx64LNUy9uEEA4MIVQHagI/Z7KtIiIiIiIiBU5U3SNDCE2B\n74BpgEUeD+CB2GA8q7YQaG1m6yLbdANuBnYAd5nZqFh+ABERERERkfwsbibXFhERERERkb1lunqk\niIiIiIiIZD8FbSIiIiIiInFMQZuIiIiIiEgcU9AmIiIiIiISxxS0iYiIiIiIxDEFbSIiIiIiInFM\nQZuIiIiIiEgcU9AmIiIiIiISxxS0iYiIiIiIxDEFbSIiIiIiInFMQZuIiIiIiEgcU9AmIiIiIiIS\nxxS0iYiIiIiIxDEFbSIiIiIiInFMQZuIiIiIiEgcU9AmIiIiIiISxxS0iYiIiIiIxDEFbSIiIiIi\nInEsQ0FbCOG8EMLMEMLsEEKXNN6/JITwWwjh1xDCzyGEppHllUMI34QQ/gghTAsh3BnrDyAiIiIi\nIpKfBTPb9wohFAJmA2cBS4FfgDZmNjPVOgeb2ebI8xOAwWZWO4RQHihvZlNCCMWBSUCL1NuKiIiI\niIhI+jKSaWsEzDGzhWa2AxgEtEi9QnLAFlEc2BVZvtzMpkSebwRmAJVi0XAREREREZGCICNBWyVg\ncarXf5FG4BVCaBlCmAEMB25K4/1qQD1gQmYaKiIiIiIiUhDFrBCJmX1qZrWBlsDjqd+LdI38CLgr\nknETERERERGRDCiSgXWWAEemel05sixNZjYuhHBUCKG0ma0JIRTBA7Z3zeyz9LYLIex7cJ2IiIiI\niEg+Z2Zhz2UZybT9AtQMIVQNIRwItAGGpV4hhFAj1fP6wIFmtiay6E1gupm9mIEG6pGFR/fu3fPE\nPvPSo6B/fp3L+Hwkn0ud09ifUz10TuPx8xa0c5mXf1cF8RHP5zOe25beIz37zbSZ2c4QQkdgFB7k\nvWFmM0IIt/nb1g+4LIRwHbAd2AK0jgRwTYGrgWkhhF8BAx4ws5H7O67Eh4SEhNxugojsQd9Lkfij\n76VI/MlP38uMdI8kEmQds8ey11I9fxp4Oo3tfgAKZ7GNkovy0x+7SH6h76VI/NH3UiT+5KfvZcwK\nkUjuy09/mPHi/9m77/CoqvSB498zmZRJL6SYDgm9gxRBMKBSFVAWBBRQEWRdOyq76v6EtazYUBcL\nCGIBARUUFhUQMRY6iIL0GtKAkEDqJJlyfn8kZENNApNMEt7P8/g4995z7n3nJkPy5pz7HrmnjiP3\n0vHknjqO3EvHk3vqOHIvHUvup2PJ/awZFS6uXVOUUrq2xCKEEEIIIYQQNU0phb5AIZJKTY8UQggh\nhBBC1A+xsbEkJSU5O4yrWkxMDEeOHKl0exlpE0IIIYQQ4ipSOprj7DCuahf7GlxspE2eaRNCCCGE\nEEKIWkySNiGEEEIIIYSoxeSZNiGEqIcyCzJ5b8t7PNr1UfKK85i8ejJ/7/53mgc3x2a38dmOz1iX\nvA6r3UpOcQ5erl54unqSXZTNnpN7eLDTg4xtN7ba4pu7bS6/pf/Gfwb8p9quIYQQQtQX8kybEELU\nQ4lHEuk/vz++7r5kFmRyU6Ob2JK2hf6N+7Pn5B7cXdy5o+UdGJSBQFMgBZYCCiwF+Lj7kF2YzZxt\nc3iw84NkmbMothXzj+v/gauLq0Ni+2r3V4xcPJL217Rn/bj1DjmnEEKIyquvz7R9/PHHzJ49m19+\n+eWy+g8YMICRI0cyevTo887l4+PDjh07iI2NdUisVX2mTUbahBCiHjJbzCTEJvDyjS8T7hNOsFcw\nB7MO8sraV+jTqA8v9H4Bpc77mQBAkbWIfZn72Jy6GY1m4Z8LeX/L+3QM74i/hz/RvtHcHHcz87bP\no2tkVwY2HkimOZNWIa0qFdt7W95jcvfJLN271JFvWQghRD3Qv39/unTpwpQpU87av3TpUiZOnEhq\naioGw8Wf8LrYz7ZzTZ06lYMHD/LJJ5+U7fv2228veq7c3Nyy1/fccw9RUVH861//qtS1HEGSNiGE\nqIcKLAV4unrSNqxt2b64wDhm3jqzwr7uRnfeGfhO2fbsQbP549gfHDl9hOyibH468hPPJT5HXEAc\nk1ZNYvLqyeQU5TD/9vkEmYIwuZoothXz05GfsNgtdArvxOBmg8vO9+eJP3mq+1Ms+HOBY9+0EEKI\nOm/s2LE8++yz5yVt8+bNY/To0ZdM2OozSdqEEKIeKrAUYDKaHHa+tmFtyxLAMW3HlO2fPWg2CsWk\nVZOYvmE6JqMJs9WM1pobYm7A5Gri7qV3c6LxCVxdXNmfuZ8CSwFNg5pSYClwWHznOllwkpScFH5O\n+pliWzHRftHc1OgmAk2B1XZNIYQQV27IkCH89a9/5ddff+X6668H4PTp0yxfvpzNmzeTk5PDgw8+\nyIoVK/Dy8uK+++7jmWeeueC5Hn30UZYsWUJ2djZNmjRh+vTpXH/99axcuZKXXnoJgK+++or4+Hi2\nbdtGr169GD16NPfee+955zIYDBw4cIAffviB+fPnYzAYePPNN+nVqxc9evRgw4YNfPnll2XtH374\nYVxcXJg+fbpD7kulkjalVD/gTUqqTc7RWk875/gg4HnADliAx7TWayvTVwghhOOZrWY8XT2r/ToG\nVfIXzzf6vnHRNutT1uPxogdBpiCKbEVMu2kanq6eDk/a7NpO4pFEdhzfwb9+/hdh3mF0jeiKj7sP\ni3cvZtraaUzsOJGdGTvJLsrm8KnDPN3jaXZn7ObhLg9XekqNEEKI6uPh4cGwYcP45JNPypK2RYsW\n0bx5c1q1asWYMWPIzc3lyJEjZGRk0KdPH8LDw7nnnnvOO1fnzp2ZMmUKvr6+vPXWWwwbNoykpCT6\n9u3L008/fd70yEs58zNi/PjxrFu37qzpkceOHWPq1Knk5OTg6+uLzWZj0aJFrFy50kF3pRJJm1LK\nAMwAbgTSgM1KqaVa6z3lmq3WWi8rbd8a+BxoXsm+QgghHMzRI21XYsWdK8grzqPAUoBSihCvEMwW\nM2ar+YrP/f3B73ku8TkALHYLOUU5dArvxPejv6fDNR3K2mmt+du3f2PC8gn0iO7BsBbDiA+Ip++8\nvoR5h7Hn5B7e7PcmLgYXjAaZhCKEEGrqlf8hSz93ecVOxo4dyy233MKMGTNwc3Pj008/5e6778Zu\nt7No0SK2b9+Op6cnMTExTJo0iU8//fSCSduoUaPKXj/22GM8//zz7N27l9atW1f9vVyicEtYWBg9\ne/bkiy++YNy4cXz33XcEBwfTrl27Kl/nYirzk6kzsF9rnQSglFoIDAbKEi+tdfk/l3pTMuJWqb5C\nCCEcz2ypmZG2ylBK4ePug4+7T9k+D6MHRdYi7NpeNlpXVS/98hLT1k7j3QHvEusfS3peOv3j++Pl\n5nXBGN4d+C7/6f8fXAwuZfvHdRiHp6snAz8biM+/fQj3CWfZyGW0CW1zWTEJIUR9cbkJlyN0796d\n4OBgvv76a6699lo2b97M119/zcmTJ7FarURHR5e1jYmJITU19YLnee211/jwww9JT08HSoqJnDx5\nslpiHjNmDO+//z7jxo1j/vz5jB492qHnr0zSFgEkl9tOoSQZO4tSagjwbyAYGFiVvkIIIRyrwFKA\nybV2jLRdiFIKD6MHZov5gknWpaw9upZ//PAP0nLT2PfgPkK9Qyvdt3zCBhDmHQbAj2N/JK84j5UH\nVnLjJzcS7hNOrH8si/6yCA+jR5XiE0IIceXOlN3fs2cPffv2pUGDBtjtdlxdXUlKSqJZs2YAJCUl\nERERcV7/X375hVdffZUff/yRFi1aABAYGFg2YnYlU+Iv1HfIkCE88MAD7Ny5k+XLl/Pqq69e9vkv\nxGFzQLTWXwNfK6WuB14Abq7qOcpXiUlISCAhIcFR4QkhxGX549gftJvZjp/u/omeMT2dHU6lma1m\n/Dz8nB3GJXm6emK2Vj1pe3396/SJ68OT3Z7E3ejukFiMBiP+Hv7c0eoOmjVoRnJOMk+seoJdGbvO\nmmYphBCiZowZM4YXXniBHTt2lBXzMBgMDB8+nGeeeYaPP/6YzMxMpk+fzlNPPXVe/7y8PFxdXQkK\nCqK4uJiXX375rLL9oaGhrF69Gq11lRO40NBQDh06dNY+d3d3hg4dyqhRo+jSpQuRkZGVOldiYiKJ\niYkVtqtM0pYKRJfbjizdd0Fa61+VUo2UUoFV7XtuaU8hhHCmLWlb6DevHwDH8447OZqqqU3PtF2M\nydVUYTGSE/kn+OPYHyilMBqMtAtrx/qU9bx808sOS9jOdaZS5qvrXiWnKKdariGEEOLSYmJi6Nat\nGzt27GDQoEFl+99++20eeughGjVqhMlkYsKECRd8nq1v37707duXJk2a4O3tzWOPPUZUVFTZ8WHD\nhjFv3jyCgoJo1KgRW7ZsuWTyVv7YuHHjGDZsGIGBgSQkJLBkyRKg5Fm82bNn89FHH1X6fZ47UDV1\n6tQLX7+i1dCVUi7AXkqKiaQDm4CRWuvd5drEaa0Plr7uACzVWkdVpm+5c+j6uDK7EKJusWs7OUU5\n+Hv4c/ui2+kV24uNqRvpG9eX0W0dOz/dUSw2C6sPrSajIIMCSwHhPuHM3zGffnH9uKf9+T/Iaoum\nM5qydMRSmjVodtb+/OJ8XAwuDFowiE2pm2ge3JxjeccI8w5jQ8oGukR0Yd24dZf9LFxl3brgViZ0\nmMCtTW+t1usIIURNU0pdsrCGuDzJyck0b96cY8eO4e3tfcm2F/salO4/L3uscKRNa21TSj0IrOJ/\nZft3K6XuLzmsZwFDlVJjgGLADAy/VN+KrimEEM7y4s8vsmTPEvrH92fN4TXMu30e249vd0ilw+oy\nfcN0Ptz2IR2u6YCXqxezj81ma/pWbmt2m7NDuyQ3Fzf6fNqHEK8QjAYjVruVU4WnSM5Oxt3oTs+Y\nnhx/4njZiJrWmm/3f0ufuD7VnrAB+Lr7ykibEEKISrHb7bz++uuMGDGiwoTtclTqmTat9Qqg6Tn7\nZpZ7/QrwSmX7CiFEbZSSk8KbG9/EoAz8Z9N/mNF/Bp6unphcTZgttTNps9gs/GfTf1g2Yhntr2kP\nQGZBJhOWT6B1SNVLGtekz//yOel56fi4+WC1WzEajPi4+9A4sDH7MvcR4x9z1hRIpRQDmwy8xBkd\ny9dNkjYhhBAVKygoIDQ0lIYNG/Ldd99VyzVkMRohxFUlLTcNk9FEgCngvGPPrnmWiR0nMrXXVBSq\nrNKgyWii0FpY06FWyhe7viAuIK4sYQMI8gxi8fDFToyqcpoHN6d5cPOLHnM2GWkTQghRGZ6enmcV\nOakOkrQJIeqF34/9jotyYW/mXlJzUhnfcTzuLu4UWguZvmE6qw6uwsPowfeHvifCJ4KDDx9kU+om\nvtrzFa/3eZ2/r/47i3YuIn1S+nmLK3sYPRw+PXL+9vk0DGhIt6huVe5rs9v4bMdnvLP5HfZl7uPL\n4V86NDZRwsfdR5I2IYQQtYIkbUKIOu/Xo78yaMEgPF09aRLUhEOnDvHoykdxc3HDx80Hi93CjP4z\nCPYK5tmezzJ9w3T+/eu/OXL6CB//8TFf7PqCIFMQe/62B38P//POb3I1ccp8ymHx5hXn8fCKhxnf\nYfxZSVtabhobUzZi0zb2ntzL7pO7MVvNFFgKyCzILCs0olDEBcbxfzf8Hz2ie5y1aLVwHF933zpX\nNVQIIUT9JEmbEKLO0lrz71//zafbP+XJbk/yjx7/AEqSn/zifKL8okjNSSXWP/asRZUbBTSi3fvt\ncDG48N7A9+gb15cY/5iLFrcwGU2kWdMqHVexrZjNqZvx8/AjLiCubJHrLHMWfu5+zP5tNnZtZ9ne\nZXyx6wsy8jNoE9qGlJwUWgS3wN3oTkP/htzY8EZ83H3wdPUkyBREsFcwJqMJs9VMQ/+GV7QwqKiY\nr7svucXVO91FCCGEqAxJ2oQQddavR3/lw20fMvOWmdwQe0PZ/nCf8LLXcYFx5/WL9I1k2chl7M7Y\nzb3t760w+TG5Vu6ZNq01qw6uYupPU8k0Z6JQpOelM+m6SbgoF1785UUaBTQiPS+d53s9z0PfPcSM\n/jO4s82dfLPvGwJMAQxoPKAKd0BUJ3mmTQhRX8XExMgf/pwsJiamSu0laRNC1FlrDq9heMvh3Njo\nxir37RbVrdLPk1XmmbYdx3cw4LMB+Lj58FjXx7irzV2YXE2sPLCS2dtmo7Vm5wM7OXTqEA08G9Aw\noCErDqxg4rUTcTG4cGebO6v8HkT1kqRNCFFfHTlyxNkhiCqSpE0IUWcdyT5C96ju1X4dk/HSJf+T\nTidx3ZzreKH3Czzc5eGzpln2je9L3/i+ZdsNAxqWvV4+ann1BCwcQpI2IYQQtUX1r04qhBDVwGq3\n8vux32no37DixlfoQtMjtda8t/k9vF/ypsW7LXi6x9M82vXRGln0WdQMHzepHimEEKJ2kJE2IUSd\n9MHWD/j92O80Dmpc7dcyGU0czz+OxWbBru1sTN3IqoOrmLl1JkvuWELvhr3PWyZA1H0y0iaEEKK2\nkN8yhBB1yp8n/uSU+RRf7v6S9we+T7RfdLVfs1NEJwI8Amj+TnOSc5JpEdyCJkFNWDNmDa1DW1f7\n9YVzSNImhBCitqhU0qaU6ge8Scl0yjla62nnHB8FTC7dzAUe0FpvLz32GDAOsAM7gHu01sWOCV8I\ncTWx2Cz0mNsDk9FEQmwCw1sOr5Hr+rr7svKulaw5vIYgzyDahbWrkesK5/Jy88JsNWOz285aMkII\nIYSoaRUmbUopAzADuBFIAzYrpZZqrfeUa3YI6Km1zi5N8GYBXZVS4cBDQDOtdbFSahEwAvjE0W9E\nCFH//X7sdyJ8IvjzgT9r/NpKqcuqUinqLoMy4O3mTV5xHn4efs4ORwghxFWsMk/Mdwb2a62TtNYW\nYCEwuHwDrfUGrXV26eYGIKLcYRfASyllBDwpSfyEEKLKfkv/jU4RnZwdhriKSDESIYQQtUFlkrYI\nILncdgpnJ2Xnug/4DkBrnQa8DhwFUoHTWuvVlxeqEOJqt+PEDtqGtnV2GOIqIs+1CSGEqA0cWpta\nKdULuIfS59uUUv6UjMrFAOGAd+nzb0IIUWVHs48S4xfj7DDEVUSSNiGEELVBZQqRpALly7NFlu47\ni1KqDSXPsvXTWp8q3X0TcEhrnVXaZgnQDfjsQheaMmVK2euEhAQSEhIqEZ4Q1ceu7axLXkfniM64\nubg5O5yrXkpOClF+Uc4OQ1xFJGkTQghRnRITE0lMTKywXWWSts1AvFIqBkinpJDIyPINlFLRwGJg\ntNb6YLlDRykpSOIBFFFSzGTzxS5UPmkTojbYmLKRHnN78M6Ad3ig0wPODueql5yTTKRvpLPDEFcR\nX3dfcotznR2GEEKIeurcgaqpU6desF2F0yO11jbgQWAVsBNYqLXerZS6Xyk1obTZP4FA4F2l1Dal\n1KbSvpuAL4FtwB+AomQ0Tog6oWtkV3655xdeX/86W9O2klecV3bsWN4x7lpyFyMXj7zEGYSj5Bbl\nkl+cT4hXiLNDEVcRH/cLFyKxa7sTohFCCHG1qtQ6bVrrFUDTc/bNLPd6PDD+In2nAhdOGYWo5ZRS\ndI/qzpCmQxi3bBxHs48yqvUofjzyI8fyjpFlzsLT1dPZYV4VNqVuomN4RwzKoY/iCnFJvm4l0yNP\nmU/h6uKKt5s3c36bwwPfPkCjgEa81Pslbmt+m7PDFEIIUc9VKmkT4mqmlOL1vq8D8MOhH/jxyI98\nNPgjOoZ3pNhWTNArQeQX5+Pl5uXkSM92+NRhTheeJsucRauQVoR6hzo7pCuyLnkd10Ve5+wwxFXG\n192X7MJsBnw2gD9PlKwPmFecx4z+M4j1j2X4l8NZ7b2a66Lke1MIIUT1kaRNiCq4sdGNZy2w7GH0\noHVIa7akbeGG2BucGFnJdC2FQikFQL/5/VAovNy8OJh1kLvb3U2n8E7sy9yHydVEgEcA9197v1Nj\nror1KeuZ0HFCxQ2FcKAovyjuX34/ncI7ceChAyRlJ9Fldhf6xPWhcVBjHunyCMv2LpOkTQghRLWS\neUZCXKHrIq9jfcp6p8awdM9SfP7tQ7N3mvFb+m9orckrzmP1mNVsnbCV/Q/tJ784n2lrp2HXdk6Z\nT/HwiocpshY5Ne7Ksms761PWy0ibqHETOk7g2KRjJN6dSKh3KJ0jOmP/PzuNgxoD0D2qO4t3L+bp\nH56WKpNCCCGqjYy0CXGFrou6jnnb55FblMu87fPIKMjgxyM/8uGgD2kY0PCi/RbsWEDbsLa0CG7B\nT0d+YkvaFmL8YwgyBRHqHUqL4BZntbfaraTkpBDrH1u2T2tNgaWAOdvm8MpNr9DAswHdP+xOobUQ\nAJPRBECwVzAfDPrgrPN9vutzknOSiQ+Md9CdqD6L/lyEr7tvnZ/iKeqmc7/vzoxmA/Rq2IuJ105k\nxqYZDGw8kO7R3Ws6PCGEEFcBSdqEuELdo7pz//L7GbJoCGaLmSZBTYj1j+WZNc/w2dCSJQlzinLI\nL86ngWcDXF1cmb5+Oo+vehx/D39cDa5kFGTQvEFzmgQ1Ibsom50ndnJXm7vwMHpwW7PbCPcJ544v\n72Br+la6R3XHx92HXRm7OJF/gkJrIWHeYcweNJsQrxBuanQTgxcOZm3yWjyMHheNO8YvhqPZR2t9\n0qa1ZtSSUXSJ6OLsUIQ4j6erJ49f9zg/HvmRTHOms8MRQghRT0nSJsQVivCNYMnwJew+uZtRrUfh\n7+FPRn4GTWY04ZrXr8HD6MEp8yncje64ubjROqQ1u0/uJunRJCJ8Ijiefxxfd1+83bzLzvnHsT94\ne+PbJBUl0Xl2ZzxdPXmq21OsvGslC/9ciL+HP9F+0UT7RRPiFXLWX/6DPIMINAUCYHI1XTTuGP8Y\nZmyaQe+Gvavv5lRAa82srbOI9I2kX3w/XAwuZfu3H9/Osbxj9JvfD4Cf7/nZaXEKUZEgUxBZ5ixn\nhyGEEKKekqRNCAfo1bAXvRr2KtsO9gom+bFkcotyKbQW4uvuS5BnEH8c+4O9mXvpFtWtbJHocJ/w\n887XNqwtcwbP4XjecTakbODAQwfKErBxHcZVGM+Z5OdS5fGf7fEsvT7uxQdbP2B8xwuu2FHtsouy\neei7h2gY0JCj2Ue5xucaTuSfYMWBFfyc9DPtwtqVtXVzcXNKjEJURqApkMwCGWkTQghRPSRpE6Ka\neLt5nzV6BiXJWNuwtpU+R6h3KKmPp1b52pVZy6xxUGMWD1/MwM8Gcl+H+84araspRdYi/D38eb7X\n84z9eizxgfF0jehKtF80yY8lY3I1seP4DjakbKjx2ISoChlpE0IIUZ0kaROiHqrsAtRdIrvg4+7D\nnpN7aB7cvJqjOl+xrRh3oztDmg3h21Hf0jOmZ9ko4RmtQ1vTOrR1jccmRFUEmgJJzkl2dhhCCCHq\nKSn5L0Q9pKj8qFmP6B78cvSXaozm4opsRbi5uOHm4kavhr3OS9iEqCsaeDbgRP4JZ4chhBCiXdNv\nUwAAIABJREFUnpKkTYh6qLIjbeDkpM1ahLuLu1OuLYQjRflFkZKT4uwwhBBC1FOV+s1OKdVPKbVH\nKbVPKTX5AsdHKaX+KP3vV6VUm3LH/JRSXyildiuldiqlpG63ENWsKs+n9YjpwS9Jzknaim3FUmBE\n1AtRvlEyPVIIIUS1qTBpU0oZgBlAX6AlMFIp1eycZoeAnlrrtsALwKxyx94CvtVaNwfaArsdEbgQ\n4uKqMtLWNKgpBZYCDp86TE5RDo989wiuz7uyOXVzNUZYoshWhLtRRtpE3RfmHUaWOYsia5GzQxFC\nCFEPVeY3u87Afq11ktbaAiwEBpdvoLXeoLXOLt3cAEQAKKV8gR5a67ml7axa6xyHRS+EuKCqJG1K\nKUa1HkWjtxvR6K1GFFgKGNp8aI1MmSy2Fcv0SFEvuBhcCPMOIzW36tVehRBCiIpUpnpkBFB+zkcK\nJYncxdwHfFf6uiFwUik1l5JRti3AI1pr82XEKoSoJBdVtYIeL/R+geEthxPiFUJ8YDzvbHqH9Snr\nSclJwa7thHqFVsuIWJG1SKZHinojyjeK5OxkGgU0cnYoQggh6hmHFiJRSvUC7gHOPPdmBDoA72it\nOwAFwN8deU0hxPleufkV1oxZU+n23m7edIvqRnxgPAAtglswf8d8us7uSvuZ7fn76ur52Mr0SFGf\nRPtFy3NtQgghqkVlRtpSgehy25Gl+85SWnxkFtBPa32qdHcKkKy13lK6/SX/S+jOM2XKlLLXCQkJ\nJCQkVCI8IcS5QrxCCGkYctn9E2ITyHgygwaeDdicupmhnw+lR0wPdp7Yyb3t7yXCN8Ihccr0SFGf\nnBlpW3N4DcnZyYxtN9bZIQkhhKjlEhMTSUxMrLCd0lpfuoFSLsBe4EYgHdgEjNRa7y7XJhr4ARit\ntd5wTv+fgPFa631KqecAT631hSpQ6opiEULUPJvdxtivx3Kq8BRF1iIaBzbmvVvec8i5F+xYwNK9\nS1n4l4UOOZ8QzjR321xeW/8aKTkp5BTlUPxsMa4urs4OSwghRB2ilEJrfV4Z8AqnR2qtbcCDwCpg\nJ7BQa71bKXW/UmpCabN/AoHAu0qpbUqpTeVO8TAwXyn1OyXPtb10he9FCFGDXAwuzLt9Ht+M+obP\nhn7Gwp0LSTyS6JBzy/RIUZ8MbzmcYM9g1oxZQ4vgFqxLXufskIQQQtQTFY601RQZaROibli6Zynj\n/zueWbfOYnDTwVVaE+5cs7bOYnPqZj4Y9IEDIxTC+Z778TlWH17N2nvXOjsUIYQQdchlj7QJIUR5\ng5sNZvmo5Qz/YvgVFygpsspIm6if7m53N6k5Uv5fCCGEY0jSJoSoss4RnTny6BFmbp1Jem46ZkvJ\nKh7fH/yeAksBFpuFPSf3MPn7yby69lUACiwFzPltDkdOHyk7jxQiEfVVA88GZJoznR2GEEKIeqIy\n1SOFEOI84T7hNAxoSPgb4XgYPVgyfAmDFg6icWBjjucfx8PoQf/4/izauYg2oW0Y8/UY3Fzc+PHI\njzQJakLfuL4U2WSdNlE/ebt5U2wrptBaiIfRw9nhCCGEqOPkmTYhxGVLz03H3ejOu5vfZd72eYR4\nhTCh4wRCvELoE9cHrTUhr4XganDllZtfIdgzmP7z+3Nv+3tZtncZcYFx9I3ry5SEKc5+K0I4XPjr\n4Wwev9lhS2QIIYSo/y72TJuMtAkhLts1PtcAMKr1KNJz0xnTdgxdIruUHVdKMbn7ZN7b8h63NbsN\nk6uJ7X/dTquQVry98W3+vvrvzB0811nhC1GtGng24GTBSUnahBBCXDEZaRNCCCGqwcjFI4n0ieTV\nPq86OxQhhBB1hFSPFEIIIWrQjP4zWLJnCa+ufRX5o6QQQogrISNtQgghRDXZl7mP9jPb8/v9v9M4\nqLGzwxFCCFHLyUibEEIIUcOaBDXhhpgb2HNyj7NDEUIIUYdJIRIhhBCiGjVr0Ixle5cB4OPuQ5Ap\niKYNmp633EWhtZDMgkwpXCKEEOI8Mj1SCCGEqEZ/nviTKYlTMFvNfLv/WwA6hXfipkY3UWgtZPWh\n1YxqPYple5exPmU9f+v0N74/9D29Y3vz3i3vOTl6IYQQNeli0yMrlbQppfoBb1IynXKO1nraOcdH\nAZNLN3OBv2qtd5Q7bgC2ACla60EXuYYkbUIIIeq13Rm7CfYKZn3yerambyU9N50hzYYwb8c8QjxD\n6BLZheN5x4kPjGfYF8PI+UcORoNMihFCiOqyP3M/yTnJ9G7Y29mhAFeQtJUmXPuAG4E0YDMwQmu9\np1ybrsBurXV2aYI3RWvdtdzxx4COgK8kbUIIIUTF4t6OY2zbsVhsFh677jECTYHODkkIIeoNm93G\nwj8XctdXdwGgn6sdeciVFCLpDOzXWidprS3AQmBw+QZa6w1a6+zSzQ1A2YR8pVQkMACYfbnBCyGE\nEFebXrG9WJu8lqTsJG785EZO5J+osE9OUQ6F1sIaiE4IIequQ6cO0WNuD15b/xp+7n4kxCY4O6QK\nVSZpiwCSy22nUC4pu4D7gO/KbU8HngRqR/oqhBBC1AGzB81m5V0rmTt4Ls0aNKP9zPZsSdty0fZZ\n5iwa/6cxgdMCuf7D68krzqvBaIUQou54dMWj9IrtxdYJW1kwdAHuLu7ODqlCDp0or5TqBdwDXF+6\nPRA4rrX+XSmVAJw31FfelClTyl4nJCSQkJDgyPCEEEKIOsfF4MKCoQtYsnsJAz8byLFJx1Dq/B+n\nG1M20iqkFctHLmfE4hHctug2vh31La4urk6IWgghaqe84jwSjyTy0ZCPMCgDnq6eFFgKnBZPYmIi\niYmJFbarTNKWCkSX244s3XcWpVQbYBbQT2t9qnR3d2CQUmoAYAJ8lFKfaK3HXOhC5ZM2IYQQQvzP\nbc1u496l95JpzqSBZ4PzjhdaC/F198XkauKTIZ9wy4JbCH41mCJbEX7ufmy4bwOx/rEUWYvYdmwb\nXSK6lCV/b6x/gzDvMEa1HlXTb0sIIardmboZFruFa2ddS7/4fmXPCTs7aTt3oGrq1KkXbFeZpG0z\nEK+UigHSgRHAyPINlFLRwGJgtNb64Jn9WuungadL29wATLpYwiaEEEKIi1NKER8Yz4GsAxdN2jyM\nHgD4efixZswaTheexsvNiydWPcG9S++lZ0xPvj/0PXtP7iXKL4qHOz9MrH8sk1ZNAiAlJ4Unuz2J\nRmPXdqlcKYSoVmm5afi5+5FlzuKHwz/w0i8v8UiXR9iZsROtNVa7FYMy0DasLXe1uQtvN28M6sJP\nd9m1nSJrESZXE/nF+fyU9BMzt84EYFv6NmzaRnxgPEaDkbmD55b1c3bSVlkV/mustbYppR4EVvG/\nkv+7lVL3lxzWs4B/AoHAu6rkz3YWrXXn6gxcCCGEuNq0DGnJA988wIu9XySjIIORrUaWTX80W81l\nSRuAq4srwV7BADzf63kW/LmAh757iPjAeI49cYxle5cx/r/jsdgsrLhzBS2CW9Dpg05MWzuNLHMW\n97S7hw8Hf+iU9ymEqPtyinJ4Y/0bdInoQp+4PrgYXICSpU/+8sVfaB3SmkU7F+GiXAjzDsPLzYs7\nW9/JupR1XHvNtbi5uOHq4opd25m3fR6TV0+mwFJApG8kb/R5gwaeDVi2dxn7svaRW5TLgawDZJmz\naBPaho2pGwGYectMQr1CuabHNbgaXDly+gidIzrj5eZVFmddSdpkcW0hhBCijjhlPsVbG99i7u9z\nyS3K5ZtR33Bd1HUAvLv5XXYc33HJBbmTTifhYfQg1DsUgNOFpymwFBDuEw5ARn4Gdm1n0c5FrDiw\ngm/v/Lb635QQol76797/8ujKRwnwCOBU4SmGtRjGzY1uZspPU9h7ci+v3PwKNze6mTDvsLKEriI2\nu40VB1YwfcN0im3FtAltQ5+4Pvi4+RDmHUZabho2baNpUFOO5R2jS2SXCs95PO84bd5vw/Enjl/p\nW3aIK1pcuyZI0iaEEEJU3j1L76FbZDfGdxwPlDyXlpydzPR+06/43DuO72DE4hHsfGDnFZ9LCIC3\nN77Ngj8XcHuz23my+5PODqdO252xm1OFp+gW1c3ZoVzS8z89T74ln5dufIkNKRt4c8ObZBRkYDKa\neL7X83QM7+jsEAHILcol/I1wcv+R6+xQgIsnbTJZXQghhKiD2oW247X1rxHmHcatTW+l0FqIydXk\nkHPH+seyK2MXuzJ20SK4hUPOKa5u249vJ8QrhIU7F0rSdoXe2fwOX+35in0P7jtrmp+zHcg6wJe7\nvsRmt5FRkMGXu75k5i0zMSgD3aK61dok0+RqosBSgNb6gpV5awtJ2oQQQog6aOK1E/Hz8GPUklGc\neOLEWYVIrpSPuw/P3fAcN31yEz/d/RONgxpXqb/VbuVY3jEifSMdEo+o+2zaRnxAPAezDlbcWFzS\nxtSN+Ln78fr61/m/G/6PZXuXkVmQSeOgxjQObEyIVwg2bSOvOI8scxZvbXiLk+aTzLtt3nlJic1u\nK5uauGzvMj7f+TkRPhEMbTGUTuGdOF14mrXJazEajFwbfi0NPBvw7uZ3+S39N6BkkWo/Dz88jB78\ncOgHhrUYhq+7L9F+0bzR9w0GNhlY4/enqowGI0aDkWJbMe7G2rtemyRtQgghRB3kbnTn7nZ3859N\n/+H3Y79jtpjLCo84wpSEKUT6RnLrglvZ+cBOXAwuaK3RaBSKub/PJSM/gwGNBxDmHYZBGfB28ybT\nnMn9y+9n+b7lrBmzho7hHfF193VYXKJustlteLp6UmgtdHYodVqhtZCdJ3aydcJW2s1sx9K9S0k6\nncSAxgOY9dss9mXuo1FAI4qsRew4sQN/D38e7PQgX+/9mj9P/EnjoMa8sf4N2oe159Ptn7LgzwUM\naDwAo8HIsr3LGNt2LK4urty55E4OnTqEyWiieXBzThacxKAMWO1WjmYf5f2B72NQBm5rdhuF1kKK\nbEU80uURukZ2dfYtuixnipFI0iaEEEKIatE5vDOb0zY7dKTtjPs63MeH2z7kho9uIN+Sz6FTh7g2\n/FoCPAI4dOoQ3aK6MWTREHKKcrDZbeQW5xJoCmRA4wG8P/B97vjyDopsRcy8ZSYjWo1waGyibrFp\nG15uXhTZipwdSp20/fh2xv93PLlFuQSYAmge3JzpfaeTV5zH/R3vx8/DDygpe79412K2pG1h0nWT\n6NWwF9F+0eQV5zH086Gk5KRgtpqJ9ovm4c4P88rNr/D5zs+J9I1k9q2zy/7wMyVhCkmnk4j2iy6r\n4Pjlri+J8o0irziPm+NudubtcLgzSVuAKcDZoVyUFCIRQggh6rAPt33ImsNrMBlNdIroxISOExx6\n/uzCbH44/AMRPhG4ubgxf8d8/Nz9eKLbE+c9Q3ehZ0I2pmzk9s9vJ/XxVIfGJeqWEV+OoFtUN57/\n+Xkynsxwdjh1ztivxxLsGUyARwBh3mGM6zCuSv2LbcWsObyGrpFd8ffwr6Yo6674t+P57s7vqjwV\nvDpIIRIhhBCiHuoc0Zlpa6fROaIzJqNjCpGU5+fhx+3Nby/bbn9N+4u2vdBD/B3DO5KRn4HFZilb\nU05cfWzahperl0yPvEz7MvcxvsN4ro++/rL6u7m40S++n4Ojqj/qwlptkrQJIYQQdVjzBs1Jy03j\nWN4xh0+PdASjwUiIVwjpeekczDrIN/u/YVfGLgJNgSTEJnBfh/ucHaKoATZ76fRIq0yPvBwpOSlE\n+UY5O4x6qy4kbQZnByCEEEKIy+dicKF9WHvWHF5TK5M2gFDvUIZ/MZzRX43GoAyM7zCe9mHteWXt\nK84OTdQQq92KyWjCardi13Znh1OnWO1WjucdJ9wn3Nmh1Ft1IWmTkTYhhBCijnug0wP8cvQXQrxC\nnB3KBd3a5FY2pGzgm1HfEOQZBJSMHEzfcOULgYu6waZLSsu7G90pshY5bE3Bq8Hr617H3egu04ur\nUb1J2pRS/YA3KRmZm6O1nnbO8VHA5NLNXOCvWusdSqlI4BMgFLADH2it33ZU8EIIIYSAEa1GMKzF\nsLL1lmqbKQlTztvn6epJviW/5oMRTmGz23BRLri7uDt0Ifj6zmKzsHDnQhb9ZZGzQ6nXPF09MVvN\nzg7jkiqcHqmUMgAzgL5AS2CkUqrZOc0OAT211m2BF4APSvdbgce11i2B64C/XaCvEEIIIa5QbU3Y\nLsbL1Yv8YknarhZnRto8jB5S9r+SPtz2IW4vuHE0+yg3NbrJ2eHUa3VhpK0yz7R1BvZrrZO01hZg\nITC4fAOt9QatdXbp5gYgonT/Ma3176Wv84DdZ44JIYQQ4url5uKGXdux2CzODkXUgLKRNqO7VJCs\npANZB3i+1/OceOIEbi5uzg6nXqsvSVsEkFxuO4VLJ173Ad+du1MpFQu0AzZWPjwhhBBC1EdKqTrx\ni5JwjLNG2qSCZKWk56Vzjfc1dW4UvS6qC/8WObQQiVKqF3APcP05+72BL4FHSkfchBBCCHGV83Lz\nIt+Sj5+Hn7NDEdXMZrdhNBjLnmkDyMjPYGv6VpoGNcXX3Rcfdx8ZUSrnWN4xwrzDnB3GVaG+JG2p\nQHS57cjSfWdRSrUBZgH9tNanyu03UpKwfaq1XnqpC02ZMqXsdUJCAgkJCZUITwghhBB1UV34RUk4\nhtVuxUWVjBh9vedrFu9ezH/3/Re7tnPKfIq84jxyi3P57s7vrtrnt2ZsmsHD3z3M9dHXk5SdxNHs\no7zY+0Vnh3VV8HT15HThaadcOzExkcTExArbVSZp2wzEK6VigHRgBDCyfAOlVDSwGBittT54Tv8P\ngV1a67cqulD5pE0IIYQQ9Zsji5FsSNmAXdvpFtXNIee7HGcSE6WU02Korc5Mj4zwjWDOtjmMbjOa\nES1H8ES3J8qm/y3etZh//vhPesX2Iq84DzcXt1pXZXJr2lY6hnd0+Hl3HN/BM2ueYfHwxQR5BuHm\n4sa7m9+lcWBjh19LnM9kNJFmSXPKtc8dqJo6deoF21WYtGmtbUqpB4FV/K/k/26l1P0lh/Us4J9A\nIPCuKvmXyqK17qyU6g7cCexQSm0DNPC01nrFFb07IYQQQtR5Z6ZHlpeem84Hv33AfR3uu+BiwtvS\nt+Hv4U/DgIbkFuVitVvZm7mXPp/2wWgwcm/7e8kpyqFJUBMmXjsRbzfvS8ZQYClgz8k95Bfnk5qb\nSrRfNI0CGlU4Lc1sMZOck0yToCYAJJ1OosmMJvSN68uSO5ZgNMhSuOWdKUTy3Z3nlT0oM6jpIF5b\n/xr+0/xxUS54u3nTMqQlg5oM4m+d/1aD0V7YhpQNXDfnOp7v9Txrk9cS7RvNr8m/cujUIVyUC8Fe\nwSwfuZyWIS2rdN6vdn/FhOUTeKzrYwxpNqQs6e8a2bU63oa4gLow6q+01s6OAQCllK4tsQghhBCi\n+qmpJb+czr51Nve2v5ddGbsY8NkAGng2wNXgyiNdHqHYVkyL4BbE+Mfww6EfGPP1GExGE95u3uQW\n55JXnEeYdxhv93ubuMA4lu1dhkLx/aHvSctNw8XgQqx/LLc3u53Woa3JKcpha9pWMgoyyCvOI/FI\nIt5u3uw4sYM+cX3ILsxmb+Ze1o9bT7MGZ69StPPETo7nH2f78e28/OvLZJmz+GjIR4xsNZKXf32Z\nw6cPc/j0YVoGt2R63+ky4lZO2/fb8vGQj2kX1q7CtqcLT5NlziKvOI/1yet5e9Pb7HxgZw1EeWkD\nPxuI2WLmaPZRnunxDCfyT9A1sisdrukAwOzfZvPO5ndo2qApcQFx3NbsNnKKcsgtzuXO1nee9f1w\n+NRh1iavZdraaZwyn+KrO76iU0QnZ721q9787fP5Zv83fDb0M2eHglIKrfV5/3hI0iaEEEIIp1h1\ncBVmi5kpP02hwFKAt5s3Y9qMYeK1EwmYFkCvhr0INAWyPnk9JwtOcn309UzoOIHuUd0ptBbi4+5D\nfnFJIRNPV8+zzl1sK2Zr2lYCTYFsTN1I4pFEdmbsxGq30i+uHyFeIRgNRmL9YxnYZGDJSFDpNL3H\nVz6O2WJm27FtuBvd6R3bmzVH1rA/cz/uRneOnD7C9onbsdqtdPqgExqNXdv55Z5faBncku4fdufe\n9vfyRLcnnHFba6VW77ZiwdAFtA5tXaV+VruVBq80KFuM3dfdl1m3zKJ/4/7VFOmF/Zb+G4MXDubA\nQwdwN7pfsI3NbmPVwVXYtI2fk35m9aHVhHiFkJSdxKTrJnFfh/vILMhk7u9z+fev/6ZHdA/GtR9H\nv/h+uLq41uj7EWf7avdXfPzHx3w94mtnhyJJmxBCCCFqJ6vdyoPfPsjMrTPJ+XsOPu4+aK2dNlL1\n3ub3eOGXF2jo35BJ100i8UgiCbEJ3NLkFlwMLmSZs2jg2QAAu7Zj13ZsdlvZL/OLdy3m4z8+ZtnI\nZU6JvzZq/k5zFg9fTIvgFlXum3Q6CRdDyXTJdcnrGPPVGGzahr+HP5G+kRRYCoj1jyUuII4scxY7\nTuzg3nb3MqbtGE4WnMTPww9fd98riv+N9W9w5PQR3u7/dpX7/pb+GwkfJZRVzYz1j2Xx8MVVTmBF\n9Vl5YCWvr3+dVaNXOTuUiyZtMuFaCCGEEE5lNBh5b+B7vNXvrbLEx5lTC2P9Y0nLTeOWxrdwW/Pb\nuK35bWcdP5OwARiUAYMynPUMW7uwdjy84mGnJp61zZmS/5cjxj+m7PWAxgPIeDKDLHMW2UXZHMg6\ngJuLG8fzjrMueR2tQlrRJaILE5ZP4Ok1T2Mymsguysbfwx9PV09MRhNj2o5hRKsR+Ln7seLACnaf\n3E2WOYvOEZ0Z0mwIHkaP82I4kHWA5g2aX1b8Ha7pwPEnjqOUws3FDYOqzDLJoiZ5unpitppr5Fp5\nxXn8d+9/2ZCygfS8dNyN7uzP3E92UfYlpw9L0iaEEEIIp1NKXXTaWU1rFNAIgEjfyMvuH+oVypLd\nSxjaYqgjQ6uzbNpWVvL/SimlCPIMIsgzqOxrBXBHqzuAktFPjaZ3w97EB8aTX5xPpjmTAksBWeYs\nHvjmAZ5Z8wx+7n408GzArU1upYFnA1785UWm/jSVuIA4/Dz8uL/j/fSM6QmUJG23Nrn1smOubVUw\nxdlqshDJYyse4/tD3/NApwe4NvxarHYrEztOxNfdl81pm1nIwgv2k6RNCCGEEKKcMxUhL3dERCnF\ni71fZOzXY1m4cyF/nviTJcOX0Dz48kZq6gOr3Vr2zGB1MygDEzpOKNv2cvPCy82rbPv3ib9TbCvm\n56SfuSHmhrLnyZ7q/hQbUjZwynyKL3Z9wdI9S+kZ05NiWzG/pf8m0xnrsZpM2v67779sGr+JaL/o\n8461Dm3NOMZdsJ8kbUIIIYQQ5SilOPzIYYI9gy/7HP0b9+f70d+z5+QeUnNS6f1Jb4Y2H0r/+P4M\nbDLQgdHWDWdK/tcWbi5u5y3ibTQYuT76egBOFpzkp6SfAJi3fR4tgltc9sirqP1qKmnLLMjEbDUT\n5RtV5b6StAkhhBBCnCPWP/aKz9E2rC1tw9oytMVQ1iWvY3PqZsYtG8cbfd+gX3w/krOTKbYVl5V6\nP3zqMOE+4bi5uNW7Z+HOLK5dVwSaAsk0Z7IlbQuTV0/mhzE/ODskUY2qO2mzaztvb3ybxbsX0zK4\n5WV9viVpE0IIIYSoRkaDkZ4xPekZ05N2Ye2Y+tNUJi6fCECIVwizB81m9aHVvPjLi4R4hVBgKWDz\n+M3nrRNXl9W2kbaKBHkGsXzfcjakbGD2rbNpE9rG2SGJauTp6kl2YTZb07bSMqQlFpuFE/kniAuM\nA+CXpF/Yl7mPm+NuJi03jSZBTQg0BV7ynFprRi0ZhdFgLFsbcs6gOZdVQRWk5L8QQgghRI0rthVT\nZC3i0RWPsj9rPzH+MdzZ+k4CPAJYtHMRaw6vYd24deetP1dXBb0SxN4H955VebM2252xmxbvtmDB\n0AWMaDXC2eGIambXdlz+5YKH0QOr3YrVbgWgW1Q3krOTybfkE+IVQm5RLn4efmit6RXbix8O/4DR\nYKRrZFd83X15stuThHiF8HPSz2xN38p7W95jcvfJhHiFMLjp4EqNsMk6bUIIIYQQdUCxrZh+8/ph\ntpqJ8YuhX3w/7m53t7PDuiL+L/tz+JHDBJgCnB1KpZzIP0Hoa6GkPp5KuE+4s8MRNaDYVoybixs2\nu43U3FTe3vg2IV4h3NHyDoK9gsv+gGK1W3ly1ZMEmgIZ1HQQRbYitqRt4UDWAeb+PhdXgytRflE0\nCWrCxI4TuSH2hirFIUmbEEIIIUQdkZGfwQ+HfyCnKIdJqyaR/FgyWmt83X3r1LNhZ/j824e0x9Pw\ncfdxdiiVYtd2Xlv3Gk91f8rZoYg6JC03DavdesHKkJV1RUmbUqof8CZgAOZoraedc3wUMLl0Mxd4\nQGu9vTJ9y51DkjYhhBBCiHP0nNuTTambcDe6E+UbxSs3v8JNjW7CzcUNrTVb07eyP3M/I1qNqLUF\nTEwvmsh8KrPeTPcUorpcLGmrsBCJUsoAzABuBNKAzUqppVrrPeWaHQJ6aq2zS5O0WUDXSvYVQggh\nhBAXkXh3IoqS3+H6zOvDwM8G0ja0LVMTprLy4EpWHFiBUgqlVK19/qquFSIRorapTPXIzsB+rXUS\ngFJqITAYKEu8tNYbyrXfAERUtq8QQgghhLi48ot8f3fndxiUgTm/zeGlX1/Cy9WLDfdtYNneZYxc\nPBKT0USQZxAxfjFE+VV9LajqUtdK/gtR21QmaYsAksttp1CSjF3MfcB3l9lXCCGEEEJchNFQ8qvb\n+I7jGd9xfNn+3g17A/DE90/g5uJGrH8s34z6xikxnktrXVKdT0bahLhsDl2nTSnVC7gHuN6R5xVC\nCCGEEBfXKKARln9aMBqMHMw6SJfZXVhzeA1WuxVXgyu9GvYCIOl0Eha7hfjA+BqLza7tKFStfd5O\niLqgMklbKlC+BEpk6b6zKKXaUPIsWz+t9amq9D1jypQpZa8TEhJISEioRHhCCCGEEOIMzqcgAAAV\nNElEQVTMKFzDgIZkmjP527d/I9wnnJ+TfqZrZFcKrYVsSdtC65DWbP/r9rP6Jmcnsy55HcNaDsOg\nDGSZs8jIzyCnKIe4wLgKFxK+FJkaKcTFJSYmkpiYWGG7CqtHKqVcgL2UFBNJBzYBI7XWu8u1iQZ+\nAEaXf76tMn3LtZXqkUIIIYQQDrApdRMdrumA0WAkOTuZQ6cO4enqia+7Lzd8dAOBpkBOFZ4i0jeS\nTuGd2JS6iX2Z+yiwFKCUwsPoQZApCJu2MbjpYGYMmHHZsZgtZgKmBVD4bKED36EQ9dNlV4/UWtuU\nUg8Cq/hf2f7dSqn7Sw7rWcA/gUDgXVUy9m3RWne+WF8Hvi8hhBBCCHGOzhH/KyEQ5Rd1VlGS7X/d\nTnpuOoGmQI6cPsLW9K2EeYcxufvksqInLgYXDMrAtvRt3LrgVm5seCO3NLkFo8GIUgq7tvPHsT84\nXXi6bOrlGWaLma/3fE18YDydIjphsVvQyB/mhbgSsri2EEIIIYS4IK01M7fO5P0t77PjxA5cDa60\nCmlFljkLu7aTlJ3ErFtmkVecx5I9S9ifuZ8CSwEdrunAzoydTOw4kfUp67FpGz+O/dHZb0eIWu+K\nFteuCZK0CSGEEELUXhabBbcX3HjlplfoHt2d6yKvY/6O+aw+tBqNpn98f7pFdcPL1YtAUyBL9y5l\n+/HteBg9eKTLI7gb3Z39FoSo9SRpE0IIIYQQV8Su7WetGyeEcKyLJW3yqRNCCCGEEJUiCZsQziGf\nPCGEEEIIIYSoxSRpE0IIIYQQQohaTJI2IYQQQgghhKjFJGkTQgghhBBCiFpMkjYhhBBCCCGEqMUk\naRNCCCGEEEKIWkySNiGEEEIIIYSoxSRpE0IIIYQQQoharFJJm1Kqn1Jqj1Jqn1Jq8gWON1VKrVNK\nFSqlHj/n2GNKqT+VUtuVUvOVUm6OCl6cLTEx0dkh1DtyTx1H7qXjyT11HLmXjif31HHkXjqW3E/H\nkvtZMypM2pRSBmAG0BdoCYxUSjU7p1km8BDw6jl9w0v3d9BatwGMwAgHxC0uQD40jif31HHkXjqe\n3FPHkXvpeHJPHUfupWPJ/XQsuZ81ozIjbZ2B/VrrJK21BVgIDC7fQGt9Umu9FbBeoL8L4KWUMgKe\nQNoVxixqkHwQhah95HMpRO0jn0shap/69LmsTNIWASSX204p3VchrXUa8DpwFEgFTmutV1c1SOE8\n9embXYj6Qj6XQtQ+8rkUovapT59LpbW+dAOlhgJ9tdYTSrfvAjprrR++QNvngFyt9Rul2/7AYmAY\nkA18CXyhtf7sAn0vHYgQQgghhBBC1HNaa3XuPmMl+qUC0eW2I0v3VcZNwCGtdRaAUmoJ0A04L2m7\nUHBCCCGEEEIIcbWrzPTIzUC8UiqmtPLjCGDZJdqXT76OAl2VUh5KKQXcCOy+7GiFEEIIIYQQ4ipT\n4Uib1tqmlHoQWEVJkjdHa71bKXV/yWE9SykVCmwBfAC7UuoRoIXWepNS6ktgG2Ap/f+s6nozQggh\nhBBCCFHfVPhMmxBCCCGEEEII56nU4tpC1Eel6wgKUSsppborpa4pXS6F0inmQtQKSqleSqk4pZRr\n6bZ8f4paQyl1m1Kqd+law+IKnfk5JJxLvgh1hFLKG5gKHANWaq23OzmkOksp1RiYBxxXSv2f1vp3\npZTSMux8WZRSPkAfYLnWusjZ8dR1SqkWwMtAOLAHyAX+6tSg6rDSfzufAjKBn7TWvzs5pDpNKdUS\neBEIA/ZRshbrnU4Nqg4r/f78J5AD/Kq1/snJIdV5pYnaDGAlkI7UUrhspX+MmQq0AP7i5HCuevIX\niDpAKRUMfA+4U/I1e1kpdUvpMfkaVoFSyoWSgjiHgB1AD6WUmyRsl0cp1YOSX9w+Bbo4OZw6TykV\nAjwErNFaXwtMAvoqpVrL92jVKaX+Qsnz1r6UJBnPKqU6lx6TkaEqUko1AO4FftRadwUeA5qq/2/v\nzoMlqwo7jn9/bwaYiSKyDIomCspAxmVCkBEKRxFFCCqCIFUhhiVxiUFciJbRQkSMLMqIUUTRxARZ\nQqBATKiiZHGDMLIIA4ZEQcVIKhkEEYHMoDLML3+c08Pl8WZ5r2+/7jf396nqmn733u66fed3b/dZ\n7jnS0207x3Ryaj6/C8wFfgV8UNIuw92rjcI8SgX3Y8CiWrGYc35q5gC7AXtJynf8kKWlbWaYA9xu\n+xgAST8GzqC0bKwe6p7NMHVgnUuAL1JqjRbXxzeHumMz12zgEOBlwJsl3d6b4iOm5BfAebavq3/f\nSxkE6inD26UZbXvgaNvfrPOGfgB4PnBjCsFTcj9wou2H6t8fAv4LeBGllSjHdHLmAG+2/X1JWwAL\nKRWKMUnjess8RqlIXAG8ElhKmUM4+ZwESWO2H5H0DUqvj9OAVwx5tzotrTQjqE6v0Jwbbxtgp16f\nYtuXAD+V9LG6ff4f10LSayXdKWmPxuL768X7Ssqk73vWEVBzLNdjgmzeaHspcCbwPGCfHMMNNz6f\ntRLm+sYmmwEvBx4Yxv7NNBPk82zg+vqD7lfAfMoPutgAE+TTth+StImko4AFwL8AJ0n6y/qanP9r\nMUE+z68Ftu2A84HXAn8hab+6fY7lOjTzOa6l91nAvrb/jlLRcJykj0naanh7O/qa+ZQ0y/bqesxe\nSamgkaSDJG02zP3sslwQRoiKE4E7gH/sLbe9jDL/3fsbmx8DHCLpaWltm5ikRcBRlNaK43rL64VI\nth8ErgGeASzqrZ7u/ZwJ1pHNFfXflcBXgMMorRuxHuvIZ7NQ8RzgHtt3TO/ezSzryOcvajZ7P+Z+\nTTnesR5ryyeA7UeBr9l+ne1zKfdkfbSuy/fROOvIZ+/7ZgFwEeV76C7gtNrlNMdyLSbKZ+N4/hy4\nuhaGX0bpVfNIeoFMbKJ81l5JY5Ruu3fW+9U/RakIu13S1sPa3y5LoW20PJUy193ewG8kHd5Y9x7g\n/bULBZQL+w2UGqWoJI1JmlP/vAs4wfZi4DmSDqvbrOnXbvtq4FbgFZIuA/56uvd5hpgwm/UeQQBs\nnwesotTKkXsznmxD8jnOlsBNtWXjs5LeOm07O7OsM5+NGuMFlK5SSNp5SPs6siZz/awtlz0/Aq5S\nGVQjnmxt+dwEwPY3bZ9j+x7gUsqP5xcPa2dH1SSun8+lVCJcC3wN+DQwd1wrZzxuwnzWSoNtgIWS\n3gV8HLgPuNz2/c3fUjFNbOcxxAfwUmBH4Kn17+3qv4cANwGzGtueBZwDvBDYB/h273V5GErBdinw\nZWD+uHUHUwpncxrLZtV/P0fpJnkesPWwP8eoPDY0m5RWjN7z+ZS+7z8ELgQ2G/bnGJXHZPNZl59E\n+XFyDeU+zC2H/TlG5TGJfI7VfxcB/0QZBe1KYAmwybA/x6g8pnD93JRyT9ZRwM3AccP+DKP0mMz1\nc9zr9gMuB7YY9mcYpceG5rN3PIEjgXn1+Xzg7Tnfn3DMJvPb86uUAXMWAFvU30s7DPszdPGRybWH\npNYWLQFeD3wD2Mb2gY31syh93H9k+/i6bC5l6O/FlJPt47Yvmu59H0WSdgNOAd5BGX56R+CfbV/e\n2ObrwPW2P9pYth3wD5QauxvrsjF3uFvKVLJZl8+trzsEONb2BdO64yOsj3yeBTwb+LDt2+qyTk9P\nMdl89o6XpEMpFQlLgTOTz8dNJZ8q91i/j9L97KO2b5nu/R5FfVw/dwfeDewEnGz70mnd8RE21etn\nXb6JS3feYMq/PefZvq+xzS7O1ClDke6Rw/O7wALb29t+C7CFpGN7Tf8u97WcDhwkadP6mtm2T6eM\nhraw6wW2cU3zO1Jq1H8C/A2l1u3lkhY0tnk3cKikPSWdLGmB7eW297d9Y+160ekCWzXpbNZuu7Mp\ncwg+s/eDuNl9smv6zOeptULhRNsH2L6t3ncw1uUCWzWpfNYC2xjlftWTbS9u5LOz34F95vMUStf8\nL9h+g+1bevmctg8wuqZy/dyE0nK5zPaiFNj6zudJve7PzQJbuvMBU/vtuQqg93cKbMOTC+w0UpnU\nucfAvY1lHwBeQ+n62KtNvxG4BFgm6TpKczaUm2y7/oP4Q8Dpkg6oi24C7pa0sP6ovYJSiFgzr4jt\nOyl9t68CVtn+QeP9xmyv7mqBrYVsvsT2w7b/tW7TG+m0kyP1tZDPR2uFwvL6frNcJJ+Tz+d3gVfY\nvtj2h+s2a+51m67PMEpayufdrkP/9yoTOnw8+8nnUmCx7WttL6nbdPa7HVrJ52OeYPCmrlZ4tfD9\n/ocAtn87fXsdE0mhbRpI2k3SlcCXJX1SZXLXh+vqrRonyQ8pTf/U2uEXUJqwVwDH2/5Gb139t3M/\niCUtkrSMUvP2A+CdKkNP30cZRWoxgO3/AJbX7ZC0hcoUCTcDO9r+SPN9O/xjo61sPmGeO9urpu9T\njI4W83l88327eK5Da/k8zva3G++pDh/PXD9b1FI+P2z7W433TD5bzmdXDer7PYYnhbYBk7QXZQCB\nL1Null0JHGT7XuBnwBuBbevmfwu8UY8PpboI+Lztl7pMDpum/dIl51Tbb7H9JcrALLvWGt9/B3ZU\nneMG+Bbw6loL/CDwGdsH214uaVbXj2eyORDJZ0sGlc+u1rZXyWdLks+BSD5bku/3jdPsYe9AB9xM\nuUn7MgBJPwAOreu+QOk7vL+k823fLekaamHa9ld6byJpdldbL+AJgy9cTen73/Ms4Df1+VWU0aM+\nIWkl8FbKqHuzgNW276/vNdbVmsxxks2WJJ8DkXy2JPkciOSzJcnnQCSfG6EU2lomaa7tR+pz2f4/\nSZc3NvkfYLWkObZ/JumLwAGUWo7nAd8DnjQBZBdPmsaFvNkldAWworkOuL+u+znw95JMaepfSel6\n8oSRozrclSfZbFHy2a7ks13JZ7uSz3Yln+1KPrshQ/63SOXm2dnAJ11mj2+u6w07/QHKfBjHNtZt\nArwJWO7GvRddVZvi1bz4ai2jOkq6gjLC3lJJu7oOO92sHVrba7sk2WxP8tm+5LM9yWf7ks/2JJ/t\nSz67I/e0tUB1pDzg34C9gN8fv02j1uiZwKWSZqsMs7qr7UdtX9A7adThkaPUGIVM0gJJb6k1QxNd\n0LcCfg2slHQhcJKk3s21q1R1+YKebLYr+WxX8tmu5LNdyWe7ks92JZ/dk0JbC3o1PravpTQx/5mk\nzcdvV2uYdqBMkH0D8Azg9gner7P9sevFfI6kPwfOBg6n9GHfHdYcw57NKc375wLXuMy39stml4vG\nBauTks12JZ/tSj7blXy2K/lsV/LZruSze1Jo61Ot7NlW0gn1wnMa8AfAnhNs/gzgQGA1cLjtD7rj\n815o3GSstabnDOC9tncH9gceBP5I0ua1mb/3mi2BU4DdbZ/ZeH2QbLYh+Ryc5LN/yefgJJ/9Sz4H\nJ/nsphTaJknS6ZJ6E7TOqzU9v6KcFK9yGcHoQuBISds2Xjdm+x5gD9tvtv2fksbG1Sx1Tq9rg6T5\nkp5ea3ouAraX9GyXG2uXAk+nTAC55jW2b7V9nO2Venyy3M7WFCWb7Us+25N8ti/5bE/y2b7ksz3J\nZ0AKbVNxKXCspJ2BMyXtU2ssLgKeL2k/22cBvwO8To/3Oe416d8IpcbI9uouNu/Xi8/x9flOki4C\nvgScK+mltq8CLgB6E2ReQxn5aF9Jz5rg/To7Gek4yWYLks+BST5bkHwOTPLZguRzYJLPSKFtMmqN\nxbXAlZRm+0uAIwBcbuT8b+BASZtSJjQ8Ctiqrn/CCdLxi9ClwHslPRV4L3Cl7b0pfdiXSNoMOBV4\niaQ9bK8EvgNcaPt/x79ZLj7JZsuSz5Yln61KPluWfLYq+WxZ8hk9KbRNTi/87wReRekf/KCkI+ry\nqyk3zh7pMqHhES6zz0fVuPh8B1hi+2hgmaSlwDJgLvAu2z8FLgM+AWD7JtvfGtZ+zwDJZguSz4FJ\nPluQfA5M8tmC5HNgks8AyDxtk1UvSqslnUA5Sd4PfI4y2eOfUvoXL7H9/bp9c5LIzusdD0lbAz8F\nFgKvB+bZPkHSO4AlwAuB+4Dfs33H8PZ45kg2+5d8Dk7y2b/kc3CSz/4ln4OTfAakpW3S/PhNsicC\n8yg30C4BPgP81vYRvZOmbpeTpqFe0Mdcbpr9LHAxsAqYI2kH4LmUIWmfYnul7Tuk3DC7IZLN/iWf\ng5N89i/5HJzks3/J5+AknwFpaZuSRo3HHwMfsf0CSZu6DqHaWz/k3ZwRJN1BmRjyx8B7gNNsf2q4\nezVzJZvtSj7blXy2K/lsV/LZruSzXclnpNA2RY2T52rgLNsXqwxLm1F5NkDj+B0MnGp7J0lb2n6g\nrp/l3DA7Jclm/5LPwUk++5d8Dk7y2b/kc3CSz25L98gpqifN5sAK4K667LGcNBumHr8x218F7pZ0\nqO0HJM2qfbFzQZ+iZLN/yefgJJ/9Sz4HJ/nsX/I5OMlnt81e/yaxDrsBt9VHTNK4i89P6rJczNuR\nbPYp+Ryo5LNPyedAJZ99Sj4HKvnsqHSP7ENG5+mfpL2BvYETc0FvT7LZjuRzMJLPdiSfg5F8tiP5\nHIzks7tSaIuhysUnRlnyGaMs+YxRlnxGtCuFtoiIiIiIiBGWgUgiIiIiIiJGWAptERERERERIyyF\ntoiIiIiIiBGWQltERGy0JD0m6RZJt0taJumvJGk9r3mupMOmax8jIiLWJ4W2iIjYmK2wvavtFwGv\nAfYHTljPa3YA/mTgexYREbGBUmiLiIhOsP0L4O3AMbCmRe0aSd+rjz3qpqcAi2sL3XskjUn6pKQb\nJN0q6W3D+gwREdFNGfI/IiI2WpIesv20cct+CewMPAystv1bSTsCF9heJGkv4H2231C3fxswz/bJ\nkjYFrgPeZPtn0/tpIiKiq2YPewciIiKmWe+etk2Bz0naBXgMmL+W7fcFXizp0Pr30+q2KbRFRMS0\nSKEtIiI6Q9LzgFW275N0AnCP7YWSZgGPrO1lwLtsXzVtOxoREdGQe9oiImJjtmakSEnzgC8AZ9RF\nWwDL6/MjgFn1+cPA5o33uAI4WtLs+j7zJc0d5E5HREQ0paUtIiI2ZnMk3ULpCvkocI7tT9d1nwcu\nkXQE8HVgRV3+fWC1pGXA2bY/I2l74JY6XcC9wEHT+BkiIqLjMhBJRERERETECEv3yIiIiIiIiBGW\nQltERERERMQIS6EtIiIiIiJihKXQFhERERERMcJSaIuIiIiIiBhhKbRFRERERESMsBTaIiIiIiIi\nRlgKbRERERERESPs/wFS2LLXMrSZ1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115408e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "google[['Close','Volatility']].plot(subplots=True, figsize=(15,6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Data Structures: How we work with Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas provides two primary data structures:\n",
    "* `Series` is built for 1-dimensional series data\n",
    "* `DataFrame` is built for 2-dimensional collections of tabular data\n",
    "\n",
    "These data structures differ and are related as follows:\n",
    "* **Series**: 1-dimensional array of **homogeneous** data\n",
    "* **DataFrame**: 2-dimensional table of **heterogeneous** data, composed of multiple **Series**\n",
    "\n",
    "Each of these also contains an integrated **`index`**:\n",
    "* **Series** objects include a second array called the **index** that can contain homogeneous values of any type like integers, strings or datetime objects. \n",
    "* **DataFrame** objects include a column **index**. The **index** objects allow for very sophisticated selection operations using square brackets (`[]`) along with some specialized selection functions.\n",
    "\n",
    "*side note: Pandas has another data structure called Panels which is not commonly used*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "format": "row"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.18.0'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Numerical computing\n",
    "import numpy as np\n",
    "\n",
    "# Draw pictures!\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Python people are weird!\n",
    "import pandas as pd\n",
    "\n",
    "# Set Jupyter notebook display options\n",
    "pd.options.display.max_rows = 8\n",
    "pd.options.display.max_columns = 8\n",
    "\n",
    "# Check version number\n",
    "pd.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Should you have an older version, use the following conda to upgrade pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## To upgrade pandas, uncomment and run the following:\n",
    "\n",
    "# !conda install pandas=0.16.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Designed for 1-dimensional, homogeneous series data\n",
    "* Composed of an array of data and an special Index object\n",
    "* Access to data contents can be done by `position` index (0,1,2,...) or by `label` index ('a','b','c',...)\n",
    "* Index label type can be integers, strings, or datetime objects. \n",
    "* Operations like *retrieving*,  *joining*, and *aligning* data are made easy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "0          gamma Cep b\n",
      "1          HD 114762 b\n",
      "2       PSR B1257+12 B\n",
      "3       PSR B1257+12 A\n",
      "             ...      \n",
      "1638         WASP-89 b\n",
      "1639         HD 1666 b\n",
      "1640        HD 95127 b\n",
      "1641      Kepler-432 c\n",
      "Name: NAME, dtype: object\n",
      "\n",
      "\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "# Example: Creating a Series from a data column from file\n",
    "\n",
    "file   = \"data/exoplanets.csv\"\n",
    "data   = pd.read_csv(file)\n",
    "series = data['NAME']\n",
    "\n",
    "print( \"\\n\" )\n",
    "print( series )\n",
    "\n",
    "print( \"\\n\" )\n",
    "print( type( series ) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     5\n",
       "1    10\n",
       "2    15\n",
       "3    20\n",
       "4    25\n",
       "dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: Creating a Series a list\n",
    "\n",
    "new_list = [5,10,15,20,25]\n",
    "pd.Series(new_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    200.0\n",
       "b    100.0\n",
       "c    700.0\n",
       "d    450.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: Creating a Series from a dictionary... index will be sorted\n",
    "\n",
    "new_dictionary = {'b': 100, 'a': 200.0, 'd':450, 'c':700}\n",
    "pd.Series(new_dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With that, you now have enough information to know the Big, Big, Secret of Pandas:\n",
    "### (Almost) All the complexity in pandas (and it is actually, somewhat complex in parts) arises from the ability to index by *POSITION* (e.g. 0, 1, 2, ith location) and also index by *LABEL* ('a', 'b', 'c', etc.). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b    100\n",
       "a    200\n",
       "d    450\n",
       "c    700\n",
       "dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: Creating a Series from an ORDERED dictionary\n",
    "#          ... to preserver order of the indices.\n",
    "\n",
    "from collections import OrderedDict\n",
    "od = OrderedDict([('b', 100), ('a',200), ('d',450), ('c',700)])\n",
    "pd.Series(od)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "[ 10.  20.  30.  40.  50.]\n",
      "<class 'numpy.ndarray'>\n",
      "\n",
      "\n",
      "0    10.0\n",
      "1    20.0\n",
      "2    30.0\n",
      "3    40.0\n",
      "4    50.0\n",
      "dtype: float64\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "# Example: Creating a Series from a NumPy array\n",
    "\n",
    "array1  = np.arange(1,6)*10.0\n",
    "series1 = pd.Series(array1)\n",
    "\n",
    "print( \"\\n\" )\n",
    "print( array1 )\n",
    "print( type(array1) )\n",
    "\n",
    "print( \"\\n\" )\n",
    "print( series1 )\n",
    "print( type(series1) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas Series data structure always has an `Index`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=5, step=1)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The default Index is just a sequence of integers:\n",
    "array1  = np.arange(1,6)*10.0\n",
    "series1 = pd.Series(array1)\n",
    "\n",
    "series1.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.0\n"
     ]
    }
   ],
   "source": [
    "# The Index is used for indexing\n",
    "\n",
    "print( series1.iloc[0] )  # \"Positional\" indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The Index can be set to a sequence of labels:\n",
    "\n",
    "index2 = ['a','b','c','d','a']\n",
    "series2 = pd.Series(np.arange(1,6)*10.0, index=index2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The Index labels are used for indexing\n",
    "\n",
    "series2.loc['b']  # \"label\" indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    10.0\n",
       "a    50.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Indexing when the label is not unique?\n",
    "\n",
    "series2.loc['a']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterating over a Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a 10.0\n",
      "b 20.0\n",
      "c 30.0\n",
      "d 40.0\n",
      "a 50.0\n"
     ]
    }
   ],
   "source": [
    "for key, value in series2.iteritems():\n",
    "    print(key, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    20.0\n",
       "2    30.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Positional indexing and slicing\n",
    "\n",
    "series1.iloc[1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b    20.0\n",
      "dtype: float64\n",
      "\n",
      "\n",
      "b    20.0\n",
      "c    30.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Be careful when slicing with integers versus labels\n",
    "\n",
    "print( series2.iloc[1:2]     )  # Positional indexing is permitted with slicing\n",
    "print(\"\\n\")\n",
    "print( series2.loc['b':'c'] )  # Ranges of labels can be sliced (slicing here is *inclusive*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    50.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: slice from the end\n",
    "series2.iloc[-1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    10.0\n",
       "b    20.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: slice from the beginning\n",
    "series2.iloc[:-3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d    40.0\n",
       "b    20.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: slice with labels, but with integer step size\n",
    "series2.loc['d':'b':-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series Arithmetic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    10.0\n",
      "1    20.0\n",
      "2    30.0\n",
      "3    40.0\n",
      "4    50.0\n",
      "dtype: float64\n",
      "0     20.0\n",
      "1     40.0\n",
      "2     60.0\n",
      "3     80.0\n",
      "4    100.0\n",
      "dtype: float64\n",
      "0     60.0\n",
      "1    120.0\n",
      "2    180.0\n",
      "3    240.0\n",
      "4    300.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Arithmetic with Series is as with numpy arrays\n",
    "print(series1)\n",
    "\n",
    "series2 = series1 * 2 \n",
    "print(series2)\n",
    "\n",
    "series3 = series2 * 3\n",
    "print(series3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series with NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    -3.141593\n",
       "1    -3.078126\n",
       "2    -3.014660\n",
       "3    -2.951193\n",
       "        ...   \n",
       "96    2.951193\n",
       "97    3.014660\n",
       "98    3.078126\n",
       "99    3.141593\n",
       "dtype: float64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# numpy ufuncs can be applied elementwise to Series\n",
    "\n",
    "series_2pi = pd.Series( np.linspace(-1*np.pi, +1*np.pi, 100) )\n",
    "series_2pi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    -1.224647e-16\n",
      "1    -6.342392e-02\n",
      "2    -1.265925e-01\n",
      "3    -1.892512e-01\n",
      "          ...     \n",
      "96    1.892512e-01\n",
      "97    1.265925e-01\n",
      "98    6.342392e-02\n",
      "99    1.224647e-16\n",
      "dtype: float64\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "# Even though pandas was built on numpy, numpy methods on Series return a Series\n",
    "\n",
    "series_sin = np.sin( series_2pi )\n",
    "print( series_sin )\n",
    "print( type( series_sin ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series with Matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x115bbb668>]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm4l3P+x/Hn+9RIlCVMUiNaJRmMX5PB1dGImIaQSZY2\nLZhso6SGkSzZRoaQJW1kzZDUJMspDZpG0iIq2lVDoZRp/fz++JxGcta+y+e+7+/rcV1dvuec+5z7\n5Vz1Pvd5fzZzziEiIsmXFzqAiIhkhwq+iEiOUMEXEckRKvgiIjlCBV9EJEeo4IuI5Ii0FHwzG2pm\nq81sVgnXPGBmC8xsppkdk477iohI2aXrCX8YcHpxHzSzM4C6zrn6QA9gSJruKyIiZZSWgu+cmwp8\nXcIlZwMjC6+dBuxrZtXTcW8RESmbbPXwawLLdnp7ReH7REQkSzRoKyKSIypm6T4rgF/s9Hatwvf9\nhJlpcx8RkXJyzllp16TzCd8K/xRlLNABwMyaAd8451YX94Wcc5H+c/PNNwfPoJzKme6cb7zhaNrU\nUb++49FHHd9+W7bP27bN8dZbjosvdhxwgOPOOx0bN+r7mc0/ZZWWJ3wzGw3kAweY2VLgZmAPX7vd\nY8658WZ2ppktBDYAndNxXxFJ3Zdfwp/+BO+8A/fcA+eeCxUqlP3z8/LglFP8n08/hX79oGFDeOwx\naNUqc7ml/NJS8J1zF5bhmp7puJeIpM+ECdCpE1xyCcyZA1WqpPb1GjaEMWPgrbf81z3nHLjzTqhc\nOR1pJVUatN0N+fn5oSOUiXKmV5Jybt8Ot90GXbv6An3vvakX+521aAEffQSrVkGzZrBs2U+vSdL3\nMy6sPP2fbDAzF7VMIkmyaRNcdBF88QW8+CIcckjm7uUc3Hcf3H8/vPoqHKM19hlhZrgsD9qKSMRt\n3AhnneVfv/12Zos9gBlcdx0MGgSnnQaTJmX2flIyFXyRHLFunR9EPfhgePZZqFQpe/du29a3ji68\n0P+gkTBU8EVywH//C2efDY0awbBhUDFbK3B2cvLJ8MIL0K4dTJ2a/fuLevgiibdtmy+yeXnwzDPl\nm3KZCZMm+TGE119XTz9d1MMXEZyDq66CtWth1KjwxR6gZUt46CE/lrByZeg0uSXAL3Yiki0PPwxT\npvgWSjZ79qU5/3yYP9+3mQoKYK+9QifKDWrpiCTU1Klw3nnw7rtQt27oND/lHHToAJs3+0FkK7Uh\nIcVRS0ckh61Y4fv2I0ZEs9iDL/CPPw4LF/oWj2SenvBFEmbrVsjPhzPOgD//OXSa0n32GZxwAowf\nD8cfHzpNPOkJXyRHDRzo967p2zd0krKpW9ePNbRrB998EzpNsukJXyRB3nsP2rSBGTOgZszOlPvj\nH33Bf/rp0EniR0/4Ijlm3Tq4+GIYMiR+xR781szTp8NLL4VOklx6whdJiMsugy1bYOjQ0El233vv\n+f34P/oIfv7z0Gnio6xP+Cr4IgkwebJfvTpnDuy3X+g0qenTx8/cefFFTdUsK7V0RHLE99/7fe0f\neij+xR7glltg3jy1djJBT/giMdenDyxeDM89FzpJ+rzzjt9Zc+5c2Gef0GmiTy0dkRwwaxaceirM\nng3Vq4dOk16XXgpVq/rDU6RkKvgiCeecX2B1wQVw+eWh06TfV19B48b+3N3jjgudJtrUwxdJuGee\ngfXroXv30Eky48AD/QHol13mz+CV1Kngi8TQ+vXQuzcMHhyNLY8zpWNHP1PnqadCJ0kGtXREYqhP\nH1i9GoYPD50k895/3+/6+emnUKVK6DTRpB6+SEItWuQ3GZszB2rUCJ0mOy66COrUgVtvDZ0kmlTw\nRRLqwguhYUO4+ebQSbJn2TJ/HOKMGVC7dug00aOCL5JA//oXnHOOPy1q771Dp8mu/v39Clz1839K\nBV8kYZyD5s39KVFdu4ZOk33r10P9+jBxIvzyl6HTRIumZYokzKuv+sPIO3cOnSSMqlWhX7/47PMf\nRSr4IjGwfTvceCPccUeyp2GWpkcPv8/O5Mmhk8STCr5IDDz/vD/F6ve/D50krEqV/EydG27wLS4p\nHxV8kYjbutXPyLn9dm0XDNC+PXz3Hbz2Wugk8aOCLxJxo0b5+fa//W3oJNFQoYKfsdO/v57yy0sF\nXyTCNm+GAQP0dL+rc87xp3uNGxc6Sbyo4ItE2KhRfiriiSeGThIteXm+zaWn/PJRwReJqK1b/ayc\nm24KnSSa2rSBbdtg7NjQSeJDBV8kokaPhl/8Ak4+OXSSaMrL80/4t9yip/yyUsEXiaBt23zf/i9/\nCZ0k2s46y/fyJ04MnSQeVPBFIuj55/0BIKecEjpJtOXl+ZW3d9wROkk8qOCLRIxzMHCgX1mrmTml\n+8MfYMUKf/C5lEwFXyRiJkzwhb5Vq9BJ4qFiRb/yduDA0EmiTwVfJGLuusufaKWn+7Lr0AFmzfL7\n5UvxVPBFIuS992DpUt+mkLKrVAmuvRbuuSd0kmjTfvgiEdKmDZx6KvTsGTpJ/KxbB4cfDh98AIcd\nFjpNdukAFJGYmTcP8vP9mbV77RU6TTxdf73fjuL++0MnyS4VfJGY6d4datbMrbNq023FCmjSBD77\nDPbfP3Sa7FHBF4mRL7+EBg38WbUHHRQ6Tbx16uS/l/36hU6SPSr4IjEyYAAsXw6PPRY6SfzNng2n\nnQaLF/vB3FyQ1TNtzayVmX1iZvPNrE8RH29uZt+Y2YzCPzem474iSfDf/8LDD8M114ROkgxNmsDR\nR8Nzz4VOEj0pF3wzywMGA6cDjYH2ZnZEEZdOcc4dV/jntlTvK5IUTz8Nxx0HRx4ZOklyXHONH7hV\ns+DH0vGE3xRY4Jxb4pzbAjwLnF3EdVpGIrIL52DQILjuutBJkuX002HjRm23sKt0FPyawLKd3l5e\n+L5dnWBmM83sNTPTs4wI8OabfkVtixahkyRLXh5cdVXuTc8sTcUs3ecD4FDn3EYzOwN4GWhQ3MX9\n+/f/3+v8/Hzy8/MznU8kiAcf9IVJ2yikX4cOfnvpzz+HOnVCp0mvgoICCgoKyv15Kc/SMbNmQH/n\nXKvCt28AnHPurhI+ZxHwK+fc2iI+plk6khM+/xx+/WtYskQLrTLl+uv9yWH33Rc6SWZlc5bOdKCe\nmdU2sz2AC4AfHTpmZtV3et0U/4PmJ8VeJJc89BB06aJin0l//COMHAkbNoROEg0pt3Scc9vMrCfw\nOv4HyFDn3Dwz6+E/7B4D2prZ5cAW4HugXar3FYmz776D4cO1u2Om1a7tj4h86ino0SN0mvC08Eok\ngEcegUmT4KWXQidJvjff9NM0Z81K7lhJVhdeiUjZOQeDB2tHzGxp0cL38adMCZ0kPBV8kSybMsUX\nfZ1Xmx1m/ofr4MGhk4Snlo5IlrVr5/vKesLPnvXrfT9/1iyoVSt0mvTT5mkiEbRypd9CYfFi2Hff\n0GlyS8+eUK2a36guaVTwRSJowAD44gsYMiR0ktwzdy60bOnXPfzsZ6HTpJcGbUUiZssWv/3xFVeE\nTpKbGjeGevVg7NjSr00qFXyRLHn1VX/W6tFHh06Suy67zE+JzVUq+CJZMmQIXH556BS57bzz/AEp\n8+eHThKGCr5IFnz2GXz4oS84Ek6lStC5Mzz6aOgkYWjQViQL+vSBbdvg3ntDJ5FFi6BpU1i6FCpX\nDp0mPTRoKxIRmzb5fXO6dw+dRAAOPxyOPx7GjAmdJPtU8EUy7O9/h6OOggbFngAh2da9e262dVTw\nRTLs0Ue1U2PUtG4NCxfCxx+HTpJdKvgiGTR/vi8qbdqETiI7+9nP/FkEjz8eOkl2adBWJIOuv97/\n9+67w+aQn1q0CP7v/2D5cthzz9BpUqNBW5HANm+GESOga9fQSaQoOwZvX3wxdJLsUcEXyZCxY6FR\nIw3WRln37n67i1yhgi+SIY8/Dt26hU4hJWnd2o+z5MrKWxV8kQxYvBg++EAra6Nujz2gQwcYOjR0\nkuzQoK1IBtx0E3z7LTzwQOgkUppPP4XmzWHZsvhum6xBW5FAtm2DYcM0WBsXDRv6cZZx40InyTwV\nfJE0mzgRDjlE2yDHSdeu8MQToVNkngq+SJoNHQqXXho6hZRH27bw3nu+rZNk6uGLpNF//uPbA0uW\n6MzauLniCqhRw4+/xI16+CIBjBrlt1FQsY+fLl382Mv27aGTZI4KvkiaOKd2Tpz96ldQtSpMnhw6\nSeao4Iukyfvvw9atcNJJoZPI7jDzT/lJnpOvHr5ImnTrBvXq+dOtJJ7WrIG6df3Cuf32C52m7NTD\nF8miDRv8JlyXXBI6iaTigAPgtNPgmWdCJ8kMFXyRNBgzBk480c+/l3i79NLktnVU8EXS4Mknff9X\n4u/UU2H1apg9O3SS9FPBF0nRjqPyWrcOnUTSoUIF6NjRT9FMGg3aiqToxhth40a4777QSSRdFi6E\n3/zGn4a1xx6h05ROg7YiWbBtmz/VqnPn0EkknerVgyOOgNdeC50kvVTwRVLw5ptQvTo0aRI6iaTb\njpW3SaKCL5KCYcP0dJ9UbdvCO+/AqlWhk6SPCr7Ibvr6a5gwAdq3D51EMqFKFTjnHL8/UlKo4Ivs\npmefhdNPh2rVQieRTOncGYYP9/skJYEKvshuGjYMOnUKnUIy6aSTYNMmmD49dJL0UMEX2Q1z58KK\nFX4ZviSXmf+hPnx46CTpoXn4Iruhd2+oWBEGDgydRDJt6VI49lj/A37PPUOnKZrm4YtkyNat8NRT\naufkikMPheOOg1deCZ0kdSr4IuU0cSIcfjg0bBg6iWTLjsHbuFPBFymn4cP1dJ9r2rSBadN8WyfO\nVPBFymHNGpg0Cdq1C51EsmmvveD88+M/J18FX6QcnnkGfvc7HVKei3bM1onznBIVfJFyUDsndzVr\n5ov9tGmhk+y+tBR8M2tlZp+Y2XwzK/JETzN7wMwWmNlMMzsmHfcVyaY5c/zBGC1ahE4iISRhTn7K\nBd/M8oDBwOlAY6C9mR2xyzVnAHWdc/WBHsCQVO8rkm3Dh0OHDv6ADMlNl1wCL7wA338fOsnuSccT\nflNggXNuiXNuC/AscPYu15wNjARwzk0D9jWz6mm4t0hWbNni59537Bg6iYRUqxYcf3x85+Sno+DX\nBJbt9PbywveVdM2KIq6JlC1b4PXXQ6eQqJg4EerWhQYNQieR0OLc1qkYOkBR+vfv/7/X+fn55Ofn\nB8nRoQNMmaJ/5KLBWvlBmzbw0kt+xXXFQBW0oKCAgoKCcn9eynvpmFkzoL9zrlXh2zcAzjl3107X\nDAHeds49V/j2J0Bz59zqIr5eZPbS6dULKlWC228PnURCWrPGP90vWaLpmBJN2dxLZzpQz8xqm9ke\nwAXA2F2uGQt0KAzWDPimqGIfNR07wsiR/txSyV2aey9JkXLBd85tA3oCrwNzgWedc/PMrIeZdS+8\nZjywyMwWAo8CV6R632xo0sSfV/rmm6GTSEjDh2uwVpJB2yOXYvBgePddGD06dBIJYfZsOPNMWLxY\n0zElusra0lHBL8WaNVCnju/f7rdf6DSSbdddB5Urw223hU4iUjzth58mBxwALVvC88+HTiLZtmUL\nPP202jmSHCr4ZdCpkz+/VHLLP/4B9epB/fqhk4ikhwp+GbRq5Xu4n34aOolkk+beS9Koh19GOsM0\nt3z1lX+6X7oU9tkndBqRkqmHn2adOmlOfi4ZPRpat1axl2RRwS+jxo2hZk14443QSSQbhg3z55iK\nJIkKfjlo8DY3zJwJX38Np5wSOolIeqmHXw5r1/o5+YsWwf77h04jmXL11X7NxS23hE4iUjbq4WdA\ntWpw+unw7LOhk0imbN7s987R3HtJIhX8curcWW2dJBs3Do480v8mJ5I0Kvjl1LIlfPGFP99UkkeD\ntZJk6uHvhn79YNMm+OtfQyeRdFq1Cho1guXLYe+9Q6cRKTv18DOoc2d/vumWLaGTSDqNGgXnnKNi\nL8mlgr8b6tf3xx6OHx86iaSLc/Dkk3DppaGTiGSOCv5u6tzZFwhJhvffh+3b4Te/CZ1EJHNU8HfT\n+ef7A85XrQqdRNLhySehSxewUrugIvGlQdsUdOniB/l69w6dRFKxYQPUqgVz58Ihh4ROI1J+GrTN\ngi5d/JNhTH4+STHGjIETT1Sxl+RTwU/BiSf6vu9774VOIqkYOlRz7yU3qOCnwOyHp3yJpwUL4JNP\n4Pe/D51EJPPUw0/RypV+Kf6yZVClSug0Ul59+/r9c7SITuJMPfwsqVEDTj4ZXnghdBIpr61bYcQI\nzb2X3KGCnwZdusATT4ROIeU1YQIcdpj/DU0kF6jgp8Hvfgeffw7z5oVOIuUxdKie7iW3qIefJn37\n+r117r03dBIpix0bpS1dClWrhk4jkhr18LOsSxd/yPnmzaGTSFmMHAnnnqtiL7lFBT9N6tf3B52P\nHRs6iZTGOT/m0rVr6CQi2aWCn0Zdu2rwNg4mT4Y99oBmzUInEckuFfw0Ou88+Pe/YfHi0EmkJI8/\nDt26aaM0yT0atE2zq66C/faDAQNCJ5GirFkDdev6WVXVqoVOI5IeGrQNpHt3P91v69bQSaQoo0b5\nabQq9pKLVPDT7Kij/GIenYYVPc790M4RyUUq+BnQvTs89ljoFLKrd9/1ayWaNw+dRCQMFfwMOP98\nv2Xy0qWhk8jOHn0UevTQYK3kLg3aZsiVV/o+8S23hE4iAGvXQp06sHAhHHhg6DQi6aVB28A0eBst\nI0ZA69Yq9pLbVPAzpEkTqF0bxo0LnUSc8+2cyy4LnUQkLBX8DLr8cnjkkdApZPJkqFDBH0kpkstU\n8DOobVuYMcP3jSWcHU/3GqyVXKdB2wzr3dv/9557wubIVTu2QV60yK+AFkmisg7aquBn2MKFcMIJ\n/szbPfcMnSb33H47LFmidRGSbJqlExH16sFxx+nM2xC2bvXtnCuuCJ1EJBpU8LNAg7dhjBsHtWrB\nMceETiISDSr4WdC6NaxYAR98EDpJbnn4YT3di+xMBT8LKlb0T/kPPRQ6Se6YPx8++shvcyEingZt\ns+TLL6FBAz+Ie8ABodMk3zXXQOXKMHBg6CQimZeVWTpmtj/wHFAbWAz8wTn3bRHXLQa+BbYDW5xz\nTUv4moks+ACdOsGRR8L114dOkmzr1/stqj/8EA49NHQakczL1iydG4A3nHMNgbeAvsVctx3Id84d\nW1KxT7qePX1fedu20EmSbeRIaNFCxV5kV6kW/LOBEYWvRwBtirnO0nCv2Dv+eKhRA159NXSS5Nq+\nHR580B81KSI/lmoR/rlzbjWAc24V8PNirnPAJDObbmY5fd7Q1VfD3/4WOkVyTZrke/cnnRQ6iUj0\nVCztAjObBFTf+V34An5jEZcX13w/0Tm30swOwhf+ec65qcXds3///v97nZ+fT35+fmkxY+O886BX\nL5g5U/PDM+GBB/zTvfbNkSQrKCigoKCg3J+X6qDtPHxvfrWZHQy87ZxrVMrn3Aysd87dV8zHEzto\nu8Odd8Knn8KwYaGTJMv8+f7JfskS/5QvkiuyNWg7FuhU+Loj8EoRQfYysyqFr/cGTgPmpHjfWOvW\nDV5+GVavDp0kWe6/3++KqWIvUrRUn/CrAc8DvwCW4KdlfmNmNYDHnXOtzexw4O/4dk9F4Gnn3J0l\nfM3EP+GDL0w1asDNN4dOkgxr1vh9i+bNg4MPDp1GJLu0W2bEzZvnpw4uWqRdNNPhjjv8orYnnwyd\nRCT7tFtmxDVq5HfRfOqp0Enib9MmGDwYrr02dBKRaFPBD6hXL7j3Xj93XHbfc8/BUUf5c4RFpHgq\n+AHl50OVKvDaa6GTxJdz8Ne/6ulepCxU8AMy80/5Ov5w9/3jH77ot2oVOolI9KngB9a2LSxdCtOm\nhU4ST3fdBX36aKGVSFmo4AdWsSL86U9w992hk8TPtGmweDG0axc6iUg8aFpmBGzYAHXqwNtv++2T\npWzOPRdOOQWuvDJ0EpGwNA8/Zm6/3W8NMGJE6dcKfPIJNG/u1zHstVfoNCJhqeDHzDffQN26/tzb\nww4LnSb6OnXy36+bbgqdRCQ8FfwY6tsX1q3T2bel+fxzaNrUr6zdb7/QaUTCU8GPodWr/QrcuXP9\nPjtStG7d/PdnwIDQSUSiQQU/pq65xk8xHDQodJJoWrLEb0mxYAFUqxY6jUg0qODH1MqV0LixnvKL\nc8UVsO++MHBg6CQi0aGCH2PXXuv319FRiD+2fDkcfbQ/POagg0KnEYkOFfwYW7XKz8efPRtq1gyd\nJjp69ID99/cnhonID1TwY65XL7/t74MPhk4SDZ99Br/+tV+roN69yI+p4Mfcf/7jZ+zMmAG1a4dO\nE16HDn7evU4IE/kpFfwE+Mtf/MZqw4eHThLWxx/7LRQWLIB99gmdRiR6VPATYN06qF8f3nzTH/CR\nq847z7dzrr8+dBKRaFLBT4hBg/ymamPHhk4SxrvvwgUX+L1ztGeOSNF0pm1CXH45fPQRTJ0aOkn2\nOecHr2+7TcVeJB1U8CNuzz19wevVK/fOvh0zBr7/Hi6+OHQSkWRQwY+Biy7yxX706NBJsmfzZrjh\nBn/Ie57+loqkhf4pxUBeHtx/v99Nc8OG0GmyY/BgaNAAfvvb0ElEkkODtjHSvr0vgrfcEjpJZq1c\n6bdQmDoVGjYMnUYk+jRLJ4GWLoVjj03+YqwOHeCQQ7SFgkhZqeAn1G23wb/+Ba+84rdRTpqpU3+Y\nhlmlSug0IvGgaZkJ1bu330/mlVdCJ0m/rVuhZ08/UKtiL5J+KvgxU6kSDBkCV10F69eHTpNegwbB\ngQdCu3ahk4gkk1o6MdWpk9818r77QidJjwUL4IQTfLuqTp3QaUTiRT38hPvySz+T5aWXfKGMM+eg\nRQs46yx/+IuIlI96+Al30EF+rnrHjrBxY+g0qXniCf//cNVVoZOIJJue8GPuoot88b///tBJds/C\nhf43lLffzu0dQUVSoZZOjli7Fpo0gaee8nvGx8mWLXDyyXDhhXq6F0mFWjo5olo1GDrUL1b68svQ\nacrn1lv9GbVXXhk6iUhu0BN+QvTtCx9+COPHx2OzsSlT4A9/gJkz4eCDQ6cRiTc94eeYW2+F776D\nu+4KnaR0K1b41bTDh6vYi2STnvATZPlyOP54388/9dTQaYq2aRM0b+6nYPbrFzqNSDJo0DZHTZ4M\n55/vWyZHHBE6zY85B5ddBl99BS++mMy9gERCUEsnRzVv7ts6rVv7whol997rz6gdPlzFXiQEFfwE\n6twZ2raFNm2ic2DKyJF+odiECVC1aug0IrlJLZ2E2r4dLr3U76E/bhxUrhwuy/jx0KWLX1zVqFG4\nHCJJpZZOjsvL81sWVK8O557rB0tDeOUVv9Hbyy+r2IuEpoKfYBUq+FZKlSq+p79uXXbv/8wz0KOH\nf8Jv1iy79xaRn1LBT7iKFX3hbdAATjrJT93MNOf83va9esEbb/ipoiISngp+DqhY0Q+YdujgNyqb\nOjVz99qwAS6+GEaNgn/+UxuiiURJSgXfzNqa2Rwz22Zmx5VwXSsz+8TM5ptZn1TuKbvHzD9xP/KI\nn6d/001+87J0+uAD/wOlQgVf7A87LL1fX0RSk+oT/mzgHGBycReYWR4wGDgdaAy0N7OILQkqn4KC\ngtARyqSonK1b+z13/v1v31dPx//Kd9/5g0vOPBOuuw5GjCjfrKA4fz+jSDnTKy45yyKlgu+c+9Q5\ntwAoaTpQU2CBc26Jc24L8Cxwdir3DS0ufwGKy3nwwX4gtVcvP12ydWt4/33fey+PtWv9Hj516/rX\nc+f6A1nKu6gq7t/PqFHO9IpLzrLIRg+/JrBsp7eXF75PAjKD9u1h3jxo2dL39xs39qt0p08vfhrn\nV1/B00/7Pezr1YNFi/xvCSNG+APIRSS6KpZ2gZlNAqrv/C7AAX92zr2aqWCSHZUqwdVX+wNI/vlP\nX8y7dfOHih9+uF8Vu/fe8M03vrhv3uzPn23d2h+grt0uReIjLSttzext4Drn3IwiPtYM6O+ca1X4\n9g2Ac84VuZGvmWmZrYhIOZVlpW2pT/jlUNzNpgP1zKw2sBK4AGhf3BcpS2gRESm/VKdltjGzZUAz\nYJyZTSh8fw0zGwfgnNsG9AReB+YCzzrn5qUWW0REyitym6eJiEhmRHalrZldZ2bbzaxa6CxFMbMB\nZvaRmX1oZv8ws0gOX5rZ3WY2z8xmmtkYM9sndKailHURXwhxWThoZkPNbLWZzQqdpThmVsvM3jKz\nuWY228yuCp2pKGZWycymFf77nm1mN4fOVBIzyzOzGWY2tqTrIlnwzawW0BJYEjpLCe52zv3SOXcs\n8BoQ1b8QrwONnXPHAAuAvoHzFKfURXwhxGzh4DB8zijbCvzJOdcYOAH4YxS/n865TcAphf++jwHO\nMLOmgWOV5Grg49IuimTBBwYBvUOHKIlz7rud3twb2B4qS0mcc28453Zkex+oFTJPccq4iC+E2Cwc\ndM5NBb4OnaMkzrlVzrmZha+/A+YR0XU5zrmNhS8r4Se4RLL/XfiAfCbwRGnXRq7gm9lZwDLn3OzQ\nWUpjZreZ2VLgQuAvofOUQRdgQugQMaOFgxliZofhn56nhU1StMI2yYfAKmCSc2566EzF2PGAXOoP\npHROyyyzEhZz3Qj0w7dzdv5YEKUtOnPO3QjcWNjXvRLon/2UZVscZ2Z/BrY450YHiEhhBi3iEwDM\nrArwInD1Lr8tR0bhb8bHFo57vWxmRzrnSm2bZJOZ/Q5Y7ZybaWb5lFIvgxR851zLot5vZkcBhwEf\nmZnh2w8fmFlT59x/shgRKD5nEUYD4wlU8EvLaWad8L/ytchKoGKU4/sZJSuAQ3d6u1bh+2Q3mVlF\nfLEf5Zx7JXSe0jjn1hUuLm1FGfrkWXYicJaZnQlUBqqa2UjnXIeiLo5US8c5N8c5d7Bzro5z7nD8\nr8/Hhij2pTGzeju92Qbfi4wcM2uF/3XvrMKBqDiIUh//fwsHzWwP/MLBEmdCBGZE6/tXlCeBj51z\nfwsdpDhmdqCZ7Vv4ujK+6/BJ2FQ/5Zzr55w71DlXB/93863iij1ErOAXwRHdv7x3mtksM5sJnIof\nJY+iB4HUmJaSAAAAfElEQVQqwKTCaVsPhw5UlOIW8YUWp4WDZjYaeBdoYGZLzaxz6Ey7MrMTgYuA\nFoVTHmcUPpRETQ3g7cJ/39OAic658YEzpUwLr0REckTUn/BFRCRNVPBFRHKECr6ISI5QwRcRyREq\n+CIiOUIFX0QkR6jgi4jkCBV8EZEc8f/sWnwIv7tTaAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115a9eb38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(series_2pi, series_sin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    1\n",
      "1    2\n",
      "2    3\n",
      "3    4\n",
      "4    5\n",
      "dtype: int64\n",
      "0    10\n",
      "1    20\n",
      "2    30\n",
      "3    40\n",
      "4    50\n",
      "dtype: int64\n",
      "0    11\n",
      "1    22\n",
      "2    33\n",
      "3    44\n",
      "4    55\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# You can add two series\n",
    "\n",
    "series1 = pd.Series([1,2,3,4,5])\n",
    "print(series1)\n",
    "\n",
    "series2 = pd.Series([10,20,30,40,50])\n",
    "print(series2)\n",
    "\n",
    "series3 = series1 + series2\n",
    "print(series3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a     NaN\n",
      "b     NaN\n",
      "c    13.0\n",
      "d    24.0\n",
      "e    35.0\n",
      "f     NaN\n",
      "g     NaN\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Even better when you assign labels to your data!\n",
    "\n",
    "series1 = pd.Series([1,2,3,4,5],      index=['a','b','c','d','e'])\n",
    "series2 = pd.Series([10,20,30,40,50], index=['c','d','e','f','g'])\n",
    "\n",
    "print( series1 + series2 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  SQL users: what just happened here?????"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24.0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# So when you apply a method like mean(), the NaNs are handled intelligently...\n",
    "\n",
    "(series1 + series2).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pandas mean:  1.5\n",
      "Numpy mean:   nan\n"
     ]
    }
   ],
   "source": [
    "# Contrast how NumPy and Pandas handle NaN:\n",
    "\n",
    "print( \"Pandas mean: \", pd.Series( [1.0, 2.0, np.nan] ).mean() )\n",
    "print( \"Numpy mean:  \", np.array(  [1.0, 2.0, np.nan] ).mean() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "What is a DataFrame?\n",
    "\n",
    "* 2-dimensional labled tabular data (think spreadsheet or SQL table) \n",
    "* **heterogeneous** data, composed of multiple **Series**\n",
    "* an **Index** or \"row label\" for selection operations\n",
    "\n",
    "DataFrame inputs:\n",
    "\n",
    "* python dict of 1D sequences (e.g. ndarrays, lists, dicts, or Series)\n",
    "* 2-D numpy.ndarray\n",
    "* pandas Series\n",
    "* another DataFrame\n",
    "\n",
    "DataFrame Operations:\n",
    "- Pandas DataFrame objects have column names:\n",
    "    - dictionary style: df['DATE']\n",
    "    - attribute style: df.DATE\n",
    "- DataFrame binary operations (+ - / *) defaults to outer join, on both columns as well as the index\n",
    "- NaN can be handled after join"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>FIRSTURL</th>\n",
       "      <th>DATE</th>\n",
       "      <th>...</th>\n",
       "      <th>RA_STRING(h:m:s)</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gamma Cep b</td>\n",
       "      <td>Campbell 1988</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1988ApJ...331..902C</td>\n",
       "      <td>1988</td>\n",
       "      <td>...</td>\n",
       "      <td>+23:39:20.98</td>\n",
       "      <td>+77:37:55.08</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>5.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HD 114762 b</td>\n",
       "      <td>Latham 1989</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1989Natur.339...38L</td>\n",
       "      <td>1989</td>\n",
       "      <td>...</td>\n",
       "      <td>+13:12:20.10</td>\n",
       "      <td>+17:31:01.66</td>\n",
       "      <td>0.894578</td>\n",
       "      <td>0.859205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>PSR B1257+12 B</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>http://adsabs.harvard.edu//abs/1992Natur.355.....</td>\n",
       "      <td>1992</td>\n",
       "      <td>...</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PSR B1257+12 A</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>http://adsabs.harvard.edu//abs/1992Natur.355.....</td>\n",
       "      <td>1992</td>\n",
       "      <td>...</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1638</th>\n",
       "      <td>WASP-89 b</td>\n",
       "      <td>Hellier 2015</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/2015AJ....150...18H</td>\n",
       "      <td>2015</td>\n",
       "      <td>...</td>\n",
       "      <td>20:55:35.98</td>\n",
       "      <td>-18:58:16.1</td>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.880000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1639</th>\n",
       "      <td>HD 1666 b</td>\n",
       "      <td>Harakawa 2015</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/2015ApJ...806....5H</td>\n",
       "      <td>2015</td>\n",
       "      <td>...</td>\n",
       "      <td>+00:20:52.34</td>\n",
       "      <td>-19:55:51.89</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>1.930000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1640</th>\n",
       "      <td>HD 95127 b</td>\n",
       "      <td>Niedzielski 2015</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/2015ApJ...803....1N</td>\n",
       "      <td>2015</td>\n",
       "      <td>...</td>\n",
       "      <td>+10:59:35.10</td>\n",
       "      <td>+43:48:52.17</td>\n",
       "      <td>1.200000</td>\n",
       "      <td>20.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1641</th>\n",
       "      <td>Kepler-432 c</td>\n",
       "      <td>Ciceri 2015</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/2015A%26A...573L...</td>\n",
       "      <td>2015</td>\n",
       "      <td>...</td>\n",
       "      <td>19:33:07.729</td>\n",
       "      <td>+48:17:09.</td>\n",
       "      <td>1.320000</td>\n",
       "      <td>4.060000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1642 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                NAME          FIRSTREF  \\\n",
       "0        gamma Cep b     Campbell 1988   \n",
       "1        HD 114762 b       Latham 1989   \n",
       "2     PSR B1257+12 B    Wolszczan 1992   \n",
       "3     PSR B1257+12 A    Wolszczan 1992   \n",
       "...              ...               ...   \n",
       "1638       WASP-89 b      Hellier 2015   \n",
       "1639       HD 1666 b     Harakawa 2015   \n",
       "1640      HD 95127 b  Niedzielski 2015   \n",
       "1641    Kepler-432 c       Ciceri 2015   \n",
       "\n",
       "                                               FIRSTURL  DATE  \\\n",
       "0     http://adsabs.harvard.edu/abs/1988ApJ...331..902C  1988   \n",
       "1     http://adsabs.harvard.edu/abs/1989Natur.339...38L  1989   \n",
       "2     http://adsabs.harvard.edu//abs/1992Natur.355.....  1992   \n",
       "3     http://adsabs.harvard.edu//abs/1992Natur.355.....  1992   \n",
       "...                                                 ...   ...   \n",
       "1638  http://adsabs.harvard.edu/abs/2015AJ....150...18H  2015   \n",
       "1639  http://adsabs.harvard.edu/abs/2015ApJ...806....5H  2015   \n",
       "1640  http://adsabs.harvard.edu/abs/2015ApJ...803....1N  2015   \n",
       "1641  http://adsabs.harvard.edu/abs/2015A%26A...573L...  2015   \n",
       "\n",
       "            ...          RA_STRING(h:m:s)  DEC_STRING(d:m:s)  STAR_MASS(msun)  \\\n",
       "0           ...              +23:39:20.98       +77:37:55.08         1.260000   \n",
       "1           ...              +13:12:20.10       +17:31:01.66         0.894578   \n",
       "2           ...              13:00:03.577        +12:40:56.4              NaN   \n",
       "3           ...              13:00:03.577        +12:40:56.4              NaN   \n",
       "...         ...                       ...                ...              ...   \n",
       "1638        ...               20:55:35.98        -18:58:16.1         0.920000   \n",
       "1639        ...              +00:20:52.34       -19:55:51.89         1.500000   \n",
       "1640        ...              +10:59:35.10       +43:48:52.17         1.200000   \n",
       "1641        ...              19:33:07.729         +48:17:09.         1.320000   \n",
       "\n",
       "      STAR_RADIUS(rsun)  \n",
       "0              5.010000  \n",
       "1              0.859205  \n",
       "2                   NaN  \n",
       "3                   NaN  \n",
       "...                 ...  \n",
       "1638           0.880000  \n",
       "1639           1.930000  \n",
       "1640          20.000000  \n",
       "1641           4.060000  \n",
       "\n",
       "[1642 rows x 17 columns]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a DataFrame from a file read:\n",
    "\n",
    "file    = \"data/exoplanets.csv\"\n",
    "df_file = pd.read_csv(file)\n",
    "df_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.349066</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.698132</td>\n",
       "      <td>0.0</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1.047198</td>\n",
       "      <td>0.0</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>2.094395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>2.443461</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>2.792527</td>\n",
       "      <td>0.0</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>3.141593</td>\n",
       "      <td>0.0</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     a         b    c  d\n",
       "0    1  0.000000  0.0  a\n",
       "1    2  0.349066  0.0  b\n",
       "2    3  0.698132  0.0  c\n",
       "3    4  1.047198  0.0  a\n",
       "..  ..       ...  ... ..\n",
       "6    7  2.094395  0.0  a\n",
       "7    8  2.443461  0.0  b\n",
       "8    9  2.792527  0.0  c\n",
       "9   10  3.141593  0.0  a\n",
       "\n",
       "[10 rows x 4 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a DataFrame from a dictionary\n",
    "\n",
    "dictionary = {'a':[1,2,3,4,5,6,7,8,9,10],\n",
    "              'b':np.linspace(0,np.pi,10),\n",
    "              'c':0.0,\n",
    "              'd':[\"a\",\"b\",\"c\",\"a\",\"b\",\"c\",\"a\",\"b\",\"c\",\"a\"]}\n",
    "\n",
    "df_dict = pd.DataFrame(dictionary)\n",
    "df_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['a', 'b', 'c', 'd'], dtype='object')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the DataFrame colums and index\n",
    "df_dict.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=10, step=1)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_dict.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use the Pandas `DataFrame.info()` method to see a bit more detail, including:\n",
    "* the name of each column\n",
    "* the Numpy dtype of each column\n",
    "* how many actual values are present\n",
    "* amount of memory used\n",
    "\n",
    "*side note: Pandas has excellent support for not-a-number (NaN) entries in DataFrames and Series.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 10 entries, 0 to 9\n",
      "Data columns (total 4 columns):\n",
      "a    10 non-null int64\n",
      "b    10 non-null float64\n",
      "c    10 non-null float64\n",
      "d    10 non-null object\n",
      "dtypes: float64(2), int64(1), object(1)\n",
      "memory usage: 400.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "df_dict.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrame Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Pandas DataFrames have a special Index, goes beyond integer indexing\n",
    "* Pandas DataFrames are indexed by column label, then (row) Index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Alice</th>\n",
       "      <th>Bob</th>\n",
       "      <th>Chuck</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Jan</th>\n",
       "      <td>1</td>\n",
       "      <td>0.108171</td>\n",
       "      <td>0.864162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feb</th>\n",
       "      <td>2</td>\n",
       "      <td>0.031362</td>\n",
       "      <td>0.500275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mar</th>\n",
       "      <td>3</td>\n",
       "      <td>0.003560</td>\n",
       "      <td>0.513126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Apr</th>\n",
       "      <td>4</td>\n",
       "      <td>0.076148</td>\n",
       "      <td>0.510653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>May</th>\n",
       "      <td>5</td>\n",
       "      <td>0.648280</td>\n",
       "      <td>0.807013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Alice       Bob     Chuck\n",
       "Jan      1  0.108171  0.864162\n",
       "Feb      2  0.031362  0.500275\n",
       "Mar      3  0.003560  0.513126\n",
       "Apr      4  0.076148  0.510653\n",
       "May      5  0.648280  0.807013"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_grades = pd.DataFrame({'Alice' : [1,2,3,4,5], \n",
    "                         'Bob':np.random.random(5), \n",
    "                         'Chuck' : np.random.random(5)},\n",
    "                         index=['Jan', 'Feb', 'Mar', 'Apr', 'May'])\n",
    "df_grades"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jan    1\n",
      "Feb    2\n",
      "Mar    3\n",
      "Apr    4\n",
      "May    5\n",
      "Name: Alice, dtype: int64\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "# Indexing is by column first (gives a Series)\n",
    "series_a = df_grades['Alice']\n",
    "\n",
    "print( series_a )\n",
    "print( type(series_a) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Indexing the column name, gives a Series, then 'positional' indexing\n",
    "df_grades['Alice'].iloc[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Indexing the column name, gives a Series, then 'label' indexing\n",
    "df_grades.loc['Jan','Alice']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterating over a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen with Series, we can iterate over DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Key: Jan , Value: 1\n",
      "Key: Feb , Value: 2\n",
      "Key: Mar , Value: 3\n",
      "Key: Apr , Value: 4\n",
      "Key: May , Value: 5\n"
     ]
    }
   ],
   "source": [
    "for key, value in df_grades['Alice'].iteritems():\n",
    "    print('Key:', '%s' % key, ', Value:', '%s' % value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrames support iteration over the `Index`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index: Jan , Row: 1.0\n",
      "Index: Feb , Row: 2.0\n",
      "Index: Mar , Row: 3.0\n",
      "Index: Apr , Row: 4.0\n",
      "Index: May , Row: 5.0\n"
     ]
    }
   ],
   "source": [
    "for index, row in df_grades.iterrows():\n",
    "    print('Index:', '%s' % index, ', Row:', '%s' % row['Alice'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrame Manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is just a brief introduction working with DataFrames. There's a *lot* more to learn..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     Alice\n",
      "Jan      1\n",
      "Feb      2\n",
      "Mar      3\n",
      "Apr      4\n",
      "May      5 \n",
      " <class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "# Extract a column as a new DataFrame\n",
    "df_col = df_grades[['Alice']]\n",
    "print( df_col, \"\\n\", type(df_col) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     Alice       Bob     Chuck\n",
      "Jan      1  0.108171  0.864162 \n",
      " <class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "# Extract a row as a new DataFrame\n",
    "df_row = df_grades.loc['Jan':'Jan']\n",
    "print( df_row, \"\\n\", type(df_row) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Jan    1\n",
       "Feb    2\n",
       "Mar    3\n",
       "Name: Alice, dtype: int64"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slicing along rows works just as with Series\n",
    "df_grades.loc['Jan':'Mar','Alice']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Alice</th>\n",
       "      <th>Bob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Jan</th>\n",
       "      <td>1</td>\n",
       "      <td>0.108171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feb</th>\n",
       "      <td>2</td>\n",
       "      <td>0.031362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mar</th>\n",
       "      <td>3</td>\n",
       "      <td>0.003560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Apr</th>\n",
       "      <td>4</td>\n",
       "      <td>0.076148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>May</th>\n",
       "      <td>5</td>\n",
       "      <td>0.648280</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Alice       Bob\n",
       "Jan      1  0.108171\n",
       "Feb      2  0.031362\n",
       "Mar      3  0.003560\n",
       "Apr      4  0.076148\n",
       "May      5  0.648280"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For column range, need to use the .ix attribute of the DataFrame\n",
    "df_grades.loc[:,'Alice':'Bob']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bob</th>\n",
       "      <th>Chuck</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mar</th>\n",
       "      <td>0.003560</td>\n",
       "      <td>0.513126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Apr</th>\n",
       "      <td>0.076148</td>\n",
       "      <td>0.510653</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Bob     Chuck\n",
       "Mar  0.003560  0.513126\n",
       "Apr  0.076148  0.510653"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For extracting row and column ranges\n",
    "df_grades.loc['Mar':'Apr', 'Bob':'Chuck']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Alice</th>\n",
       "      <th>Bob</th>\n",
       "      <th>Chuck</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Jan</th>\n",
       "      <td>1</td>\n",
       "      <td>0.108171</td>\n",
       "      <td>0.864162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feb</th>\n",
       "      <td>2</td>\n",
       "      <td>0.031362</td>\n",
       "      <td>0.500275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mar</th>\n",
       "      <td>3</td>\n",
       "      <td>0.003560</td>\n",
       "      <td>0.513126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Apr</th>\n",
       "      <td>4</td>\n",
       "      <td>0.076148</td>\n",
       "      <td>0.510653</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Alice       Bob     Chuck\n",
       "Jan      1  0.108171  0.864162\n",
       "Feb      2  0.031362  0.500275\n",
       "Mar      3  0.003560  0.513126\n",
       "Apr      4  0.076148  0.510653"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Boolean masks work too\n",
    "df_grades.loc[df_grades['Bob'] < .5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5] <class 'list'>\n",
      "[1 2 3 4 5] <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "# Pandas DataFrames can be cast to other types\n",
    "\n",
    "my_list  = list(df_grades['Alice'])\n",
    "print( my_list, type(my_list) )\n",
    "\n",
    "my_array = np.array(df_grades['Alice'])\n",
    "print( my_array, type(my_array) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary of what we've just seen:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas I/O: How We Get Data Into Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas has excellent support for file I/O (read/write) for a wide variety of common file formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.18.0'"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import modules needed for this lesson\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "%matplotlib inline\n",
    "\n",
    "# Set Jupyter notebook display options\n",
    "pd.options.display.max_rows = 6\n",
    "pd.options.display.max_columns = 8\n",
    "\n",
    "# Test the version of pandas: it should be 0.16 or greater\n",
    "pd.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The data used throughout this demonstration is from the catalog of exoplanets -- planets outside our solar system -- compiled by [exoplanets.org](http://exoplanets.org/table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Temporary Directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any time you are handling data, it is commonlly needed to handle \"tmp\" files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Remove old tmp dir, Create a new tmp dir\n",
    "import os\n",
    "if not os.path.exists(\"tmp\"):\n",
    "    os.mkdir(\"tmp\")\n",
    "\n",
    "# Define a convenience function to help us clean up\n",
    "def clean_tmp(file_name=\"tmp/exoplanet.csv\"):\n",
    "    if os.path.isfile(file_name):\n",
    "        os.remove(file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading and Inspecting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a simple demonstration of reading and inspecting data.\n",
    "\n",
    "We will see a much more detailed treatment of examining your data in the next lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "file_name = \"data/exoplanets.csv\"\n",
    "data = pd.read_csv(file_name)\n",
    "print(type(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>FIRSTURL</th>\n",
       "      <th>DATE</th>\n",
       "      <th>NUM_OBS</th>\n",
       "      <th>MASS(mjupiter)</th>\n",
       "      <th>RADIUS(rjupiter)</th>\n",
       "      <th>PERIOD(day)</th>\n",
       "      <th>ECCENTRICITY</th>\n",
       "      <th>SEMI_MAJOR_AXIS(AU)</th>\n",
       "      <th>SEPARATION(AU)</th>\n",
       "      <th>STAR_NAME</th>\n",
       "      <th>DIST(PARSEC)</th>\n",
       "      <th>RA_STRING(h:m:s)</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gamma Cep b</td>\n",
       "      <td>Campbell 1988</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1988ApJ...331..902C</td>\n",
       "      <td>1988</td>\n",
       "      <td>131.0</td>\n",
       "      <td>1.51728</td>\n",
       "      <td>NaN</td>\n",
       "      <td>905.5740</td>\n",
       "      <td>0.1200</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>gamma Cep</td>\n",
       "      <td>14.1024</td>\n",
       "      <td>+23:39:20.98</td>\n",
       "      <td>+77:37:55.08</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>5.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HD 114762 b</td>\n",
       "      <td>Latham 1989</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1989Natur.339...38L</td>\n",
       "      <td>1989</td>\n",
       "      <td>74.0</td>\n",
       "      <td>11.63510</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83.9151</td>\n",
       "      <td>0.3354</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>HD 114762</td>\n",
       "      <td>38.6548</td>\n",
       "      <td>+13:12:20.10</td>\n",
       "      <td>+17:31:01.66</td>\n",
       "      <td>0.894578</td>\n",
       "      <td>0.859205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          NAME       FIRSTREF  \\\n",
       "0  gamma Cep b  Campbell 1988   \n",
       "1  HD 114762 b    Latham 1989   \n",
       "\n",
       "                                            FIRSTURL  DATE  NUM_OBS  \\\n",
       "0  http://adsabs.harvard.edu/abs/1988ApJ...331..902C  1988    131.0   \n",
       "1  http://adsabs.harvard.edu/abs/1989Natur.339...38L  1989     74.0   \n",
       "\n",
       "   MASS(mjupiter)  RADIUS(rjupiter)  PERIOD(day)  ECCENTRICITY  \\\n",
       "0         1.51728               NaN     905.5740        0.1200   \n",
       "1        11.63510               NaN      83.9151        0.3354   \n",
       "\n",
       "   SEMI_MAJOR_AXIS(AU)  SEPARATION(AU)  STAR_NAME  DIST(PARSEC)  \\\n",
       "0             1.979300        1.979300  gamma Cep       14.1024   \n",
       "1             0.362932        0.362932  HD 114762       38.6548   \n",
       "\n",
       "  RA_STRING(h:m:s) DEC_STRING(d:m:s)  STAR_MASS(msun)  STAR_RADIUS(rsun)  \n",
       "0     +23:39:20.98      +77:37:55.08         1.260000           5.010000  \n",
       "1     +13:12:20.10      +17:31:01.66         0.894578           0.859205  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preview data after a read.\n",
    "# Notice the first column\n",
    "data.iloc[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=1642, step=1)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# There is an Index array imbedded in the DataFrame\n",
    "data.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>FIRSTURL</th>\n",
       "      <th>NUM_OBS</th>\n",
       "      <th>MASS(mjupiter)</th>\n",
       "      <th>RADIUS(rjupiter)</th>\n",
       "      <th>PERIOD(day)</th>\n",
       "      <th>ECCENTRICITY</th>\n",
       "      <th>SEMI_MAJOR_AXIS(AU)</th>\n",
       "      <th>SEPARATION(AU)</th>\n",
       "      <th>STAR_NAME</th>\n",
       "      <th>DIST(PARSEC)</th>\n",
       "      <th>RA_STRING(h:m:s)</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DATE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1988-01-01</th>\n",
       "      <td>gamma Cep b</td>\n",
       "      <td>Campbell 1988</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1988ApJ...331..902C</td>\n",
       "      <td>131.0</td>\n",
       "      <td>1.517280</td>\n",
       "      <td>NaN</td>\n",
       "      <td>905.574000</td>\n",
       "      <td>0.1200</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>gamma Cep</td>\n",
       "      <td>14.1024</td>\n",
       "      <td>+23:39:20.98</td>\n",
       "      <td>+77:37:55.08</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>5.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1989-01-01</th>\n",
       "      <td>HD 114762 b</td>\n",
       "      <td>Latham 1989</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1989Natur.339...38L</td>\n",
       "      <td>74.0</td>\n",
       "      <td>11.635100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83.915100</td>\n",
       "      <td>0.3354</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>HD 114762</td>\n",
       "      <td>38.6548</td>\n",
       "      <td>+13:12:20.10</td>\n",
       "      <td>+17:31:01.66</td>\n",
       "      <td>0.894578</td>\n",
       "      <td>0.859205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992-01-01</th>\n",
       "      <td>PSR B1257+12 B</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>http://adsabs.harvard.edu//abs/1992Natur.355.....</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.013531</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66.541900</td>\n",
       "      <td>0.0186</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>PSR B1257+12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992-01-01</th>\n",
       "      <td>PSR B1257+12 A</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>http://adsabs.harvard.edu//abs/1992Natur.355.....</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000063</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.262000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.168494</td>\n",
       "      <td>0.168494</td>\n",
       "      <td>PSR B1257+12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1993-01-01</th>\n",
       "      <td>beta Gem b</td>\n",
       "      <td>Hatzes 1993</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1993ApJ...413..339H</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2.758220</td>\n",
       "      <td>NaN</td>\n",
       "      <td>589.640010</td>\n",
       "      <td>0.0200</td>\n",
       "      <td>1.757390</td>\n",
       "      <td>1.757390</td>\n",
       "      <td>beta Gem</td>\n",
       "      <td>10.3584</td>\n",
       "      <td>+07:45:19.36</td>\n",
       "      <td>+28:01:34.72</td>\n",
       "      <td>2.080000</td>\n",
       "      <td>8.156090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1994-01-01</th>\n",
       "      <td>PSR B1257+12 C</td>\n",
       "      <td>Wolszczan 1994</td>\n",
       "      <td>http://adsabs.harvard.edu//abs/1994Sci...264.....</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.012272</td>\n",
       "      <td>NaN</td>\n",
       "      <td>98.211400</td>\n",
       "      <td>0.0252</td>\n",
       "      <td>0.416595</td>\n",
       "      <td>0.416595</td>\n",
       "      <td>PSR B1257+12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995-01-01</th>\n",
       "      <td>51 Peg b</td>\n",
       "      <td>Mayor 1995</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1995Natur.378..355M</td>\n",
       "      <td>256.0</td>\n",
       "      <td>0.461143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.230785</td>\n",
       "      <td>0.0130</td>\n",
       "      <td>0.052107</td>\n",
       "      <td>0.052107</td>\n",
       "      <td>51 Peg</td>\n",
       "      <td>15.6079</td>\n",
       "      <td>+22:57:27.85</td>\n",
       "      <td>+20:46:07.26</td>\n",
       "      <td>1.054000</td>\n",
       "      <td>1.024510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996-01-01</th>\n",
       "      <td>47 UMa b</td>\n",
       "      <td>Butler 1996</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1996ApJ...464L.153B</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.545890</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1078.000000</td>\n",
       "      <td>0.0320</td>\n",
       "      <td>2.101490</td>\n",
       "      <td>2.101490</td>\n",
       "      <td>47 UMa</td>\n",
       "      <td>14.0627</td>\n",
       "      <td>+10:59:28.22</td>\n",
       "      <td>+40:25:48.44</td>\n",
       "      <td>1.063000</td>\n",
       "      <td>1.118690</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996-01-01</th>\n",
       "      <td>70 Vir b</td>\n",
       "      <td>Marcy 1996</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1996ApJ...464L.147M</td>\n",
       "      <td>74.0</td>\n",
       "      <td>7.461440</td>\n",
       "      <td>NaN</td>\n",
       "      <td>116.688400</td>\n",
       "      <td>0.4007</td>\n",
       "      <td>0.483601</td>\n",
       "      <td>0.483601</td>\n",
       "      <td>70 Vir</td>\n",
       "      <td>17.9856</td>\n",
       "      <td>+13:28:25.95</td>\n",
       "      <td>+13:46:48.68</td>\n",
       "      <td>1.101000</td>\n",
       "      <td>1.597750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-01</th>\n",
       "      <td>16 Cyg B b</td>\n",
       "      <td>Cochran 1997</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1997ApJ...483..457C</td>\n",
       "      <td>95.0</td>\n",
       "      <td>1.639970</td>\n",
       "      <td>NaN</td>\n",
       "      <td>798.500000</td>\n",
       "      <td>0.6810</td>\n",
       "      <td>1.660270</td>\n",
       "      <td>1.660270</td>\n",
       "      <td>16 Cyg B</td>\n",
       "      <td>21.2134</td>\n",
       "      <td>+19:41:52.10</td>\n",
       "      <td>+50:31:04.51</td>\n",
       "      <td>0.956000</td>\n",
       "      <td>1.117600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      NAME        FIRSTREF  \\\n",
       "DATE                                         \n",
       "1988-01-01     gamma Cep b   Campbell 1988   \n",
       "1989-01-01     HD 114762 b     Latham 1989   \n",
       "1992-01-01  PSR B1257+12 B  Wolszczan 1992   \n",
       "1992-01-01  PSR B1257+12 A  Wolszczan 1992   \n",
       "1993-01-01      beta Gem b     Hatzes 1993   \n",
       "1994-01-01  PSR B1257+12 C  Wolszczan 1994   \n",
       "1995-01-01        51 Peg b      Mayor 1995   \n",
       "1996-01-01        47 UMa b     Butler 1996   \n",
       "1996-01-01        70 Vir b      Marcy 1996   \n",
       "1997-01-01      16 Cyg B b    Cochran 1997   \n",
       "\n",
       "                                                     FIRSTURL  NUM_OBS  \\\n",
       "DATE                                                                     \n",
       "1988-01-01  http://adsabs.harvard.edu/abs/1988ApJ...331..902C    131.0   \n",
       "1989-01-01  http://adsabs.harvard.edu/abs/1989Natur.339...38L     74.0   \n",
       "1992-01-01  http://adsabs.harvard.edu//abs/1992Natur.355.....      NaN   \n",
       "1992-01-01  http://adsabs.harvard.edu//abs/1992Natur.355.....      NaN   \n",
       "1993-01-01  http://adsabs.harvard.edu/abs/1993ApJ...413..339H     55.0   \n",
       "1994-01-01  http://adsabs.harvard.edu//abs/1994Sci...264.....      NaN   \n",
       "1995-01-01  http://adsabs.harvard.edu/abs/1995Natur.378..355M    256.0   \n",
       "1996-01-01  http://adsabs.harvard.edu/abs/1996ApJ...464L.153B      NaN   \n",
       "1996-01-01  http://adsabs.harvard.edu/abs/1996ApJ...464L.147M     74.0   \n",
       "1997-01-01  http://adsabs.harvard.edu/abs/1997ApJ...483..457C     95.0   \n",
       "\n",
       "            MASS(mjupiter)  RADIUS(rjupiter)  PERIOD(day)  ECCENTRICITY  \\\n",
       "DATE                                                                      \n",
       "1988-01-01        1.517280               NaN   905.574000        0.1200   \n",
       "1989-01-01       11.635100               NaN    83.915100        0.3354   \n",
       "1992-01-01        0.013531               NaN    66.541900        0.0186   \n",
       "1992-01-01        0.000063               NaN    25.262000        0.0000   \n",
       "1993-01-01        2.758220               NaN   589.640010        0.0200   \n",
       "1994-01-01        0.012272               NaN    98.211400        0.0252   \n",
       "1995-01-01        0.461143               NaN     4.230785        0.0130   \n",
       "1996-01-01        2.545890               NaN  1078.000000        0.0320   \n",
       "1996-01-01        7.461440               NaN   116.688400        0.4007   \n",
       "1997-01-01        1.639970               NaN   798.500000        0.6810   \n",
       "\n",
       "            SEMI_MAJOR_AXIS(AU)  SEPARATION(AU)     STAR_NAME  DIST(PARSEC)  \\\n",
       "DATE                                                                          \n",
       "1988-01-01             1.979300        1.979300     gamma Cep       14.1024   \n",
       "1989-01-01             0.362932        0.362932     HD 114762       38.6548   \n",
       "1992-01-01             0.321369        0.321369  PSR B1257+12           NaN   \n",
       "1992-01-01             0.168494        0.168494  PSR B1257+12           NaN   \n",
       "1993-01-01             1.757390        1.757390      beta Gem       10.3584   \n",
       "1994-01-01             0.416595        0.416595  PSR B1257+12           NaN   \n",
       "1995-01-01             0.052107        0.052107        51 Peg       15.6079   \n",
       "1996-01-01             2.101490        2.101490        47 UMa       14.0627   \n",
       "1996-01-01             0.483601        0.483601        70 Vir       17.9856   \n",
       "1997-01-01             1.660270        1.660270      16 Cyg B       21.2134   \n",
       "\n",
       "           RA_STRING(h:m:s) DEC_STRING(d:m:s)  STAR_MASS(msun)  \\\n",
       "DATE                                                             \n",
       "1988-01-01     +23:39:20.98      +77:37:55.08         1.260000   \n",
       "1989-01-01     +13:12:20.10      +17:31:01.66         0.894578   \n",
       "1992-01-01     13:00:03.577       +12:40:56.4              NaN   \n",
       "1992-01-01     13:00:03.577       +12:40:56.4              NaN   \n",
       "1993-01-01     +07:45:19.36      +28:01:34.72         2.080000   \n",
       "1994-01-01     13:00:03.577       +12:40:56.4              NaN   \n",
       "1995-01-01     +22:57:27.85      +20:46:07.26         1.054000   \n",
       "1996-01-01     +10:59:28.22      +40:25:48.44         1.063000   \n",
       "1996-01-01     +13:28:25.95      +13:46:48.68         1.101000   \n",
       "1997-01-01     +19:41:52.10      +50:31:04.51         0.956000   \n",
       "\n",
       "            STAR_RADIUS(rsun)  \n",
       "DATE                           \n",
       "1988-01-01           5.010000  \n",
       "1989-01-01           0.859205  \n",
       "1992-01-01                NaN  \n",
       "1992-01-01                NaN  \n",
       "1993-01-01           8.156090  \n",
       "1994-01-01                NaN  \n",
       "1995-01-01           1.024510  \n",
       "1996-01-01           1.118690  \n",
       "1996-01-01           1.597750  \n",
       "1997-01-01           1.117600  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can use a column to create a better Index\n",
    "data = pd.read_csv(file_name, parse_dates=True, index_col='DATE')\n",
    "data.head(10)     # another way to preview data after a read"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['1988-01-01', '1989-01-01', '1992-01-01', '1992-01-01',\n",
       "               '1993-01-01', '1994-01-01', '1995-01-01', '1996-01-01',\n",
       "               '1996-01-01', '1997-01-01',\n",
       "               ...\n",
       "               '2015-01-01', '2015-01-01', '2015-01-01', '2015-01-01',\n",
       "               '2015-01-01', '2015-01-01', '2015-01-01', '2015-01-01',\n",
       "               '2015-01-01', '2015-01-01'],\n",
       "              dtype='datetime64[ns]', name='DATE', length=1642, freq=None)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Notice it is not just an Index, but a DatetimeIndex; note the dtype\n",
    "data.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>FIRSTURL</th>\n",
       "      <th>NUM_OBS</th>\n",
       "      <th>MASS(mjupiter)</th>\n",
       "      <th>RADIUS(rjupiter)</th>\n",
       "      <th>PERIOD(day)</th>\n",
       "      <th>ECCENTRICITY</th>\n",
       "      <th>SEMI_MAJOR_AXIS(AU)</th>\n",
       "      <th>SEPARATION(AU)</th>\n",
       "      <th>STAR_NAME</th>\n",
       "      <th>DIST(PARSEC)</th>\n",
       "      <th>RA_STRING(h:m:s)</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DATE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>gamma Cep b</td>\n",
       "      <td>Campbell 1988</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1988ApJ...331..902C</td>\n",
       "      <td>131.0</td>\n",
       "      <td>1.517280</td>\n",
       "      <td>NaN</td>\n",
       "      <td>905.5740</td>\n",
       "      <td>0.1200</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>gamma Cep</td>\n",
       "      <td>14.1024</td>\n",
       "      <td>+23:39:20.98</td>\n",
       "      <td>+77:37:55.08</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>5.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>HD 114762 b</td>\n",
       "      <td>Latham 1989</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1989Natur.339...38L</td>\n",
       "      <td>74.0</td>\n",
       "      <td>11.635100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83.9151</td>\n",
       "      <td>0.3354</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>HD 114762</td>\n",
       "      <td>38.6548</td>\n",
       "      <td>+13:12:20.10</td>\n",
       "      <td>+17:31:01.66</td>\n",
       "      <td>0.894578</td>\n",
       "      <td>0.859205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>PSR B1257+12 B</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>http://adsabs.harvard.edu//abs/1992Natur.355.....</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.013531</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66.5419</td>\n",
       "      <td>0.0186</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>PSR B1257+12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      NAME        FIRSTREF  \\\n",
       "DATE                                         \n",
       "2015-01-01     gamma Cep b   Campbell 1988   \n",
       "2015-01-01     HD 114762 b     Latham 1989   \n",
       "2015-01-01  PSR B1257+12 B  Wolszczan 1992   \n",
       "\n",
       "                                                     FIRSTURL  NUM_OBS  \\\n",
       "DATE                                                                     \n",
       "2015-01-01  http://adsabs.harvard.edu/abs/1988ApJ...331..902C    131.0   \n",
       "2015-01-01  http://adsabs.harvard.edu/abs/1989Natur.339...38L     74.0   \n",
       "2015-01-01  http://adsabs.harvard.edu//abs/1992Natur.355.....      NaN   \n",
       "\n",
       "            MASS(mjupiter)  RADIUS(rjupiter)  PERIOD(day)  ECCENTRICITY  \\\n",
       "DATE                                                                      \n",
       "2015-01-01        1.517280               NaN     905.5740        0.1200   \n",
       "2015-01-01       11.635100               NaN      83.9151        0.3354   \n",
       "2015-01-01        0.013531               NaN      66.5419        0.0186   \n",
       "\n",
       "            SEMI_MAJOR_AXIS(AU)  SEPARATION(AU)     STAR_NAME  DIST(PARSEC)  \\\n",
       "DATE                                                                          \n",
       "2015-01-01             1.979300        1.979300     gamma Cep       14.1024   \n",
       "2015-01-01             0.362932        0.362932     HD 114762       38.6548   \n",
       "2015-01-01             0.321369        0.321369  PSR B1257+12           NaN   \n",
       "\n",
       "           RA_STRING(h:m:s) DEC_STRING(d:m:s)  STAR_MASS(msun)  \\\n",
       "DATE                                                             \n",
       "2015-01-01     +23:39:20.98      +77:37:55.08         1.260000   \n",
       "2015-01-01     +13:12:20.10      +17:31:01.66         0.894578   \n",
       "2015-01-01     13:00:03.577       +12:40:56.4              NaN   \n",
       "\n",
       "            STAR_RADIUS(rsun)  \n",
       "DATE                           \n",
       "2015-01-01           5.010000  \n",
       "2015-01-01           0.859205  \n",
       "2015-01-01                NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can sort by the Index\n",
    "data = data.set_index(data.index.sort_values(ascending=False))\n",
    "data.iloc[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Minimal Cleanup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later, we will cover data clean-up in much more detail. \n",
    "\n",
    "Here are just a few simple things to do to inspect and clean-up data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['NAME', 'FIRSTREF', 'FIRSTURL', 'NUM_OBS', 'MASS(mjupiter)',\n",
       "       'RADIUS(rjupiter)', 'PERIOD(day)', 'ECCENTRICITY',\n",
       "       'SEMI_MAJOR_AXIS(AU)', 'SEPARATION(AU)', 'STAR_NAME', 'DIST(PARSEC)',\n",
       "       'RA_STRING(h:m:s)', 'DEC_STRING(d:m:s)', 'STAR_MASS(msun)',\n",
       "       'STAR_RADIUS(rsun)'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the labels of some or all of the columns\n",
    "# data.columns[0:10]\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DATE\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "2015-01-01      NaN\n",
       "              ...  \n",
       "2000-01-01    1.145\n",
       "2000-01-01    1.533\n",
       "2000-01-01      NaN\n",
       "2000-01-01    0.998\n",
       "2000-01-01      NaN\n",
       "2000-01-01      NaN\n",
       "2000-01-01      NaN\n",
       "2000-01-01      NaN\n",
       "1999-01-01      NaN\n",
       "1999-01-01    0.157\n",
       "1999-01-01    0.969\n",
       "1999-01-01      NaN\n",
       "1999-01-01    1.065\n",
       "1999-01-01      NaN\n",
       "1998-01-01    1.040\n",
       "1998-01-01    1.560\n",
       "1997-01-01    0.231\n",
       "1997-01-01      NaN\n",
       "1997-01-01    1.462\n",
       "1997-01-01      NaN\n",
       "1997-01-01    0.955\n",
       "1996-01-01    1.430\n",
       "1996-01-01    0.900\n",
       "1995-01-01    0.139\n",
       "1994-01-01    0.944\n",
       "1993-01-01      NaN\n",
       "1992-01-01    1.040\n",
       "1992-01-01      NaN\n",
       "1989-01-01      NaN\n",
       "1988-01-01      NaN\n",
       "Name: RADIUS(rjupiter), dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['RADIUS(rjupiter)']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DATE\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "2015-01-01   NaN\n",
       "Name: RADIUS(rjupiter), dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_data = data['RADIUS(rjupiter)']\n",
    "print(type(extract_data))\n",
    "extract_data.iloc[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- zero fill: \n",
    "    ```df['var'] = 0```\n",
    "   - values from NumPy array: ```df['my_data'] = data```\n",
    "   - note: df.var construct can not create a column by that\n",
    "     name; only used to access existing columns by name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Add a column labeled 'price'\n",
    "data['price'] = 1e6 # One *MILLION* dollars!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DATE\n",
       "2015-01-01    1000000.0\n",
       "2015-01-01    1000000.0\n",
       "2015-01-01    1000000.0\n",
       "2015-01-01    1000000.0\n",
       "2015-01-01    1000000.0\n",
       "Name: price, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the updated DataFrame\n",
    "data.price.iloc[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Demonstration that you can delete columns after a file read\n",
    "del data['FIRSTURL']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>NUM_OBS</th>\n",
       "      <th>MASS(mjupiter)</th>\n",
       "      <th>RADIUS(rjupiter)</th>\n",
       "      <th>PERIOD(day)</th>\n",
       "      <th>ECCENTRICITY</th>\n",
       "      <th>SEMI_MAJOR_AXIS(AU)</th>\n",
       "      <th>SEPARATION(AU)</th>\n",
       "      <th>STAR_NAME</th>\n",
       "      <th>DIST(PARSEC)</th>\n",
       "      <th>RA_STRING(h:m:s)</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DATE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>gamma Cep b</td>\n",
       "      <td>Campbell 1988</td>\n",
       "      <td>131.0</td>\n",
       "      <td>1.517280</td>\n",
       "      <td>NaN</td>\n",
       "      <td>905.57400</td>\n",
       "      <td>0.1200</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>gamma Cep</td>\n",
       "      <td>14.1024</td>\n",
       "      <td>+23:39:20.98</td>\n",
       "      <td>+77:37:55.08</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>5.010000</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>HD 114762 b</td>\n",
       "      <td>Latham 1989</td>\n",
       "      <td>74.0</td>\n",
       "      <td>11.635100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83.91510</td>\n",
       "      <td>0.3354</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>HD 114762</td>\n",
       "      <td>38.6548</td>\n",
       "      <td>+13:12:20.10</td>\n",
       "      <td>+17:31:01.66</td>\n",
       "      <td>0.894578</td>\n",
       "      <td>0.859205</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>PSR B1257+12 B</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.013531</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66.54190</td>\n",
       "      <td>0.0186</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>PSR B1257+12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>PSR B1257+12 A</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000063</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.26200</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.168494</td>\n",
       "      <td>0.168494</td>\n",
       "      <td>PSR B1257+12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>beta Gem b</td>\n",
       "      <td>Hatzes 1993</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2.758220</td>\n",
       "      <td>NaN</td>\n",
       "      <td>589.64001</td>\n",
       "      <td>0.0200</td>\n",
       "      <td>1.757390</td>\n",
       "      <td>1.757390</td>\n",
       "      <td>beta Gem</td>\n",
       "      <td>10.3584</td>\n",
       "      <td>+07:45:19.36</td>\n",
       "      <td>+28:01:34.72</td>\n",
       "      <td>2.080000</td>\n",
       "      <td>8.156090</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      NAME        FIRSTREF  NUM_OBS  MASS(mjupiter)  \\\n",
       "DATE                                                                  \n",
       "2015-01-01     gamma Cep b   Campbell 1988    131.0        1.517280   \n",
       "2015-01-01     HD 114762 b     Latham 1989     74.0       11.635100   \n",
       "2015-01-01  PSR B1257+12 B  Wolszczan 1992      NaN        0.013531   \n",
       "2015-01-01  PSR B1257+12 A  Wolszczan 1992      NaN        0.000063   \n",
       "2015-01-01      beta Gem b     Hatzes 1993     55.0        2.758220   \n",
       "\n",
       "            RADIUS(rjupiter)  PERIOD(day)  ECCENTRICITY  SEMI_MAJOR_AXIS(AU)  \\\n",
       "DATE                                                                           \n",
       "2015-01-01               NaN    905.57400        0.1200             1.979300   \n",
       "2015-01-01               NaN     83.91510        0.3354             0.362932   \n",
       "2015-01-01               NaN     66.54190        0.0186             0.321369   \n",
       "2015-01-01               NaN     25.26200        0.0000             0.168494   \n",
       "2015-01-01               NaN    589.64001        0.0200             1.757390   \n",
       "\n",
       "            SEPARATION(AU)     STAR_NAME  DIST(PARSEC) RA_STRING(h:m:s)  \\\n",
       "DATE                                                                      \n",
       "2015-01-01        1.979300     gamma Cep       14.1024     +23:39:20.98   \n",
       "2015-01-01        0.362932     HD 114762       38.6548     +13:12:20.10   \n",
       "2015-01-01        0.321369  PSR B1257+12           NaN     13:00:03.577   \n",
       "2015-01-01        0.168494  PSR B1257+12           NaN     13:00:03.577   \n",
       "2015-01-01        1.757390      beta Gem       10.3584     +07:45:19.36   \n",
       "\n",
       "           DEC_STRING(d:m:s)  STAR_MASS(msun)  STAR_RADIUS(rsun)      price  \n",
       "DATE                                                                         \n",
       "2015-01-01      +77:37:55.08         1.260000           5.010000  1000000.0  \n",
       "2015-01-01      +17:31:01.66         0.894578           0.859205  1000000.0  \n",
       "2015-01-01       +12:40:56.4              NaN                NaN  1000000.0  \n",
       "2015-01-01       +12:40:56.4              NaN                NaN  1000000.0  \n",
       "2015-01-01      +28:01:34.72         2.080000           8.156090  1000000.0  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the results\n",
    "data.iloc[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Renaming Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PLANET</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>NUM_OBS</th>\n",
       "      <th>MASS(mjupiter)</th>\n",
       "      <th>RADIUS(rjupiter)</th>\n",
       "      <th>PERIOD(day)</th>\n",
       "      <th>ECCENTRICITY</th>\n",
       "      <th>SEMI_MAJOR_AXIS(AU)</th>\n",
       "      <th>SEPARATION(AU)</th>\n",
       "      <th>STAR_NAME</th>\n",
       "      <th>DIST(PARSEC)</th>\n",
       "      <th>RA_STRING(h:m:s)</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DATE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>gamma Cep b</td>\n",
       "      <td>Campbell 1988</td>\n",
       "      <td>131.0</td>\n",
       "      <td>1.517280</td>\n",
       "      <td>NaN</td>\n",
       "      <td>905.5740</td>\n",
       "      <td>0.1200</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>1.979300</td>\n",
       "      <td>gamma Cep</td>\n",
       "      <td>14.1024</td>\n",
       "      <td>+23:39:20.98</td>\n",
       "      <td>+77:37:55.08</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>5.010000</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>HD 114762 b</td>\n",
       "      <td>Latham 1989</td>\n",
       "      <td>74.0</td>\n",
       "      <td>11.635100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83.9151</td>\n",
       "      <td>0.3354</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>0.362932</td>\n",
       "      <td>HD 114762</td>\n",
       "      <td>38.6548</td>\n",
       "      <td>+13:12:20.10</td>\n",
       "      <td>+17:31:01.66</td>\n",
       "      <td>0.894578</td>\n",
       "      <td>0.859205</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-01-01</th>\n",
       "      <td>PSR B1257+12 B</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.013531</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66.5419</td>\n",
       "      <td>0.0186</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>0.321369</td>\n",
       "      <td>PSR B1257+12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13:00:03.577</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    PLANET        FIRSTREF  NUM_OBS  MASS(mjupiter)  \\\n",
       "DATE                                                                  \n",
       "2015-01-01     gamma Cep b   Campbell 1988    131.0        1.517280   \n",
       "2015-01-01     HD 114762 b     Latham 1989     74.0       11.635100   \n",
       "2015-01-01  PSR B1257+12 B  Wolszczan 1992      NaN        0.013531   \n",
       "\n",
       "            RADIUS(rjupiter)  PERIOD(day)  ECCENTRICITY  SEMI_MAJOR_AXIS(AU)  \\\n",
       "DATE                                                                           \n",
       "2015-01-01               NaN     905.5740        0.1200             1.979300   \n",
       "2015-01-01               NaN      83.9151        0.3354             0.362932   \n",
       "2015-01-01               NaN      66.5419        0.0186             0.321369   \n",
       "\n",
       "            SEPARATION(AU)     STAR_NAME  DIST(PARSEC) RA_STRING(h:m:s)  \\\n",
       "DATE                                                                      \n",
       "2015-01-01        1.979300     gamma Cep       14.1024     +23:39:20.98   \n",
       "2015-01-01        0.362932     HD 114762       38.6548     +13:12:20.10   \n",
       "2015-01-01        0.321369  PSR B1257+12           NaN     13:00:03.577   \n",
       "\n",
       "           DEC_STRING(d:m:s)  STAR_MASS(msun)  STAR_RADIUS(rsun)      price  \n",
       "DATE                                                                         \n",
       "2015-01-01      +77:37:55.08         1.260000           5.010000  1000000.0  \n",
       "2015-01-01      +17:31:01.66         0.894578           0.859205  1000000.0  \n",
       "2015-01-01       +12:40:56.4              NaN                NaN  1000000.0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change the label of a single column\n",
    "data = data.rename(columns={'NAME':'PLANET'})\n",
    "data.iloc[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question/Exercise\n",
    "1. What is average length of a year for the known expoplanets?\n",
    "\n",
    "2. Plot the period (year length) for the exoplanets in order of their discovery date. Why does the plot look odd?\n",
    "\n",
    "3. Interstellar real estate: Instead of planets being ONE MILLION DOLLARS each, they are 1 million dollars per Jupiter mass - but anything over 10 parsecs is out \"in the boonies\" so we subtract 25,000 dollars for each parsec beyond 10. Create a column with the price of each planet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File Formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas has an extensive set of I/O methods:\n",
    "* Can read from a wide range of flat files, including Excel and HDF5\n",
    "* Can also read SQL queries into memory\n",
    "* Sensible defaults, automatically labeled and typed `DataFrames`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pd.read_clipboard\n",
      "pd.read_csv\n",
      "pd.read_excel\n",
      "pd.read_fwf\n",
      "pd.read_gbq\n",
      "pd.read_hdf\n",
      "pd.read_html\n",
      "pd.read_json\n",
      "pd.read_msgpack\n",
      "pd.read_pickle\n",
      "pd.read_sas\n",
      "pd.read_sql\n",
      "pd.read_sql_query\n",
      "pd.read_sql_table\n",
      "pd.read_stata\n",
      "pd.read_table\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print a list of all read_* methods in pandas\n",
    "\n",
    "print(\"\".join([\"pd.%s\\n\" % reader \n",
    "               for reader in dir(pd) \n",
    "               if reader.startswith('read_')]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pd.DataFrame.to_clipboard\n",
      "pd.DataFrame.to_csv\n",
      "pd.DataFrame.to_dense\n",
      "pd.DataFrame.to_dict\n",
      "pd.DataFrame.to_excel\n",
      "pd.DataFrame.to_gbq\n",
      "pd.DataFrame.to_hdf\n",
      "pd.DataFrame.to_html\n",
      "pd.DataFrame.to_json\n",
      "pd.DataFrame.to_latex\n",
      "pd.DataFrame.to_msgpack\n",
      "pd.DataFrame.to_panel\n",
      "pd.DataFrame.to_period\n",
      "pd.DataFrame.to_pickle\n",
      "pd.DataFrame.to_records\n",
      "pd.DataFrame.to_sparse\n",
      "pd.DataFrame.to_sql\n",
      "pd.DataFrame.to_stata\n",
      "pd.DataFrame.to_string\n",
      "pd.DataFrame.to_timestamp\n",
      "pd.DataFrame.to_wide\n",
      "pd.DataFrame.to_xarray\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# DataFrame can be cast to files or other objects\n",
    "\n",
    "print(\"\".join([\"pd.DataFrame.%s\\n\" % reader \n",
    "               for reader in dir(pd.DataFrame) \n",
    "               if reader.startswith('to_')]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Below are external links to details on the file formats that pandas can read.\n",
    "\n",
    "- [CSV](https://en.wikipedia.org/wiki/Comma-separated_values)\n",
    "- Excel\n",
    "- SQL\n",
    "- [JSON](http://www.json.org/)\n",
    "- [HDF5](https://www.hdfgroup.org/HDF5/)\n",
    "- [pickle](https://docs.python.org/3/library/pickle.html)\n",
    "- [msgpack](http://msgpack.org/)\n",
    "- [Stata](https://en.wikipedia.org/wiki/Stata)\n",
    "- [Google BigQuery](https://en.wikipedia.org/wiki/BigQuery)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With that said, now is a good time to say: \n",
    "## NEVER WRITE YOUR OWN CSV READER IN PYTHON. \n",
    "(it's probably a bad idea)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![messy](img/the-more-you-know.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We will now demonstrate using pandas reading and writing to a variety of file formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## CSV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/io.html#csv-text-files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/exoplanets.csv', \n",
    "                 parse_dates=['DATE'],\n",
    "                 encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1642 entries, 0 to 1641\n",
      "Data columns (total 17 columns):\n",
      "NAME                   1642 non-null object\n",
      "FIRSTREF               1642 non-null object\n",
      "FIRSTURL               1642 non-null object\n",
      "DATE                   1642 non-null datetime64[ns]\n",
      "NUM_OBS                653 non-null float64\n",
      "MASS(mjupiter)         1641 non-null float64\n",
      "RADIUS(rjupiter)       1147 non-null float64\n",
      "PERIOD(day)            1629 non-null float64\n",
      "ECCENTRICITY           857 non-null float64\n",
      "SEMI_MAJOR_AXIS(AU)    1629 non-null float64\n",
      "SEPARATION(AU)         1638 non-null float64\n",
      "STAR_NAME              1642 non-null object\n",
      "DIST(PARSEC)           737 non-null float64\n",
      "RA_STRING(h:m:s)       1642 non-null object\n",
      "DEC_STRING(d:m:s)      1642 non-null object\n",
      "STAR_MASS(msun)        927 non-null float64\n",
      "STAR_RADIUS(rsun)      1585 non-null float64\n",
      "dtypes: datetime64[ns](1), float64(10), object(6)\n",
      "memory usage: 218.2+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>FIRSTURL</th>\n",
       "      <th>DATE</th>\n",
       "      <th>...</th>\n",
       "      <th>RA_STRING(h:m:s)</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>47 UMa b</td>\n",
       "      <td>Butler 1996</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1996ApJ...464L.153B</td>\n",
       "      <td>1996-01-01</td>\n",
       "      <td>...</td>\n",
       "      <td>+10:59:28.22</td>\n",
       "      <td>+40:25:48.44</td>\n",
       "      <td>1.063</td>\n",
       "      <td>1.11869</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       NAME     FIRSTREF                                           FIRSTURL  \\\n",
       "7  47 UMa b  Butler 1996  http://adsabs.harvard.edu/abs/1996ApJ...464L.153B   \n",
       "\n",
       "        DATE        ...          RA_STRING(h:m:s)  DEC_STRING(d:m:s)  \\\n",
       "7 1996-01-01        ...              +10:59:28.22       +40:25:48.44   \n",
       "\n",
       "   STAR_MASS(msun)  STAR_RADIUS(rsun)  \n",
       "7            1.063            1.11869  \n",
       "\n",
       "[1 rows x 17 columns]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The first exoplanet to get news headlines was in 1996.\n",
    "# Use some indexing to inspect the rows.\n",
    "# Combining head() and .iloc[] bracket indexing.\n",
    "df.head(8).iloc[-1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do some simple data extractions to prove we can."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'47 UMa b'"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract the name using [row index] and [column label]\n",
    "df.loc[7,'NAME']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.54589"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract the mass using the [column label] and [row index]\n",
    "df.loc[7,'MASS(mjupiter)']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.54589"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract the mass using and [row index] and [column index]\n",
    "# The MASS(mjupiter) column is the 5th labeled column (not counting the Index)\n",
    "# All pandas DataFrames have the method .iloc() \n",
    "df.iloc[7,5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df.to_csv('tmp/exoplanets.csv', index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV Online"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas can read a data table directly from the web"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>water_level</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>reservoir_storage</th>\n",
       "      <th>conservation_storage</th>\n",
       "      <th>percent_full</th>\n",
       "      <th>conservation_capacity</th>\n",
       "      <th>dead_pool_capacity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1966-01-01</th>\n",
       "      <td>480.88</td>\n",
       "      <td>968.23</td>\n",
       "      <td>9426</td>\n",
       "      <td>8817</td>\n",
       "      <td>36.8</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1966-02-01</th>\n",
       "      <td>492.67</td>\n",
       "      <td>1573.66</td>\n",
       "      <td>24376</td>\n",
       "      <td>23767</td>\n",
       "      <td>99.1</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1966-03-01</th>\n",
       "      <td>492.68</td>\n",
       "      <td>1574.32</td>\n",
       "      <td>24392</td>\n",
       "      <td>23783</td>\n",
       "      <td>99.2</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-03-27</th>\n",
       "      <td>492.05</td>\n",
       "      <td>1536.96</td>\n",
       "      <td>23412</td>\n",
       "      <td>22803</td>\n",
       "      <td>95.1</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-03-28</th>\n",
       "      <td>492.02</td>\n",
       "      <td>1535.35</td>\n",
       "      <td>23366</td>\n",
       "      <td>22757</td>\n",
       "      <td>94.9</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-03-29</th>\n",
       "      <td>492.19</td>\n",
       "      <td>1544.67</td>\n",
       "      <td>23628</td>\n",
       "      <td>23019</td>\n",
       "      <td>96.0</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14813 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            water_level  surface_area  reservoir_storage  \\\n",
       "date                                                       \n",
       "1966-01-01       480.88        968.23               9426   \n",
       "1966-02-01       492.67       1573.66              24376   \n",
       "1966-03-01       492.68       1574.32              24392   \n",
       "...                 ...           ...                ...   \n",
       "2016-03-27       492.05       1536.96              23412   \n",
       "2016-03-28       492.02       1535.35              23366   \n",
       "2016-03-29       492.19       1544.67              23628   \n",
       "\n",
       "            conservation_storage  percent_full  conservation_capacity  \\\n",
       "date                                                                    \n",
       "1966-01-01                  8817          36.8                  23972   \n",
       "1966-02-01                 23767          99.1                  23972   \n",
       "1966-03-01                 23783          99.2                  23972   \n",
       "...                          ...           ...                    ...   \n",
       "2016-03-27                 22803          95.1                  23972   \n",
       "2016-03-28                 22757          94.9                  23972   \n",
       "2016-03-29                 23019          96.0                  23972   \n",
       "\n",
       "            dead_pool_capacity  \n",
       "date                            \n",
       "1966-01-01                 609  \n",
       "1966-02-01                 609  \n",
       "1966-03-01                 609  \n",
       "...                        ...  \n",
       "2016-03-27                 609  \n",
       "2016-03-28                 609  \n",
       "2016-03-29                 609  \n",
       "\n",
       "[14813 rows x 7 columns]"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'http://waterdatafortexas.org/reservoirs/individual/austin.csv'\n",
    "df  = pd.read_csv(url, comment='#', index_col='date', parse_dates=True)\n",
    "df\n",
    "\n",
    "# note: you can parse dates, and ignore comment lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Excel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/io.html#excel-files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Save the DataFrame as an Excel file\n",
    "file_name = 'tmp/exoplanets.xls'\n",
    "clean_tmp(file_name)\n",
    "\n",
    "df.to_excel(file_name, index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>water_level</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>reservoir_storage</th>\n",
       "      <th>conservation_storage</th>\n",
       "      <th>percent_full</th>\n",
       "      <th>conservation_capacity</th>\n",
       "      <th>dead_pool_capacity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>480.88</td>\n",
       "      <td>968.23</td>\n",
       "      <td>9426</td>\n",
       "      <td>8817</td>\n",
       "      <td>36.8</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>492.67</td>\n",
       "      <td>1573.66</td>\n",
       "      <td>24376</td>\n",
       "      <td>23767</td>\n",
       "      <td>99.1</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>492.68</td>\n",
       "      <td>1574.32</td>\n",
       "      <td>24392</td>\n",
       "      <td>23783</td>\n",
       "      <td>99.2</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   water_level  surface_area  reservoir_storage  conservation_storage  \\\n",
       "0       480.88        968.23               9426                  8817   \n",
       "1       492.67       1573.66              24376                 23767   \n",
       "2       492.68       1574.32              24392                 23783   \n",
       "\n",
       "   percent_full  conservation_capacity  dead_pool_capacity  \n",
       "0          36.8                  23972                 609  \n",
       "1          99.1                  23972                 609  \n",
       "2          99.2                  23972                 609  "
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the Excel file\n",
    "file_name = 'tmp/exoplanets.xls'\n",
    "df = pd.read_excel(file_name, sheetnames=[0], encoding='utf-8')\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SQLite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/io.html#sql-queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "file_name = \"tmp/exoplanets.sqlite\"\n",
    "clean_tmp(file_name)\n",
    "\n",
    "from sqlalchemy import create_engine\n",
    "engine = create_engine('sqlite:///' + file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df.to_sql('table', engine, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>water_level</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>reservoir_storage</th>\n",
       "      <th>conservation_storage</th>\n",
       "      <th>percent_full</th>\n",
       "      <th>conservation_capacity</th>\n",
       "      <th>dead_pool_capacity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>480.88</td>\n",
       "      <td>968.23</td>\n",
       "      <td>9426</td>\n",
       "      <td>8817</td>\n",
       "      <td>36.8</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>492.67</td>\n",
       "      <td>1573.66</td>\n",
       "      <td>24376</td>\n",
       "      <td>23767</td>\n",
       "      <td>99.1</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>492.68</td>\n",
       "      <td>1574.32</td>\n",
       "      <td>24392</td>\n",
       "      <td>23783</td>\n",
       "      <td>99.2</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>492.40</td>\n",
       "      <td>1556.78</td>\n",
       "      <td>23953</td>\n",
       "      <td>23344</td>\n",
       "      <td>97.4</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>492.54</td>\n",
       "      <td>1565.29</td>\n",
       "      <td>24172</td>\n",
       "      <td>23563</td>\n",
       "      <td>98.3</td>\n",
       "      <td>23972</td>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   water_level  surface_area  reservoir_storage  conservation_storage  \\\n",
       "0       480.88        968.23               9426                  8817   \n",
       "1       492.67       1573.66              24376                 23767   \n",
       "2       492.68       1574.32              24392                 23783   \n",
       "3       492.40       1556.78              23953                 23344   \n",
       "4       492.54       1565.29              24172                 23563   \n",
       "\n",
       "   percent_full  conservation_capacity  dead_pool_capacity  \n",
       "0          36.8                  23972                 609  \n",
       "1          99.1                  23972                 609  \n",
       "2          99.2                  23972                 609  \n",
       "3          97.4                  23972                 609  \n",
       "4          98.3                  23972                 609  "
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_sql('table', engine)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/io.html#json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Writing to JSON file\n",
    "file_name = \"tmp/exoplanets.json\"\n",
    "clean_tmp(file_name)\n",
    "\n",
    "df.to_json(file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>conservation_capacity</th>\n",
       "      <th>conservation_storage</th>\n",
       "      <th>dead_pool_capacity</th>\n",
       "      <th>percent_full</th>\n",
       "      <th>reservoir_storage</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>water_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>8817</td>\n",
       "      <td>609</td>\n",
       "      <td>36.8</td>\n",
       "      <td>9426</td>\n",
       "      <td>968.23</td>\n",
       "      <td>480.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23767</td>\n",
       "      <td>609</td>\n",
       "      <td>99.1</td>\n",
       "      <td>24376</td>\n",
       "      <td>1573.66</td>\n",
       "      <td>492.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23127</td>\n",
       "      <td>609</td>\n",
       "      <td>96.5</td>\n",
       "      <td>23736</td>\n",
       "      <td>1548.63</td>\n",
       "      <td>492.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23065</td>\n",
       "      <td>609</td>\n",
       "      <td>96.2</td>\n",
       "      <td>23674</td>\n",
       "      <td>1546.36</td>\n",
       "      <td>492.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23189</td>\n",
       "      <td>609</td>\n",
       "      <td>96.7</td>\n",
       "      <td>23798</td>\n",
       "      <td>1550.92</td>\n",
       "      <td>492.30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        conservation_capacity  conservation_storage  dead_pool_capacity  \\\n",
       "0.0                     23972                  8817                 609   \n",
       "1.0                     23972                 23767                 609   \n",
       "10.0                    23972                 23127                 609   \n",
       "100.0                   23972                 23065                 609   \n",
       "1000.0                  23972                 23189                 609   \n",
       "\n",
       "        percent_full  reservoir_storage  surface_area  water_level  \n",
       "0.0             36.8               9426        968.23       480.88  \n",
       "1.0             99.1              24376       1573.66       492.67  \n",
       "10.0            96.5              23736       1548.63       492.26  \n",
       "100.0           96.2              23674       1546.36       492.22  \n",
       "1000.0          96.7              23798       1550.92       492.30  "
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reading the JSON file results in columns sorted by label\n",
    "\n",
    "file_name = \"tmp/exoplanets.json\"\n",
    "df = pd.read_json(file_name)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pd.read_json?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`format='table'` is used because its appendable, queryable, and supports all pandas dtypes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "file_name = \"tmp/exoplanets.hdf\"\n",
    "clean_tmp(file_name)\n",
    "\n",
    "df.to_hdf(file_name,\n",
    "          'df',\n",
    "           mode='w',\n",
    "           format='table',\n",
    "           encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>conservation_capacity</th>\n",
       "      <th>conservation_storage</th>\n",
       "      <th>dead_pool_capacity</th>\n",
       "      <th>percent_full</th>\n",
       "      <th>reservoir_storage</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>water_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>8817</td>\n",
       "      <td>609</td>\n",
       "      <td>36.8</td>\n",
       "      <td>9426</td>\n",
       "      <td>968.23</td>\n",
       "      <td>480.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23767</td>\n",
       "      <td>609</td>\n",
       "      <td>99.1</td>\n",
       "      <td>24376</td>\n",
       "      <td>1573.66</td>\n",
       "      <td>492.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23127</td>\n",
       "      <td>609</td>\n",
       "      <td>96.5</td>\n",
       "      <td>23736</td>\n",
       "      <td>1548.63</td>\n",
       "      <td>492.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23065</td>\n",
       "      <td>609</td>\n",
       "      <td>96.2</td>\n",
       "      <td>23674</td>\n",
       "      <td>1546.36</td>\n",
       "      <td>492.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000.0</th>\n",
       "      <td>23972</td>\n",
       "      <td>23189</td>\n",
       "      <td>609</td>\n",
       "      <td>96.7</td>\n",
       "      <td>23798</td>\n",
       "      <td>1550.92</td>\n",
       "      <td>492.30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        conservation_capacity  conservation_storage  dead_pool_capacity  \\\n",
       "0.0                     23972                  8817                 609   \n",
       "1.0                     23972                 23767                 609   \n",
       "10.0                    23972                 23127                 609   \n",
       "100.0                   23972                 23065                 609   \n",
       "1000.0                  23972                 23189                 609   \n",
       "\n",
       "        percent_full  reservoir_storage  surface_area  water_level  \n",
       "0.0             36.8               9426        968.23       480.88  \n",
       "1.0             99.1              24376       1573.66       492.67  \n",
       "10.0            96.5              23736       1548.63       492.26  \n",
       "100.0           96.2              23674       1546.36       492.22  \n",
       "1000.0          96.7              23798       1550.92       492.30  "
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_name = \"tmp/exoplanets.hdf\"\n",
    "df = pd.read_hdf(file_name,'df',encoding='utf-8')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## FAQ: File Formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequently asked file-format questions, and answers:\n",
    "\n",
    "- which formats provide good fidelity\n",
    "  - hdf5, pickle, msgpack\n",
    "  \n",
    "- which formats can you query\n",
    "  - hdf5, sql\n",
    "  \n",
    "- which formats can you iterate\n",
    "  - csv, hdf5, sql\n",
    "  \n",
    "- which formats provide better interoprability\n",
    "  - csv, json, excel\n",
    "  \n",
    "- which formats can you transmit over the wire\n",
    "  - json, msgpack\n",
    "  \n",
    "- which formats have better compression\n",
    "  - hdf5, pickle, msgpack\n",
    "  \n",
    "- which formats allow multiple datasets in the same file\n",
    "  - hdf5, msgpack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reading Large Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set some expectations for read times for various file formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Timing Reads: Common Formats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 loops, best of 3: 103 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_excel('tmp/exoplanets.xls', sheetnames=[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 loops, best of 3: 25.2 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_sql('table', engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 loops, best of 3: 25.3 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_json('tmp/exoplanets.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 loops, best of 3: 8.58 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_csv('tmp/exoplanets.csv', parse_dates=['DATE'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 loops, best of 3: 17.9 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_hdf('tmp/exoplanets.hdf','df')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timing Reads: Pickle and MSGPack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "df.to_pickle('tmp/exoplanets.pkl')\n",
    "df.to_msgpack('tmp/exoplanets.msgpack',encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 1.64 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_pickle('tmp/exoplanets.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 1.78 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pd.read_msgpack('tmp/exoplanets.msgpack', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading Compressed Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may encounter compressed data files, e.g. ZIP or TAR.\n",
    "\n",
    "Here we deomstrate some simple strategies for handling compressed data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning Temporary Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: clean up tmp files created in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Always check for existence before removing\n",
    "if os.path.exists(\"tmp\"):\n",
    "    os.listdir(\"tmp\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build a list of files inside, remove each one by one, then remove the dir\n",
    "if os.path.exists(\"tmp\"):\n",
    "    file_list = os.listdir(\"tmp\")\n",
    "    for file in file_list:\n",
    "        clean_tmp(os.path.join(\"tmp\", file))\n",
    "    os.rmdir(\"tmp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas: Examing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.18.0'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import modules needed for this lesson\n",
    "import pandas as pd\n",
    "\n",
    "# Set Jupyter notebook display options\n",
    "pd.options.display.max_rows = 6\n",
    "pd.options.display.max_columns = 8\n",
    "pd.options.display.width = 100\n",
    "\n",
    "# Test the version of pandas: it should be 0.16 or greater\n",
    "pd.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello world\n"
     ]
    }
   ],
   "source": [
    "print(\"Hello world\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examining Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Data into a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CSV files, being tabular data, have a natural mapping into Pandas DataFrames. \n",
    "* Here we read in beer reviews from beeradvocate.com, via [Stanford SNAP](https://snap.stanford.edu/data/web-BeerAdvocate.html). \n",
    "* Pandas can tell if a file is zipped, and will unzip in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/beer2.csv.gz',\n",
    "                  index_col=0,\n",
    "                  parse_dates=['time'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>profile_name</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>blaheath</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>GJ40</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>biegaman</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.5</td>\n",
       "      <td>28577</td>\n",
       "      <td>3818</td>\n",
       "      <td>Unearthly (Imperial India Pale Ale)</td>\n",
       "      <td>...</td>\n",
       "      <td>nick76</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The aroma has pine, wood, citrus, caramel, an...</td>\n",
       "      <td>2009-10-05 21:32:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.8</td>\n",
       "      <td>398</td>\n",
       "      <td>119</td>\n",
       "      <td>Wolaver's Pale Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>champ103</td>\n",
       "      <td>3.0</td>\n",
       "      <td>A: Pours a slightly hazy golden/orange color....</td>\n",
       "      <td>2009-10-05 21:33:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   abv  beer_id  brewer_id                            beer_name         ...         profile_name  \\\n",
       "0  7.0     2511        287                  Bell's Cherry Stout         ...             blaheath   \n",
       "1  5.7    19736       9790                   Duck-Rabbit Porter         ...                 GJ40   \n",
       "2  4.8    11098       3182         Fürstenberg Premium Pilsener         ...             biegaman   \n",
       "3  9.5    28577       3818  Unearthly (Imperial India Pale Ale)         ...               nick76   \n",
       "4  5.8      398        119                   Wolaver's Pale Ale         ...             champ103   \n",
       "\n",
       "   review_taste                                               text                time  \n",
       "0           4.5   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48  \n",
       "1           4.0   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09  \n",
       "2           3.5   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13  \n",
       "3           4.0   The aroma has pine, wood, citrus, caramel, an... 2009-10-05 21:32:37  \n",
       "4           3.0   A: Pours a slightly hazy golden/orange color.... 2009-10-05 21:33:14  \n",
       "\n",
       "[5 rows x 13 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas provides useful inpsection tools to help understand the dtypes that were generated for each column. \n",
    "\n",
    "Notice that some columns do not have 50,000 *non-null* values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 50000 entries, 0 to 49999\n",
      "Data columns (total 13 columns):\n",
      "abv                  48389 non-null float64\n",
      "beer_id              50000 non-null int64\n",
      "brewer_id            50000 non-null int64\n",
      "beer_name            50000 non-null object\n",
      "beer_style           50000 non-null object\n",
      "review_appearance    50000 non-null float64\n",
      "review_aroma         50000 non-null float64\n",
      "review_overall       50000 non-null float64\n",
      "review_palate        50000 non-null float64\n",
      "profile_name         50000 non-null object\n",
      "review_taste         50000 non-null float64\n",
      "text                 49991 non-null object\n",
      "time                 50000 non-null datetime64[ns]\n",
      "dtypes: datetime64[ns](1), float64(6), int64(2), object(4)\n",
      "memory usage: 5.3+ MB\n"
     ]
    }
   ],
   "source": [
    "# Notice the column (1) names, (2) number of non-null, (3) dtype, (4) memory usage\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>review_appearance</th>\n",
       "      <th>review_aroma</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>48389.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.00000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>7.232406</td>\n",
       "      <td>25379.008540</td>\n",
       "      <td>3505.655420</td>\n",
       "      <td>3.874010</td>\n",
       "      <td>3.778880</td>\n",
       "      <td>3.84982</td>\n",
       "      <td>3.778840</td>\n",
       "      <td>3.832480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.387132</td>\n",
       "      <td>20600.374872</td>\n",
       "      <td>5696.746136</td>\n",
       "      <td>0.596153</td>\n",
       "      <td>0.677396</td>\n",
       "      <td>0.70716</td>\n",
       "      <td>0.660929</td>\n",
       "      <td>0.711728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>6.700000</td>\n",
       "      <td>25880.000000</td>\n",
       "      <td>583.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.00000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8.600000</td>\n",
       "      <td>46302.000000</td>\n",
       "      <td>3694.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.50000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>27.000000</td>\n",
       "      <td>68386.000000</td>\n",
       "      <td>27039.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.00000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                abv       beer_id     brewer_id  review_appearance  review_aroma  review_overall  \\\n",
       "count  48389.000000  50000.000000  50000.000000       50000.000000  50000.000000     50000.00000   \n",
       "mean       7.232406  25379.008540   3505.655420           3.874010      3.778880         3.84982   \n",
       "std        2.387132  20600.374872   5696.746136           0.596153      0.677396         0.70716   \n",
       "...             ...           ...           ...                ...           ...             ...   \n",
       "50%        6.700000  25880.000000    583.000000           4.000000      4.000000         4.00000   \n",
       "75%        8.600000  46302.000000   3694.000000           4.000000      4.000000         4.50000   \n",
       "max       27.000000  68386.000000  27039.000000           5.000000      5.000000         5.00000   \n",
       "\n",
       "       review_palate  review_taste  \n",
       "count   50000.000000  50000.000000  \n",
       "mean        3.778840      3.832480  \n",
       "std         0.660929      0.711728  \n",
       "...              ...           ...  \n",
       "50%         4.000000      4.000000  \n",
       "75%         4.000000      4.500000  \n",
       "max         5.000000      5.000000  \n",
       "\n",
       "[8 rows x 8 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary statistics, using the built-in .describe() method\n",
    "\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Column selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A variety of column operations are available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['abv', 'beer_id', 'brewer_id', 'beer_name', 'beer_style', 'review_appearance',\n",
      "       'review_aroma', 'review_overall', 'review_palate', 'profile_name', 'review_taste', 'text',\n",
      "       'time'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# The column names, as an Index object\n",
    "print( df.columns )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abv',\n",
       " 'beer_id',\n",
       " 'brewer_id',\n",
       " 'beer_name',\n",
       " 'beer_style',\n",
       " 'review_appearance',\n",
       " 'review_aroma',\n",
       " 'review_overall',\n",
       " 'review_palate',\n",
       " 'profile_name',\n",
       " 'review_taste',\n",
       " 'text',\n",
       " 'time']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The column labels can be converted to list of strings\n",
    "\n",
    "list(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns in a pandas DataFrame can be accessed using the .attribute method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                          American Stout\n",
       "1                         American Porter\n",
       "2                         German Pilsener\n",
       "                       ...               \n",
       "49997    American Double / Imperial Stout\n",
       "49998                    English Pale Ale\n",
       "49999              Russian Imperial Stout\n",
       "Name: beer_style, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View column data using .attribute, column label\n",
    "\n",
    "df.beer_style"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or with dictionary-like key selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                          American Stout\n",
       "1                         American Porter\n",
       "2                         German Pilsener\n",
       "                       ...               \n",
       "49997    American Double / Imperial Stout\n",
       "49998                    English Pale Ale\n",
       "49999              Russian Imperial Stout\n",
       "Name: beer_style, dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View column data using [bracket], column label string\n",
    "\n",
    "df['beer_style']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A single column (**Series**) can be selected using the column name as an attribute (or accessor) of the **DataFrame** object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "beer_series = df.beer_name\n",
    "print( type(beer_series) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                    Bell's Cherry Stout\n",
       "1                     Duck-Rabbit Porter\n",
       "2           Fürstenberg Premium Pilsener\n",
       "3    Unearthly (Imperial India Pale Ale)\n",
       "4                     Wolaver's Pale Ale\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "beer_series.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping and Adding Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>profile_name</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>blaheath</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>GJ40</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>biegaman</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   abv  beer_id  brewer_id                     beer_name         ...         profile_name  \\\n",
       "0  7.0     2511        287           Bell's Cherry Stout         ...             blaheath   \n",
       "1  5.7    19736       9790            Duck-Rabbit Porter         ...                 GJ40   \n",
       "2  4.8    11098       3182  Fürstenberg Premium Pilsener         ...             biegaman   \n",
       "\n",
       "   review_taste                                               text                time  \n",
       "0           4.5   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48  \n",
       "1           4.0   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09  \n",
       "2           3.5   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13  \n",
       "\n",
       "[3 rows x 13 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   abv  beer_id  brewer_id                     beer_name   ...    review_taste  \\\n",
       "0  7.0     2511        287           Bell's Cherry Stout   ...             4.5   \n",
       "1  5.7    19736       9790            Duck-Rabbit Porter   ...             4.0   \n",
       "2  4.8    11098       3182  Fürstenberg Premium Pilsener   ...             3.5   \n",
       "\n",
       "                                                text                time  alcohol  \n",
       "0   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48      0.0  \n",
       "1   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09      0.0  \n",
       "2   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      0.0  \n",
       "\n",
       "[3 rows x 14 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Adding a new column\n",
    "\n",
    "df['alcohol'] = 0.0\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 50000 entries, 0 to 49999\n",
      "Data columns (total 14 columns):\n",
      "abv                  48389 non-null float64\n",
      "beer_id              50000 non-null int64\n",
      "brewer_id            50000 non-null int64\n",
      "beer_name            50000 non-null object\n",
      "beer_style           50000 non-null object\n",
      "review_appearance    50000 non-null float64\n",
      "review_aroma         50000 non-null float64\n",
      "review_overall       50000 non-null float64\n",
      "review_palate        50000 non-null float64\n",
      "profile_name         50000 non-null object\n",
      "review_taste         50000 non-null float64\n",
      "text                 49991 non-null object\n",
      "time                 50000 non-null datetime64[ns]\n",
      "alcohol              50000 non-null float64\n",
      "dtypes: datetime64[ns](1), float64(7), int64(2), object(4)\n",
      "memory usage: 5.7+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Over-writing one column with another"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "      <td>5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>8.1</td>\n",
       "      <td>21950</td>\n",
       "      <td>2372</td>\n",
       "      <td>Terrapin Coffee Oatmeal Imperial Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Poured a light sucking crude oil beckoning bl...</td>\n",
       "      <td>2009-12-25 17:23:52</td>\n",
       "      <td>8.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>4.6</td>\n",
       "      <td>5453</td>\n",
       "      <td>1306</td>\n",
       "      <td>Badger Original Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>500ml brown bottle, 4.0% ABV. Pours a crystal...</td>\n",
       "      <td>2009-12-25 17:25:06</td>\n",
       "      <td>4.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>9.4</td>\n",
       "      <td>47695</td>\n",
       "      <td>14879</td>\n",
       "      <td>Barrel Aged B.O.R.I.S. Oatmeal Imperial Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>22 oz bottle poured into a flute glass, share...</td>\n",
       "      <td>2009-12-25 17:26:06</td>\n",
       "      <td>9.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       abv  beer_id  brewer_id                                      beer_name   ...     \\\n",
       "0      7.0     2511        287                            Bell's Cherry Stout   ...      \n",
       "1      5.7    19736       9790                             Duck-Rabbit Porter   ...      \n",
       "2      4.8    11098       3182                   Fürstenberg Premium Pilsener   ...      \n",
       "...    ...      ...        ...                                            ...   ...      \n",
       "49997  8.1    21950       2372         Terrapin Coffee Oatmeal Imperial Stout   ...      \n",
       "49998  4.6     5453       1306                            Badger Original Ale   ...      \n",
       "49999  9.4    47695      14879  Barrel Aged B.O.R.I.S. Oatmeal Imperial Stout   ...      \n",
       "\n",
       "      review_taste                                               text                time  alcohol  \n",
       "0              4.5   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48      7.0  \n",
       "1              4.0   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09      5.7  \n",
       "2              3.5   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      4.8  \n",
       "...            ...                                                ...                 ...      ...  \n",
       "49997          4.5   Poured a light sucking crude oil beckoning bl... 2009-12-25 17:23:52      8.1  \n",
       "49998          3.5   500ml brown bottle, 4.0% ABV. Pours a crystal... 2009-12-25 17:25:06      4.6  \n",
       "49999          4.5   22 oz bottle poured into a flute glass, share... 2009-12-25 17:26:06      9.4  \n",
       "\n",
       "[50000 rows x 14 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['alcohol'] = df['abv']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>American Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>American Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "      <td>5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>German Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   beer_id  brewer_id                     beer_name       beer_style   ...     review_taste  \\\n",
       "0     2511        287           Bell's Cherry Stout   American Stout   ...              4.5   \n",
       "1    19736       9790            Duck-Rabbit Porter  American Porter   ...              4.0   \n",
       "2    11098       3182  Fürstenberg Premium Pilsener  German Pilsener   ...              3.5   \n",
       "\n",
       "                                                text                time  alcohol  \n",
       "0   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48      7.0  \n",
       "1   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09      5.7  \n",
       "2   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      4.8  \n",
       "\n",
       "[3 rows x 13 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dropping a column\n",
    "\n",
    "df.drop(['abv'], axis=1).head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 50000 entries, 0 to 49999\n",
      "Data columns (total 14 columns):\n",
      "abv                  48389 non-null float64\n",
      "beer_id              50000 non-null int64\n",
      "brewer_id            50000 non-null int64\n",
      "beer_name            50000 non-null object\n",
      "beer_style           50000 non-null object\n",
      "review_appearance    50000 non-null float64\n",
      "review_aroma         50000 non-null float64\n",
      "review_overall       50000 non-null float64\n",
      "review_palate        50000 non-null float64\n",
      "profile_name         50000 non-null object\n",
      "review_taste         50000 non-null float64\n",
      "text                 49991 non-null object\n",
      "time                 50000 non-null datetime64[ns]\n",
      "alcohol              48389 non-null float64\n",
      "dtypes: datetime64[ns](1), float64(7), int64(2), object(4)\n",
      "memory usage: 5.7+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## String Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Varios srtring operations can be applied to the columna data values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        Amer\n",
       "1        Amer\n",
       "2        Germ\n",
       "         ... \n",
       "49997    Amer\n",
       "49998    Engl\n",
       "49999    Russ\n",
       "Name: beer_style, dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print first 4 letter of each string in the column\n",
    "\n",
    "df.beer_style.str[0:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        14\n",
       "1        15\n",
       "2        15\n",
       "         ..\n",
       "49997    32\n",
       "49998    16\n",
       "49999    22\n",
       "Name: beer_style, dtype: int64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the length of each string in the column\n",
    "\n",
    "df.beer_style.str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         True\n",
       "1         True\n",
       "2        False\n",
       "         ...  \n",
       "49997     True\n",
       "49998    False\n",
       "49999    False\n",
       "Name: beer_style, dtype: bool"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get boolean (True/False) mask for strings matching a pattern\n",
    "\n",
    "df.beer_style.str.contains('[A|a]merican')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                          American Stout\n",
       "1                         American Porter\n",
       "3          American Double / Imperial IPA\n",
       "                       ...               \n",
       "49991                  American Brown Ale\n",
       "49996      American Double / Imperial IPA\n",
       "49997    American Double / Imperial Stout\n",
       "Name: beer_style, dtype: object"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract the rows that match the True\n",
    "\n",
    "boolean_mask = df.beer_style.str.contains('[A|a]merican')\n",
    "df.beer_style[boolean_mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# To see full documentation for str() method, uncomment the following:\n",
    "\n",
    "# help(df.beer_style.str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datetime Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `.dt` accessor, datetime manipulations can be applied to a datetime series or column elementwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       2009-10-05 21:31:48\n",
       "1       2009-10-05 21:32:09\n",
       "2       2009-10-05 21:32:13\n",
       "                ...        \n",
       "49997   2009-12-25 17:23:52\n",
       "49998   2009-12-25 17:25:06\n",
       "49999   2009-12-25 17:26:06\n",
       "Name: time, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select the `time` column, noteice the dtype=datetime\n",
    "\n",
    "df.time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the `dtype=datatime`, we can perform DataTime operations on the column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        2009-10-05\n",
       "1        2009-10-05\n",
       "2        2009-10-05\n",
       "            ...    \n",
       "49997    2009-12-25\n",
       "49998    2009-12-25\n",
       "49999    2009-12-25\n",
       "Name: time, dtype: object"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: Just show the data, not the full date-time\n",
    "\n",
    "df.time.dt.date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        21:31:48\n",
       "1        21:32:09\n",
       "2        21:32:13\n",
       "           ...   \n",
       "49997    17:23:52\n",
       "49998    17:25:06\n",
       "49999    17:26:06\n",
       "Name: time, dtype: object"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Just show the time, not the full date-time\n",
    "\n",
    "df.time.dt.time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        21\n",
       "1        21\n",
       "2        21\n",
       "         ..\n",
       "49997    17\n",
       "49998    17\n",
       "49999    17\n",
       "Name: time, dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Just show the hour\n",
    "\n",
    "df.time.dt.hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on DatetimeProperties in module pandas.tseries.common object:\n",
      "\n",
      "class DatetimeProperties(Properties)\n",
      " |  Accessor object for datetimelike properties of the Series values.\n",
      " |  \n",
      " |  Examples\n",
      " |  --------\n",
      " |  >>> s.dt.hour\n",
      " |  >>> s.dt.second\n",
      " |  >>> s.dt.quarter\n",
      " |  \n",
      " |  Returns a Series indexed like the original Series.\n",
      " |  Raises TypeError if the Series does not contain datetimelike values.\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      DatetimeProperties\n",
      " |      Properties\n",
      " |      pandas.core.base.PandasDelegate\n",
      " |      pandas.core.base.PandasObject\n",
      " |      pandas.core.base.StringMixin\n",
      " |      pandas.core.base.NoNewAttributesMixin\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  ceil(self, *args, **kwargs)\n",
      " |      floor the index to the specified freq\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      freq : freq string/object\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      index of same type\n",
      " |      \n",
      " |      Raises\n",
      " |      ------\n",
      " |      ValueError if the freq cannot be converted\n",
      " |  \n",
      " |  floor(self, *args, **kwargs)\n",
      " |      floor the index to the specified freq\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      freq : freq string/object\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      index of same type\n",
      " |      \n",
      " |      Raises\n",
      " |      ------\n",
      " |      ValueError if the freq cannot be converted\n",
      " |  \n",
      " |  normalize(self, *args, **kwargs)\n",
      " |      Return DatetimeIndex with times to midnight. Length is unaltered\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      normalized : DatetimeIndex\n",
      " |  \n",
      " |  round(self, *args, **kwargs)\n",
      " |      round the index to the specified freq\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      freq : freq string/object\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      index of same type\n",
      " |      \n",
      " |      Raises\n",
      " |      ------\n",
      " |      ValueError if the freq cannot be converted\n",
      " |  \n",
      " |  strftime(self, *args, **kwargs)\n",
      " |      Return an array of formatted strings specified by date_format, which\n",
      " |      supports the same string format as the python standard library. Details\n",
      " |      of the string format can be found in `python string format doc <https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior>`__\n",
      " |      \n",
      " |      .. versionadded:: 0.17.0\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      date_format : str\n",
      " |          date format string (e.g. \"%Y-%m-%d\")\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      ndarray of formatted strings\n",
      " |  \n",
      " |  to_period(self, *args, **kwargs)\n",
      " |      Cast to PeriodIndex at a particular frequency\n",
      " |  \n",
      " |  to_pydatetime(self)\n",
      " |  \n",
      " |  tz_convert(self, *args, **kwargs)\n",
      " |      Convert tz-aware DatetimeIndex from one time zone to another (using\n",
      " |      pytz/dateutil)\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      tz : string, pytz.timezone, dateutil.tz.tzfile or None\n",
      " |          Time zone for time. Corresponding timestamps would be converted to\n",
      " |          time zone of the TimeSeries.\n",
      " |          None will remove timezone holding UTC time.\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      normalized : DatetimeIndex\n",
      " |      \n",
      " |      Raises\n",
      " |      ------\n",
      " |      TypeError\n",
      " |          If DatetimeIndex is tz-naive.\n",
      " |  \n",
      " |  tz_localize(self, *args, **kwargs)\n",
      " |      Localize tz-naive DatetimeIndex to given time zone (using\n",
      " |      pytz/dateutil), or remove timezone from tz-aware DatetimeIndex\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      tz : string, pytz.timezone, dateutil.tz.tzfile or None\n",
      " |          Time zone for time. Corresponding timestamps would be converted to\n",
      " |          time zone of the TimeSeries.\n",
      " |          None will remove timezone holding local time.\n",
      " |      ambiguous : 'infer', bool-ndarray, 'NaT', default 'raise'\n",
      " |          - 'infer' will attempt to infer fall dst-transition hours based on\n",
      " |            order\n",
      " |          - bool-ndarray where True signifies a DST time, False signifies a\n",
      " |            non-DST time (note that this flag is only applicable for\n",
      " |            ambiguous times)\n",
      " |          - 'NaT' will return NaT where there are ambiguous times\n",
      " |          - 'raise' will raise an AmbiguousTimeError if there are ambiguous\n",
      " |            times\n",
      " |      infer_dst : boolean, default False (DEPRECATED)\n",
      " |          Attempt to infer fall dst-transition hours based on order\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      localized : DatetimeIndex\n",
      " |      \n",
      " |      Raises\n",
      " |      ------\n",
      " |      TypeError\n",
      " |          If the DatetimeIndex is tz-aware and tz is not None.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors defined here:\n",
      " |  \n",
      " |  date\n",
      " |      Returns numpy array of datetime.date. The date part of the Timestamps.\n",
      " |  \n",
      " |  day\n",
      " |      The days of the datetime\n",
      " |  \n",
      " |  dayofweek\n",
      " |      The day of the week with Monday=0, Sunday=6\n",
      " |  \n",
      " |  dayofyear\n",
      " |      The ordinal day of the year\n",
      " |  \n",
      " |  days_in_month\n",
      " |      The number of days in the month\n",
      " |      \n",
      " |      .. versionadded:: 0.16.0\n",
      " |  \n",
      " |  daysinmonth\n",
      " |      The number of days in the month\n",
      " |      \n",
      " |      .. versionadded:: 0.16.0\n",
      " |  \n",
      " |  freq\n",
      " |      get/set the frequncy of the Index\n",
      " |  \n",
      " |  hour\n",
      " |      The hours of the datetime\n",
      " |  \n",
      " |  is_month_end\n",
      " |      Logical indicating if last day of month (defined by frequency)\n",
      " |  \n",
      " |  is_month_start\n",
      " |      Logical indicating if first day of month (defined by frequency)\n",
      " |  \n",
      " |  is_quarter_end\n",
      " |      Logical indicating if last day of quarter (defined by frequency)\n",
      " |  \n",
      " |  is_quarter_start\n",
      " |      Logical indicating if first day of quarter (defined by frequency)\n",
      " |  \n",
      " |  is_year_end\n",
      " |      Logical indicating if last day of year (defined by frequency)\n",
      " |  \n",
      " |  is_year_start\n",
      " |      Logical indicating if first day of year (defined by frequency)\n",
      " |  \n",
      " |  microsecond\n",
      " |      The microseconds of the datetime\n",
      " |  \n",
      " |  minute\n",
      " |      The minutes of the datetime\n",
      " |  \n",
      " |  month\n",
      " |      The month as January=1, December=12\n",
      " |  \n",
      " |  nanosecond\n",
      " |      The nanoseconds of the datetime\n",
      " |  \n",
      " |  quarter\n",
      " |      The quarter of the date\n",
      " |  \n",
      " |  second\n",
      " |      The seconds of the datetime\n",
      " |  \n",
      " |  time\n",
      " |      Returns numpy array of datetime.time. The time part of the Timestamps.\n",
      " |  \n",
      " |  tz\n",
      " |  \n",
      " |  week\n",
      " |      The week ordinal of the year\n",
      " |  \n",
      " |  weekday\n",
      " |      The day of the week with Monday=0, Sunday=6\n",
      " |  \n",
      " |  weekofyear\n",
      " |      The week ordinal of the year\n",
      " |  \n",
      " |  year\n",
      " |      The year of the datetime\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from Properties:\n",
      " |  \n",
      " |  __init__(self, values, index, name, orig=None)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from pandas.core.base.PandasObject:\n",
      " |  \n",
      " |  __dir__(self)\n",
      " |      Provide method name lookup and completion\n",
      " |      Only provide 'public' methods\n",
      " |  \n",
      " |  __sizeof__(self)\n",
      " |      Generates the total memory usage for a object that returns\n",
      " |      either a value or Series of values\n",
      " |  \n",
      " |  __unicode__(self)\n",
      " |      Return a string representation for a particular object.\n",
      " |      \n",
      " |      Invoked by unicode(obj) in py2 only. Yields a Unicode String in both\n",
      " |      py2/py3.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from pandas.core.base.StringMixin:\n",
      " |  \n",
      " |  __bytes__(self)\n",
      " |      Return a string representation for a particular object.\n",
      " |      \n",
      " |      Invoked by bytes(obj) in py3 only.\n",
      " |      Yields a bytestring in both py2/py3.\n",
      " |  \n",
      " |  __repr__(self)\n",
      " |      Return a string representation for a particular object.\n",
      " |      \n",
      " |      Yields Bytestring in Py2, Unicode String in py3.\n",
      " |  \n",
      " |  __str__(self)\n",
      " |      Return a string representation for a particular Object\n",
      " |      \n",
      " |      Invoked by str(df) in both py2/py3.\n",
      " |      Yields Bytestring in Py2, Unicode String in py3.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from pandas.core.base.StringMixin:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from pandas.core.base.NoNewAttributesMixin:\n",
      " |  \n",
      " |  __setattr__(self, key, value)\n",
      " |      # prevent adding any attribute via s.xxx.new_attribute = ...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# To see full help documentation for the DataTime properties, uncomment the following:\n",
    "\n",
    "help(df.time.dt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much of the same indexing methods avialable in Numpy can be applied to Pandas DataFrames and Series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Column Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...can be indexed by brackets or by .attribute accessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['abv', 'beer_id', 'brewer_id', 'beer_name', 'beer_style', 'review_appearance',\n",
       "       'review_aroma', 'review_overall', 'review_palate', 'profile_name', 'review_taste', 'text',\n",
       "       'time', 'alcohol'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recall the column names: they are your \"indices\"\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                                  Bell's Cherry Stout\n",
       "1                                   Duck-Rabbit Porter\n",
       "2                         Fürstenberg Premium Pilsener\n",
       "                             ...                      \n",
       "49997           Terrapin Coffee Oatmeal Imperial Stout\n",
       "49998                              Badger Original Ale\n",
       "49999    Barrel Aged B.O.R.I.S. Oatmeal Imperial Stout\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# \"Indexing\" by brackets and any one of the column names/labels\n",
    "\n",
    "df['beer_name']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0             Bell's Cherry Stout\n",
       "1              Duck-Rabbit Porter\n",
       "2    Fürstenberg Premium Pilsener\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Indexing into rows of the column\n",
    "\n",
    "df['beer_name'][0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                                  Bell's Cherry Stout\n",
       "1                                   Duck-Rabbit Porter\n",
       "2                         Fürstenberg Premium Pilsener\n",
       "                             ...                      \n",
       "49997           Terrapin Coffee Oatmeal Imperial Stout\n",
       "49998                              Badger Original Ale\n",
       "49999    Barrel Aged B.O.R.I.S. Oatmeal Imperial Stout\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " # Index by .attribute accessor\n",
    "    \n",
    "df.beer_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0             Bell's Cherry Stout\n",
       "1              Duck-Rabbit Porter\n",
       "2    Fürstenberg Premium Pilsener\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Indexing into rows of the column\n",
    "\n",
    "df.beer_name[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns are indexed first, then rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Bell's Cherry Stout\""
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['beer_name'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0             Bell's Cherry Stout\n",
       "1              Duck-Rabbit Porter\n",
       "2    Fürstenberg Premium Pilsener\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['beer_name'][0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0             Bell's Cherry Stout\n",
       "1              Duck-Rabbit Porter\n",
       "2    Fürstenberg Premium Pilsener\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ix[0:2, 'beer_name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary: DataFrames and indexed by (1) by Column and then (2) by Row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boolean arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        False\n",
       "1        False\n",
       "2         True\n",
       "         ...  \n",
       "49997    False\n",
       "49998     True\n",
       "49999    False\n",
       "Name: abv, dtype: bool"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# boolean series\n",
    "df.abv < 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>4.8</td>\n",
       "      <td>1669</td>\n",
       "      <td>256</td>\n",
       "      <td>Great White</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Ok, for starters great white I believe will b...</td>\n",
       "      <td>2009-10-05 21:34:29</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>4.6</td>\n",
       "      <td>401</td>\n",
       "      <td>118</td>\n",
       "      <td>Dark Island</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Poured into a snifter, revealing black opaque...</td>\n",
       "      <td>2009-10-05 21:47:36</td>\n",
       "      <td>4.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49976</th>\n",
       "      <td>4.7</td>\n",
       "      <td>18552</td>\n",
       "      <td>9663</td>\n",
       "      <td>Ichnusa</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>When in Sardinia I drink this beer, and altho...</td>\n",
       "      <td>2009-12-25 15:43:48</td>\n",
       "      <td>4.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49980</th>\n",
       "      <td>4.1</td>\n",
       "      <td>51464</td>\n",
       "      <td>29</td>\n",
       "      <td>Bud Light Golden Wheat</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Not quite as good as other wheat beers, but i...</td>\n",
       "      <td>2009-12-25 15:57:12</td>\n",
       "      <td>4.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>4.6</td>\n",
       "      <td>5453</td>\n",
       "      <td>1306</td>\n",
       "      <td>Badger Original Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>500ml brown bottle, 4.0% ABV. Pours a crystal...</td>\n",
       "      <td>2009-12-25 17:25:06</td>\n",
       "      <td>4.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5399 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       abv  beer_id  brewer_id                     beer_name   ...    review_taste  \\\n",
       "2      4.8    11098       3182  Fürstenberg Premium Pilsener   ...             3.5   \n",
       "7      4.8     1669        256                   Great White   ...             4.5   \n",
       "21     4.6      401        118                   Dark Island   ...             4.0   \n",
       "...    ...      ...        ...                           ...   ...             ...   \n",
       "49976  4.7    18552       9663                       Ichnusa   ...             2.0   \n",
       "49980  4.1    51464         29        Bud Light Golden Wheat   ...             3.5   \n",
       "49998  4.6     5453       1306           Badger Original Ale   ...             3.5   \n",
       "\n",
       "                                                    text                time  alcohol  \n",
       "2       Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      4.8  \n",
       "7       Ok, for starters great white I believe will b... 2009-10-05 21:34:29      4.8  \n",
       "21      Poured into a snifter, revealing black opaque... 2009-10-05 21:47:36      4.6  \n",
       "...                                                  ...                 ...      ...  \n",
       "49976   When in Sardinia I drink this beer, and altho... 2009-12-25 15:43:48      4.7  \n",
       "49980   Not quite as good as other wheat beers, but i... 2009-12-25 15:57:12      4.1  \n",
       "49998   500ml brown bottle, 4.0% ABV. Pours a crystal... 2009-12-25 17:25:06      4.6  \n",
       "\n",
       "[5399 rows x 14 columns]"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# boolean indexing along rows\n",
    "# The index value from the full dataframe is preserved\n",
    "df[df.abv < 5]\n",
    "\n",
    "# Note the number of rows left after the operation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `.loc` accessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.loc` accessor method allows for chained selection of rows and columns. \n",
    "\n",
    "The selection pattern is row first\n",
    "\n",
    "```\n",
    "df.loc[ row , column ]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remember**: using `.loc` means that the right endpoint in a slice is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "      <td>5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   abv  beer_id  brewer_id                     beer_name   ...    review_taste  \\\n",
       "0  7.0     2511        287           Bell's Cherry Stout   ...             4.5   \n",
       "1  5.7    19736       9790            Duck-Rabbit Porter   ...             4.0   \n",
       "2  4.8    11098       3182  Fürstenberg Premium Pilsener   ...             3.5   \n",
       "\n",
       "                                                text                time  alcohol  \n",
       "0   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48      7.0  \n",
       "1   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09      5.7  \n",
       "2   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      4.8  \n",
       "\n",
       "[3 rows x 14 columns]"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.0"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0,'abv']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2009-10-05 21:32:13')"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[2,'time']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values passed to `.loc` for either rows or columns can be values in the indices or **boolean indexing arrays**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>beer_style</th>\n",
       "      <th>review_overall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>German Pilsener</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Witbier</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Scottish Ale</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49976</th>\n",
       "      <td>Euro Pale Lager</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49980</th>\n",
       "      <td>Herbed / Spiced Beer</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>English Pale Ale</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5399 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 beer_style  review_overall\n",
       "2           German Pilsener             3.0\n",
       "7                   Witbier             4.5\n",
       "21             Scottish Ale             3.5\n",
       "...                     ...             ...\n",
       "49976       Euro Pale Lager             3.0\n",
       "49980  Herbed / Spiced Beer             4.0\n",
       "49998      English Pale Ale             4.0\n",
       "\n",
       "[5399 rows x 2 columns]"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df.abv < 5, ['beer_style', 'review_overall']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using bitwise comparison operations between boolean arrays can provide complex selection logic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        False\n",
       "1        False\n",
       "2         True\n",
       "         ...  \n",
       "49997    False\n",
       "49998     True\n",
       "49999    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reviews written after June 2009 for beer with abv<5\n",
    "(df.abv < 5) & (df.time > pd.Timestamp('2009-06'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "      <td>5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>8.1</td>\n",
       "      <td>21950</td>\n",
       "      <td>2372</td>\n",
       "      <td>Terrapin Coffee Oatmeal Imperial Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Poured a light sucking crude oil beckoning bl...</td>\n",
       "      <td>2009-12-25 17:23:52</td>\n",
       "      <td>8.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>4.6</td>\n",
       "      <td>5453</td>\n",
       "      <td>1306</td>\n",
       "      <td>Badger Original Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>500ml brown bottle, 4.0% ABV. Pours a crystal...</td>\n",
       "      <td>2009-12-25 17:25:06</td>\n",
       "      <td>4.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>9.4</td>\n",
       "      <td>47695</td>\n",
       "      <td>14879</td>\n",
       "      <td>Barrel Aged B.O.R.I.S. Oatmeal Imperial Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>22 oz bottle poured into a flute glass, share...</td>\n",
       "      <td>2009-12-25 17:26:06</td>\n",
       "      <td>9.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18007 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       abv  beer_id  brewer_id                                      beer_name   ...     \\\n",
       "0      7.0     2511        287                            Bell's Cherry Stout   ...      \n",
       "1      5.7    19736       9790                             Duck-Rabbit Porter   ...      \n",
       "2      4.8    11098       3182                   Fürstenberg Premium Pilsener   ...      \n",
       "...    ...      ...        ...                                            ...   ...      \n",
       "49997  8.1    21950       2372         Terrapin Coffee Oatmeal Imperial Stout   ...      \n",
       "49998  4.6     5453       1306                            Badger Original Ale   ...      \n",
       "49999  9.4    47695      14879  Barrel Aged B.O.R.I.S. Oatmeal Imperial Stout   ...      \n",
       "\n",
       "      review_taste                                               text                time  alcohol  \n",
       "0              4.5   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48      7.0  \n",
       "1              4.0   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09      5.7  \n",
       "2              3.5   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      4.8  \n",
       "...            ...                                                ...                 ...      ...  \n",
       "49997          4.5   Poured a light sucking crude oil beckoning bl... 2009-12-25 17:23:52      8.1  \n",
       "49998          3.5   500ml brown bottle, 4.0% ABV. Pours a crystal... 2009-12-25 17:25:06      4.6  \n",
       "49999          4.5   22 oz bottle poured into a flute glass, share... 2009-12-25 17:26:06      9.4  \n",
       "\n",
       "[18007 rows x 14 columns]"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reviews written after June 2009 for beer with abv<5 OR reviews with a rating >= 4.5\n",
    "boolean_chain = ( \n",
    "                   ( (df.abv < 5)  & (df.time > pd.Timestamp('2009-06'))  )\n",
    "                 | (df.review_overall >= 4.5)\n",
    "                 )\n",
    "df[boolean_chain]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## isin with strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        False\n",
       "1        False\n",
       "2        False\n",
       "         ...  \n",
       "49997    False\n",
       "49998    False\n",
       "49999    False\n",
       "Name: beer_style, dtype: bool"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# .contains() method returns booleans...\n",
    "\n",
    "df.beer_style.str.contains('IPA')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.5</td>\n",
       "      <td>28577</td>\n",
       "      <td>3818</td>\n",
       "      <td>Unearthly (Imperial India Pale Ale)</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The aroma has pine, wood, citrus, caramel, an...</td>\n",
       "      <td>2009-10-05 21:32:37</td>\n",
       "      <td>9.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6.7</td>\n",
       "      <td>6549</td>\n",
       "      <td>140</td>\n",
       "      <td>Northern Hemisphere Harvest Wet Hop Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>I like all of Sierra Nevada's beers but felt ...</td>\n",
       "      <td>2009-10-05 21:34:31</td>\n",
       "      <td>6.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>8.0</td>\n",
       "      <td>36179</td>\n",
       "      <td>3818</td>\n",
       "      <td>Hoppe (Imperial Extra Pale Ale)</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>The aroma is papery with citrus, yeast, and s...</td>\n",
       "      <td>2009-10-05 21:43:23</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49947</th>\n",
       "      <td>7.0</td>\n",
       "      <td>709</td>\n",
       "      <td>199</td>\n",
       "      <td>Big Eye IPA</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12oz into my pint glass.\\t\\tA: Golden honey a...</td>\n",
       "      <td>2009-12-25 08:48:11</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49984</th>\n",
       "      <td>6.0</td>\n",
       "      <td>38388</td>\n",
       "      <td>3718</td>\n",
       "      <td>L'IPA Du Lièvre</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>I love this beer, seek it out when I am in Mo...</td>\n",
       "      <td>2009-12-25 16:25:45</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>8.0</td>\n",
       "      <td>7971</td>\n",
       "      <td>863</td>\n",
       "      <td>Pliny The Elder</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>This is a big hoppy monster of an IPA..If you...</td>\n",
       "      <td>2009-12-25 17:23:24</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7774 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       abv  beer_id  brewer_id                                beer_name   ...    review_taste  \\\n",
       "3      9.5    28577       3818      Unearthly (Imperial India Pale Ale)   ...             4.0   \n",
       "8      6.7     6549        140  Northern Hemisphere Harvest Wet Hop Ale   ...             4.0   \n",
       "16     8.0    36179       3818          Hoppe (Imperial Extra Pale Ale)   ...             3.0   \n",
       "...    ...      ...        ...                                      ...   ...             ...   \n",
       "49947  7.0      709        199                              Big Eye IPA   ...             4.0   \n",
       "49984  6.0    38388       3718                          L'IPA Du Lièvre   ...             4.5   \n",
       "49996  8.0     7971        863                          Pliny The Elder   ...             4.0   \n",
       "\n",
       "                                                    text                time  alcohol  \n",
       "3       The aroma has pine, wood, citrus, caramel, an... 2009-10-05 21:32:37      9.5  \n",
       "8       I like all of Sierra Nevada's beers but felt ... 2009-10-05 21:34:31      6.7  \n",
       "16      The aroma is papery with citrus, yeast, and s... 2009-10-05 21:43:23      8.0  \n",
       "...                                                  ...                 ...      ...  \n",
       "49947   12oz into my pint glass.\\t\\tA: Golden honey a... 2009-12-25 08:48:11      7.0  \n",
       "49984   I love this beer, seek it out when I am in Mo... 2009-12-25 16:25:45      6.0  \n",
       "49996   This is a big hoppy monster of an IPA..If you... 2009-12-25 17:23:24      8.0  \n",
       "\n",
       "[7774 rows x 14 columns]"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#... so .contain() can be used with .loc() as well:\n",
    "\n",
    "df[df.beer_style.str.contains('IPA')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>review_appearance</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>7573.000000</td>\n",
       "      <td>7774.000000</td>\n",
       "      <td>7774.000000</td>\n",
       "      <td>7774.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>7774.000000</td>\n",
       "      <td>7774.000000</td>\n",
       "      <td>7774.000000</td>\n",
       "      <td>7573.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>7.647368</td>\n",
       "      <td>28153.392977</td>\n",
       "      <td>3517.474016</td>\n",
       "      <td>4.042256</td>\n",
       "      <td>...</td>\n",
       "      <td>4.002508</td>\n",
       "      <td>3.945266</td>\n",
       "      <td>3.996977</td>\n",
       "      <td>7.647368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.761671</td>\n",
       "      <td>18962.917960</td>\n",
       "      <td>5405.981214</td>\n",
       "      <td>0.474063</td>\n",
       "      <td>...</td>\n",
       "      <td>0.614975</td>\n",
       "      <td>0.528179</td>\n",
       "      <td>0.596710</td>\n",
       "      <td>1.761671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>7.000000</td>\n",
       "      <td>30288.000000</td>\n",
       "      <td>689.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>7.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8.700000</td>\n",
       "      <td>46363.000000</td>\n",
       "      <td>3818.000000</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>...</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>8.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>18.000000</td>\n",
       "      <td>59772.000000</td>\n",
       "      <td>27039.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>18.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               abv       beer_id     brewer_id  review_appearance     ...       review_overall  \\\n",
       "count  7573.000000   7774.000000   7774.000000        7774.000000     ...          7774.000000   \n",
       "mean      7.647368  28153.392977   3517.474016           4.042256     ...             4.002508   \n",
       "std       1.761671  18962.917960   5405.981214           0.474063     ...             0.614975   \n",
       "...            ...           ...           ...                ...     ...                  ...   \n",
       "50%       7.000000  30288.000000    689.000000           4.000000     ...             4.000000   \n",
       "75%       8.700000  46363.000000   3818.000000           4.500000     ...             4.500000   \n",
       "max      18.000000  59772.000000  27039.000000           5.000000     ...             5.000000   \n",
       "\n",
       "       review_palate  review_taste      alcohol  \n",
       "count    7774.000000   7774.000000  7573.000000  \n",
       "mean        3.945266      3.996977     7.647368  \n",
       "std         0.528179      0.596710     1.761671  \n",
       "...              ...           ...          ...  \n",
       "50%         4.000000      4.000000     7.000000  \n",
       "75%         4.500000      4.500000     8.700000  \n",
       "max         5.000000      5.000000    18.000000  \n",
       "\n",
       "[8 rows x 9 columns]"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary statistics for hoppy (IPA) beers\n",
    "\n",
    "df[ df.beer_style.str.contains('IPA') ].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>review_appearance</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>40816.000000</td>\n",
       "      <td>42226.000000</td>\n",
       "      <td>42226.000000</td>\n",
       "      <td>42226.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>42226.000000</td>\n",
       "      <td>42226.000000</td>\n",
       "      <td>42226.000000</td>\n",
       "      <td>40816.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>7.155413</td>\n",
       "      <td>24868.231658</td>\n",
       "      <td>3503.479562</td>\n",
       "      <td>3.843035</td>\n",
       "      <td>...</td>\n",
       "      <td>3.821709</td>\n",
       "      <td>3.748200</td>\n",
       "      <td>3.802195</td>\n",
       "      <td>7.155413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.478320</td>\n",
       "      <td>20847.856022</td>\n",
       "      <td>5748.733260</td>\n",
       "      <td>0.610974</td>\n",
       "      <td>...</td>\n",
       "      <td>0.719334</td>\n",
       "      <td>0.678128</td>\n",
       "      <td>0.726893</td>\n",
       "      <td>2.478320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>6.500000</td>\n",
       "      <td>24044.000000</td>\n",
       "      <td>506.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>6.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8.600000</td>\n",
       "      <td>46290.000000</td>\n",
       "      <td>3521.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>8.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>27.000000</td>\n",
       "      <td>68386.000000</td>\n",
       "      <td>25680.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>27.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                abv       beer_id     brewer_id  review_appearance      ...       review_overall  \\\n",
       "count  40816.000000  42226.000000  42226.000000       42226.000000      ...         42226.000000   \n",
       "mean       7.155413  24868.231658   3503.479562           3.843035      ...             3.821709   \n",
       "std        2.478320  20847.856022   5748.733260           0.610974      ...             0.719334   \n",
       "...             ...           ...           ...                ...      ...                  ...   \n",
       "50%        6.500000  24044.000000    506.000000           4.000000      ...             4.000000   \n",
       "75%        8.600000  46290.000000   3521.000000           4.000000      ...             4.500000   \n",
       "max       27.000000  68386.000000  25680.000000           5.000000      ...             5.000000   \n",
       "\n",
       "       review_palate  review_taste       alcohol  \n",
       "count   42226.000000  42226.000000  40816.000000  \n",
       "mean        3.748200      3.802195      7.155413  \n",
       "std         0.678128      0.726893      2.478320  \n",
       "...              ...           ...           ...  \n",
       "50%         4.000000      4.000000      6.500000  \n",
       "75%         4.000000      4.500000      8.600000  \n",
       "max         5.000000      5.000000     27.000000  \n",
       "\n",
       "[8 rows x 9 columns]"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary statistics for NOT hoppy (IPA) beers\n",
    "\n",
    "df[ df.beer_style.str.contains('IPA')==False ].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical Operations through Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1904     240\n",
       "53863    208\n",
       "52441    158\n",
       "        ... \n",
       "41285      1\n",
       "47430      1\n",
       "53274      1\n",
       "Name: beer_id, dtype: int64"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# count unique entries of beer_id\n",
    "\n",
    "beer_ids = df.beer_id.value_counts(sort=True)  # default is sorted, but wanted to be explicit\n",
    "beer_ids\n",
    "# left is ID, right is number of times it appears"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 3 \n",
      "Beer ID, Reviews \n",
      " 1904     240\n",
      "53863    208\n",
      "52441    158\n",
      "Name: beer_id, dtype: int64 \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>8.6</td>\n",
       "      <td>52441</td>\n",
       "      <td>147</td>\n",
       "      <td>Stone 09.09.09 Vertical Epic Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Poured in to a chimay goblet.\\t\\tPours black ...</td>\n",
       "      <td>2009-10-06 00:10:46</td>\n",
       "      <td>8.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>446</th>\n",
       "      <td>8.6</td>\n",
       "      <td>52441</td>\n",
       "      <td>147</td>\n",
       "      <td>Stone 09.09.09 Vertical Epic Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Pours pitch black and completely opaque witho...</td>\n",
       "      <td>2009-10-06 07:59:01</td>\n",
       "      <td>8.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>714</th>\n",
       "      <td>8.6</td>\n",
       "      <td>52441</td>\n",
       "      <td>147</td>\n",
       "      <td>Stone 09.09.09 Vertical Epic Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Near black pour with a mountainous 3 finger h...</td>\n",
       "      <td>2009-10-06 21:31:06</td>\n",
       "      <td>8.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49715</th>\n",
       "      <td>6.8</td>\n",
       "      <td>1904</td>\n",
       "      <td>140</td>\n",
       "      <td>Sierra Nevada Celebration Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>2.5</td>\n",
       "      <td>I've tried this beer on tap about a year ago,...</td>\n",
       "      <td>2009-12-25 01:13:38</td>\n",
       "      <td>6.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49844</th>\n",
       "      <td>8.6</td>\n",
       "      <td>52441</td>\n",
       "      <td>147</td>\n",
       "      <td>Stone 09.09.09 Vertical Epic Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>thanks to colonelforbin for this one!\\t\\tPour...</td>\n",
       "      <td>2009-12-25 04:10:49</td>\n",
       "      <td>8.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49863</th>\n",
       "      <td>5.5</td>\n",
       "      <td>53863</td>\n",
       "      <td>28</td>\n",
       "      <td>Our Special Ale 2009 (Anchor Christmas Ale)</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12 oz bottle poured into a pint glass \\t\\tPou...</td>\n",
       "      <td>2009-12-25 04:38:07</td>\n",
       "      <td>5.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>606 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       abv  beer_id  brewer_id                                    beer_name   ...    review_taste  \\\n",
       "142    8.6    52441        147             Stone 09.09.09 Vertical Epic Ale   ...             4.5   \n",
       "446    8.6    52441        147             Stone 09.09.09 Vertical Epic Ale   ...             3.5   \n",
       "714    8.6    52441        147             Stone 09.09.09 Vertical Epic Ale   ...             3.5   \n",
       "...    ...      ...        ...                                          ...   ...             ...   \n",
       "49715  6.8     1904        140                Sierra Nevada Celebration Ale   ...             2.5   \n",
       "49844  8.6    52441        147             Stone 09.09.09 Vertical Epic Ale   ...             4.0   \n",
       "49863  5.5    53863         28  Our Special Ale 2009 (Anchor Christmas Ale)   ...             4.0   \n",
       "\n",
       "                                                    text                time  alcohol  \n",
       "142     Poured in to a chimay goblet.\\t\\tPours black ... 2009-10-06 00:10:46      8.6  \n",
       "446     Pours pitch black and completely opaque witho... 2009-10-06 07:59:01      8.6  \n",
       "714     Near black pour with a mountainous 3 finger h... 2009-10-06 21:31:06      8.6  \n",
       "...                                                  ...                 ...      ...  \n",
       "49715   I've tried this beer on tap about a year ago,... 2009-12-25 01:13:38      6.8  \n",
       "49844   thanks to colonelforbin for this one!\\t\\tPour... 2009-12-25 04:10:49      8.6  \n",
       "49863   12 oz bottle poured into a pint glass \\t\\tPou... 2009-12-25 04:38:07      5.5  \n",
       "\n",
       "[606 rows x 14 columns]"
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# All reviews for the top three most reviewed beers\n",
    "print( \"Top 3 \\nBeer ID, Reviews \\n\", beer_ids[0:3], \"\\n\" )\n",
    "\n",
    "df[df.beer_id.isin(beer_ids[0:3].index)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Stone 09.09.09 Vertical Epic Ale', 'Sierra Nevada Celebration Ale',\n",
       "       'Our Special Ale 2009 (Anchor Christmas Ale)'], dtype=object)"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.beer_id.isin(beer_ids[0:3].index)].beer_name.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Positional Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "      <td>5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.5</td>\n",
       "      <td>28577</td>\n",
       "      <td>3818</td>\n",
       "      <td>Unearthly (Imperial India Pale Ale)</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The aroma has pine, wood, citrus, caramel, an...</td>\n",
       "      <td>2009-10-05 21:32:37</td>\n",
       "      <td>9.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.8</td>\n",
       "      <td>398</td>\n",
       "      <td>119</td>\n",
       "      <td>Wolaver's Pale Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>A: Pours a slightly hazy golden/orange color....</td>\n",
       "      <td>2009-10-05 21:33:14</td>\n",
       "      <td>5.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   abv  beer_id  brewer_id                            beer_name   ...    review_taste  \\\n",
       "0  7.0     2511        287                  Bell's Cherry Stout   ...             4.5   \n",
       "1  5.7    19736       9790                   Duck-Rabbit Porter   ...             4.0   \n",
       "2  4.8    11098       3182         Fürstenberg Premium Pilsener   ...             3.5   \n",
       "3  9.5    28577       3818  Unearthly (Imperial India Pale Ale)   ...             4.0   \n",
       "4  5.8      398        119                   Wolaver's Pale Ale   ...             3.0   \n",
       "\n",
       "                                                text                time  alcohol  \n",
       "0   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48      7.0  \n",
       "1   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09      5.7  \n",
       "2   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      4.8  \n",
       "3   The aroma has pine, wood, citrus, caramel, an... 2009-10-05 21:32:37      9.5  \n",
       "4   A: Pours a slightly hazy golden/orange color.... 2009-10-05 21:33:14      5.8  \n",
       "\n",
       "[5 rows x 14 columns]"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `.iloc` accessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you prefer NumPy-like indexing, use the `.iloc` accessor:\n",
    "\n",
    "Similar to `.loc`, the `.iloc` accessor allows row and column by integer position or slice only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7.0</td>\n",
       "      <td>966</td>\n",
       "      <td>365</td>\n",
       "      <td>Pike Street XXXXX Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>From notes. Pours black, thin mocha head fade...</td>\n",
       "      <td>2009-10-05 21:33:48</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11.8</td>\n",
       "      <td>43670</td>\n",
       "      <td>423</td>\n",
       "      <td>Bourbon Barrel Quad (BBQ)</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Burnt amber in color with a 1/4\" head. Aroma ...</td>\n",
       "      <td>2009-10-05 21:36:03</td>\n",
       "      <td>11.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     abv  beer_id  brewer_id                     beer_name   ...    review_taste  \\\n",
       "2    4.8    11098       3182  Fürstenberg Premium Pilsener   ...             3.5   \n",
       "5    7.0      966        365       Pike Street XXXXX Stout   ...             4.5   \n",
       "10  11.8    43670        423     Bourbon Barrel Quad (BBQ)   ...             4.5   \n",
       "\n",
       "                                                 text                time  alcohol  \n",
       "2    Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13      4.8  \n",
       "5    From notes. Pours black, thin mocha head fade... 2009-10-05 21:33:48      7.0  \n",
       "10   Burnt amber in color with a 1/4\" head. Aroma ... 2009-10-05 21:36:03     11.8  \n",
       "\n",
       "[3 rows x 14 columns]"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## implied row\n",
    "df.iloc[[2,5,10]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7.0</td>\n",
       "      <td>966</td>\n",
       "      <td>365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11.8</td>\n",
       "      <td>43670</td>\n",
       "      <td>423</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     abv  beer_id  brewer_id\n",
       "2    4.8    11098       3182\n",
       "5    7.0      966        365\n",
       "10  11.8    43670        423"
      ]
     },
     "execution_count": 178,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# row, column\n",
    "df.iloc[[2,5,10],0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Index Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas containers Series and DataFrame both have an associated Index\n",
    "* the Index can be a simple column of integers, such as 0, 1, 2, ..., but that is the least useful\n",
    "* the Index can be a column of lables, like string labels, such as \"A\", \"B\", \"C\".\n",
    "* the Index can be **hierarchical** (aka \"MultiIndex\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Index` is the farthest left column shown in the DataFrame output. The values are printed in bold. In this DataFrame each row is indexed by a unique integer from 0 to 49999."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>profile_name</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2511</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>blaheath</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.7</td>\n",
       "      <td>19736</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>GJ40</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.8</td>\n",
       "      <td>11098</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>biegaman</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.5</td>\n",
       "      <td>28577</td>\n",
       "      <td>3818</td>\n",
       "      <td>Unearthly (Imperial India Pale Ale)</td>\n",
       "      <td>...</td>\n",
       "      <td>nick76</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The aroma has pine, wood, citrus, caramel, an...</td>\n",
       "      <td>2009-10-05 21:32:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.8</td>\n",
       "      <td>398</td>\n",
       "      <td>119</td>\n",
       "      <td>Wolaver's Pale Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>champ103</td>\n",
       "      <td>3.0</td>\n",
       "      <td>A: Pours a slightly hazy golden/orange color....</td>\n",
       "      <td>2009-10-05 21:33:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   abv  beer_id  brewer_id                            beer_name         ...         profile_name  \\\n",
       "0  7.0     2511        287                  Bell's Cherry Stout         ...             blaheath   \n",
       "1  5.7    19736       9790                   Duck-Rabbit Porter         ...                 GJ40   \n",
       "2  4.8    11098       3182         Fürstenberg Premium Pilsener         ...             biegaman   \n",
       "3  9.5    28577       3818  Unearthly (Imperial India Pale Ale)         ...               nick76   \n",
       "4  5.8      398        119                   Wolaver's Pale Ale         ...             champ103   \n",
       "\n",
       "   review_taste                                               text                time  \n",
       "0           4.5   Batch 8144\\tPitch black in color with a 1/2 f... 2009-10-05 21:31:48  \n",
       "1           4.0   Sampled from a 12oz bottle in a standard pint... 2009-10-05 21:32:09  \n",
       "2           3.5   Haystack yellow with an energetic group of bu... 2009-10-05 21:32:13  \n",
       "3           4.0   The aroma has pine, wood, citrus, caramel, an... 2009-10-05 21:32:37  \n",
       "4           3.0   A: Pours a slightly hazy golden/orange color.... 2009-10-05 21:33:14  \n",
       "\n",
       "[5 rows x 13 columns]"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reread the data file, to start fresh\n",
    "df = pd.read_csv('data/beer2.csv.gz',\n",
    "                  index_col=0,\n",
    "                  parse_dates=['time'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,     9,\n",
       "            ...\n",
       "            49990, 49991, 49992, 49993, 49994, 49995, 49996, 49997, 49998, 49999],\n",
       "           dtype='int64', length=50000)"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values in the `Index` in this case are unique.\n",
    "* In general, an Index need not be unique (examples to come...)\n",
    "* There are many advantages to having a unique `Index`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The current index is unique\n",
    "df.index.is_unique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Changing the Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Index` of a DataFrame can be reset to any currently existing column or a custom Pandas `Index` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abv',\n",
       " 'beer_id',\n",
       " 'brewer_id',\n",
       " 'beer_name',\n",
       " 'beer_style',\n",
       " 'review_appearance',\n",
       " 'review_aroma',\n",
       " 'review_overall',\n",
       " 'review_palate',\n",
       " 'profile_name',\n",
       " 'review_taste',\n",
       " 'text',\n",
       " 'time']"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What column names do we have to use as an Index?\n",
    "\n",
    "list(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While each review is unique, we may want to reference this data set by other clasifications like `beer_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>...</th>\n",
       "      <th>profile_name</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2511</th>\n",
       "      <td>7.0</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Cherry Stout</td>\n",
       "      <td>American Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>blaheath</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Batch 8144\\tPitch black in color with a 1/2 f...</td>\n",
       "      <td>2009-10-05 21:31:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19736</th>\n",
       "      <td>5.7</td>\n",
       "      <td>9790</td>\n",
       "      <td>Duck-Rabbit Porter</td>\n",
       "      <td>American Porter</td>\n",
       "      <td>...</td>\n",
       "      <td>GJ40</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sampled from a 12oz bottle in a standard pint...</td>\n",
       "      <td>2009-10-05 21:32:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11098</th>\n",
       "      <td>4.8</td>\n",
       "      <td>3182</td>\n",
       "      <td>Fürstenberg Premium Pilsener</td>\n",
       "      <td>German Pilsener</td>\n",
       "      <td>...</td>\n",
       "      <td>biegaman</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Haystack yellow with an energetic group of bu...</td>\n",
       "      <td>2009-10-05 21:32:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>7.6</td>\n",
       "      <td>142</td>\n",
       "      <td>Spaten Optimator</td>\n",
       "      <td>Doppelbock</td>\n",
       "      <td>...</td>\n",
       "      <td>drcarver</td>\n",
       "      <td>3.5</td>\n",
       "      <td>I have had this beer numerous times in the pa...</td>\n",
       "      <td>2009-10-05 21:43:36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51573</th>\n",
       "      <td>6.2</td>\n",
       "      <td>96</td>\n",
       "      <td>Odd Notion (Fall 2009)</td>\n",
       "      <td>Belgian Dark Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>DannyS</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Poured from a 12 oz bottle into a pint glass....</td>\n",
       "      <td>2009-10-05 21:44:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>717</th>\n",
       "      <td>5.0</td>\n",
       "      <td>256</td>\n",
       "      <td>Downtown Brown</td>\n",
       "      <td>English Brown Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>n0rc41</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Good all around beer. This is my girlfriends ...</td>\n",
       "      <td>2009-10-05 21:44:25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         abv  brewer_id                     beer_name         beer_style         ...          \\\n",
       "beer_id                                                                          ...           \n",
       "2511     7.0        287           Bell's Cherry Stout     American Stout         ...           \n",
       "19736    5.7       9790            Duck-Rabbit Porter    American Porter         ...           \n",
       "11098    4.8       3182  Fürstenberg Premium Pilsener    German Pilsener         ...           \n",
       "...      ...        ...                           ...                ...         ...           \n",
       "99       7.6        142              Spaten Optimator         Doppelbock         ...           \n",
       "51573    6.2         96        Odd Notion (Fall 2009)   Belgian Dark Ale         ...           \n",
       "717      5.0        256                Downtown Brown  English Brown Ale         ...           \n",
       "\n",
       "         profile_name  review_taste                                               text  \\\n",
       "beer_id                                                                                  \n",
       "2511         blaheath           4.5   Batch 8144\\tPitch black in color with a 1/2 f...   \n",
       "19736            GJ40           4.0   Sampled from a 12oz bottle in a standard pint...   \n",
       "11098        biegaman           3.5   Haystack yellow with an energetic group of bu...   \n",
       "...               ...           ...                                                ...   \n",
       "99           drcarver           3.5   I have had this beer numerous times in the pa...   \n",
       "51573          DannyS           4.0   Poured from a 12 oz bottle into a pint glass....   \n",
       "717            n0rc41           4.0   Good all around beer. This is my girlfriends ...   \n",
       "\n",
       "                       time  \n",
       "beer_id                      \n",
       "2511    2009-10-05 21:31:48  \n",
       "19736   2009-10-05 21:32:09  \n",
       "11098   2009-10-05 21:32:13  \n",
       "...                     ...  \n",
       "99      2009-10-05 21:43:36  \n",
       "51573   2009-10-05 21:44:19  \n",
       "717     2009-10-05 21:44:25  \n",
       "\n",
       "[20 rows x 12 columns]"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the 'beer_id' column as the Index\n",
    "\n",
    "dfb = df.set_index('beer_id', inplace=False)  # default inplace=False, more on this later\n",
    "dfb.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Notice that even these beer IDs are not unique. That's okay.\n",
    "\n",
    "dfb.index.is_unique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>profile_name</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000</td>\n",
       "      <td>50000</td>\n",
       "      <td>50000</td>\n",
       "      <td>49991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>8762</td>\n",
       "      <td>104</td>\n",
       "      <td>4124</td>\n",
       "      <td>49977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Sierra Nevada Celebration Ale</td>\n",
       "      <td>American IPA</td>\n",
       "      <td>drabmuh</td>\n",
       "      <td>I had their taster tray of seven beers. All o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>240</td>\n",
       "      <td>4324</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            beer_name    beer_style profile_name  \\\n",
       "count                           50000         50000        50000   \n",
       "unique                           8762           104         4124   \n",
       "top     Sierra Nevada Celebration Ale  American IPA      drabmuh   \n",
       "freq                              240          4324          242   \n",
       "\n",
       "                                                     text  \n",
       "count                                               49991  \n",
       "unique                                              49977  \n",
       "top      I had their taster tray of seven beers. All o...  \n",
       "freq                                                    3  "
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use .describe() to show beer_id is NOT unique\n",
    "# Compare \"count\" vs \"unique\"\n",
    "\n",
    "dfb.select_dtypes(include=['object']).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "240"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: This index is NOT unique: \n",
    "# There are 240 entries for beer_id 1904\n",
    "\n",
    "len(dfb.loc[1904])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions/Exercises:\n",
    "\n",
    "1. How many people reviewed Pliny The Elder?\n",
    "2. What was the average review overall?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hierarchical Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default Index has only 1 \"level\" and it is not named"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.indexes.numeric.Int64Index'>\n",
      "1\n",
      "[None]\n"
     ]
    }
   ],
   "source": [
    "# Inspect the Index attributes\n",
    "print( type(df.index) )\n",
    "print( df.index.nlevels )  # number/count of levels in the Index\n",
    "print( df.index.names )    # names of levels in the Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Index set to `beer_id` is named, but still has only 1 \"level\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.indexes.numeric.Int64Index'>\n",
      "1\n",
      "['beer_id']\n"
     ]
    }
   ],
   "source": [
    "# Inspect the Index attributes\n",
    "print( type(dfb.index) )\n",
    "print( dfb.index.nlevels )  # number/count of levels in the Index\n",
    "print( dfb.index.names )    # names of levels in the Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For high dimensional data, heirarchical indexing is ideal\n",
    "* Pandas supports multi-level indexing with `MultiIndex`\n",
    "* a `MultiIndex` can be built from multiple columns, e.g. `profile_name`, `beer_id`, and `time`. \n",
    "* This will aggregate the data by those columns\n",
    "* Each of these three categories is called a `level`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>01121987</th>\n",
       "      <th>29077</th>\n",
       "      <th>2009-11-30 03:44:42</th>\n",
       "      <td>9.0</td>\n",
       "      <td>11256</td>\n",
       "      <td>Corne De Brume</td>\n",
       "      <td>Scotch Ale / Wee Heavy</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Poured into a belgian beer glass, not great h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">05Harley</th>\n",
       "      <th>1307</th>\n",
       "      <th>2009-10-06 00:10:06</th>\n",
       "      <td>8.5</td>\n",
       "      <td>428</td>\n",
       "      <td>Der Weisse Bock</td>\n",
       "      <td>Weizenbock</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Can't find the date on this one.\\t\\tPurchased...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2732</th>\n",
       "      <th>2009-12-12 01:21:36</th>\n",
       "      <td>8.0</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Consecrator Doppelbock</td>\n",
       "      <td>Doppelbock</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Bottle # 8881 (02/09)\\t\\tPurchased through We...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2899</th>\n",
       "      <th>2009-10-20 22:27:01</th>\n",
       "      <td>7.1</td>\n",
       "      <td>911</td>\n",
       "      <td>Andechser Doppelbock Dunkel</td>\n",
       "      <td>Doppelbock</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Bottle # 300310\\t\\tPurchased through Kracked ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3054</th>\n",
       "      <th>2009-11-21 02:17:41</th>\n",
       "      <td>5.0</td>\n",
       "      <td>946</td>\n",
       "      <td>Piton Lager Beer</td>\n",
       "      <td>American Adjunct Lager</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Bottled in 2007.\\t\\tPurchased in St. Lucia @ ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          abv  brewer_id                      beer_name  \\\n",
       "profile_name beer_id time                                                                 \n",
       "01121987     29077   2009-11-30 03:44:42  9.0      11256                 Corne De Brume   \n",
       "05Harley     1307    2009-10-06 00:10:06  8.5        428                Der Weisse Bock   \n",
       "             2732    2009-12-12 01:21:36  8.0        287  Bell's Consecrator Doppelbock   \n",
       "             2899    2009-10-20 22:27:01  7.1        911    Andechser Doppelbock Dunkel   \n",
       "             3054    2009-11-21 02:17:41  5.0        946               Piton Lager Beer   \n",
       "\n",
       "                                                      beer_style  \\\n",
       "profile_name beer_id time                                          \n",
       "01121987     29077   2009-11-30 03:44:42  Scotch Ale / Wee Heavy   \n",
       "05Harley     1307    2009-10-06 00:10:06              Weizenbock   \n",
       "             2732    2009-12-12 01:21:36              Doppelbock   \n",
       "             2899    2009-10-20 22:27:01              Doppelbock   \n",
       "             3054    2009-11-21 02:17:41  American Adjunct Lager   \n",
       "\n",
       "                                                                ...                          \\\n",
       "profile_name beer_id time                                       ...                           \n",
       "01121987     29077   2009-11-30 03:44:42                        ...                           \n",
       "05Harley     1307    2009-10-06 00:10:06                        ...                           \n",
       "             2732    2009-12-12 01:21:36                        ...                           \n",
       "             2899    2009-10-20 22:27:01                        ...                           \n",
       "             3054    2009-11-21 02:17:41                        ...                           \n",
       "\n",
       "                                          review_overall  review_palate  review_taste  \\\n",
       "profile_name beer_id time                                                               \n",
       "01121987     29077   2009-11-30 03:44:42             5.0            4.0           3.5   \n",
       "05Harley     1307    2009-10-06 00:10:06             4.0            4.0           4.0   \n",
       "             2732    2009-12-12 01:21:36             4.0            4.0           4.5   \n",
       "             2899    2009-10-20 22:27:01             5.0            4.0           5.0   \n",
       "             3054    2009-11-21 02:17:41             3.0            2.0           3.0   \n",
       "\n",
       "                                                                                       text  \n",
       "profile_name beer_id time                                                                    \n",
       "01121987     29077   2009-11-30 03:44:42   Poured into a belgian beer glass, not great h...  \n",
       "05Harley     1307    2009-10-06 00:10:06   Can't find the date on this one.\\t\\tPurchased...  \n",
       "             2732    2009-12-12 01:21:36   Bottle # 8881 (02/09)\\t\\tPurchased through We...  \n",
       "             2899    2009-10-20 22:27:01   Bottle # 300310\\t\\tPurchased through Kracked ...  \n",
       "             3054    2009-11-21 02:17:41   Bottled in 2007.\\t\\tPurchased in St. Lucia @ ...  \n",
       "\n",
       "[5 rows x 10 columns]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sort_index is required to fully associate profile_name with beer_id\n",
    "reviews = df.set_index(['profile_name', 'beer_id', 'time']).sort_index()\n",
    "reviews.head(5)\n",
    "\n",
    "# Notice first 3 \"column\" labels: these are the Index levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the Index has 3 levels and each is named"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.indexes.multi.MultiIndex'>\n",
      "3\n",
      "['profile_name', 'beer_id', 'time']\n"
     ]
    }
   ],
   "source": [
    "# Inspect the Index attributes\n",
    "print( type(reviews.index) )\n",
    "print( reviews.index.nlevels )  # number/count of levels in the Index\n",
    "print( reviews.index.names )    # names of levels in the Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can easily subset the data using the levels of the hierarchical index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "profile_name  beer_id  time               \n",
       "01121987      29077    2009-11-30 03:44:42                          Corne De Brume\n",
       "05Harley      1307     2009-10-06 00:10:06                         Der Weisse Bock\n",
       "              2732     2009-12-12 01:21:36           Bell's Consecrator Doppelbock\n",
       "                                                              ...                 \n",
       "zplug123      53470    2009-10-28 03:46:45    He'Brew Jewbelation Bar Mitzvah (13)\n",
       "              54089    2009-11-21 03:44:34                             Life & Limb\n",
       "zzajjber      2270     2009-11-13 15:25:02                          Carlsberg Beer\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ind_column = 'beer_name'\n",
    "reviews[ind_column]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "beer_id  time               \n",
       "1307     2009-10-06 00:10:06                       Der Weisse Bock\n",
       "2732     2009-12-12 01:21:36         Bell's Consecrator Doppelbock\n",
       "2899     2009-10-20 22:27:01           Andechser Doppelbock Dunkel\n",
       "3054     2009-11-21 02:17:41                      Piton Lager Beer\n",
       "11757    2009-11-21 01:31:58              Founders Breakfast Stout\n",
       "22381    2009-11-14 23:12:35    Dark Horse Double Crooked Tree IPA\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ind_column  = 'beer_name'\n",
    "ind_profile = '05Harley'\n",
    "reviews[ind_column][ind_profile]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time\n",
       "2009-10-06 00:10:06    Der Weisse Bock\n",
       "Name: beer_name, dtype: object"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ind_column  = 'beer_name'\n",
    "ind_profile = '05Harley'\n",
    "ind_beer_id = 1307\n",
    "reviews[ind_column][ind_profile][ind_beer_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The whole multiindex is unique\n",
    "reviews.index.is_unique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using hierarchical index in not only more convenient, but faster than using fancy and boolean selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "drabmuh           242\n",
       "corby112          230\n",
       "BeerFMAndy        202\n",
       "northyorksammy    201\n",
       "mrmanning         187\n",
       "Name: profile_name, dtype: int64"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a Series, indexed by profile_name (people who reviewed beers)\n",
    "# Take the top 5 most prolific reviewers.\n",
    "top_reviewers = (reviews\n",
    "                        .index\n",
    "                        .get_level_values('profile_name')\n",
    "                        .value_counts()\n",
    "                        .head(5)\n",
    "                )\n",
    "top_reviewers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['drabmuh', 'corby112', 'BeerFMAndy', 'northyorksammy', 'mrmanning'], dtype='object')"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The .index attribute is itself a Numpy-like array\n",
    "top_reviewers.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterating over the MultiIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(reviews)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>01121987</th>\n",
       "      <th>29077</th>\n",
       "      <th>2009-11-30 03:44:42</th>\n",
       "      <td>9.0</td>\n",
       "      <td>11256</td>\n",
       "      <td>Corne De Brume</td>\n",
       "      <td>Scotch Ale / Wee Heavy</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Poured into a belgian beer glass, not great h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">05Harley</th>\n",
       "      <th>1307</th>\n",
       "      <th>2009-10-06 00:10:06</th>\n",
       "      <td>8.5</td>\n",
       "      <td>428</td>\n",
       "      <td>Der Weisse Bock</td>\n",
       "      <td>Weizenbock</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Can't find the date on this one.\\t\\tPurchased...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2732</th>\n",
       "      <th>2009-12-12 01:21:36</th>\n",
       "      <td>8.0</td>\n",
       "      <td>287</td>\n",
       "      <td>Bell's Consecrator Doppelbock</td>\n",
       "      <td>Doppelbock</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Bottle # 8881 (02/09)\\t\\tPurchased through We...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          abv  brewer_id                      beer_name  \\\n",
       "profile_name beer_id time                                                                 \n",
       "01121987     29077   2009-11-30 03:44:42  9.0      11256                 Corne De Brume   \n",
       "05Harley     1307    2009-10-06 00:10:06  8.5        428                Der Weisse Bock   \n",
       "             2732    2009-12-12 01:21:36  8.0        287  Bell's Consecrator Doppelbock   \n",
       "\n",
       "                                                      beer_style  \\\n",
       "profile_name beer_id time                                          \n",
       "01121987     29077   2009-11-30 03:44:42  Scotch Ale / Wee Heavy   \n",
       "05Harley     1307    2009-10-06 00:10:06              Weizenbock   \n",
       "             2732    2009-12-12 01:21:36              Doppelbock   \n",
       "\n",
       "                                                                ...                          \\\n",
       "profile_name beer_id time                                       ...                           \n",
       "01121987     29077   2009-11-30 03:44:42                        ...                           \n",
       "05Harley     1307    2009-10-06 00:10:06                        ...                           \n",
       "             2732    2009-12-12 01:21:36                        ...                           \n",
       "\n",
       "                                          review_overall  review_palate  review_taste  \\\n",
       "profile_name beer_id time                                                               \n",
       "01121987     29077   2009-11-30 03:44:42             5.0            4.0           3.5   \n",
       "05Harley     1307    2009-10-06 00:10:06             4.0            4.0           4.0   \n",
       "             2732    2009-12-12 01:21:36             4.0            4.0           4.5   \n",
       "\n",
       "                                                                                       text  \n",
       "profile_name beer_id time                                                                    \n",
       "01121987     29077   2009-11-30 03:44:42   Poured into a belgian beer glass, not great h...  \n",
       "05Harley     1307    2009-10-06 00:10:06   Can't find the date on this one.\\t\\tPurchased...  \n",
       "             2732    2009-12-12 01:21:36   Bottle # 8881 (02/09)\\t\\tPurchased through We...  \n",
       "\n",
       "[3 rows x 10 columns]"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index: ('01121987', 29077, Timestamp('2009-11-30 03:44:42')) , Row: 9.0\n",
      "Index: ('05Harley', 1307, Timestamp('2009-10-06 00:10:06')) , Row: 8.5\n",
      "Index: ('05Harley', 2732, Timestamp('2009-12-12 01:21:36')) , Row: 8.0\n",
      "Index: ('05Harley', 2899, Timestamp('2009-10-20 22:27:01')) , Row: 7.1\n",
      "Index: ('05Harley', 3054, Timestamp('2009-11-21 02:17:41')) , Row: 5.0\n"
     ]
    }
   ],
   "source": [
    "# Example: iterating with a hierarchical index\n",
    "\n",
    "for index, row in reviews.head().iterrows():\n",
    "    print('Index:', index, ', Row:', '%s' % row['abv'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index:beer_id 29077 , Row: 9.0\n",
      "Index:beer_id 1307 , Row: 8.5\n",
      "Index:beer_id 2732 , Row: 8.0\n",
      "Index:beer_id 2899 , Row: 7.1\n",
      "Index:beer_id 3054 , Row: 5.0\n"
     ]
    }
   ],
   "source": [
    "# Example: iterating with a hierarchical index, printing only one level\n",
    "\n",
    "for index, row in reviews.head().iterrows():\n",
    "    print('Index:beer_id', index[1], ', Row:', '%s' % row['abv'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MultiIndex .loc accessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using MultiIndex and Hierarchical indexers\n",
    "* the `.loc` accessor now requires entries for each `level`. \n",
    "* a slice or index value can be provided for each level.\n",
    "\n",
    "Usage:\n",
    "```\n",
    "df.loc[level0, level1, level2, etc.]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remember**: using `.loc` means that the right endpoint in a slice is returned. **INCLUSIVE**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">BeerFMAndy</th>\n",
       "      <th>92</th>\n",
       "      <th>2009-12-24 21:51:46</th>\n",
       "      <td>7.2</td>\n",
       "      <td>147</td>\n",
       "      <td>Arrogant Bastard Ale</td>\n",
       "      <td>American Strong Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>22 oz bottle poured into a Sierra Nevada Impe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <th>2009-10-22 03:39:21</th>\n",
       "      <td>5.7</td>\n",
       "      <td>306</td>\n",
       "      <td>Blue Moon Harvest Moon Pumpkin Ale</td>\n",
       "      <td>Pumpkin Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12 oz bottle poured into an Imperial Pint. Ma...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">northyorksammy</th>\n",
       "      <th>54918</th>\n",
       "      <th>2009-12-21 14:56:29</th>\n",
       "      <td>12.0</td>\n",
       "      <td>2097</td>\n",
       "      <td>Kuhnhenn Bonicci Barley Wine</td>\n",
       "      <td>American Barleywine</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Winner of the amateur brewing fest,Jerry Boni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54919</th>\n",
       "      <th>2009-12-21 14:59:39</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2097</td>\n",
       "      <td>Kuhnhenn Foreign Export Stout</td>\n",
       "      <td>Foreign / Export Stout</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Thick stout, a study in chocolate. Very smoot...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1062 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             abv  brewer_id                           beer_name  \\\n",
       "profile_name   beer_id time                                                                       \n",
       "BeerFMAndy     92      2009-12-24 21:51:46   7.2        147                Arrogant Bastard Ale   \n",
       "               100     2009-10-22 03:39:21   5.7        306  Blue Moon Harvest Moon Pumpkin Ale   \n",
       "...                                          ...        ...                                 ...   \n",
       "northyorksammy 54918   2009-12-21 14:56:29  12.0       2097        Kuhnhenn Bonicci Barley Wine   \n",
       "               54919   2009-12-21 14:59:39   7.0       2097       Kuhnhenn Foreign Export Stout   \n",
       "\n",
       "                                                        beer_style  \\\n",
       "profile_name   beer_id time                                          \n",
       "BeerFMAndy     92      2009-12-24 21:51:46     American Strong Ale   \n",
       "               100     2009-10-22 03:39:21             Pumpkin Ale   \n",
       "...                                                            ...   \n",
       "northyorksammy 54918   2009-12-21 14:56:29     American Barleywine   \n",
       "               54919   2009-12-21 14:59:39  Foreign / Export Stout   \n",
       "\n",
       "                                                                  ...                          \\\n",
       "profile_name   beer_id time                                       ...                           \n",
       "BeerFMAndy     92      2009-12-24 21:51:46                        ...                           \n",
       "               100     2009-10-22 03:39:21                        ...                           \n",
       "...                                                               ...                           \n",
       "northyorksammy 54918   2009-12-21 14:56:29                        ...                           \n",
       "               54919   2009-12-21 14:59:39                        ...                           \n",
       "\n",
       "                                            review_overall  review_palate  review_taste  \\\n",
       "profile_name   beer_id time                                                               \n",
       "BeerFMAndy     92      2009-12-24 21:51:46             4.5            4.0           4.0   \n",
       "               100     2009-10-22 03:39:21             2.0            2.5           2.0   \n",
       "...                                                    ...            ...           ...   \n",
       "northyorksammy 54918   2009-12-21 14:56:29             3.0            3.5           3.5   \n",
       "               54919   2009-12-21 14:59:39             3.5            4.0           3.5   \n",
       "\n",
       "                                                                                         text  \n",
       "profile_name   beer_id time                                                                    \n",
       "BeerFMAndy     92      2009-12-24 21:51:46   22 oz bottle poured into a Sierra Nevada Impe...  \n",
       "               100     2009-10-22 03:39:21   12 oz bottle poured into an Imperial Pint. Ma...  \n",
       "...                                                                                       ...  \n",
       "northyorksammy 54918   2009-12-21 14:56:29   Winner of the amateur brewing fest,Jerry Boni...  \n",
       "               54919   2009-12-21 14:59:39   Thick stout, a study in chocolate. Very smoot...  \n",
       "\n",
       "[1062 rows x 10 columns]"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.options.display.max_rows=4\n",
    "reviews.loc[top_reviewers.index, :, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The more complete usage of MulitIindex and `.loc` is to provide the index selection in a tuple whose length is the number of `level`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>BeerFMAndy</th>\n",
       "      <th>100</th>\n",
       "      <th>2009-10-22 03:39:21</th>\n",
       "      <td>5.7</td>\n",
       "      <td>306</td>\n",
       "      <td>Blue Moon Harvest Moon Pumpkin Ale</td>\n",
       "      <td>Pumpkin Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12 oz bottle poured into an Imperial Pint. Ma...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          abv  brewer_id                           beer_name  \\\n",
       "profile_name beer_id time                                                                      \n",
       "BeerFMAndy   100     2009-10-22 03:39:21  5.7        306  Blue Moon Harvest Moon Pumpkin Ale   \n",
       "\n",
       "                                           beer_style  \\\n",
       "profile_name beer_id time                               \n",
       "BeerFMAndy   100     2009-10-22 03:39:21  Pumpkin Ale   \n",
       "\n",
       "                                                                ...                          \\\n",
       "profile_name beer_id time                                       ...                           \n",
       "BeerFMAndy   100     2009-10-22 03:39:21                        ...                           \n",
       "\n",
       "                                          review_overall  review_palate  review_taste  \\\n",
       "profile_name beer_id time                                                               \n",
       "BeerFMAndy   100     2009-10-22 03:39:21             2.0            2.5           2.0   \n",
       "\n",
       "                                                                                       text  \n",
       "profile_name beer_id time                                                                    \n",
       "BeerFMAndy   100     2009-10-22 03:39:21   12 oz bottle poured into an Imperial Pint. Ma...  \n",
       "\n",
       "[1 rows x 10 columns]"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.loc[[('BeerFMAndy',100,pd.Timestamp('2009-10-22 03:39:21'))]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-axis indexers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = [9,10,11,12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>beer_name</th>\n",
       "      <th>review_overall</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">BeerFMAndy</th>\n",
       "      <th>129</th>\n",
       "      <th>2009-11-22 16:45:58</th>\n",
       "      <td>Orval Trappist Ale</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>332</th>\n",
       "      <th>2009-11-04 00:36:28</th>\n",
       "      <td>Miller Lite</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>332</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54374</th>\n",
       "      <th>2009-11-26 18:16:13</th>\n",
       "      <td>Downtown Lites</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54375</th>\n",
       "      <th>2009-11-26 18:24:53</th>\n",
       "      <td>Scooter Ale</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   beer_name  review_overall\n",
       "profile_name beer_id time                                                   \n",
       "BeerFMAndy   129     2009-11-22 16:45:58  Orval Trappist Ale             4.5\n",
       "             332     2009-11-04 00:36:28         Miller Lite             4.0\n",
       "...                                                      ...             ...\n",
       "             54374   2009-11-26 18:16:13      Downtown Lites             4.0\n",
       "             54375   2009-11-26 18:24:53         Scooter Ale             3.0\n",
       "\n",
       "[91 rows x 2 columns]"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here are the scores for all 91 of BeerFMAndy's reviews\n",
    "# for November only\n",
    "idx=pd.IndexSlice\n",
    "reviews.loc[idx['BeerFMAndy',:,'2009-11':'2009-11'],['beer_name','review_overall']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "slice?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>beer_name</th>\n",
       "      <th>review_overall</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">BeerFMAndy</th>\n",
       "      <th>129</th>\n",
       "      <th>2009-11-22 16:45:58</th>\n",
       "      <td>Orval Trappist Ale</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>332</th>\n",
       "      <th>2009-11-04 00:36:28</th>\n",
       "      <td>Miller Lite</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>332</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54374</th>\n",
       "      <th>2009-11-26 18:16:13</th>\n",
       "      <td>Downtown Lites</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54375</th>\n",
       "      <th>2009-11-26 18:24:53</th>\n",
       "      <td>Scooter Ale</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   beer_name  review_overall\n",
       "profile_name beer_id time                                                   \n",
       "BeerFMAndy   129     2009-11-22 16:45:58  Orval Trappist Ale             4.5\n",
       "             332     2009-11-04 00:36:28         Miller Lite             4.0\n",
       "...                                                      ...             ...\n",
       "             54374   2009-11-26 18:16:13      Downtown Lites             4.0\n",
       "             54375   2009-11-26 18:24:53         Scooter Ale             3.0\n",
       "\n",
       "[91 rows x 2 columns]"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Alternative syntax using slice objects\n",
    "# slice(None) == :\n",
    "reviews.loc[('BeerFMAndy',slice(None),slice('2009-11','2009-11')),['beer_name','review_overall']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>BeerFMAndy</th>\n",
       "      <th>92</th>\n",
       "      <th>2009-12-24 21:51:46</th>\n",
       "      <td>Arrogant Bastard Ale</td>\n",
       "      <td>American Strong Ale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>northyorksammy</th>\n",
       "      <th>54919</th>\n",
       "      <th>2009-12-21 14:59:39</th>\n",
       "      <td>Kuhnhenn Foreign Export Stout</td>\n",
       "      <td>Foreign / Export Stout</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                beer_name              beer_style\n",
       "profile_name   beer_id time                                                                      \n",
       "BeerFMAndy     92      2009-12-24 21:51:46           Arrogant Bastard Ale     American Strong Ale\n",
       "northyorksammy 54919   2009-12-21 14:59:39  Kuhnhenn Foreign Export Stout  Foreign / Export Stout"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.loc[idx[top_reviewers.index, [92,54919], :], ['beer_name','beer_style']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>BeerFMAndy</th>\n",
       "      <th>92</th>\n",
       "      <th>2009-12-24 21:51:46</th>\n",
       "      <td>Arrogant Bastard Ale</td>\n",
       "      <td>American Strong Ale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>northyorksammy</th>\n",
       "      <th>54919</th>\n",
       "      <th>2009-12-21 14:59:39</th>\n",
       "      <td>Kuhnhenn Foreign Export Stout</td>\n",
       "      <td>Foreign / Export Stout</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                beer_name              beer_style\n",
       "profile_name   beer_id time                                                                      \n",
       "BeerFMAndy     92      2009-12-24 21:51:46           Arrogant Bastard Ale     American Strong Ale\n",
       "northyorksammy 54919   2009-12-21 14:59:39  Kuhnhenn Foreign Export Stout  Foreign / Export Stout"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# SQL like query\n",
    "(reviews\n",
    "        .query('profile_name in @top_reviewers.index and \\\n",
    "                beer_id in [92,54919]')\n",
    "        [['beer_name','beer_style']]\n",
    " )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile_name</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arnie2709</th>\n",
       "      <th>92</th>\n",
       "      <th>2009-11-23 17:16:39</th>\n",
       "      <td>Arrogant Bastard Ale</td>\n",
       "      <td>American Strong Ale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BatsforBeer</th>\n",
       "      <th>92</th>\n",
       "      <th>2009-11-06 04:10:13</th>\n",
       "      <td>Arrogant Bastard Ale</td>\n",
       "      <td>American Strong Ale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wahhmaster</th>\n",
       "      <th>92</th>\n",
       "      <th>2009-10-07 18:02:47</th>\n",
       "      <td>Arrogant Bastard Ale</td>\n",
       "      <td>American Strong Ale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>woosterbill</th>\n",
       "      <th>92</th>\n",
       "      <th>2009-11-12 03:00:54</th>\n",
       "      <td>Arrogant Bastard Ale</td>\n",
       "      <td>American Strong Ale</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>72 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     beer_name           beer_style\n",
       "profile_name beer_id time                                                          \n",
       "Arnie2709    92      2009-11-23 17:16:39  Arrogant Bastard Ale  American Strong Ale\n",
       "BatsforBeer  92      2009-11-06 04:10:13  Arrogant Bastard Ale  American Strong Ale\n",
       "...                                                        ...                  ...\n",
       "wahhmaster   92      2009-10-07 18:02:47  Arrogant Bastard Ale  American Strong Ale\n",
       "woosterbill  92      2009-11-12 03:00:54  Arrogant Bastard Ale  American Strong Ale\n",
       "\n",
       "[72 rows x 2 columns]"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# All review for 92 and 54919\n",
    "reviews.loc[idx[:, [92, 54919], :], ['beer_name' ,'beer_style']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Categorical data types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the Numpy `dtypes` Pandas introduces a `categorical` dtype."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview of categoricals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The categorical data type is useful in the following cases:\n",
    "\n",
    "* A string variable consisting of only a few different values. \n",
    "    * Converting such a string variable to a categorical variable will save some memory.\n",
    "* The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). \n",
    "    * By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order.\n",
    "* As a signal to other python libraries that this column should be treated as a categorical variable\n",
    "    * e.g. to use suitable statistical methods or plot types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 50000 entries, 0 to 49999\n",
      "Data columns (total 13 columns):\n",
      "abv                  48389 non-null float64\n",
      "beer_id              50000 non-null int64\n",
      "brewer_id            50000 non-null int64\n",
      "beer_name            50000 non-null object\n",
      "beer_style           50000 non-null object\n",
      "review_appearance    50000 non-null float64\n",
      "review_aroma         50000 non-null float64\n",
      "review_overall       50000 non-null float64\n",
      "review_palate        50000 non-null float64\n",
      "profile_name         50000 non-null object\n",
      "review_taste         50000 non-null float64\n",
      "text                 49991 non-null object\n",
      "time                 50000 non-null datetime64[ns]\n",
      "dtypes: datetime64[ns](1), float64(6), int64(2), object(4)\n",
      "memory usage: 5.3+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical Example: Decrease Memory Use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Candidates for `categorical` dtypes are columns that have only a small fraction of unique strings. In the next cell we'll see that there are only 104 unique `beer_style`s out of the 50000 independent reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>beer_name</th>\n",
       "      <th>beer_style</th>\n",
       "      <th>profile_name</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000</td>\n",
       "      <td>50000</td>\n",
       "      <td>50000</td>\n",
       "      <td>49991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>8762</td>\n",
       "      <td>104</td>\n",
       "      <td>4124</td>\n",
       "      <td>49977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Sierra Nevada Celebration Ale</td>\n",
       "      <td>American IPA</td>\n",
       "      <td>drabmuh</td>\n",
       "      <td>I had their taster tray of seven beers. All o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>240</td>\n",
       "      <td>4324</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            beer_name    beer_style profile_name  \\\n",
       "count                           50000         50000        50000   \n",
       "unique                           8762           104         4124   \n",
       "top     Sierra Nevada Celebration Ale  American IPA      drabmuh   \n",
       "freq                              240          4324          242   \n",
       "\n",
       "                                                     text  \n",
       "count                                               49991  \n",
       "unique                                              49977  \n",
       "top      I had their taster tray of seven beers. All o...  \n",
       "freq                                                    3  "
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select all columns of dtype object \n",
    "# Find out which ones have fewest unique values\n",
    "\n",
    "df.select_dtypes(include=['object']).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>beer_style</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>American Stout</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>American Porter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>English Pale Ale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>Russian Imperial Stout</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   beer_style\n",
       "0              American Stout\n",
       "1             American Porter\n",
       "...                       ...\n",
       "49998        English Pale Ale\n",
       "49999  Russian Imperial Stout\n",
       "\n",
       "[50000 rows x 1 columns]"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['beer_style']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 50000 entries, 0 to 49999\n",
      "Data columns (total 1 columns):\n",
      "beer_style    50000 non-null object\n",
      "dtypes: object(1)\n",
      "memory usage: 781.2+ KB\n"
     ]
    }
   ],
   "source": [
    "# fancy selection returns a DataFrame (not a Series) with a single column:\n",
    "\n",
    "df[['beer_style']].info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert to categorical:\n",
    "* `beer_style` column converted to a `categorical`\n",
    "* reassign column back into the DataFrame\n",
    "* memory usage for this column has decreased!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 50000 entries, 0 to 49999\n",
      "Data columns (total 1 columns):\n",
      "beer_style    50000 non-null category\n",
      "dtypes: category(1)\n",
      "memory usage: 440.3 KB\n"
     ]
    }
   ],
   "source": [
    "df['beer_style'] = df['beer_style'].astype('category')\n",
    "df[['beer_style']].info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A slightly harder way to select IPA styles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['American Double / Imperial IPA', 'American IPA', 'Belgian IPA',\n",
       "       'English India Pale Ale (IPA)'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cats = (df\n",
    "          .beer_style\n",
    "          .cat\n",
    "          .categories[df.beer_style.cat.categories.str.contains('IPA')]\n",
    "        )\n",
    "cats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        False\n",
       "1        False\n",
       "         ...  \n",
       "49998    False\n",
       "49999    False\n",
       "Name: beer_style, dtype: bool"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generate the boolean indexer\n",
    "df.beer_style.isin(cats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>beer_name</th>\n",
       "      <th>...</th>\n",
       "      <th>profile_name</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.5</td>\n",
       "      <td>28577</td>\n",
       "      <td>3818</td>\n",
       "      <td>Unearthly (Imperial India Pale Ale)</td>\n",
       "      <td>...</td>\n",
       "      <td>nick76</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The aroma has pine, wood, citrus, caramel, an...</td>\n",
       "      <td>2009-10-05 21:32:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6.7</td>\n",
       "      <td>6549</td>\n",
       "      <td>140</td>\n",
       "      <td>Northern Hemisphere Harvest Wet Hop Ale</td>\n",
       "      <td>...</td>\n",
       "      <td>david18</td>\n",
       "      <td>4.0</td>\n",
       "      <td>I like all of Sierra Nevada's beers but felt ...</td>\n",
       "      <td>2009-10-05 21:34:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49984</th>\n",
       "      <td>6.0</td>\n",
       "      <td>38388</td>\n",
       "      <td>3718</td>\n",
       "      <td>L'IPA Du Lièvre</td>\n",
       "      <td>...</td>\n",
       "      <td>hlance</td>\n",
       "      <td>4.5</td>\n",
       "      <td>I love this beer, seek it out when I am in Mo...</td>\n",
       "      <td>2009-12-25 16:25:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>8.0</td>\n",
       "      <td>7971</td>\n",
       "      <td>863</td>\n",
       "      <td>Pliny The Elder</td>\n",
       "      <td>...</td>\n",
       "      <td>hrking</td>\n",
       "      <td>4.0</td>\n",
       "      <td>This is a big hoppy monster of an IPA..If you...</td>\n",
       "      <td>2009-12-25 17:23:24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7774 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       abv  beer_id  brewer_id                                beer_name         ...          \\\n",
       "3      9.5    28577       3818      Unearthly (Imperial India Pale Ale)         ...           \n",
       "8      6.7     6549        140  Northern Hemisphere Harvest Wet Hop Ale         ...           \n",
       "...    ...      ...        ...                                      ...         ...           \n",
       "49984  6.0    38388       3718                          L'IPA Du Lièvre         ...           \n",
       "49996  8.0     7971        863                          Pliny The Elder         ...           \n",
       "\n",
       "      profile_name  review_taste                                               text  \\\n",
       "3           nick76           4.0   The aroma has pine, wood, citrus, caramel, an...   \n",
       "8          david18           4.0   I like all of Sierra Nevada's beers but felt ...   \n",
       "...            ...           ...                                                ...   \n",
       "49984       hlance           4.5   I love this beer, seek it out when I am in Mo...   \n",
       "49996       hrking           4.0   This is a big hoppy monster of an IPA..If you...   \n",
       "\n",
       "                     time  \n",
       "3     2009-10-05 21:32:37  \n",
       "8     2009-10-05 21:34:31  \n",
       "...                   ...  \n",
       "49984 2009-12-25 16:25:45  \n",
       "49996 2009-12-25 17:23:24  \n",
       "\n",
       "[7774 rows x 13 columns]"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.beer_style.isin(cats)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categoricals: How it Works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`categorical` series or columns store only the unique strings and provide an integer `code` lookup. \n",
    "\n",
    "Notice the dtype of the code series object. Pandas will select the smallest allowable dtype for the number of unique categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        18\n",
       "1        17\n",
       "         ..\n",
       "49998    47\n",
       "49999    89\n",
       "dtype: int8"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.beer_style.cat.codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Altbier', 'American Adjunct Lager', 'American Amber / Red Ale',\n",
       "       'American Amber / Red Lager', 'American Barleywine', 'American Black Ale',\n",
       "       'American Blonde Ale', 'American Brown Ale', 'American Dark Wheat Ale',\n",
       "       'American Double / Imperial IPA',\n",
       "       ...\n",
       "       'Scotch Ale / Wee Heavy', 'Scottish Ale', 'Scottish Gruit / Ancient Herbed Ale',\n",
       "       'Smoked Beer', 'Tripel', 'Vienna Lager', 'Weizenbock', 'Wheatwine', 'Winter Warmer',\n",
       "       'Witbier'],\n",
       "      dtype='object', length=104)"
      ]
     },
     "execution_count": 225,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.beer_style.cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.beer_style.cat.ordered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section Review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examining Data\n",
    "* Reading data files\n",
    "* Inspecting the DataFrame\n",
    "* Examining Columns\n",
    "   * Selecting Columns\n",
    "   * Adding and Removing Columns\n",
    "   * String operations\n",
    "   * DateTime operations\n",
    "* Indexing\n",
    "   * Attribute and Bracket style indexing\n",
    "   * Column then row indexing\n",
    "   * Setting/Changing the Index\n",
    "   * Boolean indexing\n",
    "   * Hierarchical indexing with MultiIndex\n",
    "   * .loc() and .iloc()\n",
    "* Categorical dtype\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas GroupBy Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "pd.options.display.max_rows    = 6\n",
    "pd.options.display.max_columns = 6\n",
    "pd.options.display.width       = 80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Groupby?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Groupby is a fundamental operation in data analysis and is well supported in pandas.\n",
    "> http://pandas.pydata.org/pandas-docs/stable/groupby.html\n",
    "\n",
    "\n",
    "A “group by” involves one or more of the following:\n",
    "\n",
    "1. **Split** a table into groups\n",
    "2. **Apply** a function to each groups\n",
    "3. **Combine** the results\n",
    "\n",
    "The split and combine are often straight-forward. The apply is more interesting\n",
    "\n",
    "In the apply step, perform one or more of the following:\n",
    "\n",
    "* **Aggregation:** compute a summary statistic, e.g. count, sum, mean, or std.\n",
    "* **Transformation:** group-specific computations and return a like-indexed.\n",
    "* **Filtration:** discard some groups, filtering based on mean\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groupby Input Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In pandas the first step in a groupby looks like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "``df.groupby( grouper )``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`grouper` can be many things\n",
    "\n",
    "- **``Series``** e.g. `df['A']` used as `df.groupby(df['A'])`\n",
    "- **`string`** e.g. `\"A\"` is a column label, used as `df.groupby('A')`, which is equivalent to `df.groupby(df['A'])`\n",
    "- **function** (to be applied on the index)\n",
    "- **dict** : groups by *values*\n",
    "- **`levels=[]`**, names of levels in a MultiIndex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quick Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A very brief example of the power of pandas groupby:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>FIRSTURL</th>\n",
       "      <th>...</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DATE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1988-01-01</th>\n",
       "      <td>gamma Cep b</td>\n",
       "      <td>Campbell 1988</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1988ApJ...331..902C</td>\n",
       "      <td>...</td>\n",
       "      <td>+77:37:55.08</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>5.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1989-01-01</th>\n",
       "      <td>HD 114762 b</td>\n",
       "      <td>Latham 1989</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1989Natur.339...38L</td>\n",
       "      <td>...</td>\n",
       "      <td>+17:31:01.66</td>\n",
       "      <td>0.894578</td>\n",
       "      <td>0.859205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992-01-01</th>\n",
       "      <td>PSR B1257+12 B</td>\n",
       "      <td>Wolszczan 1992</td>\n",
       "      <td>http://adsabs.harvard.edu//abs/1992Natur.355.....</td>\n",
       "      <td>...</td>\n",
       "      <td>+12:40:56.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      NAME        FIRSTREF  \\\n",
       "DATE                                         \n",
       "1988-01-01     gamma Cep b   Campbell 1988   \n",
       "1989-01-01     HD 114762 b     Latham 1989   \n",
       "1992-01-01  PSR B1257+12 B  Wolszczan 1992   \n",
       "\n",
       "                                                     FIRSTURL  \\\n",
       "DATE                                                            \n",
       "1988-01-01  http://adsabs.harvard.edu/abs/1988ApJ...331..902C   \n",
       "1989-01-01  http://adsabs.harvard.edu/abs/1989Natur.339...38L   \n",
       "1992-01-01  http://adsabs.harvard.edu//abs/1992Natur.355.....   \n",
       "\n",
       "                  ...          DEC_STRING(d:m:s)  STAR_MASS(msun)  \\\n",
       "DATE              ...                                               \n",
       "1988-01-01        ...               +77:37:55.08         1.260000   \n",
       "1989-01-01        ...               +17:31:01.66         0.894578   \n",
       "1992-01-01        ...                +12:40:56.4              NaN   \n",
       "\n",
       "            STAR_RADIUS(rsun)  \n",
       "DATE                           \n",
       "1988-01-01           5.010000  \n",
       "1989-01-01           0.859205  \n",
       "1992-01-01                NaN  \n",
       "\n",
       "[3 rows x 16 columns]"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = \"data/exoplanets.csv\"\n",
    "data = pd.read_csv(file, parse_dates=True, index_col='DATE')\n",
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a groupby object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grouped = data.groupby(lambda x: x.year)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.DataFrameGroupBy object at 0x11cd66828>"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(grouped.groups)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>FIRSTREF</th>\n",
       "      <th>FIRSTURL</th>\n",
       "      <th>...</th>\n",
       "      <th>DEC_STRING(d:m:s)</th>\n",
       "      <th>STAR_MASS(msun)</th>\n",
       "      <th>STAR_RADIUS(rsun)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DATE</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1997-01-01</th>\n",
       "      <td>16 Cyg B b</td>\n",
       "      <td>Cochran 1997</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1997ApJ...483..457C</td>\n",
       "      <td>...</td>\n",
       "      <td>+50:31:04.51</td>\n",
       "      <td>0.956</td>\n",
       "      <td>1.11760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-01</th>\n",
       "      <td>tau Boo b</td>\n",
       "      <td>Butler 1997</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1997ApJ...474L.115B</td>\n",
       "      <td>...</td>\n",
       "      <td>+17:27:24.39</td>\n",
       "      <td>1.341</td>\n",
       "      <td>1.41847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-01</th>\n",
       "      <td>rho CrB b</td>\n",
       "      <td>Noyes 1997</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1997ApJ...483L.111N</td>\n",
       "      <td>...</td>\n",
       "      <td>+33:18:19.40</td>\n",
       "      <td>0.965</td>\n",
       "      <td>1.10084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-01</th>\n",
       "      <td>55 Cnc b</td>\n",
       "      <td>Butler 1997</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1997ApJ...474L.115B</td>\n",
       "      <td>...</td>\n",
       "      <td>+28:19:53.00</td>\n",
       "      <td>0.905</td>\n",
       "      <td>0.94300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997-01-01</th>\n",
       "      <td>upsilon And b</td>\n",
       "      <td>Butler 1997</td>\n",
       "      <td>http://adsabs.harvard.edu/abs/1997ApJ...474L.115B</td>\n",
       "      <td>...</td>\n",
       "      <td>+41:24:22.99</td>\n",
       "      <td>1.310</td>\n",
       "      <td>1.38300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     NAME      FIRSTREF  \\\n",
       "DATE                                      \n",
       "1997-01-01     16 Cyg B b  Cochran 1997   \n",
       "1997-01-01      tau Boo b   Butler 1997   \n",
       "1997-01-01      rho CrB b    Noyes 1997   \n",
       "1997-01-01       55 Cnc b   Butler 1997   \n",
       "1997-01-01  upsilon And b   Butler 1997   \n",
       "\n",
       "                                                     FIRSTURL  \\\n",
       "DATE                                                            \n",
       "1997-01-01  http://adsabs.harvard.edu/abs/1997ApJ...483..457C   \n",
       "1997-01-01  http://adsabs.harvard.edu/abs/1997ApJ...474L.115B   \n",
       "1997-01-01  http://adsabs.harvard.edu/abs/1997ApJ...483L.111N   \n",
       "1997-01-01  http://adsabs.harvard.edu/abs/1997ApJ...474L.115B   \n",
       "1997-01-01  http://adsabs.harvard.edu/abs/1997ApJ...474L.115B   \n",
       "\n",
       "                  ...          DEC_STRING(d:m:s)  STAR_MASS(msun)  \\\n",
       "DATE              ...                                               \n",
       "1997-01-01        ...               +50:31:04.51            0.956   \n",
       "1997-01-01        ...               +17:27:24.39            1.341   \n",
       "1997-01-01        ...               +33:18:19.40            0.965   \n",
       "1997-01-01        ...               +28:19:53.00            0.905   \n",
       "1997-01-01        ...               +41:24:22.99            1.310   \n",
       "\n",
       "            STAR_RADIUS(rsun)  \n",
       "DATE                           \n",
       "1997-01-01            1.11760  \n",
       "1997-01-01            1.41847  \n",
       "1997-01-01            1.10084  \n",
       "1997-01-01            0.94300  \n",
       "1997-01-01            1.38300  \n",
       "\n",
       "[5 rows x 16 columns]"
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.get_group(1997)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['NAME', 'FIRSTREF', 'FIRSTURL', 'NUM_OBS', 'MASS(mjupiter)',\n",
       "       'RADIUS(rjupiter)', 'PERIOD(day)', 'ECCENTRICITY',\n",
       "       'SEMI_MAJOR_AXIS(AU)', 'SEPARATION(AU)', 'STAR_NAME', 'DIST(PARSEC)',\n",
       "       'RA_STRING(h:m:s)', 'DEC_STRING(d:m:s)', 'STAR_MASS(msun)',\n",
       "       'STAR_RADIUS(rsun)'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What can we aggregate? How about distance away from Earth\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1988     14.102400\n",
       "1989     38.654800\n",
       "1992           NaN\n",
       "           ...    \n",
       "2013    341.894496\n",
       "2014    246.145021\n",
       "2015    279.107200\n",
       "Name: DIST(PARSEC), dtype: float64"
      ]
     },
     "execution_count": 234,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped.agg('mean')['DIST(PARSEC)']\n",
    "\n",
    "# 14 Parsecs... Pfsss!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x12501e7b8>"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcVNWZ//HPwyK4IoqAsiigo7gFdyNZShAEnbhkUWMy\nYhy3GIwz5hcDSSaQjBuZRJNM9JXXBEVMYhDNoiYOLsE2LomaCNosQjvQsijthgtolLaf3x/nlhRN\nN137vbfq+3696tW3b926/VxL6qnznHPPMXdHRETqU7e4AxARkfgoCYiI1DElARGROqYkICJSx5QE\nRETqmJKAiEgdyzsJmFk3M1tgZndHv08zszVm9nT0mJBz7FQzazKzpWY2vhKBi4hI6XoUcOxlwGJg\nl5x917n7dbkHmdlI4AxgJDAYeNDM9nPdkCAikjh5tQTMbDBwEjCz/VMdHH4qMMfdW929GWgCji4l\nSBERqYx8y0HXA18H2n+bn2xmC81sppn1ifYNAlbnHLM22iciIgnTZRIws5OBFndfyJbf/G8Ehrv7\nKGAd8MPKhCgiIpWST5/AaOAUMzsJ2B7Y2cxudfdzco75OXBPtL0WGJLz3OBo3xbMTH0EIiJFcPeO\nSvFF6bIl4O7fdPeh7j4cOAuY7+7nmNnAnMM+DSyKtu8GzjKz7cxsGLAv8GQn567oY9q0aRX/G0l8\n6Lrr66Hrrq9HuRUyOqi975vZKKANaAYuAnD3JWY2F1gCbAIu8UpELiIiJSsoCbj7w8DD0fY52zju\nGuCa0kITEZFKq+k7hjOZTNwhxELXXV903VIKi6tSY2aqEomIFMjM8Gp2DIuISO1SEhAR6cDatfDW\nW3FHUXlKAiIiHTjvPLjqqrijqDz1CYiItPPqq7DnnnDwwbBgQdzRbEl9AiIiFfa738GnPgXNzdDS\nEnc0laUkICLSzty5cPbZMGYMPPBA3NFUlpKAiEiOV16BJ5+Ek06C8ePhvvvijqiylARERHL89rcw\ncSLssENIAg88AG1tcUdVOUoCIiI55s6FM84I28OGwS67wLPPxhtTJSkJiIhEWlrg738PLYGsE0+E\n+++PL6ZKUxIQEYn89rehL2D77Tfvq/V+ASUBEZFIbiko6/jjQ0fxxo3xxFRpSgIiIsC6deHGsAkT\ntty/005wxBHw8MPxxFVpSgIiIsBvfgP//M/Qu/fWz9Vyv0DeScDMupnZ02Z2d/R7XzO738yWmdl9\nZtYn59ipZtZkZkvNbHwlAhcRKaeOSkFZtdwvUEhL4DLCkpFZU4AH3X1/YD4wFcDMDgTOAEYCE4Eb\nzaxs81yIiJTbiy+GYaDjO/nKethhYT6hVauqG1c15JUEzGwwcBIwM2f3qcDsaHs2cFq0fQowx91b\n3b0ZaAKOLku0IiIV8JvfhLmCOioFAXTrBuPG1eYUEvm2BK4Hvg7kTvs5wN1bANx9HdA/2j8IWJ1z\n3Npon4hIIm2rFJRVqyWhLheaN7OTgRZ3X2hmmW0cWvC80NOnT/9wO5PJaM1QEam6tWth8eLwTX9b\nxo+Hr30NPvgAunevTmwADQ0NNDQ0VOz8Xa4nYGZXA18EWoHtgZ2B3wFHAhl3bzGzgcBD7j7SzKYA\n7u4zotfPA6a5+xPtzqv1BEQkdj/+cRgaesstXR97yCEwcyYcc0zFw+pU1dcTcPdvuvtQdx8OnAXM\nd/d/Ae4Bzo0OmwTcFW3fDZxlZtuZ2TBgX+DJcgUsIlJO+ZSCsmpxqGgp9wlcC4wzs2XA2Oh33H0J\nMJcwkuhe4BJ95ReRJFq9Gp57Dk44Ib/ja7FfQMtLikjduv56aGyEm2/O7/h334X+/WHNGujTp+vj\nK0HLS4qIlEkhpSAIE8sddxzMn1+5mKpNSUBE6tILL0BTE4wdW9jrxo+vrX4BJQERqUt33gmnnQY9\nexb2uhNPDP0CtVLNVhIQkbpUaCko66CD4L334P/+r/wxxUFJQETqTnMzrFgR1goolFltlYSUBESk\n7txxB5x+euGloKxaGiqqJCAidafYUlDWuHHQ0ACbNpUtpNgoCYhIXVmxIowMKmWqsn79YL/94C9/\nKVtYsVESEJG6cscd8OlPQ48up8/ctlqZQkJJQETqSqmloKxa6RfQtBEiUjeefx5Gjw7TR5faEnj/\nfdhjjzBUtF+/8sSXD00bISJSpDvugM9+tvQEALDddvDJT8KDD5Z+rjgpCYhI3ShXKSirFu4XUBIQ\nkbqwfDmsWwcf+1j5zlkLU0goCYhIXciWgsq5NOS++4ay0JIl5TtntSkJiEhNcw9rBvzyl+UtBUGY\nQiLbGkirLpOAmfUysyfMbIGZNZrZtGj/NDNbY2ZPR48JOa+ZamZNZrbUzMZX8gJERNp77TW4/XY4\n7zwYPDjMFnryyWFkULmlvV8gryGiZraDu79jZt2Bx4CvAhOBt939unbHjgRuA44CBgMPAvu1Hw+q\nIaIiUi6trfDUUzBvXvhWvmRJGLlz4okwYUIo21TKG2/AkCHwyivQu3fl/k5WLENE3f2daLMX0API\nfnp3FMipwBx3b3X3ZqAJOLrEOEWkTGbOTHdHZtaaNXDTTaHE078/XHwx/OMfcPXV4QP5nntg8uTK\nJgCAXXeFQw+FRx6p7N+plLySgJl1M7MFwDrgAXd/KnpqspktNLOZZpZdcXMQsDrn5WujfSISs5df\nhgsugJdeijuS0tx1Fxx8cBijf9JJsGgRPPMMzJgBY8ZAr17VjSfNU0jkdcuEu7cBh5nZLsDvzOxA\n4Ebge+7uZnYl8EPg/EL++PTp0z/czmQyZEqZ0UlEurRoUfi5YgXstVe8sZTiL3+Br38dvvWtuCMJ\nxo+HCy+E//qv8p+7oaGBhoaG8p84UvC0EWb2H8DG3L4AM9sbuMfdDzWzKYC7+4zouXnANHd/ot15\n1CcgUmU//jH827/BLbfApElxR1O800+Hs8+Gz30u7kiC1tZQklq8GPbcs7J/q+p9AmbWL1vqMbPt\ngXHAc2Y2MOewTwPRdwzuBs4ys+3MbBiwL/BkuQIWkeI1NsKAAaElkGbLl8P++8cdxWY9eoQF69NY\nEsqnHLQnMNvMuhGSxu3ufq+Z3Wpmo4A2oBm4CMDdl5jZXGAJsAm4RF/5RZKhsRFOOSXd6+N+8EFI\nYpXu8C1Udqho2lpYmkVUpE60tcEuu4Tx81ddBY8/HndExVmxIiwIs2pV3JFs6YUX4KijoKUl3ERW\nKZpFVESKsnIl7LYbHH54ulsCSSsFZe29d0iy2c73tFASEKkTjY1wyCEwcCC8/TZs2BB3RMVZtgz+\n6Z/ijqJjY8fCn/4UdxSFURIQqRPZJGAGw4alt3M4qS0BUBIQkQRrbAw3WAGMGJHuJJDUlsDxx8Of\n/xyGjKaFkoBInci2BACGD09vv0CSy0F77AH77BPmMUoLJQGROvCPf4SO4QMOCL+ntSXwzjthXqC9\n9447ks6NHQvz58cdRf6UBETqwNKl4YM/O6dOWlsCTU0h9nIuDFNuaesXUBIQqQOLFm0uBUF6WwJJ\n7hTO+sQnQjno3XfjjiQ/SgIidSC3PwBC3XrVqnD3bZokuT8ga+edw9TSjz0WdyT5URIQqQPtk0Dv\n3qETc82a+GIqRpJHBuUaMyY9JSElAZE60D4JQDr7BdJQDoJ09QsoCYjUuPXr4a23th5Rk7Z+Afd0\nlIMAPvrR0Bn/xhtxR9I1JQGRGtfYCAcdBN3a/WtPW0vg1VfD3c79+sUdSdd69QqJ4OGH446ka0oC\nIjWuo1IQpK8lkG0FVHKGznJKS0lISUCkxnWWBIYPT1cSSEt/QJaSgIgkwrZaAmkqB6VlZFDWYYfB\niy/CSy/FHcm25bO8ZC8ze8LMFphZo5lNi/b3NbP7zWyZmd2XXYIyem6qmTWZ2VIzG1/JCxCRzrmH\nG8WyE8fl2n33MNHZ+vXVj6sYaekUzurePSx+k/QpJLpMAu7+HnC8ux8GjAImmtnRwBTgQXffH5gP\nTAUwswOBM4CRwETgRrO0VPFEasuqVbDDDh13ppqlq18gbeUgSMc8QnmVg9z9nWizF2FdYgdOBWZH\n+2cDp0XbpwBz3L3V3ZuBJuDocgUsIvnrrBSUlZZ+gaSuK9yVbL9AklfSzSsJmFk3M1sArAMecPen\ngAHu3gLg7uuA/tHhg4DVOS9fG+0TkSprP2dQe2kZJvrCC9C/f2jVpMkBB8D77yc70fbI5yB3bwMO\nM7NdgN+Z2UGE1sAWhxX6x6dPn/7hdiaTIZPJFHoKEdmGxkYYN67z50eMgKefrl48xUpbp3CW2ebW\nwIgRxZ2joaGBhoaGssaVK68kkOXub5lZAzABaDGzAe7eYmYDgZejw9YCQ3JeNjjat5XcJCAi5dfY\nCJdf3vnzw4fDHXdUL55ipa1TONeYMTBvHlx4YXGvb/8F+bvf/W55AovkMzqoX3bkj5ltD4wDlgJ3\nA+dGh00C7oq27wbOMrPtzGwYsC/wZFmjFpEubdoU5t8/8MDOj0lLx3AaO4Wzsp3DbW1xR9KxfPoE\n9gQeMrOFwBPAfe5+LzADGGdmy4CxwLUA7r4EmAssAe4FLnFPcreISG1atgyGDoXtt+/8mKFDw1j2\n99+vXlzFSHNLYOhQ2HXX0D+TRF2Wg9y9ETi8g/2vAyd08pprgGtKjk5EitbVyCCAnj1h0KAwlDTJ\nI2/S3BKAzf0Chx4adyRb0x3DIjUqnyQAyR8hlF1XeOjQuCMpXpKnkFASEKlR+SaBpPcLpGFd4a4c\nfzw88kjop0kaJQGRGlUrLYG0l4Ig3LE9bFhYezhplAREatBbb4USyvDhXR+b9JZAmjuFcyW1JKQk\nIFKDFi0KQ0PzKaGoJVAd5ZhH6N13yxNLLiUBkRqUbykINrcEkjqQO613C7f3iU/A3/4WOrqLdeml\n5YsnS0lApAZ1NWdQrj59YLvtQvkoadK0rnBXdtoJPvIReOyx4l4/ezY8+mh5YwIlAZFEeegh2LCh\n9PM0Nna8hkBnktovkKZ1hfNRbL/AokXw//4f3Hln+WNSEhBJkIsvhpkzSzuHe2HlIEjulNJpW1e4\nK2PGFJ4ENmyAz30OfvCDwhJ7vpQERBJi06bwQTxzZmn1+Zdegm7dYMCA/F+T1KUma6VTOOvYY+G5\n5/Jfzc0dLroIjjsOJk2qTExKAiIJsXIlDBkC770Hf/1r8efJtgIK+fac9JZArejVK3yg5zsz9P/8\nT3g///u/KxeTkoBIQixbFhYhOf98+PnPiz9PoaUgSO4w0VoZGZQr36GiTz8N3/52mOq7kovpKAmI\nJET2A2/SJPjtb8MNX8UoJgkktWO41spBkF/n8Jtvhn6An/608tevJCCSENkkMHBg6ED89a+LO08x\nSWDQoDASpxI3IxUrresKd2XUKGhpCVN4d8QdzjsPJkyAM8+sfDxKAiIJsWzZ5m99F1xQ3Cih1tbQ\n8XjQQYW9rnt32HtvaG4u/G9WSlrXFe5K9+6QyXReEvrJT8K1X3dddeLJZ2WxwWY238wWm1mjmV0a\n7Z9mZmvM7OnoMSHnNVPNrMnMlprZ+EpegEityK1/jx8fvi0uXFjYOZ5/HvbcM9yYVKik9QvUWqdw\nrs6Giv71r3DVVaEfoFev6sSST0ugFbjc3Q8CPgpMNrMDoueuc/fDo8c8ADMbCZwBjAQmAjea1coo\nX5HKePvtUAceNCj83r17KAkU2kFcTCkoK2n9ArXYKZyV7RfIHQr82muh/PPzn4cZR6ulyyTg7uvc\nfWG0vYGwvnD0vyodfbifCsxx91Z3bwaagKPLE65IbVq+HPbbL4zvzzrvPJgzp7C5ZkpJAklrCdRi\np3DW/vuHPo/sf++2NjjnnNAZfOqp1Y2loD4BM9sHGEVYaxhCq2Chmc3MLkZPSBCrc162ls1JQ0Q6\n0NG33qFD4ZhjCpsqoJA5g9pLWkuglstBZluOEvr+98MNZNfEsChv3knAzHYC7gQui1oENwLD3X0U\nsA74YWVCFKl9nZU+Cr1nQC2B9MgmgYcfhh/9CG6/Paz5XG1dLjQPYGY9CAngF+5+F4C75845+HPg\nnmh7LTAk57nB0b6tTJ8+/cPtTCZDJpPJM2yR2rJsGUycuPX+T30KLrkkjPg54ICtn8+1cSOsXRvK\nSsUYPjzctdzWtmVZKg4bN6Z/XeGujBkDl18Ojz8Ot9wS7hbvSENDAw353mJcBPM8Jikxs1uBV939\n8px9A919XbT978BR7n62mR0I/Ao4hlAGegDYz9v9ITNrv0ukbh15JNxwQyj/tDdlShj6+YMfbPsc\nTz0VhpYWOqIo18CB8Pe/b+6gjsszz8DZZ8PixfHGUWkjR8JnPgNXXpn/a8wMdy/bYJsuWwJmNhr4\nAtBoZgsAB74JnG1mo4A2oBm4CMDdl5jZXGAJsAm4RJ/2Ip1z3/ZImH/9Vxg9Gq6+Osz735lSSkFZ\n2X6BuJNArZeCsh59FPr2jTeGLpOAuz8GdLRI3bxtvOYaIIYuDpH0WbcOevfu/MNgv/3CzV933RVG\nj3SmHEkg2y/w8Y+Xdp5S1XKncK7dd487At0xLBK7fMbD59NBXM6WQNzqpSWQBEoCIjHL51vvZz4T\nZpVcubLzY8rVEkhCEqiXlkASKAmIxCyfb729e8MXvgA339zx8y+/DO+/X3otPwmLy2T7SNQSqA4l\nAZGY5Ts9wvnnw6xZYaRQe8UsJNORcrUE3nyz+Ndm1xVOQr28HigJiMQsd/bQbTnkEBg8GOZ1MCSj\nHKUgCENE3347PIp1330hmbzxRnGvr7V1hZNOSUAkRps2hWmDR4zI7/gLLui4g7hcScBs801jxbrp\npjCUtdipkFUKqi4lAZEYNTfDXnvlP23wmWfCn/+89YIkpcwZ1F4p00e8/jrcfz/87/+Gm99eeaXr\n17SnTuHqUhIQiVG+paCsnXYK9wrccsvmfW1t4c7agw8uT0ylDBO97bYw/cWoUXDWWTBjRuHnUEug\nupQERGJUzJz52VXH2trC7ytXhk7UPn22/bp8ldISmDULvvSlsP2tb4XfO1tGsTNqCVSXkoBIjIpJ\nAkceCTvvDA89FH4vV39AVrEtgWefDUNVx44Nv++1V1gToZB5cWp1XeEkUxIQiVGh5SAInbe5HcTl\nTgLFtgRmzYJJk8KqaFnf+EaYIjnfjuYXXoABA2pvXeEkUxIQiVGxSyh+4QthqOirr5Y/CeyzD6xa\nFb6V5+v99+FXv4Jzz91yf79+MHkyfPe7+Z1HpaDqUxIQicmGDWE1qcGDC39t375hrYFf/CIkgXJ1\nCkO4O7l/f1i9uutjs/74x7DeQUdlnMsvh3vvhaVLuz6POoWrT0lAJCYdrStciAsugJ/9LAwz7WrB\nmUIV2i+Q2yHcXp8+8LWvwbRpXZ9HLYHqUxIQiUmxpaCs7HTPI0Zse52BYhTSL7BuHTzyyLanuZ48\nOcydv2DBts9V6n8TKZySgEhMSv3WawYXXwxHHFG+mLIKaQn88pdw2mnhHobO7LgjTJ0K//Ef2z6X\nykHV12USMLPBZjbfzBabWaOZfTXa39fM7jezZWZ2n5n1yXnNVDNrMrOlZja+khcgklbl+MC77LLC\nFqLPV74tAfdtl4JyXXhh6L/4y186fr4e1hVOonxaAq3A5e5+EPBR4CtmdgAwBXjQ3fcH5gNTAaI1\nhs8ARgITgRvNNBWUSHvlKH1061b+UhDk3xJ46il47738ViLr1Qu+851wE1lHnn8+/N3uHa1jKBXT\nZRJw93XuvjDa3gAsBQYDpwKzo8NmA6dF26cAc9y91d2bgSbg6DLHLZJq7snuBM23JTBrVhgWmu/X\nvEmTYM0a+NOftn4uyf89allBfQJmtg8wCvgrMMDdWyAkCqB/dNggIHdw2dpon4hEWlrCN+Pddos7\nko7tvnuYlmL9+s6PefddmDs3fLDnq0ePcM/At74VEmEudQrHI+8kYGY7AXcCl0UtgnZv4Va/i0gn\nkv6Bl51Selslod//PnRKDxlS2LnPPBPeeQf+8Ict96tTOB498jnIzHoQEsAv3P2uaHeLmQ1w9xYz\nGwi8HO1fC+T+bzE42reV6dOnf7idyWTIZDIFBS+SVmkofWRLQp2NPpo1K8wNVKhu3eA//xO+/W04\n+eTN90ksWwYXXVR8vLWqoaGBhoaGip3fvH2brKODzG4FXnX3y3P2zQBed/cZZvYNoK+7T4k6hn8F\nHEMoAz0A7Oft/pCZtd8lUje+/vVQcpkyJe5IOnfFFaFc1VGMq1bBYYfB2rXhDuNCucOxx4a7ic88\nM/y+227Q1BSmmpDOmRnuXrbBNvkMER0NfAEYY2YLzOxpM5sAzADGmdkyYCxwLYC7LwHmAkuAe4FL\n9GkvsqWkl4Ng253Dt94aPryLSQAQyk1XXhlGC7W2hqGhWlc4Hl2Wg9z9MaCzQVsndPKaa4BrSohL\npKaloRw0YgTcccfW+93Doja//nVp5z/hBNhzzzD/0X77aV3huOiOYZEqa20N8/0kfc78zloCjzwS\nWgBHHlna+c3gqqvCaKHGRnUKx0VJQKTKmpvDN+BiSynVMnQovPRSmCY6V/YO4XJ8ax89Gg48MCSD\npLeMapWSgEiVpaEUBNCzJwwaFBZ6ydqwIQwN/eIXy/d3rrwydDCrJRAPJQGRKkvTePj200fccQd8\n4hNh9a9yOfxw+NGPwnml+pQERKosLS0B2LpfIN/J4gp12WVhIRupPiUBkSpLa0vg+edDAjv55Hhj\nkvJSEhCpsjTcI5CV2xK45ZawtnHPnrGGJGWW17QRIlIeGzbA668XPt9OXLItgQ8+gNmzw1rBUlvU\nEhCpoqamcH9AsesKV1u2JfCnP4XO4EMOiTsiKbeU/K8oUhvSVAqCsEh8794wY0ZlOoQlfioHiVRR\nmkYGZQ0fHhaJ72gKCUk/JQGRKlq+HMaNizuKwowYAcOGJXcBHCmNkoBIFS1fDl/5StxRFGbyZOjb\nN+4opFLyWk+gIn9Y6wlInXGHXXcNo200ZbIUq+rrCYhIebz8chhjrwQgSaIkIFIlaRsZJPUhn5XF\nbjKzFjN7NmffNDNbE60yll1pLPvcVDNrMrOlZja+UoGLpE0aRwZJ7cunJTALOLGD/de5++HRYx6A\nmY0EzgBGAhOBG820VpAIpGvOIKkfXSYBd38UWN/BUx19uJ8KzHH3VndvBpqAo0uKUKRGqCUgSVRK\nn8BkM1toZjPNrE+0bxCwOueYtdE+kbqnPgFJomLvE7gR+J67u5ldCfwQOL/Qk0yfPv3D7UwmQyaT\nKTIckWRrbYWVK5O/rrAkT0NDAw0NDRU7f173CZjZ3sA97n7otp4zsymAu/uM6Ll5wDR3f6KD1+k+\nAakbzz8PJ5wQ1hcWKUVc9wkYOX0AZjYw57lPA4ui7buBs8xsOzMbBuwLPFmOQEXSTKUgSaouy0Fm\ndhuQAXY3s1XANOB4MxsFtAHNwEUA7r7EzOYCS4BNwCX6ui+ikUGSXJo2QqQKvvxlOPBAuPTSuCOR\ntNO0ESIppHKQJJWSgEgVqBwkSaVykEiFbdwI/fqF9YW7d487Gkk7lYNEUia7rrASgCSRkoBIhak/\nQJJMSUCkwjRnkCSZkoBIhalTWJJMSUCkwtQSkCRTEhCpIHf1CUiyKQmIVNArr4RRQf36xR2JSMeU\nBEQqSKUgSTolAZEKUilIkk5JQKSCNDJIkk5JQKSCVA6SpFMSEKkglYMk6TSBnEiFfPAB7LgjrF8P\n228fdzRSK6o+gZyZ3WRmLWb2bM6+vmZ2v5ktM7P7zKxPznNTzazJzJaa2fhyBSqSNitWwIABSgCS\nbPmUg2YBJ7bbNwV40N33B+YDUwHM7EDgDGAkMBG40czKlrFE0uQPf4CxY+OOQmTbukwC7v4osL7d\n7lOB2dH2bOC0aPsUYI67t7p7M9AEHF2eUEXSZc4cOOusuKMQ2bZiO4b7u3sLgLuvA/pH+wcBq3OO\nWxvtE6krK1bAypUwZkzckYhsW48ynaeoHt7p06d/uJ3JZMhkMmUKRyRet98On/0s9CjXvzCpWw0N\nDTQ0NFTs/HmNDjKzvYF73P3Q6PelQMbdW8xsIPCQu480symAu/uM6Lh5wDR3f6KDc2p0kNSsQw+F\nG26Aj3887kik1sS1vKRFj6y7gXOj7UnAXTn7zzKz7cxsGLAv8GQZ4hRJjcWL4fXXYfTouCMR6VqX\njVUzuw3IALub2SpgGnAtcIeZnQe8QBgRhLsvMbO5wBJgE3CJvu5Lvbn9djjzTOimWzElBXSzmEgZ\nuYc7hG+7DY46Ku5opBbFVQ4SkTw8/XRIBEceGXckIvlREhApo+y9AbpFUtJC5SCRMmlrg332gXvv\nhYMPjjsaqVUqB4kk1OOPwy67KAFIuigJiJTJnDnw+c/HHYVIYVQOEimD1lYYNAgeewz23TfuaKSW\nqRwkkkANDTB0qBKApI+SgEgZ/PrXmjFU0knlIJESvfce7LUXPPMMDB4cdzRS61QOEkmY+++Hgw5S\nApB0UhIQKZFKQZJmKgeJlGDjxjAqaPly6N+/6+NFSqVykEiC/PGPcMwxSgCSXkoCIiXQOsKSdioH\niRTpzTfDvQEvvAC77hp3NFIvVA4SSYjf/x6OP14JQNKtpGWwzawZeBNoAza5+9Fm1he4HdgbaAbO\ncPc3S4xTJHHmzIFJk+KOQqQ0JZWDzGwFcIS7r8/ZNwN4zd2/b2bfAPq6+5QOXqtykKTWq6/CiBHw\n4ouw445xRyP1JGnlIOvgHKcCs6Pt2cBpJf4NkcS5806YOFEJQNKv1CTgwANm9pSZnR/tG+DuLQDu\nvg7Q4DmpOZo2WmpFSX0CwGh3f8nM9gDuN7NlhMSQq9Oaz/Tp0z/czmQyZDKZEsMRqby1a+HZZ2HC\nhLgjkXrQ0NBAQ0NDxc5ftiGiZjYN2ACcD2TcvcXMBgIPufvIDo5Xn4Ck0vXXhyQwa1bckUg9Skyf\ngJntYGY7Rds7AuOBRuBu4NzosEnAXSXGKJIoukFMaknRLQEzGwb8jlDu6QH8yt2vNbPdgLnAEOAF\nwhDRNzp4vVoCkjorVsCxx4ZRQT1KLaaKFKHcLYGi/zd295XAqA72vw6cUEpQHWlrg266tU1idvvt\n8NnPKgEi4HkZAAAHHElEQVRI7UjFx+rrr8Po0bBwYdyRSL3TtNFSaxKfBF57DcaODUngIx+JOxqp\nZ4sXhy8kH/tY3JGIlE+iG7WvvBISwMknw9VXg5WtCia1ZN06uPlmaGwMq3sNGbLlY489CisltrWF\nc65eHR6rVoWfjz8OZ56psqTUlsTOItrSEhLA6afD976nBCBbcoeGBvjZz8Lyjp/7XPiG/uKLmz+8\n16wJP996Kyz80j459O8fvmhkP+SzP198MUwKlz1u6NDN2yefDDvtFPfVSz0rd8dwIpPASy/BmDHh\njszvfKfKgUmirV8Ps2eHD/8ePeDLX4YvfhH69On8Ne++uzkh5D5efjkkgvYf9IMHQ+/e1bsmkULU\nfBJYuzYkgEmT4JvfjCEwSRx3ePLJ8MH/+9+Hb+MXXxz6idRClHpT00lg1aqQAC68EK64IpawpMya\nm8OH9wMPQL9+sOee4TFw4Obt7O/tyywbNsBtt4XXv/lm+OA/99xQ4xepVzWbBJqbQwK49FL493+P\nJSQpk7Y2ePBBuOEGePRROOecMLb+rbdCqW/duvCz/Xb37puTwq67wiOPwCc/GUo+J5ygDlkRqNEk\nsGJFSABf+1pIApJOb74Z6vU33BBq6pMnw9ln5zfdsju8/fbmhPDyy3DccaE+LyKb1VwSaGoKo4Cm\nTg3f+CR9Fi0KH/xz5sCJJ4YPf9XrRSojMdNGlMOyZSEBTJsGF1wQZyRSqE2bQiftT38Kzz8f+nGW\nLAmlHBFJj1hbAnvt5Vx5JXzpS7GEUJMWL4Z588KHdGvr5kdXv2dlv71v62dbG8yfD/vuG771n3Ya\n9OxZvWsUqWc11RK49lr4l3+JM4Las3Hj5hkue/YMP3v3DiNvsr9nH9nfu3cPH+7Z7wP5/LziCjjk\nkOpem4iUX+x9AiIikr/ELCojIiLpV7EkYGYTzOw5M1tuZt+o1N8REZHiVSQJmFk34KfAicBBwOfN\n7IBK/K1tqeTizEmm664vum4pRaVaAkcDTe7+grtvAuYAp1bob3WqXv8n0XXXF123lKJSSWAQsDrn\n9zXRPhERSRB1DIuI1LGKDBE1s2OB6e4+Ifp9CuDuPiPnGI0PFREpQuLnDjKz7sAyYCzwEvAk8Hl3\nX1r2PyYiIkWryB3D7v6BmU0G7ieUnG5SAhARSZ7Y7hgWEZH4pa5j2MxuMrMWM3s2Z9+hZva4mT1j\nZneZ2U7R/p5mdrOZPWtmC8zskzmvOTzav9zMfhTHtRSijNf9UHQT3wIze9rM+sVxPfkws8FmNt/M\nFptZo5l9Ndrf18zuN7NlZnafmfXJec1UM2sys6VmNj5nf2re7zJfd82+32a2W3T822b2k3bnqtn3\nu4vrLvz9dvdUPYCPAaOAZ3P2PQl8LNo+F/hetH0JoRQFsAfwt5zXPAEcFW3fC5wY97VV6bofAg6L\n+3ryvOaBwKhoeydCP9MBwAzgimj/N4Bro+0DgQWEMuc+wPNsbu2m5v0u83XX8vu9A3AccCHwk3bn\nquX3e1vXXfD7nbqWgLs/Cqxvt3u/aD/Ag8Cno+0DgfnR614B3jCzI81sILCzuz8VHXcrcFplIy9N\nOa4753WpeN/dfZ27L4y2NwBLgcGEGw9nR4fNZvN7dwowx91b3b0ZaAKOTtv7Xa7rzjllTb7f7v6O\nuz8OvJd7nlp/vzu77hwFvd+p+J8jD4vN7JRo+wxgSLT9DHCKmXU3s2HAEdFzgwg3sGWl9Wa2Qq87\n65aoqfjtKsZaEjPbh9AS+iswwN1bIPwDAvpHh7W/SXFttC+173eJ151Vq+93Z2r9/e5KQe93rSSB\n84CvmNlTwI7A+9H+mwn/IJ4CrgMeAz6IJcLKKOa6z3b3Q4CPAx83sy9WN+TCRX0ddwKXRd+U2o9m\nqMnRDWW6br3fKRHX+10TScDdl7v7ie5+FGGeov+L9n/g7pe7++HufjrQF1hO+IDM/WY8ONqXKkVc\nN+7+UvRzI3AbW5YNEsfMehD+YfzC3e+KdreY2YDo+YHAy9H+zt7X1L3fZbruWn+/O1Pr73eninm/\n05oELHqEX8z2iH52A74N/Cz6fXsz2yHaHgdscvfnoqbVm2Z2tJkZcA5wF8lX0nVH5aHdo/09gX8G\nFlX3Egp2M7DE3X+cs+9uQkc4wCQ2v3d3A2eZ2XZRGWxf4MmUvt8lX3cdvN+5Pvx3UQfvd67cz4Pi\n3u+4e8YLfRCy24uETpFVwJeArxJ61J8Drs45du9o32LCjWtDcp47AmgkdKL9OO7rqsZ1E0YV/A1Y\nGF379USjSJL4AEYTylgLCaNfngYmALsROsKXRde3a85rphJGxywFxqfx/S7XddfJ+70SeBV4K/p3\ncUCdvN9bXXex77duFhMRqWNpLQeJiEgZKAmIiNQxJQERkTqmJCAiUseUBERE6piSgIhIHVMSEBGp\nY0oCIiJ17P8DSoY683kN5ZwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x126c78470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Early detections were planets around nearby stars\n",
    "# Later detections found around stars farther away\n",
    "\n",
    "grouped.agg('mean')['DIST(PARSEC)'].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GroupBy with Real Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Choose your Groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the same dataset of beer reviews seen in previous notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/beer2.csv.gz',\n",
    "                  index_col=0,\n",
    "                  parse_dates=['time'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The unique values in the column (in this case, `beer_style`) will make up the **keys** or \"group names\" in the groupby."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                    American Stout\n",
       "1                   American Porter\n",
       "2                   German Pilsener\n",
       "3    American Double / Imperial IPA\n",
       "4           American Pale Ale (APA)\n",
       "Name: beer_style, dtype: object"
      ]
     },
     "execution_count": 237,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example values in the `beer_style` column\n",
    "\n",
    "df['beer_style'].iloc[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50000\n",
      "104\n"
     ]
    }
   ],
   "source": [
    "# The unique values in the `beer_style` column will be the keys of our groupby\n",
    "print( len( df['beer_style'] ) )\n",
    "print( len( df['beer_style'].unique() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create GroupBy Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.groupby.DataFrameGroupBy'>\n"
     ]
    }
   ],
   "source": [
    "gr = df.groupby('beer_style')\n",
    "\n",
    "print( type( gr ) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "104"
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The number of groups matches the number of unique entries in the column\n",
    "\n",
    "gr.ngroups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Scotch Ale / Wee Heavy',\n",
       " 'Hefeweizen',\n",
       " 'Rauchbier',\n",
       " 'Doppelbock',\n",
       " 'Milk / Sweet Stout']"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example of those column entries\n",
    "\n",
    "list(gr.groups)[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 267 entries, 14 to 49912\n",
      "Data columns (total 13 columns):\n",
      "abv                  259 non-null float64\n",
      "beer_id              267 non-null int64\n",
      "brewer_id            267 non-null int64\n",
      "beer_name            267 non-null object\n",
      "beer_style           267 non-null object\n",
      "review_appearance    267 non-null float64\n",
      "review_aroma         267 non-null float64\n",
      "review_overall       267 non-null float64\n",
      "review_palate        267 non-null float64\n",
      "profile_name         267 non-null object\n",
      "review_taste         267 non-null float64\n",
      "text                 266 non-null object\n",
      "time                 267 non-null datetime64[ns]\n",
      "dtypes: datetime64[ns](1), float64(6), int64(2), object(4)\n",
      "memory usage: 29.2+ KB\n"
     ]
    }
   ],
   "source": [
    "# Separating one group from many\n",
    "\n",
    "df_dunkel = gr.get_group('Dunkelweizen')\n",
    "df_dunkel.info()\n",
    "\n",
    "# note the number of rows is much less!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "267"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_dunkel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5.2</td>\n",
       "      <td>53130</td>\n",
       "      <td>1114</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>A-Very deep, dark brown with a thin tan head ...</td>\n",
       "      <td>2009-10-05 21:42:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>208</th>\n",
       "      <td>5.2</td>\n",
       "      <td>53130</td>\n",
       "      <td>1114</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Pours a very deep, dark chestnut brown which ...</td>\n",
       "      <td>2009-10-06 01:13:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>5.8</td>\n",
       "      <td>135</td>\n",
       "      <td>39</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>A - looks like a polluted hefe(in a good way)...</td>\n",
       "      <td>2009-10-06 01:54:56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>5.6</td>\n",
       "      <td>2041</td>\n",
       "      <td>703</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>Poured from a 16.9oz bottle into a US tumbler...</td>\n",
       "      <td>2009-10-06 02:20:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>5.4</td>\n",
       "      <td>17140</td>\n",
       "      <td>412</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>One of the new 500ml swing-top bottles. Pours...</td>\n",
       "      <td>2009-10-06 03:49:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     abv  beer_id  brewer_id         ...         review_taste  \\\n",
       "14   5.2    53130       1114         ...                  4.0   \n",
       "208  5.2    53130       1114         ...                  3.0   \n",
       "260  5.8      135         39         ...                  4.0   \n",
       "292  5.6     2041        703         ...                  3.5   \n",
       "372  5.4    17140        412         ...                  3.5   \n",
       "\n",
       "                                                  text                time  \n",
       "14    A-Very deep, dark brown with a thin tan head ... 2009-10-05 21:42:35  \n",
       "208   Pours a very deep, dark chestnut brown which ... 2009-10-06 01:13:42  \n",
       "260   A - looks like a polluted hefe(in a good way)... 2009-10-06 01:54:56  \n",
       "292   Poured from a 16.9oz bottle into a US tumbler... 2009-10-06 02:20:52  \n",
       "372   One of the new 500ml swing-top bottles. Pours... 2009-10-06 03:49:21  \n",
       "\n",
       "[5 rows x 13 columns]"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The ordering of the columsn is not ideal for viewing results...\n",
    "\n",
    "df_dunkel.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['beer_style',\n",
       " 'abv',\n",
       " 'beer_id',\n",
       " 'beer_name',\n",
       " 'brewer_id',\n",
       " 'profile_name',\n",
       " 'review_appearance',\n",
       " 'review_aroma',\n",
       " 'review_overall',\n",
       " 'review_palate',\n",
       " 'review_taste',\n",
       " 'text',\n",
       " 'time']"
      ]
     },
     "execution_count": 245,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For convenient viewing of groupby results\n",
    "# Rearrange the columns such that the groupby keys are first\n",
    "\n",
    "cols = ['beer_style'] + df.columns.difference(['beer_style']).tolist()\n",
    "cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>beer_style</th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1631</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Was perusing a store for some new items, and ...</td>\n",
       "      <td>2009-10-05 22:12:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>8.5</td>\n",
       "      <td>5426</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Appearance: Pours a very cloudy chestnut colo...</td>\n",
       "      <td>2009-10-05 23:02:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2566</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Poured a nice slightly cloudy golden yellow w...</td>\n",
       "      <td>2009-10-05 23:31:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2566</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Pours hazy amber-orange in color with a massi...</td>\n",
       "      <td>2009-10-05 23:36:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>9.5</td>\n",
       "      <td>646</td>\n",
       "      <td>...</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Pours a murky slightly orangish gold. Huge pe...</td>\n",
       "      <td>2009-10-06 00:44:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    beer_style   abv  beer_id         ...         review_taste  \\\n",
       "42      Tripel   8.0     1631         ...                  4.0   \n",
       "79      Tripel   8.5     5426         ...                  4.0   \n",
       "102     Tripel  10.0     2566         ...                  4.0   \n",
       "110     Tripel  10.0     2566         ...                  4.0   \n",
       "171     Tripel   9.5      646         ...                  4.5   \n",
       "\n",
       "                                                  text                time  \n",
       "42    Was perusing a store for some new items, and ... 2009-10-05 22:12:23  \n",
       "79    Appearance: Pours a very cloudy chestnut colo... 2009-10-05 23:02:42  \n",
       "102   Poured a nice slightly cloudy golden yellow w... 2009-10-05 23:31:27  \n",
       "110   Pours hazy amber-orange in color with a massi... 2009-10-05 23:36:42  \n",
       "171   Pours a murky slightly orangish gold. Huge pe... 2009-10-06 00:44:18  \n",
       "\n",
       "[5 rows x 13 columns]"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Another group: Separate the 'Tripel' group from the others\n",
    "\n",
    "df_tripel = gr.get_group('Tripel')[cols]\n",
    "df_tripel.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 809 entries, 42 to 49949\n",
      "Data columns (total 13 columns):\n",
      "beer_style           809 non-null object\n",
      "abv                  793 non-null float64\n",
      "beer_id              809 non-null int64\n",
      "beer_name            809 non-null object\n",
      "brewer_id            809 non-null int64\n",
      "profile_name         809 non-null object\n",
      "review_appearance    809 non-null float64\n",
      "review_aroma         809 non-null float64\n",
      "review_overall       809 non-null float64\n",
      "review_palate        809 non-null float64\n",
      "review_taste         809 non-null float64\n",
      "text                 809 non-null object\n",
      "time                 809 non-null datetime64[ns]\n",
      "dtypes: datetime64[ns](1), float64(6), int64(2), object(4)\n",
      "memory usage: 88.5+ KB\n"
     ]
    }
   ],
   "source": [
    "# Inspet the group\n",
    "\n",
    "df_tripel.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>beer_style</th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>...</th>\n",
       "      <th>review_taste</th>\n",
       "      <th>text</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1631</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Was perusing a store for some new items, and ...</td>\n",
       "      <td>2009-10-05 22:12:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>8.5</td>\n",
       "      <td>5426</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Appearance: Pours a very cloudy chestnut colo...</td>\n",
       "      <td>2009-10-05 23:02:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2566</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Poured a nice slightly cloudy golden yellow w...</td>\n",
       "      <td>2009-10-05 23:31:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49882</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1631</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Pours a golden amber, with a big head that st...</td>\n",
       "      <td>2009-12-25 05:02:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49945</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>7.8</td>\n",
       "      <td>1054</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>A- It's pretty. Nice golden yellow. Nice and ...</td>\n",
       "      <td>2009-12-25 08:41:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49949</th>\n",
       "      <td>Tripel</td>\n",
       "      <td>8.0</td>\n",
       "      <td>658</td>\n",
       "      <td>...</td>\n",
       "      <td>3.5</td>\n",
       "      <td>From a 750 ml caged and corked brown bottle. ...</td>\n",
       "      <td>2009-12-25 09:39:25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>809 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      beer_style   abv  beer_id         ...         review_taste  \\\n",
       "42        Tripel   8.0     1631         ...                  4.0   \n",
       "79        Tripel   8.5     5426         ...                  4.0   \n",
       "102       Tripel  10.0     2566         ...                  4.0   \n",
       "...          ...   ...      ...         ...                  ...   \n",
       "49882     Tripel   8.0     1631         ...                  4.0   \n",
       "49945     Tripel   7.8     1054         ...                  3.5   \n",
       "49949     Tripel   8.0      658         ...                  3.5   \n",
       "\n",
       "                                                    text                time  \n",
       "42      Was perusing a store for some new items, and ... 2009-10-05 22:12:23  \n",
       "79      Appearance: Pours a very cloudy chestnut colo... 2009-10-05 23:02:42  \n",
       "102     Poured a nice slightly cloudy golden yellow w... 2009-10-05 23:31:27  \n",
       "...                                                  ...                 ...  \n",
       "49882   Pours a golden amber, with a big head that st... 2009-12-25 05:02:03  \n",
       "49945   A- It's pretty. Nice golden yellow. Nice and ... 2009-12-25 08:41:01  \n",
       "49949   From a 750 ml caged and corked brown bottle. ... 2009-12-25 09:39:25  \n",
       "\n",
       "[809 rows x 13 columns]"
      ]
     },
     "execution_count": 248,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Alternatively, you could extract a \"group\" using .loc()\n",
    "\n",
    "df.loc[df.beer_style=='Tripel',cols]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Apply"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the **split-apply-combine** strategy, during the apply step, one often aggregates across a group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.groupby.DataFrameGroupBy"
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(gr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beer_style</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Altbier</th>\n",
       "      <td>5.936331</td>\n",
       "      <td>28241.698324</td>\n",
       "      <td>4724.486034</td>\n",
       "      <td>...</td>\n",
       "      <td>3.790503</td>\n",
       "      <td>3.689944</td>\n",
       "      <td>3.703911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Adjunct Lager</th>\n",
       "      <td>4.880552</td>\n",
       "      <td>9403.638070</td>\n",
       "      <td>1541.485255</td>\n",
       "      <td>...</td>\n",
       "      <td>3.071716</td>\n",
       "      <td>2.733244</td>\n",
       "      <td>2.674933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Amber / Red Ale</th>\n",
       "      <td>6.135963</td>\n",
       "      <td>23557.469145</td>\n",
       "      <td>3314.523420</td>\n",
       "      <td>...</td>\n",
       "      <td>3.833829</td>\n",
       "      <td>3.694796</td>\n",
       "      <td>3.734944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wheatwine</th>\n",
       "      <td>10.766667</td>\n",
       "      <td>44146.641618</td>\n",
       "      <td>2363.132948</td>\n",
       "      <td>...</td>\n",
       "      <td>3.722543</td>\n",
       "      <td>3.872832</td>\n",
       "      <td>3.947977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Winter Warmer</th>\n",
       "      <td>6.520407</td>\n",
       "      <td>22726.750603</td>\n",
       "      <td>2315.001609</td>\n",
       "      <td>...</td>\n",
       "      <td>3.805310</td>\n",
       "      <td>3.703540</td>\n",
       "      <td>3.767900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Witbier</th>\n",
       "      <td>5.850545</td>\n",
       "      <td>29447.023904</td>\n",
       "      <td>3096.407703</td>\n",
       "      <td>...</td>\n",
       "      <td>3.742364</td>\n",
       "      <td>3.567729</td>\n",
       "      <td>3.624170</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>104 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                abv       beer_id    brewer_id      ...       \\\n",
       "beer_style                                                          ...        \n",
       "Altbier                    5.936331  28241.698324  4724.486034      ...        \n",
       "American Adjunct Lager     4.880552   9403.638070  1541.485255      ...        \n",
       "American Amber / Red Ale   6.135963  23557.469145  3314.523420      ...        \n",
       "...                             ...           ...          ...      ...        \n",
       "Wheatwine                 10.766667  44146.641618  2363.132948      ...        \n",
       "Winter Warmer              6.520407  22726.750603  2315.001609      ...        \n",
       "Witbier                    5.850545  29447.023904  3096.407703      ...        \n",
       "\n",
       "                          review_overall  review_palate  review_taste  \n",
       "beer_style                                                             \n",
       "Altbier                         3.790503       3.689944      3.703911  \n",
       "American Adjunct Lager          3.071716       2.733244      2.674933  \n",
       "American Amber / Red Ale        3.833829       3.694796      3.734944  \n",
       "...                                  ...            ...           ...  \n",
       "Wheatwine                       3.722543       3.872832      3.947977  \n",
       "Winter Warmer                   3.805310       3.703540      3.767900  \n",
       "Witbier                         3.742364       3.567729      3.624170  \n",
       "\n",
       "[104 rows x 8 columns]"
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Aggregation methods apply to each group separately.\n",
    "\n",
    "gr.agg('mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This says apply the `mean` function to each column. \n",
    "Non-numeric columns (nuisance columns) are excluded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also select a subset of columns to perform the aggregation on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_appearance</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beer_style</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Altbier</th>\n",
       "      <td>5.936331</td>\n",
       "      <td>3.790503</td>\n",
       "      <td>3.776536</td>\n",
       "      <td>3.689944</td>\n",
       "      <td>3.703911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Adjunct Lager</th>\n",
       "      <td>4.880552</td>\n",
       "      <td>3.071716</td>\n",
       "      <td>2.783512</td>\n",
       "      <td>2.733244</td>\n",
       "      <td>2.674933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Amber / Red Ale</th>\n",
       "      <td>6.135963</td>\n",
       "      <td>3.833829</td>\n",
       "      <td>3.836431</td>\n",
       "      <td>3.694796</td>\n",
       "      <td>3.734944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wheatwine</th>\n",
       "      <td>10.766667</td>\n",
       "      <td>3.722543</td>\n",
       "      <td>3.916185</td>\n",
       "      <td>3.872832</td>\n",
       "      <td>3.947977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Winter Warmer</th>\n",
       "      <td>6.520407</td>\n",
       "      <td>3.805310</td>\n",
       "      <td>3.864441</td>\n",
       "      <td>3.703540</td>\n",
       "      <td>3.767900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Witbier</th>\n",
       "      <td>5.850545</td>\n",
       "      <td>3.742364</td>\n",
       "      <td>3.634794</td>\n",
       "      <td>3.567729</td>\n",
       "      <td>3.624170</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>104 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                abv  review_overall  review_appearance  \\\n",
       "beer_style                                                               \n",
       "Altbier                    5.936331        3.790503           3.776536   \n",
       "American Adjunct Lager     4.880552        3.071716           2.783512   \n",
       "American Amber / Red Ale   6.135963        3.833829           3.836431   \n",
       "...                             ...             ...                ...   \n",
       "Wheatwine                 10.766667        3.722543           3.916185   \n",
       "Winter Warmer              6.520407        3.805310           3.864441   \n",
       "Witbier                    5.850545        3.742364           3.634794   \n",
       "\n",
       "                          review_palate  review_taste  \n",
       "beer_style                                             \n",
       "Altbier                        3.689944      3.703911  \n",
       "American Adjunct Lager         2.733244      2.674933  \n",
       "American Amber / Red Ale       3.694796      3.734944  \n",
       "...                                 ...           ...  \n",
       "Wheatwine                      3.872832      3.947977  \n",
       "Winter Warmer                  3.703540      3.767900  \n",
       "Witbier                        3.567729      3.624170  \n",
       "\n",
       "[104 rows x 5 columns]"
      ]
     },
     "execution_count": 251,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply aggregation method only to a subset of columns\n",
    "\n",
    "review_columns = ['abv','review_overall','review_appearance',\n",
    "                  'review_palate','review_taste']\n",
    "gr[review_columns].agg('mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `.` attribute lookup works as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "beer_style\n",
       "Altbier                      5.936331\n",
       "American Adjunct Lager       4.880552\n",
       "American Amber / Red Ale     6.135963\n",
       "                              ...    \n",
       "Wheatwine                   10.766667\n",
       "Winter Warmer                6.520407\n",
       "Witbier                      5.850545\n",
       "Name: abv, dtype: float64"
      ]
     },
     "execution_count": 252,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gr.abv.agg('mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Find the `beer_style` with the greatest variance in `abv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "beer_style\n",
       "American Strong Ale                 3.506721\n",
       "American Double / Imperial Stout    2.785742\n",
       "Flanders Oud Bruin                  2.723145\n",
       "                                      ...   \n",
       "Gose                                0.185557\n",
       "Low Alcohol Beer                    0.121006\n",
       "Happoshu                            0.000000\n",
       "Name: abv, dtype: float64"
      ]
     },
     "execution_count": 253,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ex: groupby, then select abv, then aggregrate with std, then sort\n",
    "\n",
    "(df\n",
    "   .groupby('beer_style')\n",
    "   .abv\n",
    "   .std()\n",
    "   .sort_values(ascending=False)\n",
    " )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: the pattern used above is a common idiom in Pandas: a string of `.method()` calls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Multiple Aggregations on one column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beer_style</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Altbier</th>\n",
       "      <td>3.581006</td>\n",
       "      <td>0.561276</td>\n",
       "      <td>179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Adjunct Lager</th>\n",
       "      <td>2.479893</td>\n",
       "      <td>0.732925</td>\n",
       "      <td>746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Amber / Red Ale</th>\n",
       "      <td>3.653532</td>\n",
       "      <td>0.588231</td>\n",
       "      <td>1345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wheatwine</th>\n",
       "      <td>3.942197</td>\n",
       "      <td>0.484780</td>\n",
       "      <td>173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Winter Warmer</th>\n",
       "      <td>3.738938</td>\n",
       "      <td>0.579463</td>\n",
       "      <td>1243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Witbier</th>\n",
       "      <td>3.619522</td>\n",
       "      <td>0.589968</td>\n",
       "      <td>753</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>104 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                              mean       std  count\n",
       "beer_style                                         \n",
       "Altbier                   3.581006  0.561276    179\n",
       "American Adjunct Lager    2.479893  0.732925    746\n",
       "American Amber / Red Ale  3.653532  0.588231   1345\n",
       "...                            ...       ...    ...\n",
       "Wheatwine                 3.942197  0.484780    173\n",
       "Winter Warmer             3.738938  0.579463   1243\n",
       "Witbier                   3.619522  0.589968    753\n",
       "\n",
       "[104 rows x 3 columns]"
      ]
     },
     "execution_count": 254,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gr['review_aroma'].agg([np.mean, np.std, 'count'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Single Aggregation on multiple columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_appearance</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beer_style</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Altbier</th>\n",
       "      <td>5.936331</td>\n",
       "      <td>3.790503</td>\n",
       "      <td>3.776536</td>\n",
       "      <td>3.689944</td>\n",
       "      <td>3.703911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Adjunct Lager</th>\n",
       "      <td>4.880552</td>\n",
       "      <td>3.071716</td>\n",
       "      <td>2.783512</td>\n",
       "      <td>2.733244</td>\n",
       "      <td>2.674933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Amber / Red Ale</th>\n",
       "      <td>6.135963</td>\n",
       "      <td>3.833829</td>\n",
       "      <td>3.836431</td>\n",
       "      <td>3.694796</td>\n",
       "      <td>3.734944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wheatwine</th>\n",
       "      <td>10.766667</td>\n",
       "      <td>3.722543</td>\n",
       "      <td>3.916185</td>\n",
       "      <td>3.872832</td>\n",
       "      <td>3.947977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Winter Warmer</th>\n",
       "      <td>6.520407</td>\n",
       "      <td>3.805310</td>\n",
       "      <td>3.864441</td>\n",
       "      <td>3.703540</td>\n",
       "      <td>3.767900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Witbier</th>\n",
       "      <td>5.850545</td>\n",
       "      <td>3.742364</td>\n",
       "      <td>3.634794</td>\n",
       "      <td>3.567729</td>\n",
       "      <td>3.624170</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>104 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                abv  review_overall  review_appearance  \\\n",
       "beer_style                                                               \n",
       "Altbier                    5.936331        3.790503           3.776536   \n",
       "American Adjunct Lager     4.880552        3.071716           2.783512   \n",
       "American Amber / Red Ale   6.135963        3.833829           3.836431   \n",
       "...                             ...             ...                ...   \n",
       "Wheatwine                 10.766667        3.722543           3.916185   \n",
       "Winter Warmer              6.520407        3.805310           3.864441   \n",
       "Witbier                    5.850545        3.742364           3.634794   \n",
       "\n",
       "                          review_palate  review_taste  \n",
       "beer_style                                             \n",
       "Altbier                        3.689944      3.703911  \n",
       "American Adjunct Lager         2.733244      2.674933  \n",
       "American Amber / Red Ale       3.694796      3.734944  \n",
       "...                                 ...           ...  \n",
       "Wheatwine                      3.872832      3.947977  \n",
       "Winter Warmer                  3.703540      3.767900  \n",
       "Witbier                        3.567729      3.624170  \n",
       "\n",
       "[104 rows x 5 columns]"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gr[review_columns].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Multiple aggregations on multiple columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>characteristic</th>\n",
       "      <th colspan=\"3\" halign=\"left\">abv</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"3\" halign=\"left\">review_taste</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>measure</th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>std</th>\n",
       "      <th>...</th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beer_style</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Altbier</th>\n",
       "      <td>5.936331</td>\n",
       "      <td>169</td>\n",
       "      <td>1.394085</td>\n",
       "      <td>...</td>\n",
       "      <td>3.703911</td>\n",
       "      <td>179</td>\n",
       "      <td>0.608033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Adjunct Lager</th>\n",
       "      <td>4.880552</td>\n",
       "      <td>706</td>\n",
       "      <td>0.405311</td>\n",
       "      <td>...</td>\n",
       "      <td>2.674933</td>\n",
       "      <td>746</td>\n",
       "      <td>0.815884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>American Amber / Red Ale</th>\n",
       "      <td>6.135963</td>\n",
       "      <td>1256</td>\n",
       "      <td>1.299676</td>\n",
       "      <td>...</td>\n",
       "      <td>3.734944</td>\n",
       "      <td>1345</td>\n",
       "      <td>0.640439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wheatwine</th>\n",
       "      <td>10.766667</td>\n",
       "      <td>171</td>\n",
       "      <td>1.043411</td>\n",
       "      <td>...</td>\n",
       "      <td>3.947977</td>\n",
       "      <td>173</td>\n",
       "      <td>0.624275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Winter Warmer</th>\n",
       "      <td>6.520407</td>\n",
       "      <td>1228</td>\n",
       "      <td>1.079628</td>\n",
       "      <td>...</td>\n",
       "      <td>3.767900</td>\n",
       "      <td>1243</td>\n",
       "      <td>0.621533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Witbier</th>\n",
       "      <td>5.850545</td>\n",
       "      <td>716</td>\n",
       "      <td>1.561416</td>\n",
       "      <td>...</td>\n",
       "      <td>3.624170</td>\n",
       "      <td>753</td>\n",
       "      <td>0.674517</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>104 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "characteristic                  abv                    ...    review_taste  \\\n",
       "measure                        mean count       std    ...            mean   \n",
       "beer_style                                             ...                   \n",
       "Altbier                    5.936331   169  1.394085    ...        3.703911   \n",
       "American Adjunct Lager     4.880552   706  0.405311    ...        2.674933   \n",
       "American Amber / Red Ale   6.135963  1256  1.299676    ...        3.734944   \n",
       "...                             ...   ...       ...    ...             ...   \n",
       "Wheatwine                 10.766667   171  1.043411    ...        3.947977   \n",
       "Winter Warmer              6.520407  1228  1.079628    ...        3.767900   \n",
       "Witbier                    5.850545   716  1.561416    ...        3.624170   \n",
       "\n",
       "characteristic                            \n",
       "measure                  count       std  \n",
       "beer_style                                \n",
       "Altbier                    179  0.608033  \n",
       "American Adjunct Lager     746  0.815884  \n",
       "American Amber / Red Ale  1345  0.640439  \n",
       "...                        ...       ...  \n",
       "Wheatwine                  173  0.624275  \n",
       "Winter Warmer             1243  0.621533  \n",
       "Witbier                    753  0.674517  \n",
       "\n",
       "[104 rows x 15 columns]"
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = gr[review_columns].agg(['mean', 'count', 'std'])\n",
    "result.columns.names=['characteristic','measure']\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combined Series / DataFrame is the same shape as the input. \n",
    "\n",
    "Example: standardize the reviews by subtracting the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Define a function to perform the demeaning\n",
    "\n",
    "def de_mean(reviews):\n",
    "    s = reviews - reviews.mean()\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        0.65018\n",
       "1        0.65018\n",
       "2       -0.84982\n",
       "          ...   \n",
       "49997    1.15018\n",
       "49998    0.15018\n",
       "49999    1.15018\n",
       "Name: review_overall, dtype: float64"
      ]
     },
     "execution_count": 258,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Demonstrate the demeaning function acting on a Series\n",
    "\n",
    "de_mean(df.review_overall)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.763000</td>\n",
       "      <td>-14386.800000</td>\n",
       "      <td>-250.800000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>-0.100000</td>\n",
       "      <td>0.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.501562</td>\n",
       "      <td>-12613.333333</td>\n",
       "      <td>3168.025641</td>\n",
       "      <td>...</td>\n",
       "      <td>0.397436</td>\n",
       "      <td>0.038462</td>\n",
       "      <td>-0.012821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2.072727</td>\n",
       "      <td>-10567.348485</td>\n",
       "      <td>805.833333</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.803030</td>\n",
       "      <td>-0.568182</td>\n",
       "      <td>-0.287879</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>4817.388889</td>\n",
       "      <td>1236.833333</td>\n",
       "      <td>...</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>0.777778</td>\n",
       "      <td>0.555556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>-1.625862</td>\n",
       "      <td>-36774.032258</td>\n",
       "      <td>-4302.516129</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.112903</td>\n",
       "      <td>-0.838710</td>\n",
       "      <td>-0.774194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>2.165500</td>\n",
       "      <td>20759.950000</td>\n",
       "      <td>12161.450000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.925000</td>\n",
       "      <td>0.575000</td>\n",
       "      <td>0.375000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            abv       beer_id     brewer_id      ...       review_overall  \\\n",
       "0     -2.763000 -14386.800000   -250.800000      ...             0.500000   \n",
       "1     -1.501562 -12613.333333   3168.025641      ...             0.397436   \n",
       "2     -2.072727 -10567.348485    805.833333      ...            -0.803030   \n",
       "...         ...           ...           ...      ...                  ...   \n",
       "49997  0.400000   4817.388889   1236.833333      ...             0.916667   \n",
       "49998 -1.625862 -36774.032258  -4302.516129      ...            -0.112903   \n",
       "49999  2.165500  20759.950000  12161.450000      ...             0.925000   \n",
       "\n",
       "       review_palate  review_taste  \n",
       "0          -0.100000      0.450000  \n",
       "1           0.038462     -0.012821  \n",
       "2          -0.568182     -0.287879  \n",
       "...              ...           ...  \n",
       "49997       0.777778      0.555556  \n",
       "49998      -0.838710     -0.774194  \n",
       "49999       0.575000      0.375000  \n",
       "\n",
       "[50000 rows x 8 columns]"
      ]
     },
     "execution_count": 259,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use transform() to apply the demean function to grouped data\n",
    "\n",
    "df.groupby('profile_name').transform(de_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Performance: Often, it is better to work with the groupby object directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abv</th>\n",
       "      <th>beer_id</th>\n",
       "      <th>brewer_id</th>\n",
       "      <th>...</th>\n",
       "      <th>review_overall</th>\n",
       "      <th>review_palate</th>\n",
       "      <th>review_taste</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.763000</td>\n",
       "      <td>-14386.800000</td>\n",
       "      <td>-250.800000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>-0.100000</td>\n",
       "      <td>0.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.501562</td>\n",
       "      <td>-12613.333333</td>\n",
       "      <td>3168.025641</td>\n",
       "      <td>...</td>\n",
       "      <td>0.397436</td>\n",
       "      <td>0.038462</td>\n",
       "      <td>-0.012821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2.072727</td>\n",
       "      <td>-10567.348485</td>\n",
       "      <td>805.833333</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.803030</td>\n",
       "      <td>-0.568182</td>\n",
       "      <td>-0.287879</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>4817.388889</td>\n",
       "      <td>1236.833333</td>\n",
       "      <td>...</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>0.777778</td>\n",
       "      <td>0.555556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>-1.625862</td>\n",
       "      <td>-36774.032258</td>\n",
       "      <td>-4302.516129</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.112903</td>\n",
       "      <td>-0.838710</td>\n",
       "      <td>-0.774194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>2.165500</td>\n",
       "      <td>20759.950000</td>\n",
       "      <td>12161.450000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.925000</td>\n",
       "      <td>0.575000</td>\n",
       "      <td>0.375000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            abv       beer_id     brewer_id      ...       review_overall  \\\n",
       "0     -2.763000 -14386.800000   -250.800000      ...             0.500000   \n",
       "1     -1.501562 -12613.333333   3168.025641      ...             0.397436   \n",
       "2     -2.072727 -10567.348485    805.833333      ...            -0.803030   \n",
       "...         ...           ...           ...      ...                  ...   \n",
       "49997  0.400000   4817.388889   1236.833333      ...             0.916667   \n",
       "49998 -1.625862 -36774.032258  -4302.516129      ...            -0.112903   \n",
       "49999  2.165500  20759.950000  12161.450000      ...             0.925000   \n",
       "\n",
       "       review_palate  review_taste  \n",
       "0          -0.100000      0.450000  \n",
       "1           0.038462     -0.012821  \n",
       "2          -0.568182     -0.287879  \n",
       "...              ...           ...  \n",
       "49997       0.777778      0.555556  \n",
       "49998      -0.838710     -0.774194  \n",
       "49999       0.575000      0.375000  \n",
       "\n",
       "[50000 rows x 8 columns]"
      ]
     },
     "execution_count": 260,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Rather than applying the de_mean(), \n",
    "# use the built-in pandas method mean() and \n",
    "# move the arithmetic up to \"df - df.groupby()\"\n",
    "\n",
    "(\n",
    "\n",
    "    df - df.groupby('profile_name').transform('mean')\n",
    "\n",
    ").select_dtypes(exclude=['object'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 3: 6.1 s per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit df.groupby('profile_name').transform(de_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 3: 3.74 s per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit (df-df.groupby('profile_name').transform('mean')).select_dtypes(exclude=['object'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4124"
      ]
     },
     "execution_count": 263,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('profile_name').ngroups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# GroupBy with Multiple Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can group by **levels** of a MultiIndex.\n",
    "\n",
    "To understand this, we need to talk about `stack()` and `unstack()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "/* The following CSS has been adapted from\n",
       "   http://nbviewer.ipython.org/github/ketch/teaching-numerics-with-notebooks/blob/master/Styling_notebooks.ipynb\n",
       "   by Aron Ahmadia and David Ketcheson and has been reused under CC BY 4.0\n",
       "   */\n",
       "\n",
       "#site { /* main background */\n",
       "    background: #888;\n",
       "    color: #f6f6f6;\n",
       "}\n",
       "\n",
       "div.container {\n",
       "    width: 100% ! important;\n",
       "}\n",
       "\n",
       "div.cell { /* set cell width to about 80 chars */\n",
       "    width: 95%;\n",
       "}\n",
       "\n",
       "div #notebook { /* centre the content */\n",
       "    background: #fff; /* white background for content */\n",
       "    width: 100%;\n",
       "    margin: auto;\n",
       "    padding-left: 1em;\n",
       "    padding-right: 1em;\n",
       "}\n",
       "\n",
       "#notebook li { /* More space between bullet points */\n",
       "    margin-top:0.4em;\n",
       "}\n",
       "\n",
       "/* change background color of running cells */\n",
       "div.cell.code_cell.running {\n",
       "    background-color: rgba(164,188,194,.75);\n",
       "    border-width: 8px\n",
       "    border-style: solid\n",
       "    border-color: #111;\n",
       "}\n",
       "\n",
       "/* Put a solid color box around each cell and its output, visually linking them together */\n",
       "div.cell.code_cell {\n",
       "    background-color: rgba(164,188,194,.25); /* Continuum L Gray with alpha 0.25*/\n",
       "    border-radius: 10px; /* rounded borders */\n",
       "    padding: 1em;\n",
       "    margin-top: 1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render{\n",
       "\tfont-family: Georgia, Times, 'Times New Roman', serif;\n",
       "    line-height: 110%;\n",
       "    font-size: 105%;\n",
       "    width: 95%;\n",
       "    margin-left: auto;\n",
       "    margin-right: auto;\n",
       "}\n",
       "\n",
       "\n",
       "/* Formatting for header cells */\n",
       ".text_cell_render h1 {\n",
       "\tfont-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;\n",
       "    font-weight: 400;\n",
       "    font-size: 30pt;\n",
       "    line-height: 100%;\n",
       "    color: rgb(37,55,70); /* Continuum Blue*/\n",
       "    margin-bottom: 0.1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\n",
       ".text_cell_render h2 {\n",
       "\tfont-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;\n",
       "    font-weight: 700;\n",
       "    font-size: 22pt;\n",
       "    line-height: 100%;\n",
       "    color: rgb(37,55,70);\n",
       "    margin-bottom: 0.1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h3 {\n",
       "\tfont-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;\n",
       "    margin-top:12px;\n",
       "    margin-bottom: 3px;\n",
       "    font-style: italic;\n",
       "    color: rgb(37, 55, 70);\n",
       "}\n",
       "\n",
       ".text_cell_render h4 {\n",
       "\tfont-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;\n",
       "}\n",
       "\n",
       ".text_cell_render h5 {\n",
       "\tfont-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 14pt;\n",
       "    font-style: italic;\n",
       "    margin-bottom: .1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h6 {\n",
       "\tfont-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 10pt;\n",
       "    margin-bottom: 1px;\n",
       "    margin-top: 1px;\n",
       "}\n",
       "\n",
       ".CodeMirror{\n",
       "        font-family: \"PT Mono\", fixed;\n",
       "        font-size: 100%;\n",
       "}\n",
       "\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 264,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import continuum_style; continuum_style.style()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "continuum": {
   "depends": [
    "ip_conda"
   ],
   "requires": [
    "data/goog.csv"
   ],
   "tag": "pd_intro"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}