3-Exploring_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Pandas?\n",
    "\n",
    "<img src=\"https://numfocus.org/wp-content/uploads/2016/07/pandas-logo-300.png\">\n",
    "\n",
    "Pandas is a library which makes it easy to manipulate tabular data in Python. Pandas allow importing data of various file formats such as csv, excel etc. Pandas allows various data manipulation operations such as groupby, join, merge, melt, concatenation as well as data cleaning features such as filling, replacing or imputing null values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a DataFrame?\n",
    "\n",
    "DataFrame is a labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table. It is generally the most commonly used pandas object. A DataFrame is generally made up of an index and a number of columns:\n",
    "\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/3452/1*6p6nF4_5XpHgcrYRrLYVAw.png\" width=70%>\n",
    "\n",
    "<center>The anatomy of a DataFrame (<a href=\"https://medium.com/epfl-extension-school/selecting-data-from-a-pandas-dataframe-53917dc39953\">Source</a>)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let us load some data and display it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "mpg = pd.read_csv('auto-mpg.csv')\n",
    "\n",
    "mpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting your data\n",
    "\n",
    "You can use the DataFrame methods .head() and .tail() to view the first few and last few rows of a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access a particular column or `Series` you can use so called attribute access to refer to it by name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg.mpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting by index\n",
    "\n",
    "The index of a DataFrame is often meaningful, therefore the standard indexing behavior works on the index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg[5:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting by integer location\n",
    "\n",
    "Much like Excel numbers rows by number and columns by letter, pandas allows indexing both rows and columns by number. To select a particular row and/or column in a dataframe we can use the `.iloc` accessor which allows use to slice the dataframe by number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg.iloc[0:10, [1, 3]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting columns\n",
    "\n",
    "Selecting columns is similarly straightforward, to select a subset of columns we provide a list of the column names we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg[['name', 'yr', 'hp']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting by value\n",
    "\n",
    "To select by a value we can use the brack syntax and compare against a value, e.g. to select all cars which get more than 44 miles per gallon we can express that as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg[mpg.mpg>44]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now find the three cars which achieve **less than** 11 miles per gallon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select among multiple discrete values you can use the `isin` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg[mpg.origin.isin(['Europe', 'Asia'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing basic statistics\n",
    "\n",
    "To compute basic statistics on some data you can call various methods on a column, e.g. we can compute the minimum, mean, and maximum values for the mpg column by calling the min, mean and max methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg.mpg.min(), mpg.mpg.mean(), mpg.mpg.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the median horsepower (hp) value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groupby\n",
    "\n",
    "A groupby effectively groups the data by one or more variables and then computes some statistics for each group or subset, e.g. if we want to know the `mean` values for each car manufacturer's origin we express that using the following groupby:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg.groupby('origin').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pivot tables\n",
    "\n",
    "Pivot tables might be familiar to many Excel users, they provide a quick way to quickly get an overview of some dataset across multiple variables which can be laid across in the index or the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg.pivot_table('mpg', index='origin', columns='yr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading\n",
    "\n",
    "This overview only provided the most cursory summary of the things you can do with Pandas, which was mainly meant to aid you in understanding what we are doing in the next section. If you really want to learn how to use Pandas I would strongly recommend going through the section on Pandas in the [Python Data Science Handbook by Jake VanderPlas](https://jakevdp.github.io/PythonDataScienceHandbook/03.01-introducing-pandas-objects.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}