module4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to the pandas module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Built upon NumPy, the Pandas module is essential to data science. Pandas has easy-to-use tools for acquiring, cleaning, mainpulating, and displaying data. It introduces two new data structures; **series** and **dataframe**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series data structure\n",
    "- A Pandas Series is a one-dimensional, labeled data structure with all elements of the **same type**.\n",
    "- A Series can hold any data type. It is **similar to a Python list**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Pandas Dataframe\n",
    "- A dataframe is a two-dimensional, labeled data structure consisting of **rows and columns**.\n",
    "- Rows are **indexed** starting with **zero**.\n",
    "- Each **column** in a dataframe is a **series** object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acquiring data\n",
    "\n",
    "\n",
    "\n",
    "Pandas has methods for importing data from a variety of sources:\n",
    "    \n",
    "import pandas as pd\n",
    "\n",
    "    - pd.read_csv(filename) | From a CSV file\n",
    "    - pd.read_table(filename) | From a delimited text file (like TSV)\n",
    "    - pd.read_excel(filename) | From an Excel file\n",
    "    - pd.read_sql(query, connection_object) | Read from a SQL table/database\n",
    "    - pd.read_json(json_string) | Read from a JSON formatted string, URL or file.\n",
    "    - pd.read_html(url) | Parses an html URL or file and extracts tables to a list of dataframes\n",
    "    - pd.DataFrame(dict) | From a dict, keys for columns names, values for data as lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           0         1         2         3         4         5\n",
      "0   0.482669  0.964623  0.441755  0.164572  0.655034  0.276603\n",
      "1   0.562612  0.663198  0.283033  0.528836  0.625107  0.525557\n",
      "2   0.951833  0.168371  0.677403  0.829570  0.357788  0.703179\n",
      "3   0.509530  0.782079  0.672833  0.052213  0.106995  0.615498\n",
      "4   0.001125  0.697431  0.608288  0.179210  0.779170  0.284910\n",
      "5   0.234731  0.996922  0.042130  0.030645  0.995511  0.018620\n",
      "6   0.797007  0.007676  0.409448  0.335556  0.183252  0.283193\n",
      "7   0.781679  0.167867  0.556331  0.415106  0.584109  0.429637\n",
      "8   0.486461  0.449072  0.372873  0.500162  0.846352  0.303331\n",
      "9   0.054186  0.391377  0.404824  0.959971  0.006627  0.351770\n",
      "10  0.421768  0.596311  0.034870  0.383158  0.832110  0.368695\n",
      "11  0.149383  0.803241  0.300618  0.243968  0.027780  0.347651\n"
     ]
    }
   ],
   "source": [
    "# you can generate a dataframe with random data for testing\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "# make a df with 12 rows and 6 columns of floats\n",
    "df_fake = pd.DataFrame(np.random.rand(12,6))\n",
    "print(df_fake)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         AAPL_x      AAPL_y\n",
      "0    2014-01-02   77.445395\n",
      "1    2014-01-03   77.045575\n",
      "2    2014-01-06   74.896972\n",
      "3    2014-01-07   75.856461\n",
      "4    2014-01-08   75.091947\n",
      "..          ...         ...\n",
      "235  2014-12-08  113.653345\n",
      "236  2014-12-09  109.755497\n",
      "237  2014-12-10  113.960331\n",
      "238  2014-12-11  111.817477\n",
      "239  2014-12-12  110.027139\n",
      "\n",
      "[240 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "# you can read a CSV file from the Internet\n",
    "\n",
    "# Read 2014 Apple stock price into a Pandas dataframe \n",
    "df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/2014_apple_stock.csv')\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting or viewing a dataframe\n",
    "\n",
    "Pandas dataframes have many tools for accessing data\n",
    "- df.info() method: provides information about index, datatype and memory\n",
    "- df.shape attribute: returns the number of rows and columns\n",
    "- df.head(n) method: displays the first n rows\n",
    "- df.tail(n) method: displays final n rows of the dataframe\n",
    "- df.describe() method: prints summary statistics for numerical columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1704 entries, 0 to 1703\n",
      "Data columns (total 6 columns):\n",
      "country      1704 non-null object\n",
      "year         1704 non-null int64\n",
      "pop          1704 non-null float64\n",
      "continent    1704 non-null object\n",
      "lifeExp      1704 non-null float64\n",
      "gdpPercap    1704 non-null float64\n",
      "dtypes: float64(3), int64(1), object(2)\n",
      "memory usage: 80.0+ KB\n",
      "None\n",
      "\n",
      "Dataframe shape (rows,columns)\n",
      "(1704, 6)\n"
     ]
    }
   ],
   "source": [
    "print(df.info())\n",
    "print('\\nDataframe shape (rows,columns)')\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       country  year         pop continent  lifeExp   gdpPercap\n",
      "0  Afghanistan  1952   8425333.0      Asia   28.801  779.445314\n",
      "1  Afghanistan  1957   9240934.0      Asia   30.332  820.853030\n",
      "2  Afghanistan  1962  10267083.0      Asia   31.997  853.100710\n",
      "3  Afghanistan  1967  11537966.0      Asia   34.020  836.197138\n",
      "4  Afghanistan  1972  13079460.0      Asia   36.088  739.981106\n",
      "5  Afghanistan  1977  14880372.0      Asia   38.438  786.113360\n",
      "       country  year         pop continent  lifeExp   gdpPercap\n",
      "1700  Zimbabwe  1992  10704340.0    Africa   60.377  693.420786\n",
      "1701  Zimbabwe  1997  11404948.0    Africa   46.809  792.449960\n",
      "1702  Zimbabwe  2002  11926563.0    Africa   39.989  672.038623\n",
      "1703  Zimbabwe  2007  12311143.0    Africa   43.487  469.709298\n"
     ]
    }
   ],
   "source": [
    "print(df.head(6))         # display first 6 rows\n",
    "print(df.tail(4))         # display last 4 rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             year           pop      lifeExp      gdpPercap\n",
      "count  1704.00000  1.704000e+03  1704.000000    1704.000000\n",
      "mean   1979.50000  2.960121e+07    59.474439    7215.327081\n",
      "std      17.26533  1.061579e+08    12.917107    9857.454543\n",
      "min    1952.00000  6.001100e+04    23.599000     241.165877\n",
      "25%    1965.75000  2.793664e+06    48.198000    1202.060309\n",
      "50%    1979.50000  7.023596e+06    60.712500    3531.846989\n",
      "75%    1993.25000  1.958522e+07    70.845500    9325.462346\n",
      "max    2007.00000  1.318683e+09    82.603000  113523.132900\n"
     ]
    }
   ],
   "source": [
    "print(df.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          country  year         pop continent  lifeExp   gdpPercap\n",
      "0     Afghanistan  1952   8425333.0      Asia   28.801  779.445314\n",
      "1     Afghanistan  1957   9240934.0      Asia   30.332  820.853030\n",
      "2     Afghanistan  1962  10267083.0      Asia   31.997  853.100710\n",
      "3     Afghanistan  1967  11537966.0      Asia   34.020  836.197138\n",
      "4     Afghanistan  1972  13079460.0      Asia   36.088  739.981106\n",
      "...           ...   ...         ...       ...      ...         ...\n",
      "1699     Zimbabwe  1987   9216418.0    Africa   62.351  706.157306\n",
      "1700     Zimbabwe  1992  10704340.0    Africa   60.377  693.420786\n",
      "1701     Zimbabwe  1997  11404948.0    Africa   46.809  792.449960\n",
      "1702     Zimbabwe  2002  11926563.0    Africa   39.989  672.038623\n",
      "1703     Zimbabwe  2007  12311143.0    Africa   43.487  469.709298\n",
      "\n",
      "[1704 rows x 6 columns]\n",
      "             year           pop      lifeExp      gdpPercap\n",
      "count  1704.00000  1.704000e+03  1704.000000    1704.000000\n",
      "mean   1979.50000  2.960121e+07    59.474439    7215.327081\n",
      "std      17.26533  1.061579e+08    12.917107    9857.454543\n",
      "min    1952.00000  6.001100e+04    23.599000     241.165877\n",
      "25%    1965.75000  2.793664e+06    48.198000    1202.060309\n",
      "50%    1979.50000  7.023596e+06    60.712500    3531.846989\n",
      "75%    1993.25000  1.958522e+07    70.845500    9325.462346\n",
      "max    2007.00000  1.318683e+09    82.603000  113523.132900\n",
      "\n",
      "Life Expectancy Median  60.7125\n"
     ]
    }
   ],
   "source": [
    "# A life expectancy dataset\n",
    "\n",
    "url='http://bit.ly/2cLzoxH'\n",
    "df = pd.read_csv(url)\n",
    "print(df)\n",
    "print(df.describe())\n",
    "print('\\nLife Expectancy Median ',df['lifeExp'].median())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(142, 6)\n",
      "      year             country  lifeExp\n",
      "11    2007         Afghanistan   43.828\n",
      "23    2007             Albania   76.423\n",
      "35    2007             Algeria   72.301\n",
      "47    2007              Angola   42.731\n",
      "59    2007           Argentina   75.320\n",
      "...    ...                 ...      ...\n",
      "1655  2007             Vietnam   74.249\n",
      "1667  2007  West Bank and Gaza   73.422\n",
      "1679  2007          Yemen Rep.   62.698\n",
      "1691  2007              Zambia   42.384\n",
      "1703  2007            Zimbabwe   43.487\n",
      "\n",
      "[142 rows x 3 columns]\n"
     ]
    }
   ],
   "source": [
    "is_2007 = df['year'] == 2007                 # a boolean for filtering\n",
    "df_2007 = df[is_2007]                        # df_2007 is a subset of df for 2007 \n",
    "print(df_2007.shape)\n",
    "df2 = df_2007[['year','country','lifeExp']]  # a new dataframe with these 3 columns\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "year              2007\n",
      "country    Afghanistan\n",
      "lifeExp         39.613\n",
      "dtype: object\n",
      "\n",
      "year           2007\n",
      "country    Zimbabwe\n",
      "lifeExp      82.603\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(df2.min())\n",
    "print()\n",
    "print(df2.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}