1 - Reading Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do any data science, the first thing you're going to need is data. Let's start to look at a few common ways of reading data in Python. Along the way, we'll take a look at some language constructs that we'll use throughout the day and really start to get familiar with coding in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File handling in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's look at file objects in Python. We get a file handle by using the **open** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file = open(\"data/health_inspection_chi_sample.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "File objects are lazy **iterators** (here, *stream objects*). Lazy means that they only do things, in this case read data, when you ask them to. You can call **next** on iterator objects to explicitly get the next item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = next(csv_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the next item is a string which is the line in a file. In this case, it's the headers of the `csv` file.\n",
    "\n",
    "The nice thing about **iterators** is that you can iterate through them with a for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for line in csv_file:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After consuming the file, we need to close the open file-handle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can avoid that by using what's called a **context manager** in Python. This is a really nice abstraction. You use context managers via the **with** statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/health_inspection_chi_sample.csv\") as csv_file:\n",
    "    for line in csv_file:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using the `open` function as a context manager, we get an automatic call to close the open file when we exit the context (determined by the indentation level). When working with files non-interactively, you'll almost always want to use open as a context manager."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Write some code that iterates through the file `data/health_inspection_chi_sample.json` twice. Only call `open` once, however, then close the file. Can you find out, programatically, how many characters are in the file (hint: have a look at the `tell` method of the file object)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/read_json_twice.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading Data: CSVs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We peaked at this file above. We might already see some things we're going to have to clean up. Let's look at how we may do this. As always, Python has *batteries included*. The `csv` built-in module is great for basic data-munging needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's open a csv file and use csv's `reader` function to read in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file = open(\"data/health_inspection_chi_sample.csv\")\n",
    "\n",
    "reader = csv.reader(csv_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`csv.reader` also returns a lazy iterator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = next(reader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint(headers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = next(reader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The biggest difference in using `csv.reader` vs. iterating through the file is that it automatically splits the csv on commas and returns the line of the file split into a list.\n",
    "\n",
    "You can control this behavior through a `Dialect` object. By default, `csv.reader` uses a Dialect object called \"excel.\" Here let's look at the attributes of the excel dialect. Don't worry too much about the code used to achieve this. We'll look more at this later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "items = csv.excel.__dict__.items()\n",
    "\n",
    "pprint({key: value for key, value in items if not key.startswith(\"_\")})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might also find `DictReader` to be useful. \n",
    "\n",
    "First, let's back up in the file to the beginning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file.seek(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = csv.DictReader(csv_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint(next(reader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One last handy trick with the `csv` module is to use `Sniffer` to determine the csv dialect for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"data/health_inspection_chi_sample.csv\"\n",
    "\n",
    "with open(file_name) as csv_file:\n",
    "    \n",
    "    try:\n",
    "        dialect = csv.Sniffer().sniff(csv_file.read(1024))\n",
    "    except csv.Error as err:\n",
    "        # log that this file format couldn't be deduced\n",
    "        print(f\"The format of {file_name} could not be detected.\")\n",
    "    else:\n",
    "        csv_file.seek(0)\n",
    "    \n",
    "        dta = csv.reader(csv_file, dialect=dialect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A big part of increased productivity is saving yourself some work later. The first rule of dealing with data is probably something like \"The data is against you. Act accordingly.\"\n",
    "\n",
    "We set this block of code up to be pretty paranoid. Software engineers call this **defensive programming**. It's a good habit to get into when you're doing any data science work in Python. \n",
    "\n",
    "There are three things to note here. First, is the use of `Sniffer` at all. If `file_name` is a standard csv format, we'll be able to read it.\n",
    "\n",
    "The second is using a `try/except/else` block. In a `try/except` block, any exception that is raised will trigger the code in the except block. If no exception is raised, the *optional* else block will run.\n",
    "\n",
    "Let's take a look at a toy example to fix ideas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    1/0\n",
    "except:\n",
    "    print(\"Something went wrong.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    1/0\n",
    "except ZeroDivisionError as err:\n",
    "    print(\"Something went wrong.\")\n",
    "    raise(err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    1/0\n",
    "except FileNotFoundError:\n",
    "    print(\"This error isn't raised, so we're not here.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final thing to note in the block above is the use of `print` to provide some information about what went wrong. Logging is another really good habit to get into, and print statements are the dead simplest way to log the behavior of your code.\n",
    "\n",
    "In practice, you probably don't want to use `print`. You want to use the [logging](https://docs.python.org/3/library/logging.html) module, but we're not going to talk about best practices in logging anymore today."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading Data: json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps the second most common file format after CSVs is the `JSON` format. JSON stands for JavaScript Object Notation. When reading data from an API, for example, you will often encounter json files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each line in the file `data/health_inspection_chi_sample.json` is a single json object that represents the same data above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -n 1 data/health_inspection_chi_sample.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `json` module to read and manipulate json data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since each line is a json object here, we need to iterate over the file and parse each line. We use the `json.loads` function here for \"load string.\" The similar function `json.load` takes a file-like object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dta = []\n",
    "\n",
    "with open(\"data/health_inspection_chi_sample.json\") as json_file:\n",
    "    for line in json_file:\n",
    "        line = json.loads(line)\n",
    "        dta.append(line)\n",
    "\n",
    "pprint(dta[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`json.loads` places each json object into a Python dictionary, helpfully filling in `None` for `null` for missing values and otherwise preserving types. It also, works recursively as we see in the `location` field."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aside: List Comprehensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at another Pythonic concept, introduced a bit above, called a **list comprehension**. This is what's called *syntactic sugar*. It's a concise way to create a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[i for i in range(1, 6)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could have made this list by writing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_list = []\n",
    "\n",
    "for i in range(1, 6):\n",
    "    result_list.append(i)\n",
    "\n",
    "result_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List comprehensions can contain logic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = ['a', 'b', 'c', 'd', '_e', '_f']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[i for i in x if not i.startswith('_')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use a an else clause. Notice the slightly different syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[i if not i.startswith('_') else 'skipped' for i in x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List comprehensions can be nested, though it's usually best practices not to go overboard. They can quickly become difficult to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = [\n",
    "    [1, 2, 3, 4],\n",
    "    [5, 6, 7, 8],\n",
    "    [9, 10, 11, 12],\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we write this from outside to in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[element for row in matrix for element in row]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create **dictionary comprehensions**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pairs = [\n",
    "    (\"first_initial\", \"J\"), \n",
    "    (\"last_name\", \"Doe\"), \n",
    "    (\"address\", \"1234 Main Street\")\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{key: value for key, value in pairs}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Returning to the code that we introduced above, we can take further control over how a file with json objects is read in by using the `object_hook` argument. Say we wanted to remove the `location` field above. We don't need the `geoJSON` formatted information. We could do so with the `object_hook`. Write a function called `remove_entry` that removes the `'location'` field from each record in the `'data/health_inspection_chi_sample.json'` file.\n",
    "\n",
    "Pass this function to the `object_hook` argument of `json.loads`. Be careful, `object_hook` will get called recursively on nested json objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/object_hook_json.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading Data: pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's take a look at using [pandas]() to read data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Introducing Pandas\n",
    "\n",
    "First, a few words of introduction for **pandas**. Pandas is a Python package providing fast, flexible, and expressive data structures designed to work with relational or labeled data. It is a high-level tool for doing practical, real world data analysis in Python.\n",
    "\n",
    "You reach for pandas when you have:\n",
    "\n",
    "* Tabular data with heterogeneously-typed columns\n",
    "* Ordered and unordered (not necessarily fixed-frequency) time series data.\n",
    "* Arbitrary matrix data with row and column labels\n",
    "\n",
    "Almost any dataset can be converted to a pandas data structure for cleaning, transformation, and analysis.\n",
    "\n",
    "Pandas is built on top of **numpy**, which we'll look a bit more at later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import the pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set some display options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"max.rows\", 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas has facilities for reading csv files and files containing JSON records (and other formats). We can use `read_csv` for csv files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_csv(\"data/health_inspection_chi_sample.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`read_csv` is one of the best/worst functions in all of Python. It's great because it does just about everything. It's terrible, because it does just about everything. Chances are if you have a special case that pandas `read_csv` will accomodate your needs. Go ahead and have a look at the `docstring`.\n",
    "\n",
    "\n",
    "```python\n",
    "pd.read_csv?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`read_csv` returns a pandas DataFrame. We'll take a deeper dive into DataFrames next when we start to clean this data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The JSON counterpart to `read_csv` is `read_json`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Use `pd.read_json` to read in the Chicago health inspections json sample in the `data` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type your solution Here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/read_json.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading Data: web data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we've seen some ways that we can read data from disk. As Data Scientists, we often need to go out and grab data from the Internet.\n",
    "\n",
    "Generally Python is \"batteries included\" and reading data from the Internet is no exception, but there are some *great* packages out there. [requests](http://docs.python-requests.org/en/master/) is one of them for making HTTP requests.\n",
    "\n",
    "Let's look at how we can use the [Chicago Data Portal](https://data.cityofchicago.org/) API to get this data in the first place. (I originally used San Francisco for this, but the data was just too clean to be terribly interesting.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the requests library to perform a GET request to the API, passing an optional query string via `params` to limit the returned number of records. The parameters are documented as part of the [Socrata Open Data API](https://dev.socrata.com/consumers/getting-started.html) (SODA)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\n",
    "    \"https://data.cityofchicago.org/resource/cwig-ma7x.json\", \n",
    "    params=\"$limit=10\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Requests returns a [Reponse](http://docs.python-requests.org/en/master/api/#requests.Response) object with many helpful methods and attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dta = pd.read_json(response.content, orient='records')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `head` method to peak at the first 5 rows of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dta.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, pandas can also load data directly from a URL, but I encourage you to reach for `requests` as often as you need it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Try passing the URL above to `pd.read_json`. What happens?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/read_url_json.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how you can split a string across lines. This can be a very handy tip for improving readability, by splitting a string and putting it in parentheses, we preserve a single string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\"super \"\n",
    " \"long \"\n",
    " \"string \"\n",
    " \"split \"\n",
    " \"across \"\n",
    " \"lines\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pandas DataReader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the core I/O functionality in pandas, there is also the [pandas-datareader](https://pandas-datareader.readthedocs.io/en/latest/) project. This package provides programmatic access to data sets from\n",
    "\n",
    "* Yahoo! Finance (deprecated)\n",
    "* Google Finance\n",
    "* Enigma\n",
    "* Quandl\n",
    "* FRED\n",
    "* Fama/French\n",
    "* World Bank\n",
    "* OECD\n",
    "* Eurostat\n",
    "* EDGAR Index (deprecated)\n",
    "* TSP Fund Data\n",
    "* Nasdaq Trader Symbol Definitions\n",
    "* Morningstar\n",
    "* Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Further Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we need to be resourceful in order to get data. Knowing how to scrape the web can really come in handy.\n",
    "\n",
    "We're not going to go into details today, but you'll likely find libraries like [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/), [lxml](http://lxml.de/), and [mechanize](https://mechanize.readthedocs.io/en/latest/) to be helpful. \n",
    "\n",
    "There's also a `read_html` function in pandas that will quickly scrape HTML tables for you and put them into a DataFrame. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}