02-api-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making API requests: a case study with Yelp\n",
    "\n",
    "Laundromats can give interesting information about a neighborhood: neighborhoods where housing units are old or small tend to have a higher density of laundromats. Plus, I'm a big fan of [this song](https://www.youtube.com/watch?v=nxaCSy5MNEA). So in this example we'll be querying the Yelp API to find laundromats in Newark, NJ.\n",
    "\n",
    "### Importing packages\n",
    "\n",
    "A package is a bundle of software utilities that have a defined scope, In this exercise you'll be making API requests using the Python `requests` package. Also, you'll be cleaning up and exporting your data with the `pandas` package. Let's import those packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd  # pandas is commonly imported with the alias \"pd\", to save some typing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Yelp API Key setup\n",
    "\n",
    "In order to access the Yelp API you'll need to _authenticate_. If you haven't alreadu, please follow the steps [here](https://www.yelp.com/developers/documentation/v3/authentication) to obtain an API Key.\n",
    "\n",
    "Once you've obtained your key, you'll want to store it in a variable so you can include it in your API request. Please copy paste it inside the quotation marks below and **do not share it with anyone**.\n",
    "\n",
    "_Security note: exposing an API key or any other authentication secret in code is **bad practice**, as it could be copied by a malicious user who could steal your identity. Ideally, you'd want to read it from a secure file stored in your computer using an encryption mechanism. However, since this exercise is designed to run on a remote computer, we'll be pasting it into the following cell. If you have concerns this, please stop now._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store your API key in a variable\n",
    "API_KEY = \"PASTE-YOUR-API-KEY-INSIDE-THESE-QUOTATION-MARKS\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the base URL for the request\n",
    "base_url = \"https://api.yelp.com/v3/businesses/search\"\n",
    "\n",
    "# Set up the request headers -- API key is used here\n",
    "headers = {\"Authorization\": \"Bearer \" + API_KEY}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now your credentials are set up in the `headers` object. We'll use this object when making the API request, as you'll see shortly.\n",
    "\n",
    "Besides the user authentication, other important info in the request is the data to be queried. The [Yelp API documentation](https://www.yelp.com/developers/documentation/v3) has lots of information regarding all the endpoints you can reach and the parameters that each expect. \n",
    "\n",
    "In this example we'll be working with the `/businessess/search` endpoint. You can find its documentation [here](https://www.yelp.com/developers/documentation/v3/business_search). We'll be using only a few parameters:\n",
    "- `\"location\"` this is a string text (must be specified within quotation marks) with the location of the query\n",
    "- `\"term\"` is the \"search term\" (e.g. food, restaurants, etc.)\n",
    "- `\"limit\"` is the max number of entries to return. If this parameter is not set, it defaults to 20. The max is 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the request parameters\n",
    "params = {\n",
    "    \"location\": \"Newark, NJ\",\n",
    "    \"term\": \"laundromat\",\n",
    "    \"limit\": 5\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now execute the request, saving the response in a variable called `response`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\n",
    "    base_url,\n",
    "    headers=headers,\n",
    "    params=params\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic! Now, let's inspect the `response` object. First, look at its attribute `url`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response.url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's inspect the `text` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look, there are ratings, zipcodes, longitudes and latitudes... That looks like the data we requested! But it's a little messy. Let's use the `json()` method to see it in a more structured way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent! That data is structured as a _dictionary_, a very common data structure in python and other programming languages. Dictionaries have \"keys\" and \"values\", where each key is mapped to a single value. Let's take a quick look at this dictionary! First, let's save it into a variable so we can access it more easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save dictionary containing response data\n",
    "data_dict = response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's first inspect the keys of the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there are three keys in this dictionary. Let's see the values for each of those keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the value associated with the \"region\" key\n",
    "data_dict[\"region\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the value associated with the \"businesses\" key\n",
    "data_dict[\"businesses\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the value associated with the \"total\" key\n",
    "data_dict[\"total\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at you mastering dictionaries! ;)\n",
    "\n",
    "You've seen how the dictionary stores different portions of the requested data under the different keys. The key `\"businesses\"` seems to be containing most of the relevant information, though. So let's turn that part of the data into an even more convenient format: a DataFrame. \n",
    "\n",
    "DataFrames are advanced tables with associated functions to perform data transformations. They also allow you to visualize table contents more easily. Let's do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the businesses data into a dataframe\n",
    "data_df = pd.DataFrame(data_dict[\"businesses\"])\n",
    "\n",
    "# Inspect the resulting dataframe\n",
    "data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ah-ha! Now that data is so much easier to read. And furthermore, you can perform operations on it much more easily.\n",
    "\n",
    "You may have noticed that each instance of the `\"coordinates\"` column contains a dictionary with keys `\"longitude\"` and `\"latitude\"`. To make your analyses easier down the road, let's extract each of those coordinates into their own columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract latitude and longitude into new columns\n",
    "data_df[\"latitude\"] = data_df[\"coordinates\"].apply(lambda x: x[\"latitude\"])\n",
    "data_df[\"longitude\"] = data_df[\"coordinates\"].apply(lambda x: x[\"longitude\"])\n",
    "\n",
    "# # An alternative, more generic way of achieving the same thing\n",
    "# data_df = pd.concat([data_df, data_df[\"coordinates\"].apply(pd.Series)], axis=1)\n",
    "\n",
    "# Inspect transformed dataframe\n",
    "data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at those `\"latitude\"` and `\"longitude\"` columns on the right side! Now you're in a good shape to move this into ArcGIS. All you have to do is to save this as a CSV and load it there. Let's do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save requested data as a CSV\n",
    "data_df.to_csv(\"./my_requested_data.csv\", index=False)  # index=False is used to avoid writing the row index in the file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's verify that the CSV was indeed created by printing it to screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ./my_requested_data.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic job! Now if this code is run in your local computer you can use the CSV file you created for your own analysis in a different program. If this is running in the cloud, all you have to do is select the text above, paste into a text editor, and save it as a `.csv` file. A bit hacky, but it works!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}