HARVESTING_DATA_PYT_DS_SAISOFT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PYT-DS:  Harvesting Data\n",
    "\n",
    "We're used to reading JSON and CSV files over the internet, using Pandas.  However, if you have control of a server, there's no reason you can't make scripts such as the one below fetch everything for you under the hood, using URL requests.\n",
    "\n",
    "By the time the data surfaces in the Notebook, it's already an up-to-date Dataframe, sorted and massaged.  I'm exposing the pipeline here, but it's easy to imagine the Notebook actually starting around the last cell, having already done the job behind the scenes, of harvesting data.\n",
    "\n",
    "As a data scientist, your role may be as much about storing data for convenient access, in a usable form, as it is about end user analysis of said data.  Your role may be part DBA (database administrator) at the end of the day.  That's not a bad thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Created on Thu Jun 29 10:17:02 2017\n",
    "Rewritten for get_data package on Oct 25, 2017\n",
    "\n",
    "@author: Kirby Urner\n",
    "\n",
    "Decorated generator used IN PLACE OF:\n",
    "    \n",
    "class Url:\n",
    "    \n",
    "    def __init__(self, the_url):\n",
    "        self.url = the_url\n",
    "        \n",
    "    def __enter__(self):\n",
    "        self.rq = urlopen(self.url)\n",
    "        return self.rq\n",
    "    \n",
    "    def __exit__(self, *oops):\n",
    "        if oops[0]:\n",
    "            print(\"Failed to connect\")\n",
    "            return False\n",
    "        self.rq.close()\n",
    "        return True\n",
    "\"\"\"\n",
    "\n",
    "from urllib.request import urlopen\n",
    "import json\n",
    "from contextlib import contextmanager\n",
    "\n",
    "PREFIX = \"http://thekirbster.pythonanywhere.com/\"\n",
    "\n",
    "@contextmanager\n",
    "def url(target):\n",
    "    try:\n",
    "        yield urlopen(target)\n",
    "    except:\n",
    "        print(\"Failed to connect\")\n",
    "        raise\n",
    "           \n",
    "def get_chems():\n",
    "    \"\"\"\n",
    "    Get the element data from the web using API\n",
    "    \n",
    "    Typical record:\n",
    "    [1, \"H\", \"Hydrogen\", 1.008, \"diatomic nonmetal\", 1498013115, \"KTU\"]\n",
    "    \"\"\"\n",
    "    global chems\n",
    "    with url(PREFIX + \"api/elements?elem=all\") as httpreq:\n",
    "        data = json.loads(httpreq.read())  # getting JSON data\n",
    "        chems = pd.DataFrame(data)\n",
    "\n",
    "get_chems()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chems.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not quite how we want to see the data.  We need to flip it, or swap axes.  Transpose will do.  Then lets change the column names.  Finally, we'll sort."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chems = chems.T\n",
    "chems.columns=[\"Protons\", \"Symbol\", \"Name\", \"Mass\", \"Category\", \"Changed\", \"Initials\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chems.sort_values(\"Protons\")\n",
    "chems.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chems = chems.sort_index()  # lets get this in index order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chems.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LAB:\n",
    "\n",
    "Now that you have all the periodic table, why not try some of your tricks?  How many categories are there and how many elements in each category.  Remember GroupBy?  What other tricks (like magic spells) do you remember?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}