step1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ODI Queensland workshop - Web Scraping \n",
    "\n",
    "## QUT DMRC - 2015\n",
    "\n",
    "### Extract road sign name from a single item on a single page\n",
    "\n",
    "This notebook gets a page from the regulatory road sign page| on the Queensland Government Transport and Motoring website and then extracts one of the fields we are interested in from it.\n",
    "\n",
    "This notebook is broken up into a series of cells. We can run each cell in turn by clicking in the cell and then pressing ```<shift><enter>``` or choosing \"run\" from the \"Cell\" menu at the top of the page. The cell runs and any outputs it creates are shown below it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we import the Python modules we are going to use to get the information from the website."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bs4\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next steps build up the URL that has the information we want. The sections of the url that we will want to change to get more pages of information are kept seperate so we can change them more easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this is the base_url\n",
    "base_url = \"http://www.qld.gov.au/transport/safety/signs/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# select which page to scrape based on the type of road sign\n",
    "sign_type = \"regulatory\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build the url\n",
    "thepage = base_url + sign_type + '/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets check what the variable ```thepage``` is set to. You can show the value of any variable in a notebook by putting it in the last line of a notebook cell and running the cell. Jupyter will try to display it in a clear way, often clearer than the default 'print' layout. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "thepage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These steps get the page using [Requests](http://docs.python-requests.org/en/latest/) and then process it using [BeautifulSoup](http://www.crummy.com/software/BeautifulSoup/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# call the url\n",
    "stuff = requests.get(thepage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "For some websites it might be necessary to set the User-Agent header. For example we could pretends to be a standard Mozilla browser by using:\n",
    "\n",
    "    hdrs = {\"User-Agent\": \"Mozilla/5.0\"}\n",
    "    stuff = requests.get(thepage, headers=hdrs)\n",
    "    \n",
    "But for this website we don't need to do that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# check that the page url was valid\n",
    "stuff.ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# transform to soup using lxml parser\n",
    "soup = bs4.BeautifulSoup(stuff.text, \"lxml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# find the table with the signs - it is the first table on the page\n",
    "signs_table = soup.find('table')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# extract all the rows from the table\n",
    "lotsofitems = signs_table.findAll('tr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# show the table header\n",
    "lotsofitems[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# show the first sign\n",
    "lotsofitems[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# extract the sign name from the first item\n",
    "sign_heading = lotsofitems[1].find(\"strong\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# check it out\n",
    "sign_heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# extract the name of the sign inside the <strong></strong> tags\n",
    "sign_name = sign_heading.get_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#check it out\n",
    "sign_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we don't want to extract any other information from the image, we can combine the steps just to get the element we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# combine extraction into single step\n",
    "sign_name = lotsofitems[1].find(\"strong\").get_text()\n",
    "sign_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to move onto the second notebook - [Extract all sign names on a single page](step2.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}