web-scraping-intro-step1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to web scraping using Python",
    "\n",
    "## QUT DMRC - 2016\n",
    "\n",
    "### Extract album title from a single item on a single page\n",
    "\n",
    "This notebook gets a page from the Metacritic website and then extracts one of the fields we are interested in from it.\n",
    "\n",
    "This notebook is broken up into a series of cells. We can run each cell in turn by clicking in the cell and then pressing ```<shift><enter>``` or choosing \"run\" from the \"Cell\" menu at the top of the page. The cell runs and any outputs it creates are shown below it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we import the Python modules we are going to use to get the information from the website."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bs4\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next steps build up the URL that has the information we want. The sections of the url that we will want to change to get more pages of information are kept seperate so we can change them more easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this is the base_url\n",
    "base_url = \"http://www.metacritic.com/browse/albums/artist\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# select which page to scrape based on the first letter of the artist names\n",
    "lett = \"/a\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build the url (only scrape the first page - page 0)\n",
    "thepage = base_url+lett+\"?page=0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets check what the variable ```thepage``` is set to. You can show the value of any variable in a notebook by putting it in the last line of a notebook cell and running the cell. Jupyter will try to display it in a clear way, often clearer than the default 'print' layout. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "thepage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These steps get the page using [Requests](http://docs.python-requests.org/en/latest/) and then process it using [BeautifulSoup](http://www.crummy.com/software/BeautifulSoup/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the bot pretends to be a standard Mozilla browser\n",
    "hdrs = {\"User-Agent\": \"Mozilla/5.0\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# call the url\n",
    "stuff = requests.get(thepage, headers=hdrs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# transform to soup using html.parser parser\n",
    "soup = bs4.BeautifulSoup(stuff.text, \"html.parser\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#find all div-tags of class \"product_wrap\"\n",
    "lotsofitems = soup.find_all(\"div\",class_=[\"product_wrap\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# have a look at the first item in the list\n",
    "lotsofitems[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# extract the first div-tag from the first item\n",
    "thetitle = lotsofitems[0].find(\"div\",class_=\"product_title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# check it out\n",
    "thetitle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# the albumname is the text part of this tag\n",
    "temptemp = thetitle.get_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# check it out\n",
    "temptemp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# it's poorly formatted so we need to clean it up a bit by first splitting the string into a list of words\n",
    "temptemp = temptemp.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# check it out\n",
    "temptemp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# and then we need to join the words back together with single spaces between them\n",
    "clean_title = \" \".join(temptemp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# tada!\n",
    "clean_title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to move onto the second notebook - [Extract all album titles on a single page](web-scraping-intro-step2.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}