web-scraping-intro-step2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to web scraping using Python",
    "\n",
    "## QUT DMRC - 2016\n",
    "\n",
    "###  Extract all album titles on a single page\n",
    "\n",
    "This notebook extends the previous step to get all of the titles from a single page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bs4\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this is the base_url\n",
    "base_url = \"http://www.metacritic.com/browse/albums/artist\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# select which page to scrape based on the first letter of the artist names\n",
    "lett = \"/a\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build the url (only scrape the first page - page 0)\n",
    "thepage = base_url+lett+\"?page=0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the bot pretends to be a standard Mozilla browser\n",
    "hdrs = {\"User-Agent\": \"Mozilla/5.0\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# call the url\n",
    "stuff = requests.get(thepage, headers=hdrs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# transform to soup using html.parser parser\n",
    "soup = bs4.BeautifulSoup(stuff.text, \"html.parser\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#try to find all div-tags of class \"product_wrap\"\n",
    "lotsofitems = soup.find_all(\"div\",class_=[\"product_wrap\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now process ```lotsofitems``` in a new way to get all the items instead of just one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# now let's do the same thing as in the previous step but for all items in the page\n",
    "\n",
    "thelist = []\n",
    "\n",
    "for an_item in lotsofitems: \n",
    "    \n",
    "    # extract the first div-tag from the item\n",
    "    thetitle = an_item.find(\"div\",class_=\"product_title\")\n",
    "\n",
    "    # extract and clean up the albumname\n",
    "    temptemp = thetitle.get_text()\n",
    "    temptemp = temptemp.split()\n",
    "    theitem = \" \".join(temptemp)\n",
    "    \n",
    "    # add the item to a list\n",
    "    thelist += [theitem]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "thelist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to move onto the third notebook - [Extract all review data from a single page](web-scraping-intro-step3.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}