web-scraping-intro-step5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to web scraping using Python",
    "\n",
    "## QUT DMRC - 2016\n",
    "\n",
    "### Store the data in a dataframe and save to disk\n",
    "\n",
    "This notebook extends the ``get_itemlist`` function to save the data in a dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bs4\n",
    "import requests\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scrape the page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this is the base_url\n",
    "base_url = \"http://www.metacritic.com/browse/albums/artist\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# select which page to scrape based on the first letter of the artist names\n",
    "lett = \"/a\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build the url (only scrape the first page - page 0)\n",
    "thepage = base_url+lett+\"?page=0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the bot pretends to be a standard Mozilla browser\n",
    "hdrs = {\"User-Agent\": \"Mozilla/5.0\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# call the url\n",
    "stuff = requests.get(thepage, headers=hdrs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# transform to soup using html.parser parser\n",
    "soup = bs4.BeautifulSoup(stuff.text, \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define column labels for the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# columns labels\n",
    "colnames = [\"artistname\", \"albumname\", \"release_date\", \"mc_score\", \"user_score\", \"url\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Function definitions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# processes a beautiful_soup data structure and returns new album_reviews in a dataframe\n",
    "def get_itemlist(thesoup):\n",
    "    \n",
    "    #try to find all div-tags of class \"product_wrap\"\n",
    "    lotsofitems = thesoup.find_all(\"div\",class_=[\"product_wrap\"])\n",
    "    \n",
    "    thelist = []\n",
    "    for an_item in lotsofitems: \n",
    "        theitem = []\n",
    "        \n",
    "        # artistname\n",
    "        temptemp = an_item.find(\"li\",class_=\"product_artist\")\n",
    "        theitem += [temptemp.find(\"span\",class_=[\"data\"]).get_text()]\n",
    "\n",
    "        thetitle = an_item.find(\"div\",class_=\"product_title\")\n",
    "\n",
    "        # albumname\n",
    "        temptemp = thetitle.get_text()\n",
    "        temptemp = temptemp.split()\n",
    "        theitem += [\" \".join(temptemp)]\n",
    "        \n",
    "        # release_date\n",
    "        temptemp = an_item.find(\"li\",class_=\"release_date\")\n",
    "        theitem += [temptemp.find(\"span\",class_=[\"data\"]).get_text()]\n",
    "        \n",
    "        # mc_score\n",
    "        theitem += [an_item.find(\"div\",class_=\"metascore_w\").get_text()]\n",
    "\n",
    "        # user_score\n",
    "        temptemp = an_item.find(\"li\",class_=\"product_avguserscore\")\n",
    "        theitem += [temptemp.find(\"span\",class_=[\"data\"]).get_text()]\n",
    "        \n",
    "        # url\n",
    "        theitem += [\"http://www.metacritic.com\"+thetitle.a.attrs[\"href\"]]\n",
    "\n",
    "        # not all albums have both expert reviews and user reviews. Those albums\n",
    "        # that has data missing, use \"tbd\" instead. We only want to add items\n",
    "        # that have both user_score and mc_score\n",
    "        if not \"tbd\" in theitem:\n",
    "            thelist += [theitem]\n",
    "\n",
    "    return pd.DataFrame(thelist,columns=colnames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Call the function\n",
    "\n",
    "Call the function passing it the ``soup`` variable and store the result in the variable ```album_reviews```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "album_reviews = get_itemlist(soup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# have a look\n",
    "album_reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# save the data as a csv file\n",
    "album_reviews.to_csv(\"reviews.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to move onto the sixth notebook - [Restructure the code for clarity](web-scraping-intro-step6.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}