index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# QUT DMRC Workshop\n",
    "\n",
    "# Introduction to web scraping with Python\n",
    "\n",
    "This notebook gets a page from a website, extracts data from that page and stores that data as a csv file. It should be fairly straight-forward to adapt the notebook to fit your particular web scraping project.\n",
    "##  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import the Python modules required to extract data from the website.\n",
    "from bs4 import BeautifulSoup\n",
    "from requests import get\n",
    "\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step #1: Get the soup from the website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this is the URL - change this URL to the website you would like to scrape.\n",
    "the_url = \"http://www.spiegel.de/international/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# make the call to the URL\n",
    "stuff = get(the_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# transform to beautiful soup using html.parser parser\n",
    "soup = BeautifulSoup(stuff.text, \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step #2: Find the tags in the beautiful soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Example: Search for p tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Search for p tags\n",
    "lotsofitems = soup.find_all(\"p\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# How many items did you find?\n",
    "len(lotsofitems)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Have a look at the first one in the list (The index of the first item in the list is zero \"0\")\n",
    "lotsofitems[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Example: Search for div tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Search for div tags\n",
    "lotsofitems = soup.find_all(\"div\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# How many did we find?\n",
    "len(lotsofitems)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Step #3: Find tags with specific attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Find ```div``` tags with specific ```class``` values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Find all div-tags of a certain class.\n",
    "# \n",
    "# 'class' is a reserved word in Python so you have to use 'class_' instead.\n",
    "# In some cases you need use the attribute 'id' instead of 'class' to filter out the data...\n",
    "# ...you are looking for.\n",
    "# \n",
    "# Note that this is just an example, you need to look at the source code of the page\n",
    "# you are scraping and change the class value to something that makes sense for your particular case.\n",
    "lotsofitems = soup.find_all(\"div\",class_=\"teaser\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# How many did we find\n",
    "len(lotsofitems)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# have a look at the first item in the list\n",
    "# Try changing the index (between the []) to any number that is lower than the number of items in the list\n",
    "lotsofitems[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Check out the text found in the first data item in the list\n",
    "# Try changing the index (between the []) to any number that is lower than the number of items in the list\n",
    "lotsofitems[0].get_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##  \n",
    "### Step #4 - Extract data from the selected tags\n",
    "Dig deeper into the html code structure<br>\n",
    "Depending on the structure of the web page you are scraping and the data you want to extract,\n",
    "you might need to dig deeper into the structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# In this cell we extract the first div-tag from the first item found in the previous step.\n",
    "#\n",
    "# Try changing the index (between the []) to any number that is lower than the number of items in the list\n",
    "thedata = lotsofitems[0].find(\"span\",class_=\"headline\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Extract the text from this tag\n",
    "temptext = thedata.get_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Clean up the string\n",
    "clean_text = temptext.strip()\n",
    "print(clean_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##  \n",
    "### Step #5: Extract data from all relevant data entities on the page. Save to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# open a file to save the data\n",
    "f = open(\"mydata.csv\",\"a\",encoding=\"utf8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# if the file is empty, add a title row\n",
    "if f.tell()==0:\n",
    "    f.write('\"timestamp\",\"out1\",\"out2\",\"out3\"\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# iterate across 'lotsofitems' and extract three attributes from all relevant data entities on the page.\n",
    "for an_item in lotsofitems:\n",
    "\n",
    "    thedata = an_item.find(\"span\",class_=\"headline\")\n",
    "    temptext = thedata.get_text()\n",
    "    out1 = temptext.strip()\n",
    "\n",
    "    thedata = an_item.find(\"p\",class_=\"article-intro\")\n",
    "    temptext = thedata.get_text()\n",
    "    out2 = temptext.strip()\n",
    "\n",
    "    thedata = an_item.find(\"a\")\n",
    "    temptext = thedata[\"href\"]\n",
    "    out3 = temptext.strip()\n",
    "    \n",
    "    f.write('\"'+str(datetime.now())[:19]+'\",\"'+out1+'\",\"'+out2+'\",\"'+out3+'\"\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# close the file when we're done\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  That's all!\n",
    "### Now open the [dashboard](../tree) and locate the csv file that was saved in the cells above.\n",
    "* Click on the \"mydata.csv\" file in the file list to open the file in a browser tab.\n",
    "* Then chose \"Download as...\" in the \"File\" menu to download the csv file to your computer.\n",
    " \n",
    "### You may also want to explore [eight notebooks](web-scraping-intro-toc.ipynb) that gradually builds a scraper with considerably more functionality than what has been introduced in this notebook.\n",
    "\n",
    "\n",
    "##  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}