workbook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to web scraping\n",
    "\n",
    "This workshop is a one-hour beginner's introduction to web scraping. \n",
    "\n",
    "This notebook deliberately has more content that we can reasonably cover in one hour. **The most important material is in bold**, and we'll focus on that material in person. To get the most out of this workshop, I'd suggest spending some time working through it in full after the workshop.\n",
    "\n",
    "We'll cover the following topics:\n",
    "\n",
    "[Motivation](#motivation)\n",
    "\n",
    "_Why would you want to scrape data from the web?_\n",
    "\n",
    "[How the Web works](#web)\n",
    "\n",
    "_A high-level appreciation of how the Web works will help us to scrape data effectively._\n",
    "\n",
    "[Making a request](#request)\n",
    "\n",
    "_How can we ask other computers on the Internet to send us data using Python?_\n",
    "\n",
    "[Parsing HTML](#parsing)\n",
    "\n",
    "_Web pages are just files in a special format. Extracting information out of these files involves parsing HTML._\n",
    "\n",
    "[Terms of Service](#terms)\n",
    "\n",
    "_Don't go scraping willy-nilly!_\n",
    "\n",
    "[Further resources](#resources)\n",
    "\n",
    "_So you want to learn more about web scraping._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motivation<a id='motivation'></a>\n",
    "\n",
    "It's 2019. The web is everywhere.\n",
    "\n",
    "* If you want to buy a house, real estate agents have [websites](https://www.wendytlouie.com/) where they list the houses they're currently selling. \n",
    "* If you want to know whether to where a rain jacket or shorts, you check the weather on a [website](https://weather.com/weather/tenday/l/Berkeley+CA+USCA0087:1:US). \n",
    "* If you want to know what's happening in the world, you read the news [online](https://www.sfchronicle.com/). \n",
    "* If you've forgotten which city is the capital of Australia, you check [Wikipedia](https://en.wikipedia.org/wiki/Australia).\n",
    "\n",
    "**The point is this: there is an enormous amount of information (also known as data) on the web.**\n",
    "\n",
    "If we (in our capacities as, for example, data scientists, social scientists, digital humanists, businesses, public servants or members of the public) can get our hands on this information, **we can answer all sorts of interesting questions or solve important problems**.\n",
    "\n",
    "* Maybe you're studying gender bias in student evaluations of professors. One option would be to scrape ratings from [Rate My Professors](https://www.ratemyprofessors.com/) (provided you follow their [terms of service](https://www.ratemyprofessors.com/TermsOfUse_us.jsp#use))\n",
    "* Perhaps you want to build an app that shows users articles relating to their specified interests. You could scrape stories from various news websites and then use NLP methods to decide which articles to show which users.\n",
    "* [Geoff Boeing](https://geoffboeing.com/) and [Paul Waddell](https://ced.berkeley.edu/ced/faculty-staff/paul-waddell) recently published [a great study](https://arxiv.org/pdf/1605.05397.pdf) of the US housing market by scraping millions of Craiglist rental listings. Among other insights, their study shows which metropolitan areas in the US are more or less affordable to renters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How the Web works<a id='web'></a>\n",
    "\n",
    "Here's our high-level description of the web.\n",
    "\n",
    "**The internet is a bunch of computers connected together.** Some computers are laptops, some are desktops, some are smart phones, some are servers owned by companies. Each computer has its own address on the internet. Using these addresses, **one computer can ask another computer for some information (data). We say that the first computer sends a _request_ to the second computer, asking for some particular information. The second computer sends back a _response_**. The response could include the information requested, or it could be an error message. Perhaps the second computer doesn't have that information any more, or the first computer isn't allowed to access that information.\n",
    "\n",
    "<img src='img/computer-network.png' />\n",
    "\n",
    "We said that there is an enormous amount of information available on the web. When people put information on the web, they generally have two different audiences in mind, two different types of consumers of their information: humans and computers. If they want their information to be used primarily by humans, they'll make a website. This will let them lay out the information in a visually appealing way, choose colours, add pictures, and make the information interactive. If they want their information to be used by computers, they'll make a web API. A web API provides other computers structured access to their data. We won't cover APIs in this workshop, but you should know that i) APIs are very common and ii) if there is an API for a website/data source, you should use that over web scraping. Many data sources that you might be interested in (e.g. social media sites) have APIs.\n",
    "\n",
    "**Websites are just a bunch of files on one of those computers. They are just plain text files, so you can view them if you want. When you type in the address of a website in your browser, your computer sends a request to the computer located at that address. The request says \"hey buddy, please send me the file(s) for this website\". If everything goes well, the other computer will send back the file(s) in the response**. Everytime you navigate to a new website or page in your browser, this process repeats.\n",
    "\n",
    "<img src='img/request-response.png' />\n",
    "\n",
    "**There are three main languages that that website files are written with: HyperText Markup Language (HTML), Cascading Style Sheets (CSS) and JavaScript (JS)**. They normally have `.html`, `.css` and `.js` file extensions. Each language (and thus each type of file) serves a different purpose. **HTML files are the ones we care about the most, because they are the ones that contain the text you see on a web page**. CSS files contain the instructions on how to make the content in a HTML visually appealing (all the colours, font sizes, border widths, etc.). JavaScript files have the instructions on how to make the information on a website interactive (things like changing colour when you click something, entering data in a form). In this workshop, we're going to focus on HTML.\n",
    "\n",
    "\n",
    "**It's not too much of a simplification to say:**\n",
    "\n",
    "\\begin{equation}\n",
    "\\textrm{Web scraping} = \\textrm{Making a request for a HTML file} + \\textrm{Parsing the HTML response}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a request<a id='request'></a>\n",
    "\n",
    "**The first step in web scraping is to get the HTML of the website we want to scrape. The [requests](http://docs.python-requests.org/en/master/) library is the easiest way to do this in Python.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "url = 'https://en.wikipedia.org/wiki/Canberra'\n",
    "\n",
    "response = requests.get(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, it looks like everything worked! Let's see our beautiful HTML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Huh, that's weird. Doesn't look like HTML to me.\n",
    "\n",
    "What the `requests.get` function returned (and the thing in our `response` variable) was a Response object. It itself isn't the HTML that we wanted, but rather a collection of metadata about the request/response interaction between your computer and the Wikipedia server.\n",
    "\n",
    "For example, it knows whether the response was successful or not (`response.ok`), how long the whole interaction took (`response.elapsed`), what time the request took place (`response.headers['Date']`) and a whole bunch of other metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.headers['Date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Of course, what we really care about is the HTML content. We can get that from the `Response` object with `response.text`. What we get back is a string of HTML, exactly the contents of the HTML file at the URL that we requested.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = response.text\n",
    "print(html[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "Get the HTML for [the Wikipedia page about HTML](https://en.wikipedia.org/wiki/HTML). \n",
    "Print out the first 1000 characters and compare it to the HTML you see when you view the source HTML in your broswer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "url = 'https://en.wikipedia.org/wiki/HTML'\n",
    "response = requests.get(url)\n",
    "html = response.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "Write a function called `get_html` that takes a URL as an argument and returns the HTML contents as a string. Test your function on the page for [Sir Tim Berners-Lee](https://en.wikipedia.org/wiki/Tim_Berners-Lee)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "def get_html(url):\n",
    "    response = requests.get(url)\n",
    "    return response.text\n",
    "\n",
    "url = 'https://en.wikipedia.org/wiki/Tim_Berners-Lee'\n",
    "html = get_html(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "What happens if the request doesn't go so smoothly? Add a defensive measure to your function to check that the response recieved was successful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "def get_html(url):\n",
    "    response = requests.get(url)\n",
    "    assert response.ok, \"Whoops, this request didn't go as planned!\"\n",
    "    return response.text\n",
    "    \n",
    "\n",
    "url = 'https://en.wikipedia.org/wiki/Tim_Berners-Lee'\n",
    "html = get_html(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing HTML<a id='parsing'></a>\n",
    "\n",
    "**The second step in web scraping is parsing HTML. This is where things can get a little tricky.** \n",
    "\n",
    "**Imagine you're in the field of education, in fact your specialty is studying higher education institutions. You're wondering how different disciplines change over time. Is it true that disciplines are incorporating more computational techniques as the years go on? Is that true for all disciplines or only some? Can we spot emerging themes across a whole university?**\n",
    "\n",
    "**To answer these questions, we're going to need data. We're going to collect a dataset of all courses registered at UC Berkeley, not just those being taught this semester but all courses currently approved to be taught. These are listed on [this page](http://guide.berkeley.edu/courses/), called the Academic Guide. Well, actually they're not directly listed on that page. That page lists the departments/programs/units that teach currently approved courses. If we click on each department (for the sake of brevity, I'm just going to call them all \"departments\"), we can see the list of all courses they're approved to teach. For example, [here's](http://guide.berkeley.edu/courses/aerospc/) the page for Aerospace Studies. We'll call these pages departmental pages.**\n",
    "\n",
    "### Challenge\n",
    "\n",
    "View the source HTML of [the page listing all departments](http://guide.berkeley.edu/courses/), and see if you can find the part of the HTML where the departments are listed. There's a lot of other stuff in the file that we don't care too much about. You could try `Crtl-F`ing for the name of a department you can see on the webpage.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**Solution**\n",
    "\n",
    "You should see something like this:\n",
    "\n",
    "\n",
    "```\n",
    "<div id=\"atozindex\">\n",
    "<h2 class=\"letternav-head\" id='A'><a name='A'>A</a></h2>\n",
    "<ul>\n",
    "<li><a href=\"/courses/aerospc/\">Aerospace Studies (AEROSPC)</a></li>\n",
    "<li><a href=\"/courses/africam/\">African American Studies (AFRICAM)</a></li>\n",
    "<li><a href=\"/courses/a,resec/\">Agricultural and Resource Economics (A,RESEC)</a></li>\n",
    "<li><a href=\"/courses/amerstd/\">American Studies (AMERSTD)</a></li>\n",
    "<li><a href=\"/courses/ahma/\">Ancient History and Mediterranean Archaeology (AHMA)</a></li>\n",
    "<li><a href=\"/courses/anthro/\">Anthropology (ANTHRO)</a></li>\n",
    "<li><a href=\"/courses/ast/\">Applied Science and Technology (AST)</a></li>\n",
    "<li><a href=\"/courses/arabic/\">Arabic (ARABIC)</a></li>\n",
    "<li><a href=\"/courses/arch/\">Architecture (ARCH)</a></li>\n",
    "```\n",
    "\n",
    "**This is HTML. HTML uses \"tags\", code that surrounds the raw text which indicates the structure of the content. The tags are enclosed in `<` and `>` symbols. The `<li>` says \"this is a new thing in a list and `</li>` says \"that's the end of that new thing in the list\". Similarly, the `<a ...>` and the `</a>` say, \"everything between us is a hyperlink\". In this HTML file, each department is listed in a list with `<li>...</li>` and is also linked to its own page using `<a>...</a>`. In our browser, if we click on the name of the department, it takes us to that department's own page. The way the browser knows where to go is because the `<a>...</a>` tag tells it what page to go to. You'll see inside the `<a>` bit, there's a `href=...`. That tells us the (relative) location of the page it's linked to.**\n",
    "\n",
    "\n",
    "\n",
    "### Challenge\n",
    "\n",
    "Look at HTML source of [the page for the Aerospace Studies department](http://guide.berkeley.edu/courses/aerospc/), and try to find the part of the file where the information on each course is. Again, try searching for it using `Crtl-F`.\n",
    "\n",
    "**Solution**\n",
    "\n",
    "\n",
    "```\n",
    "<div class=\"courseblock\">\n",
    "\n",
    "<button class=\"btn_toggleCoursebody\" aria-expanded=\"false\" aria-controls=\"cb_aerospc1a\" data-toggle=\"#cb_aerospc1a\">\n",
    "\n",
    "<a name=\"spanaerospc1aspanspanfoundationsoftheu.s.airforcespanspan1unitspan\"></a>\n",
    "<h3 class=\"courseblocktitle\">\n",
    "<span class=\"code\">AEROSPC 1A</span> \n",
    "<span class=\"title\">Foundations of the U.S. Air Force</span> \n",
    "<span class=\"hours\">1 Unit</span>\n",
    "</h3>\n",
    "```\n",
    "\n",
    "The content that we care about is enclosed within HTML tags. It looks like the course code is enclosed in a `span` tag, which has a `class` attribute with the value `\"code\"`. What we'll have to do is extract out the information we care about by specifying what tag it's enclosed in.\n",
    "\n",
    "But first, we're going to need to get the HTML of the first page.\n",
    "\n",
    "### Challenge\n",
    "\n",
    "Get the HTML content of `http://guide.berkeley.edu/courses/` and store it in a variable called `academic_guide_html`. You can use the `get_html` function you wrote before.\n",
    "\n",
    "Print the first 500 characters to see what we got back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "academic_guide_url = 'http://guide.berkeley.edu/courses/'\n",
    "academic_guide_html = get_html(academic_guide_url)\n",
    "print(academic_guide_html[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, we've got the HTML contents of the Academic Guide site we want to scrape. Now we can parse it. [\"Parsing\"](https://en.wikipedia.org/wiki/Parsing) means to turn a string of data into a structured representation. When we're parsing HTML, we're taking the Python string and turning it into a tree. The Python package `BeautifulSoup` does all our HTML parsing for us. We give it our HTML as a string and it returns a parsed HTML tree. Here, we're also telling BeautifulSoup to use the `lxml` parser behind the scenes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "academic_guide_soup = BeautifulSoup(academic_guide_html, 'lxml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We said before that all the departments were listed on the Academic Guide page with links to their departmental page, where the actual courses are listed. So we can find all the departments by looking in our parsed HTML for all the links. Remember that the links are represented in the HTML with the `<a>...</a>` tag, so we ask our `academic_guide_soup` to find us all the tags called `a`. What we get back is a list of all the `a` elements in the HTML page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "links = academic_guide_soup.find_all('a')\n",
    "# print a random link element\n",
    "links[48]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we have a list of `a` elements, each one represents a link on the Academic Guide page. But there are other links on this page in addition to the ones we care about, for example, a link back to the UC Berkeley home page. How can we filter out all the links we don't care about?\n",
    "\n",
    "### Challenge\n",
    "\n",
    "Look through the list `links`, or the HTML source, and figure out how we can identify just the links that we care about, namely the links to departmental pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "import re\n",
    "\n",
    "def is_departmental_page(link):\n",
    "    \"\"\"\n",
    "    Return true if `link` points to a departmental page.\n",
    "    \n",
    "    By examining the source HTML by eye, I noticed that \n",
    "    the links we care about (i.e. the departmental pages) \n",
    "    all point to a relative path that starts with \"/courses/\".\n",
    "    This function uses that idea to determine if the link is \n",
    "    a departmental page.\n",
    "    \"\"\"\n",
    "    # some links don't have a href attribute, only a name attribute\n",
    "    # we don't care about them\n",
    "    try:\n",
    "        href = link.attrs['href'] \n",
    "    except KeyError:\n",
    "        return False\n",
    "    pattern = r'/courses/(.*)/'\n",
    "    match = re.search(pattern, href)\n",
    "    return bool(match)\n",
    "\n",
    "print(links[0])\n",
    "print(is_departmental_page(links[0]))\n",
    "print()\n",
    "print(links[48])\n",
    "print(is_departmental_page(links[48]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use our new `is_departmental_page` function to filter out the links we don't care about. How many departments do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "departmental_page_links = [link for link in links if is_departmental_page(link)]\n",
    "len(departmental_page_links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each link in our `departmental_page_links` list contains a HTML element representing a link. Each element contains not only the relative location of the link but also the text that is linked (i.e. the words on the page that are underlined and you can click on to go to the linked page). In BeautifulSoup, we can get that text by asking for it with `element.text`, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "departmental_page_links[0].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "From the `departmental_page_links`, we can extract out the name and the code for each department. Try doing this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "import re\n",
    "\n",
    "def extract_department_name_and_code(departmental_link):\n",
    "    \"\"\"\n",
    "    Return the (name, code) for a department.\n",
    "    \n",
    "    The easiest way to do this is to use regular expressions. \n",
    "    We're not going to cover regular expressions in this workshop, \n",
    "    but here's how to do it anyway.\n",
    "    \"\"\"\n",
    "    text = departmental_link.text\n",
    "    pattern = r'([^(]+) \\((.*)\\)'\n",
    "    match = re.search(pattern, text)\n",
    "    if match:\n",
    "        return match.group(1), match.group(2)\n",
    "\n",
    "extract_department_name_and_code(links[48])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From each link in our `departmental_page_links` list, we can get the relative link that it points to like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "departmental_page_links[0].attrs['href']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "Write a function that extracts out the relative link of a link element.\n",
    "\n",
    "*Hint: This has a similar solution to our `is_departmental_page` function from before.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "def extract_relative_link(departmental_link):\n",
    "    \"\"\"\n",
    "    We noted above that all the departmental links point to \"/courses/something/\", \n",
    "    where the \"something\" looks a lot like their code. This function \n",
    "    extracts out that \"something\", so we can add it to the base URL of \n",
    "    the Academic Guide page and get full paths to each departmental page.\n",
    "    \"\"\"\n",
    "    href = departmental_link.attrs['href']\n",
    "    pattern = r'/courses/(.*)/'\n",
    "    match = re.search(pattern, href)\n",
    "    if match:\n",
    "        return match.group(1)\n",
    "\n",
    "extract_relative_link(departmental_page_links[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright! Now we've identified all the departmental links on the Academic Guide page, we've found their name and code, and we know the relative link they point to. Next, we can use this relative link to construct the full URL they point to, which we'll then use to scrape the HTML for each departmental page.\n",
    "\n",
    "Let's write a function that takes a departmental link and returns the absolute URL of its departmental page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def construct_absolute_url(departmental_link):\n",
    "    relative_link = extract_relative_link(departmental_link)\n",
    "    return academic_guide_url + relative_link\n",
    "\n",
    "construct_absolute_url(departmental_page_links[37])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize so far, we've gone from the URL of the Academic Guide website, found all the departments that offer approved courses, identified their name and code and the link to their departmental page which lists all the courses they teach. \n",
    "\n",
    "Now we want to find the get the HTML for each departmental page and scrape it for all the courses they offer. Let's focus on one page for now, the Aerospace Studies page. Once we select the link, we use our functions from above to: i) get the name (I guess we already know it's Aerospace, but whatever) and code, get the full URL, get the HTML for that URL and then parse the HTML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aerospace_link = departmental_page_links[0]\n",
    "aerospace_name, aerospace_code = extract_department_name_and_code(aerospace_link)\n",
    "aerospace_url = construct_absolute_url(aerospace_link)\n",
    "aerospace_html = get_html(aerospace_url)\n",
    "aerospace_soup = BeautifulSoup(aerospace_html, 'lxml')\n",
    "print(aerospace_html[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right at the start of this section on parsing HTML, we saw the HTML for a departmental page. Here it is again.\n",
    "\n",
    "```\n",
    "<div class=\"courseblock\">\n",
    "\n",
    "<button class=\"btn_toggleCoursebody\" aria-expanded=\"false\" aria-controls=\"cb_aerospc1a\" data-toggle=\"#cb_aerospc1a\">\n",
    "\n",
    "<a name=\"spanaerospc1aspanspanfoundationsoftheu.s.airforcespanspan1unitspan\"></a>\n",
    "<h3 class=\"courseblocktitle\">\n",
    "<span class=\"code\">AEROSPC 1A</span> \n",
    "<span class=\"title\">Foundations of the U.S. Air Force</span> \n",
    "<span class=\"hours\">1 Unit</span>\n",
    "</h3>\n",
    "```\n",
    "\n",
    "It looks like each course is listed in a `div` element that has a `class` attribute with value `\"courseblock\"`. We can use this information to identify all the courses on a page and then extract out the information from them. You've seen how to do this before, here it is again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aerospace_courseblocks = aerospace_soup.find_all(class_='courseblock')\n",
    "len(aerospace_courseblocks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the Aerospace department has seven current courses they're approved to teach (at the time of writing). Looking at the page in our browser, that looks right to me! So now we have a list called `aerospace_courseblocks` that holds seven elements that each refer to one course taught by the Aerospace department. Now we can extract out any information we care about. We just have to look at the page in our browser, decide what information we care about, then look at the HTML source to see where that information is kept in the HTML structure. Finally, we write a function for each piece of information we want to extract out of a course.\n",
    "\n",
    "### Challenge\n",
    "Write functions to take a courseblock and extract:\n",
    "- The course code (e.g. AEROSPC 1A)\n",
    "- The coure name\n",
    "- The number of units\n",
    "- The textual description of the course"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "def extract_course_code(courseblock):\n",
    "    span = courseblock.find(class_='code')\n",
    "    return span.text\n",
    "\n",
    "def extract_course_title(courseblock):\n",
    "    span = courseblock.find(class_='title')\n",
    "    return span.text\n",
    "\n",
    "def extract_course_units(courseblock):\n",
    "    span = courseblock.find(class_='hours')\n",
    "    return span.text\n",
    "\n",
    "def extract_course_description(courseblock):\n",
    "    span = courseblock.find(class_='coursebody')\n",
    "    return span.text\n",
    "\n",
    "def extract_one_course(courseblock):\n",
    "    course = {}\n",
    "    course['course_code'] = extract_course_code(courseblock)\n",
    "    course['course_title'] = extract_course_title(courseblock)\n",
    "    course['course_units'] = extract_course_units(courseblock)\n",
    "    course['course_description'] = extract_course_description(courseblock)\n",
    "    return course\n",
    "\n",
    "first_aerospace_course = extract_one_course(aerospace_courseblocks[0])\n",
    "for value in first_aerospace_course.values():\n",
    "    print(value)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a function to scrape these four pieces of information from every course from every department and save it as a csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_one_department(department_link):\n",
    "    department_name, department_code = extract_department_name_and_code(department_link)\n",
    "    department_url = construct_absolute_url(department_link)\n",
    "    department_html = get_html(department_url)\n",
    "    department_soup = BeautifulSoup(department_html, 'lxml')\n",
    "    department_courseblocks = department_soup.find_all(class_='courseblock')\n",
    "    result = []\n",
    "    for courseblock in department_courseblocks:\n",
    "        course = extract_one_course(courseblock)\n",
    "        course['department_name'] = department_name\n",
    "        course['department_code'] = department_code\n",
    "        result.append(course)\n",
    "    return result\n",
    "\n",
    "aerospace_courses = scrape_one_department(aerospace_link)\n",
    "for value in aerospace_courses[0].values():\n",
    "    print(value)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def scrape_all_departments(be_nice=True):\n",
    "    academic_guide_url = 'http://guide.berkeley.edu/courses/'\n",
    "    academic_guide_html = get_html(academic_guide_url)\n",
    "    academic_guide_soup = BeautifulSoup(academic_guide_html, 'lxml')\n",
    "    links = academic_guide_soup.find_all('a')\n",
    "    departmental_page_links = [link for link in links if is_departmental_page(link)]\n",
    "    \n",
    "    result = []\n",
    "    for departmental_page_link in departmental_page_links:\n",
    "        department_result = scrape_one_department(departmental_page_link)\n",
    "        result.extend(department_result)\n",
    "        if be_nice:\n",
    "            time.sleep(1)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "result = scrape_all_departments(be_nice=False)\n",
    "df = pd.DataFrame(result)\n",
    "print(str(len(df)) + ' courses scraped')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9360 courses scraped! (At the time of writing). Wow, that was a lot easier than doing it by hand!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Terms of Service<a id='terms'></a>\n",
    "\n",
    "As you've seen, web scraping involves making requests from other computers for their data. It costs people money to maintain the computers that we request data from: it needs electricity, it requires staff, sometimes you need to upgrade the computer, etc. But we didn't pay anyone for using their resources.\n",
    "\n",
    "Because we're making these requests programmatically, we could make many, many requests per second. For example, we could put a request in a never-ending loop which would constantly request data from a server. But computers can't handle too much traffic, so eventually this might crash someone else's computer. Moreover, if we make too many requests when we're web scraping, that might restrict the number of people who can view the web page in their browser. This isn't very nice.\n",
    "\n",
    "Websites often have Terms of Service, documents that you agree to whenever you visit a site. Some of these terms prohibit web scraping, because it puts too much strain on their servers, or they just don't want their data accessed programmatically. Whatever the reason, we need to respect a websites Terms of Service. **Before you scrape a site, you should always check its terms of service to make sure it's allowed.**\n",
    "\n",
    "Often, there are better ways of accessing the same data. For the Wikipedia sites we scraped, there's actually an [API](https://www.mediawiki.org/wiki/REST_API) that we could have used. In fact, Wikipedia would prefer that we access their data that way. There's even a [Python package](https://pypi.org/project/wikipedia/) that wraps around this API to make it even easier to use. Furthermore, Wikipedia actually makes all of its content available for [direct download](https://dumps.wikimedia.org/). **The point of the story is: before web scraping, see if you can get the same data elsewhere.** This will often be easier for you and preferred by the people who own the data.\n",
    "\n",
    "Moreover, if you're affiliated with an institution, you may be breaching existing contracts by engaging in scraping. UC Berkeley's Library [recommends](http://guides.lib.berkeley.edu/text-mining) following this workflow:\n",
    "\n",
    "<img src='img/workflow.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further resources<a id='resources'></a>\n",
    "\n",
    "### Resources for learning\n",
    "\n",
    "* Work through this notebook in full.\n",
    "    * _We glossed over a lot of details. As your next step, I'd suggest spending as much time as you need to understand every line of text and code in this notebook._\n",
    "\n",
    "* [Web-scraping with Python](http://shop.oreilly.com/product/0636920078067.do)\n",
    "\n",
    "    * _A great textbook for learning more about web scraping using Python._\n",
    "\n",
    "* [Fantastic Data and Where To Find Them: An introduction to APIs, RSS, and Scraping](https://www.youtube.com/watch?v=A42voDYkFZw)\n",
    "\n",
    "    * This is a recorded video of a workshop on collecting data via the web at PyCon, a Python conference._\n",
    "\n",
    "* [D-Lab workshops](http://dlab.berkeley.edu/training)\n",
    "\n",
    "    * _We teach workshops on web scraping and NLP-related tools throughout the semester. Check this page for the latest scheduled workshops._\n",
    "\n",
    "* [D-Lab consulting](http://dlab.berkeley.edu/consulting)\n",
    "\n",
    "    * _We also offer free consulting for members of UC Berkeley's community. Reach out to us if you ever need a hand with a web scraping or NLP project!_\n",
    "\n",
    "\n",
    "### A few libraries to be aware of\n",
    "\n",
    "* [requests-HTML](http://html.python-requests.org/)\n",
    "\n",
    "    * _Did you see how easy it was to request data using the `requests` library? Well the author of that library, Kenneth Reitz, has another library for parsing HTML. I'm not that familiar with it, but it looks promising if it's by Reitz!_\n",
    "\n",
    "* [furl](https://github.com/gruns/furl)\n",
    "\n",
    "    * _Lets you extract out different parts of a URL._\n",
    "\n",
    "* [cssutils](http://cthedot.de/cssutils/)\n",
    "\n",
    "    * _We didn't talk about CSS at all, but you can also scrape data depending on its visual characteristics. This is a great library for parsing CSS files, but you can get some of the same functionality with BeautifulSoup._\n",
    "\n",
    "* [scrapy](https://scrapy.org/)\n",
    "\n",
    "    * _Need to do some serious web scraping? You'll wanna check out Scrapy._\n",
    "\n",
    "* [newspaper](https://newspaper.readthedocs.io/en/latest/)\n",
    "\n",
    "    * _If you know you're focussed on newspaper articles, this is a great little library for parsing common formats._"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}