03_Libraries_and_Tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Libraries and Tools\n",
    "\n",
    "Below is an overview of the libraries and other tools used for webscraping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Urllib](https://docs.python.org/3/library/urllib.html)\n",
    "\n",
    "Urllib is part of the Python standard library. It has several modules used for handling URLs (Uniform Resource Locators). \n",
    "\n",
    "* [`urllib.request`](https://docs.python.org/3/library/urllib.request.html#module-urllib.request) defines functions and classes which help in opening URLs (mostly HTTP) in a complex world — basic and digest authentication, redirections, cookies and more.\n",
    "* [`urllib.error`](https://docs.python.org/3/library/urllib.error.html#module-urllib.error) defines the exception classes for exceptions raised by urllib.request.\n",
    "* [`urllib.parse`](https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse) defines a standard interface to break URL strings into components (addressing scheme, network location, path, etc.) and to build URLs from the components.\n",
    "* [`urllib.robotparser`](https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser) uses a `RobotFileParser` class to answer questions about weather or not a particualr user agent can fetch a URL based on the [`robots.txt`](https://www.robotstxt.org/) file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Requests](https://requests.readthedocs.io/en/master/)\n",
    "\n",
    "Requests is a library that makes it easier to deal with HTTP requests. It is easy to get a web page with the `requests.get(url)` function. Requests can also generate POST requests to submit form data, handle sessions and cookies, handle SSL verification, and more.\n",
    "\n",
    "[Requests Toolbelt](https://toolbelt.readthedocs.io/en/latest/) is a library built to extend the Requests library. It is developed by the Requests core developers and contains tools that you might need, but don't fit into the Requests library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [LXML](https://lxml.de/)\n",
    "\n",
    "LXML is a fast library for parsing XML documents (including HTML). It's built on top of C libraries [libxml2](http://xmlsoft.org/) and [libxslt](http://xmlsoft.org/XSLT/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [cssselect](https://github.com/scrapy/cssselect)\n",
    "\n",
    "To use CSS selectors in LXML, you need to install cssselect. It is developed by the team that develops Scrapy and translates [CSS3 selectors](https://www.w3.org/TR/selectors-3/) to [XPath 1.0](https://www.w3.org/TR/1999/REC-xpath-19991116/) Expressions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)\n",
    "\n",
    "BeautifulSoup is a parsing library that can use different parsers (such as LXML). It is easy to learn and allows you to quickly find specific elements on a page. BeautifulSoup has the ability to infer structure and can fix broken HTML and XML tags. BeautifulSoup will only parse static contnet, so it can not handle javascript generated websites. However, often enough it's easy to inspect the page and find the API calls which can be digested with Requests and other libraries like [json](https://docs.python.org/3/library/json.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Selenium](https://selenium.dev/selenium/docs/api/py/)\n",
    "\n",
    "Selenium is a web driver designed for rendering and testing webpages. It allows you to programatically interact with the web pages, which means you are also able to extract content from the page. The advantage to Selenium is that it will execute javascript and render all the content on the page. However, this will slow down the scraping process and might not be suitable for large scale projects.\n",
    "\n",
    "To run selenium, you need to have a web brower installed. Additionaly, you must download the appropriate web driver and place it in Selenium's path for Selenium to control the browser. The web drivers for common browsers can be found at the links below.\n",
    "\n",
    "* [Chrome](https://chromedriver.chromium.org/downloads)\n",
    "* [Firefox](https://github.com/mozilla/geckodriver/releases)\n",
    "* [Opera](https://github.com/operasoftware/operachromiumdriver/releases)\n",
    "* [Internet Explorer](https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver)\n",
    "* [Edge](https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/)\n",
    "* [Safari](https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Scrapy](https://docs.scrapy.org/en/latest/)\n",
    "\n",
    "Scrapy is a web scraping framework designed for building spiders that crawl and process webpages. It combines the power of Reqeusts and BeautifulSoup in one place to allow you to quickly build scraping pipelines. Scrapy is built on top of [Twisted](https://twistedmatrix.com/trac/) which allows asynchronous requests to speed up the process. \n",
    "\n",
    "Like BeautifulSoup, Scrapy does not inherently render javascript. However, the creators of Scrapy have developed a headless lightweight browser called Splash specifically designed for web scraping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Useful Sites and References    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Web Scraping Sandbox](http://toscrape.com/)\n",
    "* [Fake Bookstore](http://books.toscrape.com/)\n",
    "* Real Quotes. There are several different configurations of this page to practice different scraping techniques.\n",
    "    * [Default: Microdata and pagination](http://quotes.toscrape.com/)\n",
    "    * [Scroll: Infinite scrolling pagination](http://quotes.toscrape.com/scroll)\n",
    "    * [JavaScript: JavaScript generated content](http://quotes.toscrape.com/js)\n",
    "    * [Tableful: A table based messed-up layout](http://quotes.toscrape.com/tableful)\n",
    "    * [Login: Login with CSRF token (any user/passwd works)](http://quotes.toscrape.com/login)\n",
    "    * [ViewState: An AJAX based filter form with ViewStates](http://quotes.toscrape.com/search.aspx)\n",
    "    * [Random: A single random quote](http://quotes.toscrape.com/random)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Real Python](https://realpython.com/)\n",
    "* [Python Web Scraping Tutorials](https://realpython.com/tutorials/web-scraping/)\n",
    "    * [Practical Introduction to Web Scraping in Python](https://realpython.com/python-web-scraping-practical-introduction/)\n",
    "    * [Beautiful Soup: Build a Web Scraper With Python](https://realpython.com/beautiful-soup-web-scraper-python/)\n",
    "    * [Modern Web Automation With Python and Selenium](https://realpython.com/modern-web-automation-with-python-and-selenium/)\n",
    "    * [Web Scraping with Scrapy and MongoDB](https://realpython.com/web-scraping-with-scrapy-and-mongodb/)\n",
    "    * [Web Scraping and Crawling with Scrapy and MongoDB](https://realpython.com/web-scraping-and-crawling-with-scrapy-and-mongodb/)\n",
    "* [Python’s Requests Library](https://realpython.com/python-requests/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [ScrapingHub](https://scrapinghub.com/)\n",
    "* Info\n",
    "    * [How to Build a Web Scraper: Python Web Scraping Libraries & Frameworks Explained](https://info.scrapinghub.com/web-scraping-guide/python-web-scraping-libraries-and-frameworks)\n",
    "    * [The Beginners Guide to Web Scraping](https://info.scrapinghub.com/web-scraping-guide/beginners-guide-to-web-scraping)\n",
    "* [Blog](https://blog.scrapinghub.com/)\n",
    "    * 2019-08-22 [Learn how to configure and utilize proxies with Python Requests module](https://blog.scrapinghub.com/python-requests-proxy)\n",
    "    * 2019-08-08 [How to set up a custom proxy in Scrapy?](https://blog.scrapinghub.com/scrapy-proxy)\n",
    "    * 2018-12-13 [Do What is Right Not What is Easy!](https://blog.scrapinghub.com/gdpr-web-scraping-iiap-europe-data-protection-congress)\n",
    "    * 2017-04-19 [Deploy your Scrapy Spiders from GitHub](https://blog.scrapinghub.com/2017/04/19/deploy-your-scrapy-spiders-from-github)\n",
    "    * 2016-10-27 [An Introduction to XPath: How to Get Started](https://blog.scrapinghub.com/2016/10/27/an-introduction-to-xpath-with-examples)\n",
    "    * 2016-09-28 [How to Run Python Scripts in Scrapy Cloud](https://blog.scrapinghub.com/2016/09/28/how-to-run-python-scripts-in-scrapy-cloud)\n",
    "    * 2016-08-25 [How to Crawl the Web Politely with Scrapy](https://blog.scrapinghub.com/2016/08/25/how-to-crawl-the-web-politely-with-scrapy)\n",
    "    * 2016-06-22 [Scraping Infinite Scrolling Pages](https://blog.scrapinghub.com/2016/06/22/scrapy-tips-from-the-pros-june-2016)\n",
    "    * 2016-05-18 [How to Debug your Scrapy Spiders](https://blog.scrapinghub.com/2016/05/18/scrapy-tips-from-the-pros-may-2016-edition)\n",
    "    * 2016-02-29 [Splash 2.0 Is Here with Qt 5 and Python 3](https://blog.scrapinghub.com/2016/02/29/splash-2-0-here-with-qt-5-and-python-3)\n",
    "    * 2012-10-26 [How to Fill Login Forms Automatically](https://blog.scrapinghub.com/2012/10/26/filling-login-forms-automatically)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [ScrapeHero](https://www.scrapehero.com)\n",
    "* [Tutorials](https://www.scrapehero.com/web-scraping-tutorials/)\n",
    "    * [What is web scraping – Part 1 – Beginner’s guide](https://www.scrapehero.com/a-beginners-guide-to-web-scraping-part-1-the-basics/)\n",
    "    * [Beginners guide to Web Scraping: Part 2 – Build a web scraper for Reddit using Python and BeautifulSoup](https://www.scrapehero.com/a-beginners-guide-to-web-scraping-part-2-build-a-scraper-for-reddit/)\n",
    "    * [Web Scraping Tutorial for Beginners – Part 3 – Navigating and Extracting Data](https://www.scrapehero.com/web-scraping-tutorial-for-beginners-part-3-navigating-and-extracting-data/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [W3 Specifications]()\n",
    "\n",
    "* CSS\n",
    "    * [Selectors Level 4](https://www.w3.org/TR/selectors-4/)\n",
    "    * [Selectors Level 3](https://www.w3.org/TR/selectors-3/)\n",
    "* XPath\n",
    "    * [XML Path Language 3.1](https://www.w3.org/TR/2017/REC-xpath-31-20170321/)\n",
    "    * [XML Path Language 3.0](http://www.w3.org/TR/2014/REC-xpath-30-20140408/)\n",
    "    * [XML Path Language 2.0](http://www.w3.org/TR/2010/REC-xpath20-20101214/)\n",
    "    * [XML Path Language 1.0](http://www.w3.org/TR/1999/REC-xpath-19991116/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [WhatIsMyBrowser](https://www.whatismybrowser.com/)\n",
    "\n",
    "* [Database of User Agent Strings](https://developers.whatismybrowser.com/useragents/explore/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Interesting Links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is Scraping Legal?\n",
    "\n",
    "* [Is Web Scraping Illegal? Depends on What the Meaning of the Word Is](https://www.imperva.com/blog/is-web-scraping-illegal/)\n",
    "* [Better Online Ticket Sales (BOTS)_Act](https://www.congress.gov/bill/114th-congress/senate-bill/3183)\n",
    "* [QVC Can't Stop Web Scraping](https://www.forbes.com/sites/ericgoldman/2015/03/24/qvc-cant-stop-web-scraping/#16f010903ca3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing Out Response Headers\n",
    "\n",
    "* [httpstat.us](https://httpstat.us/): A super simple service for generating different HTTP Codes\n",
    "    * https://httpstat.us/200 will generate an `OK` response\n",
    "    * https://httpstat.us/404 will generate a `NOT FOUND` response\n",
    "* [httpbin.org](http://httpbin.org/): Developed by Kenneth Reitz, the developer of Requests. Test out different HTTP Verbs and other request objects."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}