Home
See also: Scrapy homepage, Official documentation, Scrapy snippets on Snipplr
If you're new to Scrapy, start by reading Scrapy at a glance.
These are guides contributed by the Scrapy community. If you know of any guide not included here please feel free to add it.
-
Extracting data from websites using Scrapy - April 29, 2016
-
Scrapy Tips from the Pros series:
- January 2016: extracting microdata, parsing JavaScript code snippets
- February 2016: SitemapSpider, per-spider settings
- March 2016: multiple sessions with cookiejars, fallback XPath/CSS selectors
- April 2016: dealing with ASP.Net pages
- May 2016: debugging Scrapy spiders
-
Building a web crawler with Scrapy - September 6, 2013
-
Scrapy after the tutorials - Part 1 and Part 2 - March 8, 2014
-
How to do basic web scraping using Scrapy on a Windows Azure virtual machine - August 19, 2013
-
Scraping iTunes Charts Using Scrapy - June 12, 2013
-
SearchHub: Indexing web sites in Solr with Scrapy - June 13, 2013
-
Using Parsley extraction language with Scrapy - August 2013
-
Running Scrapy on Amazon EC2 - March 26, 2013
-
How to automatically search and download torrents with Python and Scrapy - March 11, 2013
-
Scraping Craigslist with Scrapy (includes video) - Nov 5, 2012
-
How to Install Scrapy 0.14 in a 64 bit Windows 7 Environment - November 2012 (quite old)
-
Scrape multi-pages content with Scrapy - February 13, 2011
-
Calling Scrapy from a Python script - September 27, 2011
-
Scrapy and Django (1) - February 8, 2012
-
Scrapy and Django (2) - July 27, 2012
-
Scraping Google Scholar with Scrapy and MongoDB - October 10, 2011
-
Recursively scraping a blog with Scrapy - March 13, 2012
-
Setup Macports Python and Scrapy successfully - September 8, 2010
-
How to use Scrapy with TOR (scrapy-users message) - May 18, 2012
-
Convert relative paths to absolute paths - June 2011
-
How to use Scrapy, Tor with multiple user agents Part 1 and Part 2 - March & April, 2013
-
How to Run Scrapy Spiders on Cloud Using Heroku and Redis - January 2015
-
Web Scraping With Scrapy and MongoDB - Part 1 and Part 2 - updated September 2015
- Learning Scrapy - ISBN 139781784399788, 270 pages Packt (Jan 2016)
- Python Scrapy: Scrape web data using Python
- Scrapy: it GETs the web - PyCon US 2013 talk
- Installing Scrapy on Windows (video tutorial)
- Recursively scraping Craigslist (includes video) - Nov 8, 2012
- Scraping the Web with Scrapy
- Karthik Ananth: Scrapy Workshop
- Scrapy / Python playlist on Youtube channel
English slides:
- Scrapy - a flexible crawler to power your search - give by Shane Evans in Feb 2013 Cambridge Search Meetup
- Web Crawling & Metadata Extraction in Python
- Crawling the web for fun and profit
- Scrapy for dummies
- Web scraping 1 2-3 with python + scrapy (Summer BarCampHK 2012 version)
- Collecting web information with open source tools
- When big data meet python @ COSCUP 2012
- How to scrape any website's content using Scrapy
Spanish slides:
Chinese slides:
Portuguese Slides:
- Django Dynamic Scraper - a web application (written in django) for runnning and controlling Scrapy spiders
- Slybot - A supervised learning crawler based on Scrapely
- scrapy-sentry - Logs Scrapy exceptions into Sentry
- ScrapyGraphite - Output scrapy statistics to carbon/graphite
- scrapy-mongo - A pipeline to store scrapy items in a MongoDB database
- scrapy-boilerplate - small set of utilities to simplify writing low-complexity spiders
- scrapy-inline-requests - provides a decorator to write spider callbacks which performs multiple requests without the need to write multiple callbacks for each request
- scrapy-redis - providesRedis-backed components for Scrapy
- scrapyz - Create simple spiders easily.
- Scrapy-related libraries on PyPI
- Scrapy_cn - provided a demo to solve encoding problems(utf-8).
- elite-proxies-scrapy-middleware - get new proxies from your EliteProxies account
- scrapydo - Crochet-based blocking API for Scrapy.
See http://scrapy.org/companies/
- see Release notes in the official documentation
- SEPs are available in scrapy/sep.