Skip to content

IIIypuk09/scrapy-webdriver

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

scrapy-webdriver

Scrape using Selenium webdriver.

Not well tested. Has probably lots of bugs, unhandled situations, and scrapy integration problems. Use at your own risk.

That being said, it works for me, in all its non-blocking glory.

Installation

For now it's not on pypi, but this should work:

pip install https://github.com/sosign/scrapy-webdriver/archive/master.zip

Or something like this, in setup.py:

setup(
    install_requires=[
        'scrapy_webdriver',
        ...,
    ],
    dependency_links=[
        'https://github.com/sosign/scrapy-webdriver/archive/master.zip#egg=scrapy_webdriver',
    ],
    ...,
    )

Configuration

Add something like this in your scrapy project settings:

DOWNLOAD_HANDLERS = {
    'http': 'scrapy_webdriver.download.WebdriverDownloadHandler',
    'https': 'scrapy_webdriver.download.WebdriverDownloadHandler',
}

SPIDER_MIDDLEWARES = {
    'scrapy_webdriver.middlewares.WebdriverSpiderMiddleware': 543,
}

WEBDRIVER_BROWSER = 'PhantomJS'  # Or any other from selenium.webdriver
                                 # or 'your_package.CustomWebdriverClass'
                                 # or an actual class instead of a string.
# Optional passing of parameters to the webdriver
WEBDRIVER_OPTIONS = {
    'service_args': ['--debug=true', '--load-images=false', '--webdriver-loglevel=debug']
}

Usage

In order to have webdriver handle your downloads, use the provided class scrapy_webdriver.http.WebdriverRequest in place of the stock scrapy Request, like so:

from scrapy_webdriver.http import WebdriverRequest
yield WebdriverRequest('http://www.example.com')

Parameters not supported (yet?) are: method, body, headers, cookies.

Hacking

Pull requests much welcome. Just make sure the tests still pass, and add to them as necessary:

python setup.py test

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published