Single site web crawler using aiohttp.
Install from source:
./setup.py install
$ aio_crawler --help
Usage: aio_crawler [OPTIONS] SITE_ADDRESS
Crawl the website and print results to stdout.
Options:
-c, --concurrency INTEGER Number of parallel downloads.
-t, --timeout FLOAT Timeout of each single request.
-v, --verbose
--help Show this message and exit.
It's strongly advised to use a virtualenv.
Install dependencies and the CLI hook:
./setup.py develop
Install test dependencies:
pip install -e '.[tests]'
Python 3.5.
If your OS does not ship Python 3.5, use pyenv. It's miserable but better than nothing.