Panther

A distributed web crawler written in python.

See my post for more info.

Requirements

Docker for Desktop

Usage

git clone this repo

Run docker-compose build on your machine. Once the process is finished, run docker-compose up to start the service.

URLs can be added to the frontier-queue via localhost:5000, and workers can be monitored via localhost:5001. Finally, the corpus of documents can be monitored via localhost:5002.

Contributing

1. Fork it (https://github.com/yourname/yourproject/fork)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am 'Add some fooBar')
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request