Lolcrawler is a headless browser crawler created for BugBounty and penetration-testing/RedTeaming. Beware, this code is really slow but should be able to run for several days and find some really interesting urls/paths.
The crawler is using several different methods for trying to find links and urls such as BeautifulSoup, jsbeautifier, urlextract and linkfinder.py.
You need to install docker to run this crawler due to the headless selenium browser.
Animated GIF running the crawler in verbose mode:
Download and run from Docker Hub:
docker pull jonaslejon/lolcrawler
mkdir output
docker run --rm -v `pwd`/output:/output -it jonaslejon/lolcrawler -u https://tor.triop.se -o /output/crawl.log
Download, build and run:
git clone https://github.com/jonaslejon/lolcrawler.git
cd lolcrawler
docker build -t jonaslejon/lolcrawler .
docker run --rm -v `pwd`/output:/output -it jonaslejon/lolcrawler -u https://tor.triop.se -o /output/crawl.log
Run the following command in the folder:
docker rmi jonaslejon/lolcrawler ; docker build -t jonaslejon/lolcrawler .
I was looking for a tool similar to the Burp Suite crawler that I can run and forget about. Also parts of this code is powering the backend of WPSec.com.