Onionscan backed by elasticsearch.
Fair warning, I've been on a get all the X into ES kick lately, and this is just the latest value of X. It was also hacked together in a late caffeine fueled night. Don't expect it to be too stable, but it runs pretty well for me and has provided some fun/interesting data to build dashboards with or build out other scans from ES.
I blame @jms_dot_py and this post of his for giving me the idea.
It should be noted that there are a few extraneous moving parts to this. There are reasons, but for a PoC could be removed. E.g., redis in favor of just throwing json at logstash, beanstalk instead of a file/mem based queue, etc.
These notes should be enough to show you what you need. Installing beanstalk, redis, elasticsearch, logstash, kibana, their prerequisites and some python libraries.
$ sudo apt-get install tor git bison libexif-dev screen python-pip golang
$ sudo apt-get install beanstalkd
$ pip install stem pyyaml beanstalkc
$ go get github.com/s-rah/onionscan
$ mkdir ~/gocode
$ echo 'GOLANG=~/gocode' >> .bash_profile
$ tor --hash-password mysecrettorpassword
$ sudo bash -c 'cat << EOF >> /etc/tor/torrc
> ControlPort 9051
> ControlListenAddress 127.0.0.1
> HashedControlPassword 16:5A4ED0F4254848636082D9CBB95379E8BBAD2245D24A091B230B661D7A
> EOF'
$ sudo service tor restart
$ sudo apt-get install redis-server
$ pip install redis
$ sudo add-apt-repository ppa:webupd8team/java -y
$ sudo apt-get update && sudo apt-get install oracle-java8-installer -y
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch -y
$ sudo /bin/systemctl daemon-reload
$ sudo /bin/systemctl enable elasticsearch.service
$ pip install elasticsearch
$ echo "deb http://packages.elastic.co/kibana/4.4/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list
$ sudo apt-get update && sudo apt-get -y install kibana
$ sudo service kibana start
$ echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
$ sudo apt-get update && sudo apt-get install logstash -y
$ sudo cp logstash.conf /etc/logstash/conf.d/elasticonion.conf'
$ sudo service logstash restart