HR-Scanner

A simple domain names scanner. The app knows to scan websites and extract X-Recruiting header, if presents.

Tutorial

https://medium.com/@alexanderravikovich/scan-10m-websites-for-x-recruiting-header-in-go-fa85816da994

Usage

mkdir -p ${GOPATH}/src/github.com/spaiz/
cd ${GOPATH}/src/github.com/spaiz/
git clone git@github.com:spaiz/hrscanner.git .
cd hrscanner

Install dependency manager use in the project

go get -u github.com/kardianos/govendor

Install dependencies:

govendor sync

Install app locally:

go install

Use this helper to build statically compiled binary on the MacOS (DOcker needed)

./bin/build

To run the app:

cd ${GOPATH}/src/github.com/spaiz/hrscanner/data/
unzip dns-servers.txt.zip && rm dns-servers.txt.zip
unzip uniq-domains.txt.zip && uniq-domains.txt.zip
cd ${GOPATH}/src/github.com/spaiz/hrscanner/
hrscanner -workers=500

U can previously compiled binary inside the Docker locally or upload the image to AWS ECR and run it as a Task

docker build -t hrscanner -f Dockerfile .
docker run -it hrscanner
# or with arguments
docker run -it hrscanner hrscanner -workers=500

Important

Increase ulimit on your system if you want to scan huge number of websites

U can always download updated list of DNS servers from:

https://public-dns.info//nameservers.csv