Turmeric Stained Data

Contains some IPs that look like they host domains. Some of the IPs are proxies to any domain specified in TLS SNI.

Other investigations might include looking for domains found in DO ranges versus those results from Shodan's API.

DO443.txt

Masscan Results across Digital Ocean allocated IP space for port 443.

testTargets.txt

The top 1,000 CloudFlare hosted target domains found in the Alexa Top 1 Million. Generated by ALEXATOP.

digitaloceanResults/

A directory containing CURRYFINGER output from the command;

parallel -j 50 ./CURRYFINGER -url https://{} -threads 1000 -show=false -timeout 10s -file DO443.txt :::: test1000.txt 2>/dev/null | grep ^match > results.txt

failedIPResults.txt

An interesting subset of all findings which initially reported TLS errors - but subsequently succeded with CURRYFINGER's SNI/Host setting.

shodanResults

A directory containing two sets of files for the 1,000 targets in testTargets.txt.

First, an enumeration of potential target IPs from Shodan;

for line in `cat test1000.txt`; do
    curl "https://api.shodan.io/shodan/host/search?key=[ELIDED]]&query=ssl%3A\"$line\"" | jq ".matches|.[].ip_str" | tee $line.res.txt
    sleep 1s
done

Which can be found in *.res.txt files.

Second, using CURRYFINGER to look for matches;

for line in `cat test1000.txt`; do
	cat shodanScan/$line.res.txt | tr -d '"' | ./CURRYFINGER -url https://$line -threads 1000 -show=false -timeout 10s | tee $line.finger.txt
done

Which can be found in *.finger.txt files.

These files contain positive results for CloudFlare ranges - and those matches should be filtered before further analysis.

Misc.

Here's a fine way to import with python pandas;

pd.read_table('./allIPResults.txt', sep=" ", 
                         header=None, names=[0,'ip','url','matchpct',4,5,'mbytes',7], usecols=['ip','url','matchpct','mbytes'])

Cites

Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

OK.