Not Crawling

[amiremami]

On BBOT v1.1.7.3251rc , on this site, bbot is not crawling at all.

debug.log

[stryker2k2]

I'm tinkering around with it now. @TheTechromancer would know better than me. But... looking at your debug log, there seems to be a big lack of 'URL_UNVERIFIED' from the excavate module to feed everything else; which would allowing everything else to spider out.

An hour ago, I ran bbot -t <target> -m httpx to keep it very simple. An hour ago, I got nothing.

But now when I run the same command, I get:

[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | Module    | Produced                          | Consumed                              |
[INFO] aggregate: +===========+===================================+=======================================+
[INFO] aggregate: | excavate  | 54 (54 URL_UNVERIFIED)            | 2 (2 HTTP_RESPONSE)                   |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | speculate | 3 (1 OPEN_TCP_PORT, 1 ORG_STUB, 1 | 78 (9 DNS_NAME, 2 HTTP_RESPONSE, 1    |
[INFO] aggregate: |           | URL_UNVERIFIED)                   | IP_ADDRESS, 2 URL, 64 URL_UNVERIFIED) |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | httpx     | 2 (2 URL)                         | 3 (2 OPEN_TCP_PORT, 1 URL_UNVERIFIED) |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | host      | 1 (1 DNS_NAME)                    | 0                                     |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+

I could absolutely do something with this now if I added more modules and config settings.

I don't know what the answer is. But, I'm guessing either normal network hiccup... or something fishy with excavate and/or httpx.

I'll keep looking into it.