BBOT 2.0

[TheTechromancer]

This is the main PR for BBOT 2.0 which will be merged into dev at its release.

Beginning with #1058, features destined for BBOT 2.0 will be merged into this branch.

Merged Features:

• Presets #1058
• DNS Engine - Offload DNS to Dedicated Process #1231
• Intercept Modules #1242
• Shared Ansible Dependencies #1270
• Faster Regexes #1278
• Implement Radix Tree #1295
• Helperify Massdns #1303
• Stdout Output Module #1234
• Fix omitted event types #1331
• Shorten ingress queue #1332
• Clean up debug messages #1333
• Improve --brief stdout #1334
• HTTP Engine - Offload Web Requests to Dedicated Process #1340
• Ensure BBOT config file gets created on startup #1350
• better pickle/unpickle error handling #1351
• Misc Minor BBOT 2.0 Tweaks #1358
• Paramminer Extract Validation / Tests #1359
• Include "host" as attribute in event JSON #1364
• Fix whitelisting scope bug #1366
• Adding custom headers CLI argument / tests #1368
• Fix subdomain enum duplicate query bug #1373
• Portscanner Rework #1375
• Regex Performance Improvements, Tests #1393
• Module Metadata: Author Name + Created Date #1395
• Event Context #1383

New Features

• Presets (-p / --presets)
• New discovery_context attribute on every event which contains a description of exactly how the event was discovered.
  • Also, a new discovery_path attribute that contains the complete chain of parent contexts all the way back to the scan target.
• New --json command-line flag: alias for -c modules.stdout.format=json
• New --brief command-line flag for cleaner output
• Importing Scanner is now easier:

from bbot import Scanner

• New dns.brute helper:

async def handle_event(self, event):
    query = self.make_query(event)
    self.info(f"Brute-forcing subdomains for {query} (source: {event.data})")
    for hostname in await self.helpers.dns.brute(self, query, self.subdomain_list):
        await self.emit_event(hostname, "DNS_NAME", source=event)

• New request_batch helper:

async for url, response in self.helpers.request_batch(urls):
    ...

• New async regex helper (you only need to use this if your regex is slow or you're searching a lot of text, i.e. a web response):

r = self.helpers.re.compile('\d+')
matches = await self.helpers.re.findall(r, text)

Breaking Changes

• --current-config is now --current-preset
• --help-all has gone away. To list module options, you now do --list-module-options
• httpx module has been replaced with http
• Massdns threads are now a global option:

# before
bbot -c modules.massdns.max_resolvers=5000

# after
bbot -c dns.brute_threads=5000

• DNS, web, and scope settings are now nested:

scope:
  report_distance: 0
  search_distance: 0
  dns_distance: 1

dns:
  resolution: true
  resolve_threads: 25
  max_resolve_distance: 5
  brute_threads: 1000

web:
  user_agent: "BBOT"
  spider_distance: 0
  spider_depth: 0

• All module configs are now directly under modules (output_modules and internal_modules have gone away):

# before
bbot -c output_modules.splunk.url=http://example.com

# after
bbot -c modules.splunk.url=http://example.com

• Event source attribute has been renamed to parent:

Feature Checklist:

• Presets #1058
  • Presets - add yaml parsing feedback / validation #1269
• DNS Engine - Offload DNS to Dedicated Process #1231
• Better error handling in multiprocess pool #1297
• Subdomain enum bug w/ multiple queries:
• Parallelize excavate regexes (Excavate / Ajaxpro: Slow Regexes #1252)
  • Write tests for regex helpers
  • Aho-Corasick Algorithm (Yara?)
• Only tag DNS events with DNS-related tags (cname-record, cname-wildcard, etc.)
  • Include DNS children in json output
• Offload HTTP work into dedicated process #1223
  • Axe httpx
• Perf optimization: don't create dummy events in target.py
• Event Discovery Context #1352
• Fix Error in keyboard listen task: 'bool' object has no attribute '_queue'
• Tests for custom target types, e.g. ORG:evilcorp
• Tests for url_extension_blacklist and url_extension_httpx_only
• Don't shuffle output module queues
• Clean up spider-danger / URL_UNVERIFIED situation (URL_UNVERIFIEDs shouldn't be deduped by spider-danger #1208) - most likely will depend on HTTP engine rework / httpx replacement
• Clean up console output to make tool friendlier to newcomers (see https://twitter.com/h4x0r_dz/status/1781752033424367690)
  • Possible interactive / stats-only interface?
• Rename engine processes
• Implement radix tree cloudcheck#15
• Preset - Javascript Secrets #1236
• Preset: cloud bucket enumeration
• Preset cache
• Stdout Output Module #1234
• Intercept Modules #1242
• Shared Ansible Dependencies #1270
• Graduate Agent to Standalone Tool #1195
  • remove agent-specific logic from bbot core
  • move it into its own repo
• Portscanner Rework #913
• Consider using AnyIO #1080
• Massdns Rework #1255
• Benchmark uvloop #942
• Asyncify regexes #1184
• Usability / code readability improvements:
  • Rename event.source to be event.parent. event.source is confusing.
  • Presets: Rename output.ndjson back to .json #1227
• Author, created date for modules
  • Docs table
  • Tests
• Slick "how it works" graphic for README
  • Simple graphic showing the circular flow of events in BBOT:
    • DNS resolution, whitelisting/blacklisting, excavate/speculate, etc.
  • In docs: Detailed graph showing all watched/produced interactions between modules (autogenerate?)
• Make a review pass on documentation
  • Full start-to-finish tutorial on how to write a module (including writing/debugging tests)
• Solve these mysteries (they seemed to start happening after the HTTP engine merge):
  • Caused by unwanted processes spawning?

FAILED bbot/test/test_step_2/module_tests/test_module_generic_ssrf.py::TestGeneric_SSRF::test_module_run - AssertionError: Failed to detect Generic SSRF (XXE)

2024-05-28 05:45:00,577 [INFO] bbot.modules.github_workflows base.py:1270 The logs for emberjs/rfcs/9216396978 have expired and are no longer available (status: 0)
2024-05-28 05:45:01,082 [ERROR] bbot.core.httpengine engine.py:217 Error in EngineServer HTTPEngine.download(('https://api.github.com/repos/emberjs/rfcs/actions/runs/9003778394/logs',), {'he
aders': {}, 'raise_error': True, 'warn': False, 'filename': PosixPath('/root/.bbot/scans/insecure_christian/workflow_logs/emberjs/rfcs/run_9003778394.zip')}): Client error '403 Forbidden' fo
r url 'https://api.github.com/repos/emberjs/rfcs/actions/runs/9003778394/logs'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403
2024-05-28 05:45:01,097 [TRACE] bbot.core.httpengine logger.py:132 Traceback (most recent call last):
  File "/root/bbot/bbot/core/engine.py", line 211, in run_and_return
    result = await command_fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/bbot/bbot/core/helpers/web/engine.py", line 139, in download
    response.raise_for_status()
  File "/root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.11/lib/python3.11/site-packages/httpx/_models.py", line 759, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://api.github.com/repos/emberjs/rfcs/actions/runs/9003778394/logs'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/40

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 89.41284% with 577 lines in your changes are missing coverage. Please review.

Project coverage is 92%. Comparing base (cf7fc00) to head (1691332).
Report is 2 commits behind head on dev.

Files	Patch %	Lines
bbot/core/helpers/dns/engine.py	80%	72 Missing ⚠️
bbot/scanner/scanner.py	77%	59 Missing ⚠️
bbot/core/engine.py	79%	54 Missing ⚠️
bbot/cli.py	68%	50 Missing ⚠️
bbot/scanner/preset/preset.py	92%	43 Missing ⚠️
bbot/core/helpers/web/engine.py	81%	36 Missing ⚠️
bbot/core/helpers/process.py	0%	34 Missing ⚠️
bbot/modules/portscan.py	80%	32 Missing ⚠️
bbot/core/modules.py	88%	24 Missing ⚠️
bbot/modules/base.py	82%	23 Missing ⚠️
... and 40 more

Additional details and impacted files

@@           Coverage Diff           @@
##             dev   #1235     +/-   ##
=======================================
- Coverage     92%     92%     -0%     
=======================================
  Files        321     340     +19     
  Lines      20660   23048   +2388     
=======================================
+ Hits       18987   21150   +2163     
- Misses      1673    1898    +225     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.