bug: Global Variable - custom_header does not parse correctly

[Zeetaz]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When trying to run a scan utilzing the global variable "custom_header" it will be parsed differently depending on which tool it is assigned to.

For example, running the following YAML scanner config:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: 'User-Agent: Hello_I_Will_Die_PoC'

fetch_url: {
  'uses_tools': ['katana'],
}

It will break:

celery_1       |   warnings.warn(
celery_1       |
celery_1       | fetch_url                          | WARNING | Task fetch_url is RUNNING
celery_1       | fetch_url                          | ERROR | 'str' object has no attribute 'items'
celery_1       | Traceback (most recent call last):
celery_1       |   File "/usr/src/app/reNgine/celery_custom_task.py", line 129, in __call__
celery_1       |     self.result = self.run(*args, **kwargs)
celery_1       |   File "/usr/src/app/reNgine/tasks.py", line 1812, in fetch_url
celery_1       |     f'{key}: {value}' for key, value in custom_header.items()
celery_1       | AttributeError: 'str' object has no attribute 'items'
celery_1       | fetch_url                          | WARNING | Task fetch_url status is FAILED | Error: AttributeError("'str' object has no attribute 'items'")

If you instead were to run a vulnerability scan instead for fetch_url it will function correctly for nuclei.

It is possible to make this work as a temporary solution:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: {
  '"User-Agent': 'Hello_Working_PoC"'
}

fetch_url: {
  'uses_tools': ['katana'],
}

But this reuqires you to run two separate scans for vulnerability scans and fetching URL's... I am also not sure if it will break other tools or if it is only katana that is currently broken an expects "items" instead of strings.

Expected Behavior

The expected behavior, for most tools and for katan, would be to run the configuration as:

custom_header = 'User-Agent: testing'
etc, etc.

And it should be appended to the tool (katan in this case) as

-H "User-Agent: testing"

Steps To Reproduce

1. Run the following scanner configuration on any target:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: 'User-Agent: Hello_I_Will_Die_PoC'

fetch_url: {
 'uses_tools': ['katana'],
}

2. Note that the scan fails as it does not expect a str value.

3. Run the following - modified - configuration:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: {
  '"User-Agent': 'Hello_Working_PoC"'
}

fetch_url: {
  'uses_tools': ['katana'],
}

4. Note that this will work, however it does not follow the expected syntax. This is also an extremely bad work-around as it will not work with nuclei or any other tools as they won't parse it the same.

(it will append the header to katana as -H "User-Agent: Hello_Working_PoC")

Environment

- reNgine: 2.0.3

Anything else?

No response

[github-actions]

👋 Hi @Zeetaz,
Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki
For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE
Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

[Zeetaz]

May also add that it appears as if custom_header: 'User-Agent: hello' will only work for nuclei as of right now.
Dalfox and katana for example will run it just as -H User-Agent: hello without any quotes.

[oz9un]

this issue still exists unfortunately, having the same problem...