Inefficient Targets file handling

[rsrdesarrollo]

Describe the bug
If you run nuclei with -l option (Targets) with a really big file, nuclei will try to load the full file in RAM, failing if that's not possible.

A better approach, instead of loading all the file, load bunch of lines from the file and start runner for that set of lines.

Also, if instead of a file you use process substitution with something like this:

nuclei -l <(for p in {1..999}; do curl "http://really.slow.web/get_some_targets?page=$p"; done) -t cves

Nuclei wont start the scan untill all request to the really.slow.web service are done, loosing a loot of time.

Nuclei version
v2.2.0

Screenshot of the error or bug

nuclei/v2/internal/runner/runner.go

Line 136 in 31e9c04

if options.Targets != "" {

[rsrdesarrollo]

fixing this might cause nuclei unable detect duplicates or, at least, being dup detection an optional (not mandatory) feature.

[ehsandeep]

@rsrdesarrollo thank you for opening an issue for this, yes as you pointed out, there are couples of important features including, dedup, stats calculations, and few more features depends on the knowledge of the input data.

do you have an example or numbers of the target you supplied to experience this fail or out of memory error?

[rsrdesarrollo]

Hi, my case right now is not about RAM allocation, but about time efficiency in the case of getting target URLs from a slow remote web-server (something similar to the example of process substitution).

But theoretically, in a VPS with 1GB of RAM, a file of 1GB will cause serious problems. Supposing that the medium URL size is about 64 bytes that will be like 16 million URLs. Which, when said out loud, sounds like a lot of :).

But, as I said my main concern is about making the most of the time spent on charging resources from a remote end-point.

[Mzack9999]

The file is never loaded in RAM as we developed https://github.com/projectdiscovery/hmap for that purpose (it works as a disk-based key-value store). Are you still able to reproduce this with the latest version? If yes, can you tell us how long the list was and the configuration of your VPS (e.g., RAM, disk size, OS)? Thanks!