[FEATURE REQUEST] Optionally ignore directory listings

[libklein]

Is your feature request related to a problem? Please describe.
Feroxbuster always stops exploring a directory when a directory listing is detected. I think that this behavior is desired in the common case, but may fail to detect files in special cases, i.e., those ignored by the directory listing (see https://serverfault.com/questions/345542/force-apache-directory-listing-even-if-directoryindex-files-are-present). I believe that a tool like feroxbuster should cover such cases - detecting hidden files is it's primary purpose after all.

Describe the solution you'd like
A flag or config option to force feroxbuster to ignore directory listings, i.e., recurse into directories which have a (potentially empty) directory listing.

Additional context
This issue occured during the latest stream from 0xTib3rius, specifically when he tackled THM Box Valley.

I'm happy to provide a PR if the maintainers agree with this request.

[Tib3rius]

Adding my support to this. I was going to make my own request but this pretty much covers everything I talked about on stream. I think the initial reason for skipping over directory listings is sound, but as has been pointed out, you can configure directory listings to only show certain files or that match a certain pattern.

For that reason I think feroxbuster needs an option to override the default behavior at the very least.

[epi052]

hey y'all, sorry for the late reply. I recently moved across the country and have been focused on real life things.

first, i have no problem with this being incorporated one way or another, thank you both for finding the issue and then making the suggestion!

some random thoughts:

• should the default just be flipped? up for debate imo
• if it's an option, it can be tied to the --thorough meta-flag as well
• if it's an option, and it's not active on a run where a dir listing is found, let's display the fact that the flag exists and could be used to (potentially) delve deeper
• A flag or config option to force... in my experience, if we make something a setting that's only available in the toml, i'll eventually see an issue raised to add it as a cli option, better to just go whole hog from jump

here's the checklist i use when i add a new command line option, in case we go that route

---

• Add new command line option checklist
  • config/container.rs
    • add value to Configuration struct
    • add value to Configuration::default
    • update Configuration docstring
    • add value to Configuration::parse_cli_args
    • add value to Configuration::merge_config
  • config/tests.rs
    • update Configuration::tests::setup_config_test with new value
    • update Configuration::tests::default_configuration with new value
    • add test to check config parsing
  • ferox-config.toml.example
    • add new value
  • banner/container.rs
    • add new BannerEntry to Banner struct
    • update Banner::new with a BannerEntry
    • update print_to with whether to print or not
  • tests/test_banner.rs
    • add new test so that the new BannerEntry is displayed and examined under test
  • Cargo.toml
    • bump version - minor release +1 - patch release = 0
  • README.md
    • update default config section (if needed)
    • update example config file section
    • update help output section
    • consider adding to an example (new or existing)
    • update comparison table
  • parser.rs
    • add the option
  • scan_manager/tests.rs
    • add serialized version of new option to the feroxstates_feroxserialize_implementation test
  • wherever.rs
    • implement the new feature

[Tib3rius]

There's definitely an argument for flipping the default, I just have no idea how common it is to mask files from appearing. It probably makes more sense to keep things the way they are, since ultimately it does save time (especially if recursive scans are enabled and there are several directory listings found).

Agreed with making it an actual command line option, and definitely agree with the warning / notice to let users know about it.

[libklein]

I would not change the default behaviour. First, I think that this is a case seldom observed in "the real world". Second, changing default behaviour is very intrusive for users of the software. I'd add a flag to explicitly enable emit, emit a warning if eagerly aborting scans due to directory listings. I agree that it should be enabled with the --thorough flag as well. Finally, I'd decouple the flag from --extract-links. I think an argument could be made to change the flag's behaviour depending on whether links should be extracted or not, but I'd refrain from doing this as it is, in my experience, not worth the confusion caused by this. Most people do not read the documentation ;).

[epi052]

Most people do not read the documentation

preach!

That all sounds great to me.

@libklein please let me know if you have any questions while making changes

thank you both again!