New flags for verbosity tweaking + default minimized look

[caribpa]

Hi,

First I want to thank the ffuf team for the amazing tool! 🎉 It is my go-to for the 99% of web-fuzzing cases, I'll create an issue later explaining what I'm missing ❤️

This PR adds some flags for tweaking the verbosity as well as minimizes the default look 🔹

Basically, my terminal is 90 columns and I have issues with the request/progress status information wrapping (I am aware of #278 btw), but at the same time I want to see some relevant information like where the redirections would go without having to toggle full verbosity (-v).

In a nutshell, this is the look before this PR (latest dev build 3c78f89):

$ ffuf -u https://github.com/ffuf/ffuf/FUZZ -w <(echo '/')
[banner]
________________________________________________

/                       [Status: 200, Size: 229209, Words: 29546, Lines: 2575, Duration: 48ms]
:: Progress: [1/1] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

This is after this PR:

$ ffuf -u https://github.com/ffuf/ffuf/FUZZ -w <(echo '/')
[banner]
________________________________________________

/                                               [C:200 - S:229451 - W:29609 - L:2581 - T:95ms]
:: P[1/1] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

You can recover the old verbosity, with minor changes in the Request status (Status => Code; Duration => Time), using the new -vs (verbose status) flag:

$ ffuf -u https://github.com/ffuf/ffuf/FUZZ -w <(echo '/') -vs
[banner]
________________________________________________

/                             [Code: 200, Size: 229210, Words: 29546, Lines: 2575, Time: 48ms]
:: Progress: [1/1] :: Job [1/1] :: 36 req/sec :: Duration: [0:00:01] :: Errors: 0 ::

Now the whole reason I did this: the new -vr (verbose redirect) flag:

$ ffuf -u https://github.com/ffuf/ffuf/wiki/FUZZ -w <(echo '/') -vr
[banner]
________________________________________________

/              --> /ffuf/ffuf/wiki/             [C:302 - S:100 - W:5 - L:1 - T:76ms]
:: P[1/1] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

If it is too long to fit in the same line it gets smartly printed on a new line:

$ ffuf -u https://github.com/ffuf/ffuf/wiki/FUZZ -w <(echo -e '/\n//////////////') -vr
[banner]
________________________________________________

/              --> /ffuf/ffuf/wiki/             [C:302 - S:100 - W:5 - L:1 - T:50ms]
//////////////                                  [C:302 - S:113 - W:5 - L:1 - T:52ms]
               --> /ffuf/ffuf/wiki//////////////
:: P[2/2] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

Note that the arrow (-->) in the new line tries to be aligned with the expected position, but it will give up alignment to avoid wrapping into another line:

# Normal redirection printed in a new line:
$ ffuf -u https://github.com/ffuf/ffuf/wiki/FUZZ -w <(echo '//////////////') -vr
[banner]
________________________________________________

//////////////                                  [C:302 - S:113 - W:5 - L:1 - T:52ms]
               --> /ffuf/ffuf/wiki//////////////
:: P[1/1] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

# The following redirection doesn't fit, so the alignment of the arrow is given up:
$ ffuf -u https://goo.glFUZZ -w <(echo '/') -vr
[banner]
________________________________________________

/                                               [C:301 - S:0 - W:1 - L:1 - T:124ms]
     --> https://developers.googleblog.com/2018/03/transitioning-google-url-shortener.html
:: P[1/1] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

This functionality assumes a 90 column terminal and it is hard-coded for now (there's a TODO in the code) as this will make sense to implement once #278 gets merged to reuse the terminal size auto-detector.

If you noticed, the redirections that happen in the same domain (well, schema+domain) will get trimmed to save space but they can be displayed with the -vu (verbose URLs) flag:

$ ffuf -u https://github.com/ffuf/ffuf/wiki/FUZZ -w <(echo -e '/\n////////////') -vr -vu
[banner]
________________________________________________

/        --> https://github.com/ffuf/ffuf/wiki/ [C:302 - S:100 - W:5 - L:1 - T:176ms]
////////////                                    [C:302 - S:111 - W:5 - L:1 - T:171ms]
               --> https://github.com/ffuf/ffuf/wiki////////////
:: P[2/2] :: J[1/1] :: 2 r/s :: D[0:00:15] :: E:0 ::

Note that the first arrow (-->) is not aligned to avoid displacing the Request status bar on the right.

Also, the behavior of toggling verbose mode if more than one input is found no longer happens:

$ ffuf -u https://github.COM/ffuf/ffuf/wiki/FUZZ -w <(echo 'com'):COM,<(echo -e '/\n/////') -vr
[banner]
________________________________________________

{COM: com}{FUZZ: /} --> /ffuf/ffuf/wiki/        [C:302 - S:100 - W:5 - L:1 - T:51ms]
{COM: com}{FUZZ: /////}                         [C:302 - S:104 - W:5 - L:1 - T:51ms]
               --> /ffuf/ffuf/wiki/////
:: P[2/2] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

Look how the first arrow (-->) is not aligned due to having been displaced by the inputs but it still moves to a newline (second result) if the total length would displace the Request status bar on the right.

Of course, the -vi (verbose input) flag was added to recover the old behavior (multi-line output) without toggling full verbosity as well:

$ ffuf -u https://github.COM/ffuf/ffuf/wiki/FUZZ -w <(echo 'com'):COM,<(echo -e '/\n/////') -vi
[banner]
________________________________________________

[C:302 - S:100 - W:5 - L:1 - T:170ms]
| URI | /ffuf/ffuf/wiki//
    * COM: com
    * FUZZ: /

[C:302 - S:104 - W:5 - L:1 - T:188ms]
| URI | /ffuf/ffuf/wiki//////
    * COM: com
    * FUZZ: /////

:: P[2/2] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

It shall be noted that the multi-line output will be automatically enabled (though in reduced verbosity) if the -od (output directory) flag is provided (this is the old behavior) as I didn't find a satisfactory way of displaying the name of the file with the saved request in the single-line output mode:

$ ffuf -u https://github.com/ffuf/ffuf/wiki/FUZZ -w <(echo -e '/\n/////') -vr -od /tmp
[banner]
________________________________________________

[C:302 - S:104 - W:5 - L:1 - T:170ms]
| URI | /ffuf/ffuf/wiki//////
| --> | /ffuf/ffuf/wiki/////
| RES | 531e72245ea26b09749725eb58dee0f8

[C:302 - S:100 - W:5 - L:1 - T:2126ms]
| URI | /ffuf/ffuf/wiki//
| --> | /ffuf/ffuf/wiki/
| RES | 239c1582fffe7e08fa2d31ffb3a8897b

:: P[2/2] :: J[1/1] :: 26 r/s :: D[0:00:02] :: E:0 ::

Silent mode (-s) can also display the redirections if the -vr flag is provided (the smart arrow-alignment/line-wrap doesn't happen in this mode):

$ ffuf -u https://github.COM/ffuf/ffuf/wiki/FUZZ -w <(echo 'com'):COM,<(echo -e '/\n////\n/////') -vr -s
{COM: com}{FUZZ: ////} --> /ffuf/ffuf/wiki////
{COM: com}{FUZZ: /} --> /ffuf/ffuf/wiki/
{COM: com}{FUZZ: /////} --> /ffuf/ffuf/wiki/////

I am not married to the {} around the inputs but I chose them to distinguish them from the status bars ([]) for scripting purposes.

And -v (verbose) toggles all verbose flags to display the same as in the old behavior:

$ ffuf -u https://github.com/ffuf/ffuf/wiki/FUZZ -w <(echo -e '/\n////') -v
[banner]
________________________________________________

[Code: 302, Size: 100, Words: 5, Lines: 1, Time: 58ms]
| URL | https://github.com/ffuf/ffuf/wiki//
| --> | https://github.com/ffuf/ffuf/wiki/
    * FUZZ: /

[Code: 302, Size: 103, Words: 5, Lines: 1, Time: 59ms]
| URL | https://github.com/ffuf/ffuf/wiki/////
| --> | https://github.com/ffuf/ffuf/wiki////
    * FUZZ: ////

:: Progress: [2/2] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

Pheww, that took longer to write than expected 😅!

Hope you find this PR interesting, feel free to point out any improvements/bugs/fixes, etc 🥳

[joohoi]

Thanks for the PR and sorry it took so long to get around to reviewing it.

In general I like the way this PR goes.

I'm listing my thoughts about the changes below, here are the first things (I didn't go through the full list of changes yet, but thought I'd post this right away instead of collecting a full list and posting it up later). I'll continue on the review and my own experimentation around the TUI later on. I want to underline that these are not absolute truths in any way or form, just my personal opinion that I'm posting up here for discussion starters.

Progress bar

I'd probably go even more minimal than what was suggested, some of the things there are self-explanatory (like the ticking duration timer and the incrementing progress counter), while others (job for example) isn't intuitive enough for people to benefit from the prefix, so instead of:

:: P[1/1] :: J[1/1] :: 0 r/s :: D[0:00:00] :: E:0 ::

I'd suggest format:

:: [125/12301203] : 2/6 :: 756 r/s :: [0:12:00] :: E:123 ::

(note the single delimiter between progress and job counter, as they are somewhat related)

Result status lines

I would prefer a bit different format for this, just as a personal preference as I think it looks cleaner.

So instead of suggested:

[C:200 - S:229451 - W:29609 - L:2581 - T:95ms]

I'd prefer

[ 200 | S:229451 | W:29609 | L:2581 | 95ms ]

as I think it looks cleaner, especially when there are multiple rows displayed. In addition; I don't think the first (status code) value needs a "title" as it's self-explanatory, (and in many cases colored with -c for better readability!) and the same goes with the last data point - the delay, as it has the postfix for time unit.

Multi-line redirect alignment

I would prefer the "starting point" for the redirection target to be static, as it's visually easier to spot small anomalies this way, and at least easier for me to look at. As an example:

//////////////                                  [C:302 - S:113 - W:5 - L:1 - T:52ms]
               --> /some/path
///////                                         [C:302 - S:113 - W:5 - L:1 - T:52ms]
        --> /other/path
/////////                                       [C:302 - S:113 - W:5 - L:1 - T:52ms]
          --> /some/path

vs.

//////////////                                  [C:302 - S:113 - W:5 - L:1 - T:52ms]
  --> /some/path
///////                                         [C:302 - S:113 - W:5 - L:1 - T:52ms]
  --> /other/path
/////////                                       [C:302 - S:113 - W:5 - L:1 - T:52ms]
  --> /some/path

Redirect target trimming

While I agree that in many cases it might make sense in terms of fitting to the screen to trim out the redirect target schama + domain, I'm slightly drawn towards printing the whole actual target personally to reflect the actual content of Location - header. Also, (i didn't look at the implementation yet) in case trimming, the port needs to be taken into account as well.

[caribpa]

Hey, no worries, I've also been really busy! 😁

Yes, I agree with your thoughts about the status and progress bar. I even wanted to minimize the UI even more but I wasn't sure if you wanted to keep some of the obvious titles to make extracting info from ffuf output easier.

And yes, the inconsistent redirect formatting I proposed is not user-friend at all... I've been using ffuf with my changes since last year and this inconsistency is really annoying 😅

Regarding the redirect target trimming I find it somewhat useful, especially when the target domain is long and the URL path as well:

$ ffuf -u https://this.is.a.really.long/target/path/which/will/probably/wrap/the/redirection/FUZZ -w <dict> -vr
[banner]
________________________________________________

/                                               [C:301 - S:0 - W:1 - L:1 - T:101ms]
     --> https://this.is.a.really.long/target/path/which/will/probably/wrap/the/redirection/

[...]

Nevertheless, I agree that it could be better as it doesn't help much when the path is really long.

Ideally I'd prefer it to be to be a bit smarter, something like this:

$ ffuf -u https://this.is.a.really.long/target/path/which/will/probably/wrap/the/redirection/FUZZ -w <dict> -vr
[banner]
________________________________________________

/                                               [C:301 - S:0 - W:1 - L:1 - T:101ms]
     --> /
/test                                           [C:301 - S:0 - W:1 - L:1 - T:104ms]
     --> ../test
/admin                                          [C:301 - S:0 - W:1 - L:1 - T:104ms]
     --> https://this.is.a.really.long/admin
[...]

That said, I have no problem with showing the full Location by default (-vr) and having another flag (-vrr), or flags (one for trimming only schema+domain, other for trimming all until FUZZ keywords, etc), for showing it trimmed (or the other way around) 🙂

Regarding your concern about whether the code doing the trimming is taking into account the port or not, the answer is yes, it does. It basically trims everything up to the first slash after the schema://. You could even have a HTTP Basic Auth and a port in the URL (http://user:pass@domain:port/path) and it will get trimmed without issues.