[FEATURE] Add audit logging functionality

[denandz]

This PR adds an audit log which writes every request and response, along with the config, to a JSON lines formatted log file (https://jsonlines.org/). The audit log is not affected by filter logic, and can be used to log everything sent/received by FFUF regardless of what the user specifies as a matcher/filter.

The JSON lines format was chosen so requests/responses can be written as they're sent/received. This avoids memory pressure and ensures immediate audit logging that isn't lost if FFUF crashes.

The audit logging feature has the following use cases:

• Audit logging when attacking sensitive infrastructure. Allows us to answer the question of "what requests exactly did you send when this service crashed", which prior to this pull request would have required piping all fuzz runs through a separate proxy for logging.
• Hunting for one-time discrepancies, such as finding hidden functions/header-values in JIT compiled languages like dotnet by analysing subtle timing discrepancies associated with JIT compilation across multiple runs. A messed up matcher/filter requiring a re-run of a fuzzing session would miss this.
• Performing statistical analysis and "machine-learning" based analysis on fuzz run output after-the-fact.

The audit logging output also enables future functionality where the audit log can be re-read by FFUF. Such as replaying the audit log to refine matcher/filters without sending all the fuzz-run requests again, and sending requests to replay proxies after the fact. This is going to require some more development and is still on my to-do list.

Fixes: #759

Example of the output below:

$ ./ffuf -w wlist -u http://127.0.0.1:8000/FUZZ -audit-log foo.json 

        /'___\  /'___\           /'___\       
       /\ \__/ /\ \__/  __  __  /\ \__/       
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\      
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/      
         \ \_\   \ \_\  \ \____/  \ \_\       
          \/_/    \/_/   \/___/    \/_/       

       v2.1.0-dev
________________________________________________

 :: Method           : GET
 :: URL              : http://127.0.0.1:8000/FUZZ
 :: Wordlist         : FUZZ: /home/doi/go/src/github.com/ffuf/ffuf/wlist
 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 10
 :: Threads          : 40
 :: Matcher          : Response status: 200-299,301,302,307,401,403,405,500
________________________________________________

:: Progress: [4/4] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

The filter didn't match on any responses; however, the foo.json audit log has complete requests/responses and config:

$ jq . < foo.json 
{
  "Type": "*ffuf.Config",
  "Data": {
    "auditlog": "foo.json",
    "autocalibration": false,
    "autocalibration_keyword": "FUZZ",
    "autocalibration_perhost": false,
    "autocalibration_strategies": [
      "basic"
    ],
    "autocalibration_strings": [],
    "colors": false,
    "cmdline": "./ffuf -w wlist -u http://127.0.0.1:8000/FUZZ -audit-log foo.json",
    "configfile": "",
    "postdata": "",
    "debuglog": "",
    "delay": {
      "value": "0.00"
    },
    "dirsearch_compatibility": false,
    "encoders": [],
    "extensions": [],
    "fmode": "or",
    "follow_redirects": false,
    "headers": {},
    "ignorebody": false,
    "ignore_wordlist_comments": false,
    "inputmode": "clusterbomb",
    "cmd_inputnum": 100,
    "inputproviders": [
      {
        "name": "wordlist",
        "keyword": "FUZZ",
        "value": "/home/doi/go/src/github.com/ffuf/ffuf/wlist",
        "encoders": "",
        "template": ""
      }
    ],
    "inputshell": "",
    "json": false,
    "matchers": null,
    "mmode": "or",
    "maxtime": 0,
    "maxtime_job": 0,
    "method": "GET",
    "noninteractive": false,
    "outputdirectory": "",
    "outputfile": "",
    "outputformat": "",
    "OutputSkipEmptyFile": false,
    "proxyurl": "",
    "quiet": false,
    "rate": 0,
    "raw": false,
    "recursion": false,
    "recursion_depth": 0,
    "recursion_strategy": "default",
    "replayproxyurl": "",
    "requestfile": "",
    "requestproto": "https",
    "scraperfile": "",
    "scrapers": "all",
    "sni": "",
    "stop_403": false,
    "stop_all": false,
    "stop_errors": false,
    "threads": 40,
    "timeout": 10,
    "url": "http://127.0.0.1:8000/FUZZ",
    "verbose": false,
    "wordlists": [
      "/home/doi/go/src/github.com/ffuf/ffuf/wlist"
    ],
    "http2": false,
    "client-cert": "",
    "client-key": ""
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
    "Host": "",
    "Url": "http://127.0.0.1:8000/asdva",
    "Headers": {},
    "Data": "",
    "Input": {
      "FFUFHASH": "MmJmMDgx",
      "FUZZ": "YXNkdmE="
    },
    "Position": 1,
    "Raw": "",
    "Timestamp": "2024-03-21T16:43:21.895628032+13:00"
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
   ...yoink...
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
...yoink...
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
    ...yoink...
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
    "Headers": {
      "Connection": [
        "close"
      ],
      "Content-Length": [
        "469"
      ],
      "Content-Type": [
        "text/html;charset=utf-8"
      ],
      "Date": [
        "Thu, 21 Mar 2024 03:43:21 GMT"
      ],
      "Server": [
        "SimpleHTTP/0.6 Python/3.9.2"
      ]
    },
    "Data": "PCFET...yoink...Cg==",
    "ContentLength": 469,
    "ContentWords": 96,
    "ContentLines": 15,
    "ContentType": "text/html;charset=utf-8",
    "Cancelled": false,
    "Request": {
      "Method": "GET",
      "Host": "127.0.0.1:8000",
      "Url": "http://127.0.0.1:8000/asdva",
      "Headers": {
        "User-Agent": "Fuzz Faster U Fool v2.1.0-dev"
      },
      "Data": "",
      "Input": {
        "FFUFHASH": "MmJmMDgx",
        "FUZZ": "YXNkdmE="
      },
      "Position": 1,
      "Raw": "",
      "Timestamp": "2024-03-21T16:43:21.897260281+13:00"
    },
    "Raw": "",
    "ResultFile": "",
    "ScraperData": {},
    "Duration": 3196397,
    "Timestamp": "2024-03-21T16:43:21.900456678+13:00"
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
 ...yoink...
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
 ...yoink...
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
...yoink...
  }
}

[denandz]

Yo @joohoi anything else you need to get this merged? There is some future functionality around statistical analysis of response data I'd like to work on but it's relying on this feature getting merged.

[joohoi]

Hi,
in general this looks good!

In order to reflect the reality better, there's couple of potential improvements to be made, lmk if you agree.

I think the spot where the request is written to audit log should be changed. ffuf adds for example User-Agent header later in the Execute function of the runner and that isn't caught by the current behavior.

Also, it would be good to save the raw requests (Request struct has a spot for the raw data, but it's currently only written if -od if defined), that behavior can be changed in following spots:

Request:
https://github.com/ffuf/ffuf/blob/master/pkg/runner/simple.go#L149-L151

Response:
https://github.com/ffuf/ffuf/blob/master/pkg/runner/simple.go#L171-L175

[denandz]

Oh nice! Good catch with the request data. I'll get those changes rolled in

[denandz]

Have tweaked the logic so we're logging the request object after any modifications are made by the runner. Have also tweaked the raw request/response logic to log that data.

How set are you on storing the raw request/response as a string? I can see the json getting messy when working with binary data in requests and responses. Might be an idea to move it to a byte[] and then it'll serialize out as base64.