How does response processing work?

[atn1990]

I wanted to start using response processing on my end and currently understand how it works on the request side by using rqst_ctx_callbacks but is there something equivalent for the response? Are there examples in tests or documentation you can point me to so I can get started?

[tinselcity]

There's a test server you can play with and inspect the code for, that gets built with the library called waflz_server. There's a quick start guide in the README.md about how to get started playing with the server: here. There's some examples here of running and testing with the server (even works in a proxy mode). I tested the example just now and it appears to still work fine. The server itself is a pretty good example of how to write the rqst_ctx_callbacks for a server (in this case using an embedded http(s) server library is2

The callbacks are defined/located here. There's some info in the architectural documentation about the how and the why for the callbacks as well here. The processing of either the ACL of the WAF profile can happen separately or together with a an entity called scopes which stitches together ACL/WAF/RL configuration based on some url path specification.

But if you'd just like to test an Access Control List (ACL) by itself with waflz_server here's a basic example:

• Run waflz_server with an acl json

~/gproj/waflz/build>jq '.' ./bad_file_ext.acl.json 
{
  "disallowed_extensions": [
    ".xxx"
  ]
}

• Running (listens on port 12345 by default):

rmorrison@pcu~/gproj/waflz/build>./util/waflz_server/waflz_server -a ./bad_file_ext.acl.json 
...

• curl in another term

>curl -s localhost:12345/index.xxx | jq '.'
{
  "rule_msg": "File extension is not allowed by policy",
  "sub_event": [
    {
      "rule_id": 80005,
      "rule_msg": "File extension is not allowed by policy",
      "rule_target": [
        {
          "name": "RklMRV9FWFQ=",
          "param": "disallowed_extensions"
        }
      ],
      "rule_op_name": "",
      "rule_op_param": "",
      "rule_tag": [
        "HTTP POLICY"
      ],
      "matched_var": {
        "name": "RklMRV9FWFQ=",
        "value": "Lnh4eA=="
      }
    }
  ],
  "acl_config_id": "",
  "acl_config_name": "",
  "config_last_modified": ""
}

waflz_server is pretty flexible and supports running in different modes and with a lot of different configuration. See waflz_server --help for example:

Usage: waflz_server [options]
Options:
  -h, --help          display this help and exit.
  -v, --version       display the version number and exit.
  
Config Modes: -specify only one
  -f, --profile       waf profile
  -a, --acl           access control list (acl)
  -e, --rules         rules
  -m, --modsecurity   modsecurity rules
  -l, --limit         limit.
  -b, --scopes        scopes (file or directory)
  
Engine Configuration:
  -r, --ruleset-dir   waf ruleset directory
  -g, --geoip-db      geoip-db
  -s, --geoip-isp-db  geoip-isp-db
  -d  --config-dir    configuration directory (REQUIRED for scopes)
  -x, --random-ips    randomly generate ips
  -c, --challenge     json containing browser challenges
  -n, --bot-js        js to insert in custom browser challenges
  
KV DB Configuration:
  -L, --lmdb          lmdb for rl counting
  -I, --interprocess  lmdb across multiple process (if --lmdb)
  
Server Configuration:
  -p, --port          port (default: 12345)
  -j, --action        apply actions instead of reporting
  -z, --bg            load configs in background thread
  -o, --output        write json alerts to file
  
Server Mode: choose one or none
  -w, --static        static file path (for serving)
  -y, --proxy         run server in proxy mode
  
Debug Options:
  -t, --trace         tracing (error/rule/match/all)
  -T, --server-trace  server tracing  (error/warn/debug/verbose/all)
  -A, --audit-mode    load and exit

Let me know if you need any more info.
Good luck!

[tinselcity]

Ah! Yeah good point.

The open sourcewaflz library does not support response processing. I think the big use cases for response processing were things like data leakage and any scrubbing a site might want a waf doing prior to serving a response. The response processing rules weren't popular in the OWASP CRS and they've even gone so far as to drop capabilities like CC# checking with Luhn Algorithm (a capability that existed in earlier iterations of modsecurity but were dropped due to ineffectiveness or lack of general usage).

For the current uses of response processing in the OWASP CRS, see the *-LEAKAGES.* .conf files: https://github.com/coreruleset/coreruleset/tree/v4.0/dev/rules

Looks like one other use-case covered by outbound or response processing is Web Shells

Generally speaking when we were designing waflz it wasn't initially a big customer use-case, so it was omitted. I think the Edgio security product (proprietary closed source) has moved on quite a bit and does support it and a lot more, but for waflz which was intended to a be simple server agnostic waf library, it was left out.

[atn1990]

I found this branch add_response_callbacks with some code that I might want to use and I was wondering what shape that was in. Does it do what I want and if not how much work do I have to do to get it there? I opened a pull request #310 to bring it up to the latest master.

[atn1990]

I compiled our code with #311 and rebuilt it but I ran into some build issues that might be from your side:

build: /sd/workspace/src/screwdriver.cd/sd-local/local-build/waflz/src/core/resp_ctx.cc: In member function ‘int32_t ns_waflz::resp_ctx::init_phase_4()’:
build: /sd/workspace/src/screwdriver.cd/sd-local/local-build/waflz/src/core/resp_ctx.cc:356:33: error: ‘ULONG_MAX’ was not declared in this scope
build:          if (m_content_length == ULONG_MAX)
build:                                  ^~~~~~~~~
build: /sd/workspace/src/screwdriver.cd/sd-local/local-build/waflz/src/core/resp_ctx.cc:356:33: note: ‘ULONG_MAX’ is defined in header ‘<climits>’; did you forget to ‘#include <climits>’?
build: /sd/workspace/src/screwdriver.cd/sd-local/local-build/waflz/src/core/resp_ctx.cc:23:1:
build: +#include <climits>
build:  #include <string.h>
build: /sd/workspace/src/screwdriver.cd/sd-local/local-build/waflz/src/core/resp_ctx.cc:356:33:
build:          if (m_content_length == ULONG_MAX)
build:                                  ^~~~~~~~~
build: make[2]: *** [src/CMakeFiles/waflz_obj.dir/core/resp_ctx.cc.o] Error 1
build: make[2]: *** Waiting for unfinished jobs....
build: make[1]: *** [src/CMakeFiles/waflz_obj.dir/all] Error 2
build: make: *** [all] Error 2

Are there any tests of this new code to ensure it will work for us?

[atn1990]

I opened up #312 to remedy the issue.

[atn1990]

Thank you for accepting my PR for fixing that compilation issue and now our code works with no issues and all tests pass with the new code.

I wanted to follow up on the PR #311 and ask about how to use this new functionality. I was told by my colleague that your team informed them on how to use the rqst_ctx_callbacks structure and which callbacks could be left unimplemented. Can you let me know the same for resp_ctx_callbacks if all are actually needed to be implemented or not?

Along those same lines, what is the equivalent of acl or profile process function call that uses resp_ctx_callbacks but for the response? The usage of it isn't as clear from the tests added in the PR #311 and some more pointers would be very appreciated.