Compression possible with Transfer-Encoding

[bwbroersma]

Thanks to:

• internet.nl website test triggering OWASP CRS WAF #1347

I noticed currently the compression is only checked by adding a Accept-Encoding:

Internet.nl/checks/tasks/http_headers.py

Line 733 in a469e4c

put_headers = {"Accept-Encoding": "compress, deflate, exi, gzip, pack200-gzip, x-compress, x-gzip"}

To check if the server will return a Content-Encoding:

Internet.nl/checks/tasks/http_headers.py

Lines 24 to 39 in a469e4c

	class HeaderCheckerContentEncoding:
	"""
	Class for checking the Content-Encoding HTTP header.
	"""
	def __init__(self):
	self.name = "Content-Encoding"
	def check(self, value, results, domain):
	"""
	Check if the header has any value.
	"""
	if value:
	results["http_compression_enabled"] = True

However it's also possible to have compression with Transfer-Encoding (hop-by-hop), see HTTP/1.1 - 7.2. Transfer Codings for Compression - RFC 9112. Note that in HTTP/2, Transfer-Encoding MUST not be present, see HTTP/2 - 8.2.2. Connection-Specific Header Fields - RFC 9113.

The IANA registered values that should be checked for in HTTP/1.1 are: (x-)compress, deflate and (x)-gzip.

BTW 'officially' Content-Encoding shouldn't even be used for on-the-fly compression, but reality is different:

The HTTP syntax error has been fixed in trunk, but the problem motivating this report is a no-win situation no matter how it is "fixed". The only good answer is "don't use mod_deflate" because changing content-encoding on the fly in an inconsistent manner (neither "never" nor "always) makes it impossible for later requests regarding that content (e.g., PUT or conditional GET) to be handled correctly. This is, of course, why performing on-the-fly content-encoding is a stupid idea, and why I added Transfer-Encoding to HTTP as the proper way to do on-the-fly encoding without changing the resource.

See the 2009-02-12 comment by Roy T. Fielding (one of the HTTP authors).

It could be tested by (in case of HTTP/1.1) by also sending the header:

TE: compress, deflate, gzip, x-compress, x-gzip

And then check for any compression value (chunked is okay) in Transport-Encoding, see HTTP/1.1 - 7.4. Negotiating Transfer Codings - RFC 9112.