Double content-type in response seems to affect security.txt outcome

[stitch]

Security.txt seems to be configured on this domain:
https://networking4all.com/.well-known/security.txt
https://www.networking4all.com/.well-known/security.txt

The internet.nl test says:
"Your web server does not offer a security.txt file in the right location, it could not be retrieved, or its content is not syntactically valid." https://internet.nl/site/networking4all.com/2771318/#control-panel-31

What seems to be the issue is that the response of the host contains "Content-Type" twice. The code seems to expect this once and might no be able to understand this header is set twice. Or the charset bit is not used:

< Content-Type: text/plain
< Content-Length: 447
< Last-Modified: Thu, 23 Nov 2023 14:27:04 GMT
< Connection: keep-alive
< ETag: "655f6138-1bf"
< Content-Type: text/plain; charset=utf-8

[bwbroersma]

The tech details are more specific:

Error: Media type in Content-Type header must be 'text/plain'.

Just tested, both (a single) Content-Type with either text/plain or text/plain; charset=utf-8 are okay.
Content-Type should of course be set once, but normally only the first header should be considered, I think.

I wish I would easily be able to replicate it with my nginx magic, but setting a double Content-Type header is not even possible with njs, since only the last one set is used.

[bwbroersma]

Double checked, and double Content-Type seems not to be allowed, so I think the current behavior, which indicates something is wrong with the Content-Type in the tech details, is good enough?

Of course the code could check and warn/inform about invalid headers in general, but that probably is a pretty deep rabbit hole to go into.

---

A double Content-Type seems invalid, since RFC 7231 § 3.1.1.5 defines:

Content-Type = media-type

and RFC 7230 page 24 states:

A sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list [i.e., #(values)] or the header field is a well-known exception (as noted below).

The only well-known is Set-Cookie.

---

How the internet.nl code works:

Internet.nl/checks/tasks/securitytxt.py

Lines 104 to 105 in fca192a

	elif media_type.lower() != "text/plain":
	errors.append({"msgid": "invalid_media"})

where media_type is the result of parse_headers from cgi

Internet.nl/checks/tasks/securitytxt.py

Lines 80 to 81 in fca192a

	if content_type:
	media_type, params = parse_header(content_type)

which is the content-type header from requests:

Internet.nl/checks/tasks/securitytxt.py

Line 60 in fca192a

response.headers.get("Content-Type", ""),

---

To check the response of Requests:

from cgi import parse_header
import requests
r = requests.get('https://networking4all.com/.well-known/security.txt')
parse_header(r.headers['content-type'])
('text/plain, text/plain', {'charset': 'utf-8'})