Thanks for the report, Miro!

It looks like I attempted to replace cgi in 6b7c2cd exactly a year ago and then, reverted it with 7338b83 in a few days and suppressed the deprecation warning in a follow-up (97ae5b7).

I don't remember the exact reason, but it probably made the CI fail somehow and I didn't have an immediate solution.

It seems like the change needs to be re-introduced with some edits and the deprecation warning ignore reverted. This probably requires some investigation.

I tried reintroducing the change and a bunch of tests failed. I'll try and find some time to help look at this.

The original patch ran into the difference in fundamental use of email.message.EmailMessage and cgi.parse_headers.

The email.message module is trying to create a higher level object that is intelligent about the headers it has defined. The cgi.parse_header function is a very low level unintelligent function that simply parses header content based on structure and does not inspect the contents.

Here's the cgi.parse_header code: https://github.com/python/cpython/blob/3.12/Lib/cgi.py#L226C1-L256C22
Here's the email.message.EmailMessage.get_content_type() code: https://github.com/python/cpython/blob/3.12/Lib/email/message.py#L596-L618

Because of the intelligence of the email.message object cherrypy can't use it in the very generic way that the cgi.parse_header function was being used. The email module is forcing text/plain as the return value because a slash is not part of the value and the suggested replacement funnels the value through logic that assumes a Content-Type header and forces the presence of a slash as opposed to cgi.parse_header which treats the value as any generic header.

So in the case that the value passed to the email module is ISO-8859-1, 0.7, utf-8, *, or gzip as test_tools.py::ToolTests.testCombinedTools tests, the content type will always be text/plain and therefore the return value always text/plain the way the suggested replacement is written.

I have looked though the email module and http modules. The closest thing I have found so far to what cgi.parse_header was doing is the email.message._splitparm function.

email.message._splitparam('application/json; charset="utf8"')
('application/json', 'charset="utf8"')
email.message._splitparam('gzip')
('gzip', None)
email.message._splitparam('0.7')
('0.7', None)

Even then, there's more massaging to do compared to the cgi module:

cgi.parse_header('application/json; charset="utf8"')
('application/json', {'charset': 'utf8'})
cgi.parse_header('gzip')
('gzip', {})

The email.message._parseparam() function seems to split multiple params into a list of k=v pairs.

So, using a combination of _parseparam and _splitparam and further parsing the k=v pairs we could recreate what cgi.parse_header was doing.

Is there any consideration to just replace the httputil.parse() function with a copy of the code from the cgi.parse_header code in the cherrypy code base?

I'll try and mock up a patch for each scenario to validate them.

Yeah, I also did some research and here's the logs of the differences:

$ python -c 'import cherrypy, sys; accept_header = cherrypy.lib.httputil.AcceptElement.from_str("iso-8859-5"); print(f"{accept_header=}")'
elementstr='iso-8859-5' ===> initial_value='text/plain' &&& params={}
accept_header=<cherrypy.lib.httputil.AcceptElement object at 0x7f3b6aa03e50>
$ python -c 'import cherrypy, sys; accept_header = cherrypy.lib.httputil.AcceptElement.from_str("0.8"); print(f"{accept_header=}")'
elementstr='0.8' ===> initial_value='text/plain' &&& params={}
accept_header=<cherrypy.lib.httputil.AcceptElement object at 0x7fde09f03e50>
$ python -c 'import cherrypy, sys; accept_header = cherrypy.lib.httputil.AcceptElement.from_str("unicode-1-1"); print(f"{accept_header=}")'
elementstr='unicode-1-1' ===> initial_value='text/plain' &&& params={}
accept_header=<cherrypy.lib.httputil.AcceptElement object at 0x7fae2f503e50>

$ python -c 'import cherrypy, sys; accept_header = cherrypy.lib.httputil.AcceptElement.from_str("iso-8859-5"); print(f"{accept_header=}")'
elementstr='iso-8859-5' ===> initial_value='iso-8859-5' &&& params={}
accept_header=<cherrypy.lib.httputil.AcceptElement object at 0x7f9f7d703e50>
$ python -c 'import cherrypy, sys; accept_header = cherrypy.lib.httputil.AcceptElement.from_str("0.8"); print(f"{accept_header=}")'
elementstr='0.8' ===> initial_value='0.8' &&& params={}
accept_header=<cherrypy.lib.httputil.AcceptElement object at 0x7f294b547f90>
$ python -c 'import cherrypy, sys; accept_header = cherrypy.lib.httputil.AcceptElement.from_str("unicode-1-1"); print(f"{accept_header=}")'
elementstr='unicode-1-1' ===> initial_value='unicode-1-1' &&& params={}
accept_header=<cherrypy.lib.httputil.AcceptElement object at 0x7eff5d603e50>

So it sounds like you're right and there's no easy way to reproduce the original behavior with the email module.

I guess, it'd be a good idea to vendor the removed function from CPython, putting it into https://github.com/cherrypy/cherrypy/blob/main/cherrypy/_cpcompat.py.

But before doing that, could you please, add tests for AcceptElement and HeaderElement that pass against the current implementation? And only then, make a separate PR or commit that replaces cgi.parse_headers() with a vendored copy?

It'd make sure there's a documented way of how HeaderElement is supposed to behave (as code).

Sounds good. I'll work on making some progress on it.

@radez hey, did you manage to get to that?

I've got a fix but I've not had a chance to work on the unit tests yet.

@radez it'd be a bit more transparent if you could make a draft PR out of it.

Sure, lemme post what I have. I had intended to get the tests in a draft state too and push it all up at once but work has taken a front seat and I wasn't able to make it happen all at once. I'll put a patch up today.