New HTTPHeaderDict class for response headers with multiple header value support

[the-allanc]

I personally wanted something like this while writing my own Requests-related library, but I can see other people also have wanted something like this - #3957 and #4214 both mention it in terms of handling Set-Cookie headers.

I've written an subclass implementation of CaseInsensitiveDict which is compatible with all the current usage (according to the tests), but adds new methods for clients that want to deal directly with unjoined headers. I've also added quite a number of tests and made slight changes to the documentation in relation to it.

[codecov-io]

Codecov Report

Merging #4282 into master will increase coverage by 0.26%.
The diff coverage is 98.36%.

@@            Coverage Diff             @@
##           master    #4282      +/-   ##
==========================================
+ Coverage   88.66%   88.93%   +0.26%     
==========================================
  Files          18       18              
  Lines        2021     2079      +58     
==========================================
+ Hits         1792     1849      +57     
- Misses        229      230       +1

Impacted Files	Coverage Δ
requests/adapters.py	92.99% <100%> (ø)	⬆️
requests/structures.py	98.97% <98.3%> (-1.03%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdb9b75...5cbd6dd. Read the comment docs.

[kennethreitz]

I generally like this.

This should be a PR against the proposed/3.0.0 branch, though, not master.

[kennethreitz]

I also wouldn't be opposed to the name CaseInsensitiveMultiDict

[kennethreitz]

I haven't checked, but I'd like this API to be compatible with Flask/Werkzeug's MultiDict, if possible.

[kennethreitz]

So yeah, requested changes:

• Make this against the proposed/3.0.0 branch, instead of master:
• Rename to CaseInsensitiveMultiDict
• Use same API as Flask/Werkzeug's MultiDict.

[kennethreitz]

In general though, this looks real solid! ✨🍰✨

[kennethreitz]

Good job with the tests too 🍪

[kennethreitz]

You get a cookie.

[kennethreitz]

I also really like that this is a subclass of CaseInsensitiveDict.

[the-allanc]

With regards to MultiDict in werkzeug, it uses getlist and setlist. HTTPHeaderDict in urllib3 also uses getlist (it has no setlist equivalent). I was going to use the same naming convention, except that I'm storing and returning tuples.

I did have concerns about the lists being modified outside of HTTPHeaderDict (which tries to enforce that only string types are present). So it's either a case of::

• Leaving the names as they are.
• Using "list" names even though we're returning tuples.
• Rename the methods and return lists (either copies or the underlying sequence).

[the-allanc]

Renaming HTTPHeaderDict to MultiCaseInsensitiveDict is a bit more contentious - because it's not a generalised class (it's intended for string values only).

The main intent was to keep the exact same behaviour that we had in requests for response header processing, but to handle the concatenation of values ourselves (rather than letting urllib3 to do it for us). The concatenation of values is specific to HTTP headers (comma separation).

I could refactor part of the code and extract the Multi... part of the class into a separate class, and then inherit from that. We would need to decide what happens for general item access (I note that MultiDict returns just the first item).

[kennethreitz]

I think what you have is good, then, unless we cast to lists.

[kennethreitz]

Which seems silly to me.

[kennethreitz]

I just wanted to provide a very familiar API to those who have used similar APIs before.

[kennethreitz]

I've just always liked the name CaseInsensitiveMultiDict, and wish we had it. This is good though.

[the-allanc]

OK, I've renamed the methods to getlist and setlist, and they now return lists, so that's more consistent with other implementations. Both urllib3 and werkzeug provide copies of the sequences, so we also do the same (we still use tuples under the hood, which makes me happy).

I was going to try and extract some of the code into a CaseInsensitiveMultiDict class, but there's too much nuance in trying to determine the best behaviour for such a generalised class (especially for the extend method, which tries to distinguish between items and sequences - how should it behave if you're using the dictionary to store sequences of sequences?)

I've made sure that this class is entirely backwards compatible with what we were using before - I wrote some of the tests using CaseInsensitiveDict and they still worked when I switched to HTTPHeaderDict for responses. Are you sure that you still want it against the proposed/3.0.0 branch?

[kennethreitz]

@the-allanc definitely :)

P.S. good work!

[the-allanc]

In that case, I'm closing this pull request in favour of #4290.