Multiple Set-Cookie headers folded into one value

[DanHoerst]

When Python Requests is used to contact a server which returns multiple Set-Cookie response headers, Python Requests combines them into one header field with the values separated by commas.

Expected Result

Using Nginx as my origin server with the following config:

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
        add_header Set-Cookie cookieone=test;
        add_header Set-Cookie cookietwo=testing;
    }

Contacting this origin server I expect to receive two separate Set-Cookie headers with their corresponding values. As per RFC6265:

Origin servers SHOULD NOT fold multiple Set-Cookie header fields into
a single header field. The usual mechanism for folding HTTP headers
fields (i.e., as defined in [RFC2616]) might change the semantics of
the Set-Cookie header field because the %x2C (",") character is used
by Set-Cookie in a way that conflicts with such folding.

When using the browser my response from this origin has multiple Set-Cookie headers:

Actual Result

Python Requests folds the multiple Set-Cookie headers into a single header field separated by comma(s). For example:

>>> import requests
>>> r = requests.get('http://localhost')
>>> r
<Response [200]>
>>> r.headers
{'Content-Length': '612', 'Set-Cookie': 'cookieone=test, cookietwo=testing', 'Accept-Ranges': 'bytes', 'Server': 'nginx/1.13.9', 'Last-Modified': 'Tue, 20 Feb 2018 12:21:20 GMT', 'Connection': 'keep-alive', 'ETag': '"5a8c12c0-264"', 'Date': 'Mon, 26 Feb 2018 14:21:59 GMT', 'Content-Type': 'text/html'}
>>> r.cookies
<RequestsCookieJar[Cookie(version=0, name='cookieone', value='test', port=None, port_specified=False, domain='localhost.local', domain_specified=False, domain_initial_dot=False, path='/', path_specified=False, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='cookietwo', value='testing', port=None, port_specified=False, domain='localhost.local', domain_specified=False, domain_initial_dot=False, path='/', path_specified=False, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={}, rfc2109=False)]>

Reproduction Steps

• Set up Nginx locally (using Docker docker run -itd -p 80:80 nginx)
• Use the Nginx location config above with multiple Set-Cookie headers
• Call the Nginx url from the browser and notice multiple Set-Cookie headers (screenshot above)
• Call the Nginx url from requests and notice a single, folded, Set-Cookie header (code above)

System Information

➜ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "2.7.10"
  },
  "platform": {
    "release": "17.4.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "20000000"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

Jupyterhub is an example of where this is a problem in the wild: jupyterhub/jupyterhub#1657

[nateprewitt]

Hi @DanHoerst, thanks for bringing this up. This is actually a duplicate of #3957 and yes it is a current issue with Requests header parsing implementation. As noted in the tracking issue, the cookies are still available separately in the cookie jar which is the preferred interface for this. I'm gonna close this as a dupe and we can follow up in the original as needed.

[sigmavirus24]

There's also the underlying header structure that urllib3 uses which is always available and represents values like this correctly.

[DanHoerst]

@sigmavirus24 Thanks. Do you have an example of how to access that from a Requests response object?

[nateprewitt]

@DanHoerst, response.raw.headers.getlist('Set-Cookie') should give you a list of the raw strings urllib3 gets.