json() crashes interpreter with a JSONDecodeError

[justlurking]

I have reproduced this with requests v. 2.18.4 and 2.14.2, along with Python 3.6.1 and 3.6.2. I have used my system-installed Python/requests (macOS Sierra with Python 3.6.1 downloaded from python.org), Python / requests installed as part of conda 4.3.25, and the latest Python and requests installed from conda-forge.

Same results everywhere.

Expected Result

Expected to see a dictionary of JSON values, as indicated here: http://docs.python-requests.org/en/master/

Actual Result

Python interpreter crashes.

Reproduction Steps

Python 3.6.2 | packaged by conda-forge | (default, Jul 23 2017, 23:01:38) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://pypi.python.org/pypi')
>>> r.status_code
200
>>> r.encoding
'utf-8'
>>> r.json()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/psimpson/miniconda3/envs/test/lib/python3.6/site-packages/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Users/psimpson/miniconda3/envs/test/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Users/psimpson/miniconda3/envs/test/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/psimpson/miniconda3/envs/test/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.0.3"
  },
  "idna": {
    "version": ""
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "16.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "100020cf",
    "version": "17.2.0"
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}

[nateprewitt]

Hey @justlurking, this is actually working as expected. The .json() method is only usable on Response content that is in the json format. The webpage you're requesting is returning an HTML document and therefore won't work.

You'll need to either modify your Accept headers and attempt to retrieve valid json data, or add code to handle the case of json data not being readily available. I'm closing this since things are working as expected.

[justlurking]

Thanks for the super quick and informative reply, and apologies for wasting your time. I should have realized it might be something like that given the multiple versions I tested and lack of outcry on GitHub. 🤦

I imagine the best thing to do would be to import json.JSONDecodeError and do something like this?

try:
    r.json()
except JSONDecodeError as e:
    <handle exception>

[nateprewitt]

So this should actually be a subclass of ValueError so I think you'll be able to catch that if you don't want to import. It's up to you on the level of specificity you'd like.

That general idea is the approach I'd suggest though. Hope that helps!