json.decoder.JSONDecodeError on csv (res) content

[gparonitti]

Environment details

• OS type and version: macOS 10.15.7 (19H1419)
• Python version: 3.8.5
• pipenv version: 2020.8.13
• pip version: 20.2.3
• google-api-python-client version: 2.26.1

Steps to reproduce

Trying to use generateCSV api from adsense.accounts.reports.saved.generateCsv, but it might be a more general issue with non json format content.

Code example

...
saved_report_id = 'accounts/pub-xxx/reports/yyy_zzz'

def main(argv):
  # Authenticate and construct service.
  credentials = adsense_util.get_adsense_credentials()
  with discovery.build('adsense', 'v2', credentials = credentials) as service:
    try:
      # Let the user pick account if more than one.
      account_id = adsense_util.get_account_id(service)

      # Retrieve report.
      # if saved_report_id:
      result = service.accounts().reports().saved().generateCsv(name=saved_report_id, dateRange='LAST_30_DAYS') .execute()

      print(result)
...

Stack trace

Traceback (most recent call last):
  File "src/generate_report.py", line 88, in <module>
    main(sys.argv)
  File "src/generate_report.py", line 63, in main
    print(result.execute())
  File "[...]/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "[...]/lib/python3.8/site-packages/googleapiclient/http.py", line 939, in execute
    return self.postproc(resp, content)
  File "[...]/lib/python3.8/site-packages/googleapiclient/model.py", line 218, in response
    return self.deserialize(content)
  File "[...]/lib/python3.8/site-packages/googleapiclient/model.py", line 281, in deserialize
    body = json.loads(content)
  File "[...]/.pyenv/versions/3.8.5/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "[...]/.pyenv/versions/3.8.5/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/[...]/.pyenv/versions/3.8.5/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Possible cause

The code involved is in googleapiclient/model.py and the class is JsonModel(BaseModel) specifically this part of code (275-283):

    def deserialize(self, content):
        try:
            content = content.decode("utf-8")
        except AttributeError:
            pass
        body = json.loads(content)
        if self._data_wrapper and isinstance(body, dict) and "data" in body:
            body = body["data"]
        return body

The problem appears when the content of response is not in json format (with generateCsv api the content is csv text) so it produces the JSONDecodeError.

Possible patch

One of the following could work:

• In case the request asks for csv generation, the response should be routed to a correct new callable postprocessor method that deals with csv format
• change the above lines of code in this way:

    def deserialize(self, content):
        try:
            content = content.decode("utf-8")
        except AttributeError:
            pass
        try:
            body = json.loads(content)
        except json.decoder.JSONDecodeError:
            body = content
        if self._data_wrapper and isinstance(body, dict) and "data" in body:
            body = body["data"]
        return body

[parthea]

Hi @gparonitti ,

Thank you for reporting this bug and proposing a fix. PRs are welcome! (Please include a test as well to avoid future regressions.)

[gparonitti]

@parthea np.
I was in doubt whether to open a pull request because I don't know all the details of the code flow, and I'm not sure if this patch will work other than as a mere work around for my use case. It seemed to me semantically wrong to return a type that is not in json(-ish) format from a class called JsonModel. Anyway, I'll do the pr and the test asap.

[gparonitti]

Here the pr #1574. Thanks @parthea for the prompt feedback :)