JSONDecodeError in client.query()

[harmv]

This is the same issue as #567, I could not re-open that so instead I filed a new one.

I have more details and a patch.
(The patch is probably not correct, but it indicates where the problem might be).

steps to reproduce.

Use python influxdb to do an old style query (not flux) to influxdb2.

(it doesn't matter which query you do, it always fails)

result

The request raises an exception.

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  File "requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: [Errno Expecting value] ��results���statement_id�series���name�sensors�columns��time�io1�port_id�io2�io3�mode�firmware�connected�connections�errors�CTZNZmode�values����b�k��513��1.1.0�T: 0
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "django/views/decorators/http.py", line 40, in inner
    return func(request, *args, **kwargs)
  File "sensors/views/api.py", line 253, in node_last
    data = get_data(node)
  File "sensors/views/api.py", line 176, in get_data
    results = db.query(q)
  File "influxdb/client.py", line 537, in query
    data = response.json()
  File "requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

notes

If you use an older version of python influxdb client it works fine
If you query influxdb 1.8 it works fine

details

The problem arises from using content type x-msgpack. (See #733)
The client sends Accept: application/x-msgpack to the influxdb server in the http request to inform it would really want messagepack data.

However.. the influxdb2 server somehow responds with Content-Type: application/octet-stream instead of application/x-msgpack.

The influxdb client fails to parse the reply, because it didn't recognize it as x-msgpack json.

If you use an older influxdb client, all works fine, because it send Accept: application/json header instead to the server.
If you query influxdb 1.8 server, all works fine, because that replies with a Content-Type: application/x-msgpack

patch

The following "patch" in influxdb client makes stuff working again.

diff --git a/influxdb/client.py b/influxdb/client.py
index a0f571f..8ad91db 100644
--- a/influxdb/client.py
+++ b/influxdb/client.py
@@ -345,7 +345,7 @@ class InfluxDBClient(object):
                     raise
 
         type_header = response.headers and response.headers.get("Content-Type")
-        if type_header == "application/x-msgpack" and response.content:
+        if type_header in ("application/x-msgpack", "application/octet-stream") and response.content:
             response._msgpack = msgpack.unpackb(
                 packed=response.content,
                 ext_hook=_msgpack_parse_hook,

The client now correctly parses the response as messagepack, and all works fine. No exceptions, just neatly parsed replies.

Influxdb documentation does specify what a client should send (Accept: ) but headers for responses are not clearly documented: ref https://docs.influxdata.com/influxdb/v2.0/api/v1-compatibility/

• InfluxDB version: e.g. 1.7.7 (output of the influx version command)

  $ dpkg --list |grep influxdb
  ii influxdb2 2.0.3 amd64 Distributed time-series database.

• InfluxDB-python version: e.g. 5.2.2 (output of the python -c "from __future__ import print_function; import influxdb; print(influxdb.__version__)" command)

  $ pip freeze |grep influx
  influxdb==5.3.1

• Python version: e.g. 3.7.4 (output of the python --version command)

  Python 3.8.10

• Operating system version: e.g. Windows 10, Ubuntu 18.04, macOS 10.14.5

  Ubuntu 20.04.3 LTS

[harmv]

Raised as issue too in influxdb2

If you want influxdb-python to handle influxdb2 versions with this problem, this ticket needs to be fixed on influxdb-python too.

[harmv]

workaround, without changing influxdb-python code:

pass headers to force use of json instead of MessagePack

# workaround for https://github.com/influxdata/influxdb/issues/23146
# influxdb-python crashes when issuing request to influxdb, as it gets the incorrect
# Content-Type in the reply
# The headers below,  force the reply to be in json (instead of in MessagePack)
headers = dict(Accept='application/json')

db = InfluxDBClient(host=settings.INFLUX_DB['HOST'],
                         port=settings.INFLUX_DB['PORT'],
                         database=settings.INFLUX_DB['NAME'],
                         username=settings.INFLUX_DB['USER'],
                         password=settings.INFLUX_DB['PASSWD'],
                         headers=headers)