Kombu 5.2.0 Object of type bytes is not JSON serializable

[mrezzamoradi]

I have a Celery setup (celery==5.1.2) that used to work fine with Kombu 5.1.0 and Rabbitmq 3.9.8
but now the producers sometimes throw this error:

Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/serialization.py", line 39, in _reraise_errors
    yield
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/serialization.py", line 210, in dumps
    payload = encoder(data)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/utils/json.py", line 68, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "<my python 3.9>/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "<my python 3.9>/lib/python3.9/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "<my python 3.9>/lib/python3.9/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/utils/json.py", line 58, in default
    return super().default(o)
  File "<my python 3.9>/lib/python3.9/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/rezamorandi/oneflow/API/flow/middleware/proxyfix.py", line 26, in __call__
    return self.app(environ, start_response)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/flask/app.py", line 2457, in wsgi_app
    ctx.auto_pop(error)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/flask/ctx.py", line 452, in auto_pop
    self.pop(exc)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/flask/ctx.py", line 415, in pop
    self.app.do_teardown_request(exc)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/flask/app.py", line 2299, in do_teardown_request
    func(exc)
  File "/Users/rezamorandi/oneflow/API/flow/request_hooks.py", line 299, in teardown
    TransactionLog.execute(exception=exception)
  File "/Users/rezamorandi/oneflow/API/flow/models/transaction_log.py", line 231, in execute
    signature.delay()  # Do not use fire() to make sure the task isn't suppressed
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/celery/canvas.py", line 172, in delay
    return self.apply_async(partial_args, partial_kwargs)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/celery/canvas.py", line 219, in apply_async
    return _apply(args, kwargs, **options)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/celery/app/task.py", line 572, in apply_async
    return app.send_task(
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/celery/app/base.py", line 767, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/celery/app/amqp.py", line 510, in send_task_message
    ret = producer.publish(
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/messaging.py", line 166, in publish
    body, content_type, content_encoding = self._prepare(
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/messaging.py", line 254, in _prepare
    body) = dumps(body, serializer=serializer)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/serialization.py", line 210, in dumps
    payload = encoder(data)
  File "<my python 3.9>/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/serialization.py", line 43, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/exceptions.py", line 21, in reraise
    raise value.with_traceback(tb)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/serialization.py", line 39, in _reraise_errors
    yield
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/serialization.py", line 210, in dumps
    payload = encoder(data)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/utils/json.py", line 68, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "<my python 3.9>/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "<my python 3.9>/lib/python3.9/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "<my python 3.9>/lib/python3.9/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
  File "<my python 3.9>/envs/myenv/lib/python3.9/site-packages/kombu/utils/json.py", line 58, in default
    return super().default(o)
  File "<my python 3.9>/lib/python3.9/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
kombu.exceptions.EncodeError: Object of type bytes is not JSON serializable

[open-collective-bot[bot]]

Hey @mrezzamoradi 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[mrezzamoradi]

I just found out that the new version is using python internal json module as the default encoder, wherein the older version (5.1.0) it was using simplejson; and python builtin json can not dump byte data.

[auvipy]

I just found out that the new version is using python internal json module as the default encoder, wherein the older version (5.1.0) it was using simplejson; and python builtin json can not dump byte data.

I think we should explicitly mention this to celery kombu 5.2 release notes. @celery/core-developers @celery/regular-contributors

[dstromberg]

Piggybacking, on an old thread - hope that's OK.

Is there a good way to persuade Celery+Kombu to use simplejson instead of the standard library's json module?

I'm on:

tact@tact_pub_api:/app$ python3.10 -m pip list -v | egrep -i 'celery|kombu'
celery                  5.2.3       /usr/local/lib/python3.10/site-packages pip
kombu                   5.2.4       /usr/local/lib/python3.10/site-packages pip
tact@tact_pub_api:/app$ python2.7 -m pip list -v | egrep -i 'celery|kombu'
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
celery                        3.1.26.post2 /usr/local/lib/python2.7/dist-packages pip
kombu                         3.0.37       /usr/local/lib/python2.7/dist-packages pip

...and I'm getting a traceback much like the OP had.

We (my team and application) may want to switch to the standard library's json module eventually, but for now it seems like just using simplejson makes more sense.

[dstromberg]

More info at https://stackoverflow.com/questions/73901352/how-might-i-persuade-kombu-to-use-simplejson-instead-of-the-json-module . If you don't mind, please reply there.