Pandas DataFrame.to_msgpack and read_msgpack are deprecated

[arnevdk]

Steps to reproduce

When creating a timeflux.nodes.zmq.Pub node with the msgpack serializer like

module: timeflux.nodes.zmq
class: Pub
params:
    topic: eeg
    serializer: msgpack

timeflux errors with

'DataFrame' object has no attribute 'to_msgpack'
Traceback (most recent call last):
  File "<...>/timeflux/core/worker.py", line 57, in _run
    scheduler.run()
  File "<...>//timeflux/core/scheduler.py", line 22, in run
    self.next()
  File "<...>//timeflux/core/scheduler.py", line 59, in next
    self._nodes[step["node"]].update()
  File "<...>//timeflux/nodes/zmq.py", line 177, in update
    self._socket.send_serialized(
  File "<...>//site-packages/zmq/sugar/socket.py", line 854, in send_serialized
    frames = serialize(msg)
  File "<...>//timeflux/core/message.py", line 31, in msgpack_serialize
    return [topic, data.to_msgpack()]
  File "/home/arne/.virtualenvs/timeflux-workshop/lib64/python3.9/site-packages/pandas/core/generic.py", line 5989, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'to_msgpack'

Issue

When initialized using serialize='msgpack or deserialize='msgpack', timeflux.nodes.zmq.Pub and timeflux.nodes.zmq.Sub respectively retrieve timeflux.core.message.msgpack_serialize and timeflux.core.message.msgpack_deserialize methods, which in turn call pandas.DataFrame.to_msgpack and pandas.read_msgpack. These methods have been deprecated since Pandas 0.25 and were removed in newer versions:
https://pandas.pydata.org/pandas-docs/version/0.25.0/reference/api/pandas.DataFrame.to_msgpack.html
https://pandas.pydata.org/pandas-docs/version/0.25.3/reference/api/pandas.read_msgpack.html

Proposed fixes

• Change the implementations in timeflux.core.message to use the equivalent methods from pandas-msgpack (also no longer maintained)
• Remove the msgpack serializer and implement another alternative serializer, preferrably one supporting numpy datatypes as I encoutered this bug while looking for a solution to issues I experienced similar to Use a better JSON serialization lib #74 where numpy datatype port output cannot be serialized. A good candidate is pyarrow, in favor of which pandas has deprecated msgpack: https://pandas.pydata.org/pandas-docs/version/2.0/user_guide/pyarrow.html

[mesca]

Hi Arne,

Thanks for the report. I believe msgpack is seldom used by Timeflux users, so I agree we should deprecate it.

Actually, and as you probably noticed, the code for pyarrow (de)serialization already exists, so a quick implementation would require a conditional import of the module, and an optional dependency to pyarrow.

Outputs from Timeflux nodes are objects consisting of two components:

• A DataFrame structure with a Pandas-compatible API ;
• An arbitrary Python dictionary containing metadata such as sample rate, context, etc.

So we still need a way to serialize the metadata. We could use Pickle, but that would limit interoperability to Python applications. Metadata can contain complex datatypes that could be more efficiently handled by libraries such as orjson, but we would introduce another dependency. Would a simple JSON serialization using the standard library work for you? If so, we could just log a warning if metadata (de)serialization fails.

If that sounds like an acceptable solution for you, feel free to open a PR!