Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot serialize small floats / denormalized numbers? #333

Open
SmooreInsight opened this issue Feb 28, 2024 · 1 comment
Open

Cannot serialize small floats / denormalized numbers? #333

SmooreInsight opened this issue Feb 28, 2024 · 1 comment

Comments

@SmooreInsight
Copy link

SmooreInsight commented Feb 28, 2024

Disclaimer: I am new to the wilderness of backend and most of the terminology I am using was learned from using Google Gemini as a tutor. If I don't make sense, that's why. I am not clear on what I'm doing, but I do want to be. Suggested learning resources for a deeper dive are welcome.

I have a float64 numpy array obtained from a bin file with the following in python:
file = bin_of_npfloat64
dt = np.dtype('float64')
arr = np.fromfile(
file,
dtype = dt
)
arr = np.ascontiguousarray(arr)
serialized_data = msgpack.packb(arr)

At first I had errors like this from trying to serialize with msgpack to later create a protobuf, but realized that my python version was 3.1 and might need updating for a lot of things.
raise TypeError("Cannot serialize %r" % (obj,))
TypeError: Cannot serialize array([9.88131292e-324, 2.76676762e-322, 7.50979782e-322,...

Now, after the update, I get this error (edited for simplicity).
File ~/anaconda3/lib/python3.11/site-packages/msgpack/init.py:36, in packb(o, **kwargs)
30 def packb(o, **kwargs):
31 """
32 Pack object o and return packed bytes
33
34 See :class:Packer for options.
35 """
---> 36 return Packer(**kwargs).pack(o)
File ... packer.pyx:294, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:300, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:297, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:291, in msgpack._cmsgpack.Packer.pack()
...
TypeError: can not serialize 'numpy.ndarray' object'

Gemini suggests it's a limitation of msgpack being able to handle small values. The small values seemed so extremely small that I first asked my coworker if they were actually our data and not corruption or due to approximations that are unnecessary, but the more experienced coworker insisted that nothing should have changed in how the float64 is handled.

Is this a limitation of what msgpack can handle?

@methane
Copy link
Member

methane commented Feb 29, 2024

This is repository for msgpack spec, not for msgpack-python.

Anyway, msgpack-python doesn't support numpy.
There is msgpack-numpy. But it is not my product so I can not support it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants