snappy error for variable-length data

[ifokeev]

I'm getting very often:

{"level":"error","timestamp":"2021-05-18T10:53:29.489Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:53:35.467Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:53:41.476Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:53:53.545Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:53:59.447Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:01.434Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:05.378Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:11.468Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:23.359Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:25.866Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:31.764Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:35.473Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:41.432Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:45.369Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:47.497Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:49.420Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}
{"level":"error","timestamp":"2021-05-18T10:54:53.438Z","msg":"Error returned from query scanner: snappy: corrupt input\n"}

How is it possible to fix this error? only by removing the compression?
Thanks

[Bexanderthebex]

@ifokeev can you describe reproductions steps in order to arrive to this scenario?

[goodboy]

We have been seeing this same issue for quite a while now as per:
pikers/piker#443

I did a ton of digging as well down to the external snappy go lib which seems to be the root cause.

Recently after deploying to a new nixos install i've found that a similar issue will always show up when querying large data sets

purerpc.grpclib.exceptions.UnknownError: RPC failed with status UNKNOWN (2): EOF

It all seems to be gRPC related for which we have our own async client you can find under the same org.

---

NB: we're probably going to be dropping mkts outright just because there tends to be a lot of unreliability in terms of pushing / pulling large data sets in general.

Going to be tinkering with the new release of arcticdb in the coming weeks: https://github.com/man-group/ArcticDB

[goodboy]

How is it possible to fix this error? only by removing the compression?

No, tried that and it just results in an even more vague rpc error...

See the details of the deep dive here:
pikers/piker#443 (comment)