Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove LArray #12810

Open
gortiz opened this issue Apr 8, 2024 · 0 comments
Open

Remove LArray #12810

gortiz opened this issue Apr 8, 2024 · 0 comments
Labels

Comments

@gortiz
Copy link
Contributor

gortiz commented Apr 8, 2024

This issue is a sibling of #12809.

TL;DR:

Pinot uses LArray, which is unsafe, platform dependent and deprecated. It also doesn't work with Java >= 15, so we had to create our own buffer library to be able to run Pinot with Java >= 15.

This issue proposes to remove LArray dependency and always use our own library instead

Context

As we may know Java buffers length has been limited to 2GBs. In Apache Pinot we decided to skip that limit by using LArray, a third party buffer library that uses JNI to create larger buffers. That library is not maintained anymore (see xerial/larray#75 (comment) or xerial/larray#80

This library has several problems, including:

  • It may produce segmentation faults or buffer overflows (there is no index check when using it).
  • It doesn't work in Mac using arm architecture.
  • It doesn't support Java >= 15.

In order to solve these problems, we introduced a new buffer implementation called unsafe. The name comes from the fact that uses sun.misc.Unsafe, but AFAIK it is safer than LArray. It works and that is what we use by default if at runtime we detect we are using Java >= 15.

We should remove LArray and just use our library instead. In the future a version built on top of the official Java Foreign Memory API should be used, but it can only be used when the runtime is Java 22 (or maybe 21 with enabled preview). See #12809 to know more about that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants