You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, there are two problems with the CMake project in the repo:
The spark-rapids-jni project is built into libcudf.so library. This is bad. It should have its own name, instead of reusing libcudf which leads to name clash with the true libcudf. I would recommend to name libsparkrapidsjni.so or something similar.
Unit tests are both dynamic-linked with libcudf.so (the library built from spark-rapids-jni, not the true libcudf library) and static-linked with libcudf.a (the true libcudf library). Indeed, it only needs to dynamic-linked with spark-rapids-jni since this library already embeds libcudf inside it.
Fixing these issues is not urgent for now as they don't cause any bugs but they should be addressed soon when we can, to improve the code cleanness.
The text was updated successfully, but these errors were encountered:
Building to libcudf.so is intentional for multiple reasons:
cudf's native dependency loader expects to find libcudf.so
RAPIDS accelerated UDFs will be dynamically linked against libcudf.so, so at runtime they need to be able to find this library when the dynamic linker loads them.
Since we are statically linking the CUDA runtime, we only want one .so artifact so there aren't multiple CUDA contexts
If there's another solution that addresses these problems without clobbering the lbcudf,.so name I'd be happy to discuss them. Using a dynamic CUDA runtime seems like the easiest solution for this, as it allows us to provide multiple .so artifacts without duplicating the CUDA contexts. However we explicitly moved away from dynamically linking the CUDA runtime since we had issues with the correct CUDA runtime being properly installed in all the environments and it adds an additional step for users to setup the environment. As it is now, they just need the CUDA driver installed and we're good to go.
Currently, there are two problems with the CMake project in the repo:
libcudf.so
library. This is bad. It should have its own name, instead of reusinglibcudf
which leads to name clash with the true libcudf. I would recommend to namelibsparkrapidsjni.so
or something similar.libcudf.so
(the library built from spark-rapids-jni, not the true libcudf library) and static-linked withlibcudf.a
(the true libcudf library). Indeed, it only needs to dynamic-linked with spark-rapids-jni since this library already embeds libcudf inside it.Fixing these issues is not urgent for now as they don't cause any bugs but they should be addressed soon when we can, to improve the code cleanness.
The text was updated successfully, but these errors were encountered: