Building llama_cpp container does not produce the compiled C++ version of llama.cpp executables

[amasciotta]

Hello! I am using the prebuilt container dustynv/llama_cpp, which contains the built C++ executables inside /opt/llama.cpp.

However, trying to rebuild it from scratch to get some fixes from the main repo, I obtain a container that has no /opt/llama.cpp folder.

I suspect that #422 broke something...

[dusty-nv]

Hi @amasciotta, yes it builds llama_cpp_python from https://github.com/abetlen/llama-cpp-python which has llama.cpp as submodule. For now if you need to retain the C++ stuff, you just may want to build it, it is straightforward to compile these days with -DLLAMA_CUBLAS=on -DLLAMA_CUDA_F16=1

[amasciotta]

I will definitely try to build it myself. Any other suggestion on how to set compile options to work best on Jetson Xavier and Orin?

Finally, a note: LLAMA_CUBLAS is deprecated, in favour of LLAMA_CUDA.