inference shell file

[czb2133]

It is really surprising that the ckpt is accessible in hugging face. Can you provide a python file for inference?

[smellslikeml]

Hey @czb2133 , absolutely! We recommend using the gguf files with abetlen/llama-cpp-python to load and run inference.
Check out the colab notebook here.

Also, if you want to host your own server, we included a Triton Server deployment using Docker. Check out these instructions.