We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Thank you for your great work! I am trying to deploy llama-2-7b-chat model to Jetson Orin NX 8G.
I followed the instructions in Tinychat but found that when loading llama-2-7b-chat 4bit g128, it got killed due to out of memory issue.
Tinychat
Then, I followed the demo in NVIDIA Jetson website (https://www.jetson-ai-lab.com/tutorial_text-generation.html) and downloaded Llama-2-7B-Chat-GPTQ (https://huggingface.co/TheBloke/Llama-2-7B-Chat-GPTQ). It worked well but still got killed after several conversation.
As you mentioned in Readme, llama-2-7b-chat can work in Jetson Orin NX 8G. Is there anything that I missed? Any thought?
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Thank you for your great work! I am trying to deploy llama-2-7b-chat model to Jetson Orin NX 8G.
I followed the instructions in
Tinychatbut found that when loading llama-2-7b-chat 4bit g128, it got killed due to out of memory issue.Then, I followed the demo in NVIDIA Jetson website (https://www.jetson-ai-lab.com/tutorial_text-generation.html) and downloaded Llama-2-7B-Chat-GPTQ (https://huggingface.co/TheBloke/Llama-2-7B-Chat-GPTQ). It worked well but still got killed after several conversation.
As you mentioned in Readme, llama-2-7b-chat can work in Jetson Orin NX 8G. Is there anything that I missed? Any thought?
The text was updated successfully, but these errors were encountered: