Skip to content

A LLaMA2-7b chatbot with memory running on CPU, and optimized using smooth quantization, 4-bit quantization or Intel® Extension For PyTorch with bfloat16.

License

Notifications You must be signed in to change notification settings

aahouzi/llama2-chatbot-cpu

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

75 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

LLaMA2 chatbot on CPU

🧐 Description

  • This project is a Streamlit chatbot with Langchain deploying a LLaMA2-7b-chat model on Intel® Server and Client CPUs.

  • The chatbot has a memory that remembers every part of the speech, and allows users to optimize the model using Intel® Extension for PyTorch (IPEX) in bfloat16 with graph mode or smooth quantization (A new quantization technique specifically designed for LLMs: ArXiv link), or 4-bit quantization. The user can expect up to 4.3x speed-up compared to stock PyTorch in default mode.

  • IMPORTANT: The CPU needs to support bfloat16 ops in order to be able to use such optimization. On top of software optimizations, I also introduced some hardware optimizations like non-uniform memory access (NUMA). User needs to ask for access to LLaMA2 models by following this link. When getting approval from Meta, you can generate an authentification token from your HuggingFace account, and use it to load the model.

📜 Getting started

  1. Start by cloning the repository:
git clone https://github.com/aahouzi/llama2-chatbot-cpu.git
cd llama2-chatbot-cpu
  1. Create a Python 3.9 conda environment:
conda create -y -n llama2-chat python=3.9
  1. Activate the environment:
conda activate llama2-chat
  1. Install requirements for NUMA:
conda install -y gperftools -c conda-forge
conda install -y intel-openmp
sudo apt install numactl
  1. Install the app requirements:
pip install -r requirements.txt

🚀 Start the app

  • Default mode (no optimizations):
bash launcher.sh --script=app/app.py --port=<port> --physical_cores=<physical_cores> --auth_token=<auth_token>
  • IPEX in graph mode with FP32:
bash launcher.sh --script=app/app.py --port=<port> --physical_cores=<physical_cores> --auth_token=<auth_token> --ipex --jit
  • IPEX in graph mode with bfloat16:
bash launcher.sh --script=app/app.py --port=<port> --physical_cores=<physical_cores> --auth_token=<auth_token> --dtype=bfloat16 --ipex --jit
  • Smooth quantization:
bash launcher.sh --script=app/app.py --port=<port> --physical_cores=<physical_cores> --auth_token=<auth_token> --sq
  • 4-bit quantization:
bash launcher.sh --script=app/app.py --port=<port> --physical_cores=<physical_cores> --auth_token=<auth_token> --int4

💻 Chatbot demo

📪 Contact

For any information, feedback or questions, please contact me

Releases

No releases published

Packages

No packages published