01-local-llm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7157c88-cee7-4ffc-96de-3889ebdc9272",
   "metadata": {},
   "source": [
    "<img src=\"images/ragna-logo.png\" width=\"200px\" align=\"right\"/>\n",
    "\n",
    "# Set up an offline Large Language Model\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "93e56c60-3aa0-42cf-8b8e-07e94f29dc5f",
   "metadata": {},
   "source": [
    "## What is a Large Language Model (LLM)?\n",
    "\n",
    "A \"language model\" is a machine learning model designed to understand and generate (predict) natural language. For example, auto-completion of text in input fields often use language models.\n",
    "\n",
    "A \"large language model\" is a language model based on the [Transformer architecture](https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)), trained on large amounts of (general) data and consists of several billion parameters. With this scale and complexity, LLMs are capable of various text processing and generation tasks like document summarization, answering common questions, text-based content creation.\n",
    "\n",
    "Popular LLMs include Open AI's GPT, Google's Gemini, Anthropic's Claude, and more."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9023605d-8321-4854-8ce0-462847b479dc",
   "metadata": {},
   "source": [
    "## What is a \"local\" or \"offline\" LLM\n",
    "\n",
    "Large Language Models (LLMs) like OpenAI's GPT are proprietary, can only be accessed through the OpenAI API or services like ChatGPT. While easy to use, these can be concerning for data privacy, vendor lock-in, and cost-related reasons.\n",
    "\n",
    "Offline, local, or open weight LLMs are models that can be self-hosted on your local computers.\n",
    "\n",
    "Today, we're running it on a cloud platform, but each of you have access to essentially an individual machine. This allows us to have a standard tutorial environment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87cbb3e8-c0df-4cac-bc12-f36a87a7ba9e",
   "metadata": {},
   "source": [
    "## LLM: Llama3\n",
    "\n",
    "In this tutorial, we'll use the [Llama3-8B model](https://ai.meta.com/blog/meta-llama-3/), which is released by Meta under a permissive license."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26fbc7e7-24a8-40a7-9fcf-60bcfda88451",
   "metadata": {},
   "source": [
    "### Quantization\n",
    "\n",
    "> Quantization is a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types like 8-bit integer (int8) instead of the usual 32-bit floating point (float32).\n",
    "> \n",
    "> Reducing the number of bits means the resulting model requires less memory storage, consumes less energy (in theory), and operations like matrix multiplication can be performed much faster with integer arithmetic. It also allows to run models on embedded devices, which sometimes only support integer data types.\n",
    "> \n",
    "> ~ [Hugging Face Documentation](https://huggingface.co/docs/optimum/en/concept_guides/quantization)\n",
    "\n",
    "\n",
    "In its original float32 representation, an LLM needs roughly 4 GB of VRAM for each billion of parameters. For example, LLama3-8B with its roughly 8 billion parameters needs 32GB to load the weights.\n",
    "\n",
    "By quantizing the float32 representation into a lower number `n` of bits per weight (bpw), this can be drastically reduced to `n / 8` GB of VRAM for each billion of parameters. For example, with 6 bpw, which is what we are going to use, we only need 6 GB to load the weights.\n",
    "\n",
    "There are a number of quantization schemes / file formats (`exl2`, `gtpq`, `gguf`, `awq`) and libraries (Exllamav2, llama.cpp, AutoGPTQ) to create and use quantized weights. For this tutorial we are going to use Exllamav2 with the corresponding `exl2` weights."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4018c61f-65a8-4461-866c-4a609765a563",
   "metadata": {},
   "source": [
    "## Exllamav2\n",
    "\n",
    "`exllamav2` is a quantization and inference library.\n",
    "\n",
    "We have downloaded the quantized versions of Llama3 in the `shared/pycon/models` directory, available from the root of your Nebari file system."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a8ce6bf-8e4b-4cb5-8662-b1d835320a10",
   "metadata": {},
   "source": [
    "### Side note: Instructions for local users 💻\n",
    "\n",
    "To download and use the model on your local computer (i.e., outside this tutorial at PyCon):\n",
    "\n",
    "1. Install `Exllamav2` with the [instructions in the project repository](https://github.com/turboderp/exllamav2#installation).\n",
    "2. In your local terminal: `git lfs install` and `git clone https://huggingface.co/turboderp/Llama-3-8B-Instruct-exl2`\n",
    "3. View all branches:`git branch --all`\n",
    "4. Check-out the branch with relevant weights `git checkout remotes/origin/6.0bpw`\n",
    "5. Note the model directory path, and use it in the inference and Ragna scripts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de228be0-ec86-4c4a-8a6a-bc1262197f6c",
   "metadata": {},
   "source": [
    "## Use Llama3 for inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c3e81b-78bc-4919-8e5b-80379099de67",
   "metadata": {},
   "source": [
    "Let's run the [example inference script](https://github.com/turboderp/exllamav2/blob/master/examples/inference.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7fefb78-992f-4df8-bf8d-2f737108a214",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf325fc8-d9c5-47d2-b4e6-3236d619519f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from pathlib import Path\n",
    "\n",
    "import torch\n",
    "from exllamav2 import ExLlamaV2, ExLlamaV2Cache, ExLlamaV2Config, ExLlamaV2Tokenizer\n",
    "from exllamav2.generator import ExLlamaV2BaseGenerator, ExLlamaV2Sampler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3342e3a-af00-4680-b26d-59f046947091",
   "metadata": {},
   "source": [
    "### Initialize model and cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61a49e16-6634-420a-9012-1af1cffafa3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "models_directory = Path.home() / \"shared/pycon/models\"\n",
    "model_directory = models_directory / \"turboderp/Llama-3-8B-Instruct-exl2\"\n",
    "\n",
    "print(f\"Loading model: {model_directory}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "252b594c-112b-4fce-90a7-32b5140dc8a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = ExLlamaV2Config()\n",
    "config.model_dir = model_directory\n",
    "config.prepare()\n",
    "model = ExLlamaV2(config)\n",
    "cache = ExLlamaV2Cache(model, lazy=True)\n",
    "model.load_autosplit(cache)\n",
    "tokenizer = ExLlamaV2Tokenizer(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9897dbd1-ccca-4eeb-8ea6-2d5534e0d6c9",
   "metadata": {},
   "source": [
    "### Initialize generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7659d4f8-6b00-4305-a727-de47785d70a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "generator = ExLlamaV2BaseGenerator(model, cache, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea5d31a8-8ad6-4dbc-aa96-a40e684fb6b1",
   "metadata": {},
   "source": [
    "### Generate some text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cd48260-2df6-4fcd-80f4-5574d8cdc15e",
   "metadata": {},
   "outputs": [],
   "source": [
    "settings = ExLlamaV2Sampler.Settings()\n",
    "settings.temperature = 0.85\n",
    "settings.top_k = 50\n",
    "settings.top_p = 0.8\n",
    "settings.token_repetition_penalty = 1.01\n",
    "settings.disallow_tokens(tokenizer, [tokenizer.eos_token_id])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01c8604f-2cb1-40cc-9467-dfe505601afd",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"Our story begins in PyCon US, where once\"\n",
    "\n",
    "max_new_tokens = 150\n",
    "\n",
    "generator.warmup()\n",
    "time_begin = time.time()\n",
    "\n",
    "output = generator.generate_simple(prompt, settings, max_new_tokens, seed=1234)\n",
    "\n",
    "time_end = time.time()\n",
    "time_total = time_end - time_begin\n",
    "\n",
    "print(output)\n",
    "print()\n",
    "print(\n",
    "    f\"Response generated in {time_total:.2f} seconds, {max_new_tokens} tokens, {max_new_tokens / time_total:.2f} tokens/second\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecbc3c78-b86b-47a4-b431-4c0a22c039aa",
   "metadata": {},
   "source": [
    "Instead of waiting for the full generation to complete, we can also stream the answer back in individual generated chunks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5249e90b-c178-4773-b58d-a1c165906efa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from exllamav2.generator import ExLlamaV2Sampler, ExLlamaV2StreamingGenerator\n",
    "\n",
    "generator = ExLlamaV2StreamingGenerator(model, cache, tokenizer)\n",
    "\n",
    "generator.begin_stream_ex(tokenizer.encode(prompt), settings)\n",
    "\n",
    "print(prompt, end=\"\")\n",
    "for _ in range(max_new_tokens):\n",
    "    result = generator.stream_ex()\n",
    "    if result[\"eos\"]:\n",
    "        break\n",
    "    print(result[\"chunk\"], end=\"\")\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c2e88ab-3934-44d4-87a2-b01c32457af0",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "_❗️ **Warning:** Make sure to stop the Jupyter Kernel (in the JupyterLab Menu Bar, click on \"Kernel\" -> \"Shut down Kernel\") before proceeding to prevent the \"insufficient VRAM\" error._\n",
    "\n",
    "<br>\n",
    "\n",
    "**✨ Next: [Basics of RAG-powered chat app](02-rag-basics.ipynb) →**\n",
    "\n",
    "<br>\n",
    "\n",
    "💬 _Wish to continue discussions after the tutorial? Contact the presenters: [@pavithraes](https://github.com/pavithraes), [@dharhas](https://github.com/dharhas), [@ahuang11](https://github.com/ahuang11)_\n",
    "\n",
    "<hr>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pycon-pycon-ragna",
   "language": "python",
   "name": "conda-env-pycon-pycon-ragna-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}