ragas_quickstart.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "2e63f667",
      "metadata": {
        "id": "2e63f667"
      },
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/rajshah4/LLM-Evaluation/blob/main/ragas_quickstart.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>\n",
        "\n",
        "Welcome to the ragas quickstart. We're going to get you up and running with ragas as qickly as you can so that you can go back to improving your Retrieval Augmented Generation pipelines while this library makes sure your changes are improving your entire pipeline.\n",
        "\n",
        "to kick things of lets start with the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "57585b55",
      "metadata": {
        "id": "57585b55"
      },
      "outputs": [],
      "source": [
        "%pip -q install ragas pandas datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c77789bb",
      "metadata": {
        "id": "c77789bb"
      },
      "source": [
        "Ragas also uses OpenAI for running some metrics so make sure you have your openai key ready and available in your environment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "0b7179f7",
      "metadata": {
        "id": "0b7179f7",
        "outputId": "cc6abd54-0786-43a3-b35a-1a95af7d3882",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your OPENAI API Key··········\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import getpass\n",
        "\n",
        "open_ai_key = getpass.getpass('Enter your OPENAI API Key')\n",
        "os.environ['OPENAI_API_KEY'] = open_ai_key"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "06c9fc7d",
      "metadata": {
        "id": "06c9fc7d"
      },
      "source": [
        "## The Data\n",
        "\n",
        "Ragas performs a `ground_truth` free evaluation of your RAG pipelines. This is because for most people building a gold labeled dataset which represents in the distribution they get in production is a very expensive process.\n",
        "\n",
        "Hence to work with ragas all you need are the following data\n",
        "- question: `list[str]` - These are the questions you RAG pipeline will be evaluated on.\n",
        "- answer: `list[str]` - The answer generated from the RAG pipeline and give to the user.\n",
        "- contexts: `list[list[str]]` - The contexts which where passed into the LLM to answer the question.\n",
        "\n",
        "Ideally your list of questions should reflect the questions your users give, including those that you have been problamatic in the past.\n",
        "\n",
        "Here we're using an example dataset from on of the baselines we created for the [Financial Opinion Mining and Question Answering (fiqa) Dataset](https://sites.google.com/view/fiqa/) we created. If you want to want to know more about the baseline, feel free to check the `experiements/baseline` section"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "b658e02f",
      "metadata": {
        "id": "b658e02f",
        "outputId": "7fc6c8b6-8638-4b2d-c149-1c7457e3f839",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    baseline: Dataset({\n",
              "        features: ['question', 'ground_truths', 'answer', 'contexts'],\n",
              "        num_rows: 30\n",
              "    })\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "# data\n",
        "from datasets import load_dataset\n",
        "\n",
        "fiqa_eval = load_dataset(\"explodinggradients/fiqa\", \"ragas_eval\")\n",
        "fiqa_eval"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "84aa640f",
      "metadata": {
        "id": "84aa640f"
      },
      "source": [
        "## Metrics\n",
        "\n",
        "Ragas measures your pipeline's performance against two dimensions\n",
        "\n",
        "1. Faithfulness: measures the factual consistency of the generated answer against the given context.\n",
        "2. Relevancy: measures how relevant retrieved contexts and the generated answer are to the question.\n",
        "\n",
        "Through repeated experiments, we have found that the quality of a RAG pipeline is highly dependent on these two dimensions. The final `ragas_score` is the harmonic mean of these two factors.\n",
        "\n",
        "now lets import these metrics and understand more about what they denote"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from ragas.metrics import (\n",
        "    answer_relevancy,\n",
        "    faithfulness,\n",
        "    context_recall,\n",
        "    context_precision\n",
        ")"
      ],
      "metadata": {
        "id": "TDZlAWMa6zJy"
      },
      "id": "TDZlAWMa6zJy",
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "ef8c5e60",
      "metadata": {
        "id": "ef8c5e60"
      },
      "source": [
        "here you can see that we are using 4 metrics, but what do the represent?\n",
        "\n",
        "1. answer_relevancy - a measure of how relevant the answer is to the question\n",
        "\n",
        "2. faithfulness - the factual consistency of the answer to the context base on the question.\n",
        "\n",
        "3. context_recall: measures the ability of the retriever to retrieve all the necessary information needed to answer the question.\n",
        "\n",
        "4. context_precision - a measure of how relevant the retrieved context is to the question. Conveys quality of the retrieval pipeline.\n",
        "\n",
        "**Note:** *by default these metrics are using OpenAI's API to compute the score. If you using this metric make sure you set the environment key `OPENAI_API_KEY` with your API key. You can also try other LLMs for evaluation, check the [llm guide](./guides/llms.ipynb) to learn more*\n",
        "\n",
        "If you're interested in learning more, feel free to check the [docs](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8d6ecd5a",
      "metadata": {
        "id": "8d6ecd5a"
      },
      "source": [
        "## Evaluation\n",
        "\n",
        "Running the evalutation is as simple as calling evaluate on the `Dataset` with the metrics of your choice."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "22eb6f97",
      "metadata": {
        "id": "22eb6f97",
        "outputId": "bcc5e271-06f8-4551-c8aa-e43e95bdf929",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [context_precision]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:02<00:00,  2.17s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [faithfulness]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:03<00:00,  3.79s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [answer_relevancy]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:01<00:00,  1.21s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [context_recall]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:04<00:00,  4.12s/it]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'ragas_score': 0.2931, 'context_precision': 0.4706, 'faithfulness': 0.6667, 'answer_relevancy': 0.9769, 'context_recall': 0.1111}"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "from ragas import evaluate\n",
        "\n",
        "result = evaluate(\n",
        "    fiqa_eval[\"baseline\"].select(range(1)),\n",
        "    metrics=[\n",
        "        context_precision,\n",
        "        faithfulness,\n",
        "        answer_relevancy,\n",
        "        context_recall\n",
        "    ],\n",
        ")\n",
        "\n",
        "result"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a2dc0ec2",
      "metadata": {
        "id": "a2dc0ec2"
      },
      "source": [
        "and there you have the it, all the scores you need. `ragas_score` gives you a single metric that you can use while the other onces measure the different parts of your pipeline.\n",
        "\n",
        "now if we want to dig into the results and figure out examples where your pipeline performed worse or really good you can easily convert it into a pandas array and use your standard analytics tools too!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "8686bf53",
      "metadata": {
        "id": "8686bf53",
        "outputId": "0ded27a0-2d51-48c3-de57-a71f02e22970",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 115
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            question  \\\n",
              "0  How to deposit a cheque issued to an associate...   \n",
              "\n",
              "                                            contexts  \\\n",
              "0  [Just have the associate sign the back and the...   \n",
              "\n",
              "                                              answer  \\\n",
              "0  \\nThe best way to deposit a cheque issued to a...   \n",
              "\n",
              "                                       ground_truths  context_precision  \\\n",
              "0  [Have the check reissued to the proper payee.J...           0.470588   \n",
              "\n",
              "   faithfulness  answer_relevancy  context_recall  \n",
              "0      0.666667          0.976913        0.111111  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-263074ab-ae68-417a-96c6-9eca49057380\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truths</th>\n",
              "      <th>context_precision</th>\n",
              "      <th>faithfulness</th>\n",
              "      <th>answer_relevancy</th>\n",
              "      <th>context_recall</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>How to deposit a cheque issued to an associate...</td>\n",
              "      <td>[Just have the associate sign the back and the...</td>\n",
              "      <td>\\nThe best way to deposit a cheque issued to a...</td>\n",
              "      <td>[Have the check reissued to the proper payee.J...</td>\n",
              "      <td>0.470588</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>0.976913</td>\n",
              "      <td>0.111111</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-263074ab-ae68-417a-96c6-9eca49057380')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-263074ab-ae68-417a-96c6-9eca49057380 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-263074ab-ae68-417a-96c6-9eca49057380');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "df = result.to_pandas()\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f668fce1",
      "metadata": {
        "id": "f668fce1"
      },
      "source": [
        "And thats it!\n",
        "\n",
        "You can check out the [ragas in action] notebook to get a feel of what is like to use it while trying to improve your pipelines.\n",
        "\n",
        "if you have any suggestion/feedbacks/things your not happy about, please do share it in the [issue section](https://github.com/explodinggradients/ragas/issues). We love hearing from you 😁"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}