lab/L5-Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "52824b89-532a-4e54-87e9-1410813cd39e",
   "metadata": {},
   "source": [
    "# LangChain: Evaluation\n",
    "\n",
    "## Outline:\n",
    "\n",
    "* Example generation\n",
    "* Manual evaluation (and debuging)\n",
    "* LLM-assisted evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b7ed03ed-1322-49e3-b2a2-33e94fb592ef",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28008949",
   "metadata": {},
   "source": [
    "## Create our QandA application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "974acf8e-8f88-42de-88f8-40a82cb58e8b",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.document_loaders import CSVLoader\n",
    "from langchain.indexes import VectorstoreIndexCreator\n",
    "from langchain.vectorstores import DocArrayInMemorySearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9ec1106d",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "file = 'OutdoorClothingCatalog_1000.csv'\n",
    "loader = CSVLoader(file_path=file)\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b31c218f",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "index = VectorstoreIndexCreator(\n",
    "    vectorstore_cls=DocArrayInMemorySearch\n",
    ").from_loaders([loader])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a2006054",
   "metadata": {
    "height": 183
   },
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(temperature = 0.0)\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm, \n",
    "    chain_type=\"stuff\", \n",
    "    retriever=index.vectorstore.as_retriever(), \n",
    "    verbose=True,\n",
    "    chain_type_kwargs = {\n",
    "        \"document_separator\": \"<<<<>>>>>\"\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "791ebd73",
   "metadata": {},
   "source": [
    "### Coming up with test datapoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fb04a0f9",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=\": 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.\", metadata={'source': 'OutdoorClothingCatalog_1000.csv', 'row': 10})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fe4a88c2",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=': 11\\nname: Ultra-Lofty 850 Stretch Down Hooded Jacket\\ndescription: This technical stretch down jacket from our DownTek collection is sure to keep you warm and comfortable with its full-stretch construction providing exceptional range of motion. With a slightly fitted style that falls at the hip and best with a midweight layer, this jacket is suitable for light activity up to 20° and moderate activity up to -30°. The soft and durable 100% polyester shell offers complete windproof protection and is insulated with warm, lofty goose down. Other features include welded baffles for a no-stitch construction and excellent stretch, an adjustable hood, an interior media port and mesh stash pocket and a hem drawcord. Machine wash and dry. Imported.', metadata={'source': 'OutdoorClothingCatalog_1000.csv', 'row': 11})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[11]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d548aef",
   "metadata": {},
   "source": [
    "### Hard-coded examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c2d59bf2",
   "metadata": {
    "height": 234
   },
   "outputs": [],
   "source": [
    "examples = [\n",
    "    {\n",
    "        \"query\": \"Do the Cozy Comfort Pullover Set\\\n",
    "        have side pockets?\",\n",
    "        \"answer\": \"Yes\"\n",
    "    },\n",
    "    {\n",
    "        \"query\": \"What collection is the Ultra-Lofty \\\n",
    "        850 Stretch Down Hooded Jacket from?\",\n",
    "        \"answer\": \"The DownTek collection\"\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ce3e4f",
   "metadata": {},
   "source": [
    "### LLM-Generated examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d44f8376",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "from langchain.evaluation.qa import QAGenerateChain\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "34e87816",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "example_gen_chain = QAGenerateChain.from_llm(ChatOpenAI())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "62abae09",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "new_examples = example_gen_chain.apply_and_parse(\n",
    "    [{\"doc\": t} for t in data[:5]]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "97ab28b5",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': \"What is the weight of each pair of Women's Campside Oxfords?\",\n",
       " 'answer': \"The approximate weight of each pair of Women's Campside Oxfords is 1 lb. 1 oz.\"}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_examples[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0ebe4228",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=\": 0\\nname: Women's Campside Oxfords\\ndescription: This ultracomfortable lace-to-toe Oxford boasts a super-soft canvas, thick cushioning, and quality construction for a broken-in feel from the first time you put them on. \\n\\nSize & Fit: Order regular shoe size. For half sizes not offered, order up to next whole size. \\n\\nSpecs: Approx. weight: 1 lb.1 oz. per pair. \\n\\nConstruction: Soft canvas material for a broken-in feel and look. Comfortable EVA innersole with Cleansport NXT® antimicrobial odor control. Vintage hunt, fish and camping motif on innersole. Moderate arch contour of innersole. EVA foam midsole for cushioning and support. Chain-tread-inspired molded rubber outsole with modified chain-tread pattern. Imported. \\n\\nQuestions? Please contact us for any inquiries.\", metadata={'source': 'OutdoorClothingCatalog_1000.csv', 'row': 0})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faf25f2f",
   "metadata": {},
   "source": [
    "### Combine examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ada2a3fc",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "examples += new_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9cdf5cf5",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The Cozy Comfort Pullover Set, Stripe does have side pockets.'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.run(examples[0][\"query\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f3cb08",
   "metadata": {},
   "source": [
    "## Manual Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fcaf622e",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "import langchain\n",
    "langchain.debug = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8a142638",
   "metadata": {
    "height": 30,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"query\": \"Do the Cozy Comfort Pullover Set have side pockets?\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain] Entering Chain run with input:\n",
      "\u001b[0m[inputs]\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"question\": \"Do the Cozy Comfort Pullover Set have side pockets?\",\n",
      "  \"context\": \": 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.<<<<>>>>>: 73\\nname: Cozy Cuddles Knit Pullover Set\\ndescription: Perfect for lounging, this knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out. \\n\\nSize & Fit \\nPants are Favorite Fit: Sits lower on the waist. \\nRelaxed Fit: Our most generous fit sits farthest from the body. \\n\\nFabric & Care \\nIn the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features \\nRelaxed fit top with raglan sleeves and rounded hem. \\nPull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg. \\nImported.<<<<>>>>>: 151\\nname: Cozy Quilted Sweatshirt\\ndescription: Our sweatshirt is an instant classic with its great quilted texture and versatile weight that easily transitions between seasons. With a traditional fit that is relaxed through the chest, sleeve, and waist, this pullover is lightweight enough to be worn most months of the year. The cotton blend fabric is super soft and comfortable, making it the perfect casual layer. To make dressing easy, this sweatshirt also features a snap placket and a heritage-inspired Mt. Katahdin logo patch. For care, machine wash and dry. Imported.<<<<>>>>>: 265\\nname: Cozy Workout Vest\\ndescription: For serious warmth that won't weigh you down, reach for this fleece-lined vest, which provides you with layering options whether you're inside or outdoors.\\nSize & Fit\\nRelaxed Fit. Falls at hip.\\nFabric & Care\\nSoft, textured fleece lining. Nylon shell. Machine wash and dry. \\nAdditional Features \\nTwo handwarmer pockets. Knit side panels stretch for a more flattering fit. Shell fabric is treated to resist water and stains. Imported.\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain > 4:llm:ChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"System: Use the following pieces of context to answer the users question. \\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\\n----------------\\n: 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.<<<<>>>>>: 73\\nname: Cozy Cuddles Knit Pullover Set\\ndescription: Perfect for lounging, this knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out. \\n\\nSize & Fit \\nPants are Favorite Fit: Sits lower on the waist. \\nRelaxed Fit: Our most generous fit sits farthest from the body. \\n\\nFabric & Care \\nIn the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features \\nRelaxed fit top with raglan sleeves and rounded hem. \\nPull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg. \\nImported.<<<<>>>>>: 151\\nname: Cozy Quilted Sweatshirt\\ndescription: Our sweatshirt is an instant classic with its great quilted texture and versatile weight that easily transitions between seasons. With a traditional fit that is relaxed through the chest, sleeve, and waist, this pullover is lightweight enough to be worn most months of the year. The cotton blend fabric is super soft and comfortable, making it the perfect casual layer. To make dressing easy, this sweatshirt also features a snap placket and a heritage-inspired Mt. Katahdin logo patch. For care, machine wash and dry. Imported.<<<<>>>>>: 265\\nname: Cozy Workout Vest\\ndescription: For serious warmth that won't weigh you down, reach for this fleece-lined vest, which provides you with layering options whether you're inside or outdoors.\\nSize & Fit\\nRelaxed Fit. Falls at hip.\\nFabric & Care\\nSoft, textured fleece lining. Nylon shell. Machine wash and dry. \\nAdditional Features \\nTwo handwarmer pockets. Knit side panels stretch for a more flattering fit. Shell fabric is treated to resist water and stains. Imported.\\nHuman: Do the Cozy Comfort Pullover Set have side pockets?\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain > 4:llm:ChatOpenAI] [1.73s] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"The Cozy Comfort Pullover Set, Stripe does have side pockets.\",\n",
      "        \"generation_info\": null,\n",
      "        \"message\": {\n",
      "          \"content\": \"The Cozy Comfort Pullover Set, Stripe does have side pockets.\",\n",
      "          \"additional_kwargs\": {},\n",
      "          \"example\": false\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 628,\n",
      "      \"completion_tokens\": 14,\n",
      "      \"total_tokens\": 642\n",
      "    },\n",
      "    \"model_name\": \"gpt-3.5-turbo\"\n",
      "  }\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain] [1.73s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"text\": \"The Cozy Comfort Pullover Set, Stripe does have side pockets.\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain] [1.73s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"output_text\": \"The Cozy Comfort Pullover Set, Stripe does have side pockets.\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA] [1.97s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"result\": \"The Cozy Comfort Pullover Set, Stripe does have side pockets.\"\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The Cozy Comfort Pullover Set, Stripe does have side pockets.'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.run(examples[0][\"query\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b3d6bef0",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "# Turn off the debug mode\n",
    "langchain.debug = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bdbdce",
   "metadata": {},
   "source": [
    "## LLM assisted evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a4dca05a",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "predictions = qa.apply(examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6012a3e0",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "from langchain.evaluation.qa import QAEvalChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "724b1c0b",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(temperature=0)\n",
    "eval_chain = QAEvalChain.from_llm(llm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8b46ae55",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "graded_outputs = eval_chain.evaluate(examples, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "3437cfbe",
   "metadata": {
    "height": 149
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example 0:\n",
      "Question: Do the Cozy Comfort Pullover Set have side pockets?\n",
      "Real Answer: Yes\n",
      "Predicted Answer: The Cozy Comfort Pullover Set, Stripe does have side pockets.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 1:\n",
      "Question: What collection is the Ultra-Lofty 850 Stretch Down Hooded Jacket from?\n",
      "Real Answer: The DownTek collection\n",
      "Predicted Answer: The Ultra-Lofty 850 Stretch Down Hooded Jacket is from the DownTek collection.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 2:\n",
      "Question: What is the weight of each pair of Women's Campside Oxfords?\n",
      "Real Answer: The approximate weight of each pair of Women's Campside Oxfords is 1 lb. 1 oz.\n",
      "Predicted Answer: The weight of each pair of Women's Campside Oxfords is approximately 1 lb. 1 oz.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 3:\n",
      "Question: What are the dimensions of the small and medium Recycled Waterhog Dog Mat?\n",
      "Real Answer: The dimensions of the small Recycled Waterhog Dog Mat are 18\" x 28\" and the dimensions of the medium Recycled Waterhog Dog Mat are 22.5\" x 34.5\".\n",
      "Predicted Answer: The small Recycled Waterhog Dog Mat has dimensions of 18\" x 28\" and the medium size has dimensions of 22.5\" x 34.5\".\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 4:\n",
      "Question: What are some features of the Infant and Toddler Girls' Coastal Chill Swimsuit?\n",
      "Real Answer: The swimsuit features bright colors, ruffles, and exclusive whimsical prints. It is made of four-way-stretch and chlorine-resistant fabric, ensuring that it keeps its shape and resists snags. The swimsuit is also UPF 50+ rated, providing the highest rated sun protection possible by blocking 98% of the sun's harmful rays. The crossover no-slip straps and fully lined bottom ensure a secure fit and maximum coverage. Finally, it can be machine washed and line dried for best results.\n",
      "Predicted Answer: The Infant and Toddler Girls' Coastal Chill Swimsuit is a two-piece swimsuit with bright colors, ruffles, and exclusive whimsical prints. It is made of four-way-stretch and chlorine-resistant fabric that keeps its shape and resists snags. The swimsuit has UPF 50+ rated fabric that provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The crossover no-slip straps and fully lined bottom ensure a secure fit and maximum coverage. It is machine washable and should be line dried for best results.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 5:\n",
      "Question: What is the fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts?\n",
      "Real Answer: The body of the Refresh Swimwear V-Neck Tankini Contrasts is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.\n",
      "Predicted Answer: The Refresh Swimwear V-Neck Tankini Contrasts is made of 82% recycled nylon with 18% Lycra® spandex for the body and 90% recycled nylon with 10% Lycra® spandex for the lining.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 6:\n",
      "Question: What is the fabric composition of the EcoFlex 3L Storm Pants?\n",
      "Real Answer: The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.\n",
      "Predicted Answer: The fabric composition of the EcoFlex 3L Storm Pants is 100% nylon, exclusive of trim.\n",
      "Predicted Grade: CORRECT\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, eg in enumerate(examples):\n",
    "    print(f\"Example {i}:\")\n",
    "    print(\"Question: \" + predictions[i]['query'])\n",
    "    print(\"Real Answer: \" + predictions[i]['answer'])\n",
    "    print(\"Predicted Answer: \" + predictions[i]['result'])\n",
    "    print(\"Predicted Grade: \" + graded_outputs[i]['text'])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce542854",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be5b2aae",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "319798ba",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89a504ad",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dedd758b",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36885b20",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65c6cfb6",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ad3c7cc",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26ac493e",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f94cdacd",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}