Velvet Excel

Run and measure Vertex AI Search Agent accuracy using Excel.

Note

This is not an official Google product

How it works

1. Install the Excel Addin

Follow the Instructions

2. Creating the Placeholder tables and create access token

3. Put in the Data in Test Table and Run the Tests

Why?

• 🧑‍💻 Run your search acceptance test cases without leaving Excel.
• ✅ Enables Business and Compliance teams to measure search accuracy without any training or developer skills using the tool they use every day.
• 🗒️ Allows comparing of different test runs using different settings to determine the correct values for a specific use case.
• 🚀 Accelerates search application validation from weeks/months to hours.
• 🤖 Coz testing manually is hard, boring and no fun!

Features

• Excel Office Add-on. Works with Excel Web (Microsoft 365) and Excel Desktop.

• Creates a test case table that measures accuracy for the following metrics:

  • Summary Match (True/False): Semantically match the actual summary returned with the expected summary. This uses the PaLM2 model to measure semantic similarity using the following prompt.
  • First Link Match (True/False): The actual first document link returned matches the expected document link.
  • Link in Top 2: Match if the top 2 actual document links returned are in the top 2 expected links

• Ability to run hundreds of test cases in batch mode.

• Ability to change the following Vertex AI Search Agent settings from within Excel:

  • Search Types (any one of):
    • Extractive Answers: maxExtractiveAnswerCount (1-5)
    • Extractive Segments: maxExtractiveSegmentCount (1-5)
    • Snippets: maxSnippetCount (1-5)
  • Search Summary Settings:

    • Summary Result Count: (1-5)

    • Use Semantic Chunks: (True/False)

    • Summary Model gemini-1.0-pro-002/answer_gen/v1 (default)

    • Ignore Adversarial Query: (True/False)

    • Ignore Non Summary Seeking Query: (True/False)

  • Other
    • Summary Match Additional Prompt: Additional prompt to pass to the PaLM2 model for semantic similarity matching. Useful when the need to match exact monetary quantities (millions, billions, etc).

Installation

Only sideloading is supported at this time.

Excel Web (Microsoft 365)

• Download the manfest.xml file to your local machine.
• Open Excel in your browser.
• Click on the "Add-ins" button and select "More Add-Ins".

• Click on the "Upload My Add-in" link

• Select the manifest.xml file from your local machine and click on the "Open" button.

• You should now see the Velvet Add-in listed in the Add-ins menu.

To remove it just clear your browser cache.

Excel Desktop

• Download the manfest.xml file to your local machine.
• Place the manifest.xml file in the following location:
  • Windows: Follow instructions here.
  • Mac:
    • Use Finder to sideload the manifest file. Open Finder and then enter Command+Shift+G to open the Go to folder dialog.
    • Navigate to the following location: /Users/your-mac-username/Library/Containers/Microsoft Excel/Data/Documents/wef
    • If the wef folder doesn't exist on your computer, create it.
    • Save the manifest.xml file in the wef folder.
• Open Excel.
• Confirm that the Velvet Add-in is listed in the Home Ribbon.

Development

Requirements

• NodeJS 20+
• gcloud sdk