Testing the custom application endpoint that is using AzureOpenAI in the backend is failing using the code snippet provided by giskard to wrap the custom endpoint.


The application endpoint does not require any API key at this time and the giskard.scan requires to pass an OPENAI_API_KEY. If a dummy key is set as environment variable, the code fails with API connection error.

import os
import giskard
from giskard.llm import set_llm_model

def call_my_api():
  return [requests.post('https://api-url.com/v1/chatbot/request', json=json_data, headers='Content-type': 'application/json'})]

set_llm_model("gpt4")

# Create a giskard.Model object. Don’t forget to fill the `name` and `description`
giskard_model = giskard.Model(
    call_my_api,  # our langchain.LLMChain object
    model_type="text_generation",
    name="My Generic Assistant",
    description="A generic assistant that kindly answers questions.",
    feature_names=["messages"],
)

scan_results = giskard.scan(giskard_model)
display(scan_results)  # in your notebook

Error:

---------------------------------------------------------------------------
APIConnectionError                        Traceback (most recent call last)
Cell In[22], line 1
----> 1 scan_results = giskard.scan(giskard_model)
      2 display(scan_results)  # in your notebook

File /opt/homebrew/lib/python3.11/site-packages/giskard/scanner/__init__.py:64, in scan(model, dataset, features, params, only, verbose, raise_exceptions)
     35 """Automatically detects model vulnerabilities.
     36 
     37 See :class:`Scanner` for more details.
   (...)
     61     A scan report object containing the results of the scan.
     62 """
     63 scanner = Scanner(params, only=only)
---> 64 return scanner.analyze(
     65     model, dataset=dataset, features=features, verbose=verbose, raise_exceptions=raise_exceptions
     66 )

File /opt/homebrew/lib/python3.11/site-packages/giskard/scanner/scanner.py:100, in Scanner.analyze(self, model, dataset, features, verbose, raise_exceptions)
     77 """Runs the analysis of a model and dataset, detecting issues.
     78 
     79 Parameters
   (...)
     96     A report object containing the detected issues and other information.
     97 """
...
    988     'HTTP Request: %s %s "%i %s"', request.method, request.url, response.status_code, response.reason_phrase
    989 )
    991 try:

APIConnectionError: Connection error.