giskard.scan is failing for azure openai & langchain

[AshishGoelTR]

Issue Type

Bug

Source

source

Giskard Library Version

2.10.0

Giskard Hub Version

N/A

OS Platform and Distribution

No response

Python version

3.11

Installed python packages

No response

Current Behaviour?

I am running the sample code from here (https://docs.giskard.ai/en/latest/open_source/scan/scan_llm/index.html) and using an Azure OpenAI with langchain but the scan fails with the error below.

cc:  @GianRomani @AndreeaIugaTR

Standalone code OR list down the steps to reproduce the issue

Use the link below to access the jupyter notebook:
https://goonlinetools.com/snapshot/code/#m3ar9v8ihyqpr9qg3i2eos

Relevant log output

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[9], line 1
----> 1 report = gsk.scan(giskard_model, demo_dataset, only="jailbreak", raise_exceptions=True)

File [/opt/homebrew/lib/python3.11/site-packages/giskard/scanner/__init__.py:64](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/lib/python3.11/site-packages/giskard/scanner/__init__.py:64), in scan(model, dataset, features, params, only, verbose, raise_exceptions)
     35 """Automatically detects model vulnerabilities.
     36 
     37 See :class:`Scanner` for more details.
   (...)
     61     A scan report object containing the results of the scan.
     62 """
     63 scanner = Scanner(params, only=only)
---> 64 return scanner.analyze(
     65     model, dataset=dataset, features=features, verbose=verbose, raise_exceptions=raise_exceptions
     66 )

File [/opt/homebrew/lib/python3.11/site-packages/giskard/scanner/scanner.py:123](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/lib/python3.11/site-packages/giskard/scanner/scanner.py:123), in Scanner.analyze(self, model, dataset, features, verbose, raise_exceptions)
    121 with warnings.catch_warnings():
    122     warnings.simplefilter("ignore")
--> 123     issues, errors = self._run_detectors(
    124         detectors, model, dataset, features, verbose=verbose, raise_exceptions=raise_exceptions
    125     )
    127 issues = self._postprocess(issues)
    129 # Scan completed
...
     35 if not self.config.punctuation_sensitive:
---> 36     text = text.translate(str.maketrans("", "", string.punctuation))
     37 return text

AttributeError: 'AIMessage' object has no attribute 'translate'

-------------------------------------

[alexcombessie]

Hey @kevinmessiaen - Do you have any news on this issue?

[kevinmessiaen]

It seems that the model prediction method passed in the scan return a object of type AIMessage from langchain instead of a str.

Please make sure to map AIMessage to str in the prediction_function