Error - benchllm with Azure OpenAi service

[FJLopezGarcia]

Hello,
I am trying to use benchllm with Azure OpenAi service. I have updated the Chat example you have but I am getting this error:
InvalidRequestError: The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.

But sometimes the code works and perfom the evaluation properly:

below you have the code I am running. Do you know how to fix it? Thanks!!

`
from dotenv import load_dotenv
import os
import openai
import benchllm
from benchllm.input_types import ChatInput

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_DEPLOYMENT_ENDPOINT = os.getenv("OPENAI_DEPLOYMENT_ENDPOINT")

openai.api_type = "azure"
openai.api_version = "2023-07-01-preview"
openai.api_base = OPENAI_DEPLOYMENT_ENDPOINT
openai.api_key = OPENAI_API_KEY

def chat(messages: ChatInput):
try:
response = openai.ChatCompletion.create(
engine="Chat358K",
messages=messages
)
content = response.choices[0].message.content.strip()
print("Model Response:", content)
return content
except openai.OpenAIError as e:
print("OpenAI Error:", e)
return str(e)

@benchllm.test(suite="./tests")
def gpt_3_5(input: ChatInput):
print("input:", input)
return chat(input)
`

[FJLopezGarcia]

Do you have any update? Did you have the chance to review this issue? Thanks a lot

[Elio-Khater]

Hello,
We are trying to connect BenchLLM to Azure Open Ai, and we are encountering the exact same problem that @FJLopezGarcia is facing. Is there any updates on the matter?