Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AzureOpenAIEmbeddings with custom endpoint fails #897

Open
a-romero opened this issue Apr 24, 2024 · 0 comments
Open

AzureOpenAIEmbeddings with custom endpoint fails #897

a-romero opened this issue Apr 24, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@a-romero
Copy link

[ x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
Not able to use Embeddings based metrics as I'm getting the message above. I am using a custom Azure OpenAI endpoint and leverage langchain, which otherwise works perfectly well with my setup.

I wonder if this might be a regression of:
#545

Ragas version: 0.1.7
Python version: 3.9.6

Code to Reproduce

from ragas.metrics import (
    answer_similarity,
    answer_correctness
)

from ragas import evaluate, adapt


chat_model = AzureChatOpenAI(
    openai_api_base=xxxxxx,
    openai_api_version=xxxxxx,
    deployment_name=xxxxxx,
    openai_api_key=xxxxxx,
    max_tokens=5000,
    temperature=0.0
    )

embeddings_model = AzureOpenAIEmbeddings(
    azure_deployment=xxxxxx,
    azure_endpoint=xxxxxx,
    api_version=xxxxxx,
    api_key=xxxxxx
)

metrics = [
    answer_similarity,
    answer_correctness
]

result = evaluate(
    dataset_evaluation_1,
    metrics=metrics,
    llm=chat_model,
    embeddings=embeddings_model
)

Error trace
The issue seems to be that it is trying to connect to the Azure OpenAI stardard endpoint (to which I don't have network access) rather than the one configured in the code:

OSError: Tunnel connection failed: 407 Proxy Authentication Required.
The above exception was the direct cause of the following exception:
urllib3.exceptions.ProxyError: ('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Authentication Required'))

urlib3.exception.MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Authentication Required'))

Additional context
Have tried wrapping the model in LangchainEmbeddingsWrapper and that still results in the 407 because it tries to access the default Azure OpenAI endpoint (openaipublic.blob.core.windows.net) rather than my custom one.

Have also tried passing the models per metric:

for metric in metrics:
   metric.embeddings=embeddings_model

and that results in the error:
'AzureOpenAIEmbeddings' object has no attribute 'embed_texts'

@a-romero a-romero added the bug Something isn't working label Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant