Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GSK-3513] Fix RAGAS metric computation #1925

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

pierlj
Copy link
Contributor

@pierlj pierlj commented May 10, 2024

According to this issue, some RAGAS metrics are not properly computed by RAGAS (this includes context recall, context precision and faithfulness).

To fix it:

  • added a retrieved_documents argument to the evaluate method
  • allowed the answer_fn to return retrieved documents alongside the answer to a question

Copy link

linear bot commented May 10, 2024

Copy link

sentry-io bot commented May 10, 2024

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: giskard/rag/evaluate.py

Function Unhandled Issue
evaluate ValueError: Must provide either the api_version argument or the OPENAI_API_VERSION environment variable ...
Event Count: 1
evaluate TypeError: string indices must be integers, not 'str' main in model_pr...
Event Count: 1
evaluate NameError: name 'doc_chat' is not defined _main...
Event Count: 1
evaluate AttributeError: 'EvaluateRAG' object has no attribute 'custom_model' src.evalua...
Event Count: 1

Did you find this useful? React with a 👍 or 👎

@pierlj pierlj requested a review from mattbit May 10, 2024 11:05
@mattbit
Copy link
Member

mattbit commented May 10, 2024

@pierlj looks good, can you add a test on the ragas metrics to make sure they are calculated correctly?

Copy link
Member

@mattbit mattbit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for me, @pierlj do you want to make a last check?

@mattbit mattbit added the Lockfile Temporary label to update pdm.lock label May 21, 2024
@pierlj
Copy link
Contributor Author

pierlj commented May 21, 2024

Good for me, @pierlj do you want to make a last check?

Yep, I will have a look

@github-actions github-actions bot removed the Lockfile Temporary label to update pdm.lock label May 21, 2024
@mattbit mattbit enabled auto-merge May 21, 2024 09:46
@mattbit mattbit self-assigned this May 24, 2024
Copy link

sonarcloud bot commented May 24, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants