Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation for data structures GradingResult and ProviderResponse #769

Closed
efung opened this issue May 9, 2024 · 1 comment
Closed

Comments

@efung
Copy link
Contributor

efung commented May 9, 2024

For the following structures, the documentation differs from what we've tested. Happy to make a PR to update the docs, if someone can confirm the following. I'm only using Python custom assertions and providers.

GradingResult (Typescript reference)

  • score documented to be a float between 0 and 1, but we've been able to return a dictionary.
  • reason is allowed to be None ( I guess null in Typescript)

ProviderResponse (Typescript reference)

  • The Python documentation says that output must be Optional[str]. However, the Typescript docs says it could also be a dictionary, and we confirmed that works in Python.
@typpo
Copy link
Collaborator

typpo commented May 11, 2024

Thanks for catching these errors!

In the Typescript definition, GradingResult expects a numeric score and a non-null reason. We should treat that as the source of truth because the rest of the code depends on these types. So I would consider this a validation issue in the Python assertion (maybe javascript too).

For ProviderResponse, it does look like output can be an object. So the Python docs are wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants