You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For the following structures, the documentation differs from what we've tested. Happy to make a PR to update the docs, if someone can confirm the following. I'm only using Python custom assertions and providers.
The Python documentation says that output must be Optional[str]. However, the Typescript docs says it could also be a dictionary, and we confirmed that works in Python.
The text was updated successfully, but these errors were encountered:
In the Typescript definition, GradingResult expects a numeric score and a non-null reason. We should treat that as the source of truth because the rest of the code depends on these types. So I would consider this a validation issue in the Python assertion (maybe javascript too).
For ProviderResponse, it does look like output can be an object. So the Python docs are wrong.
For the following structures, the documentation differs from what we've tested. Happy to make a PR to update the docs, if someone can confirm the following. I'm only using Python custom assertions and providers.
GradingResult (Typescript reference)
score
documented to be afloat
between 0 and 1, but we've been able to return a dictionary.reason
is allowed to beNone
( I guessnull
in Typescript)ProviderResponse (Typescript reference)
output
must beOptional[str]
. However, the Typescript docs says it could also be a dictionary, and we confirmed that works in Python.The text was updated successfully, but these errors were encountered: