-
Notifications
You must be signed in to change notification settings - Fork 131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support case when model returns empty Actions list #779
Conversation
585b903
to
ad5d50c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you test it outside of unit tests with a real LLM? I remember there potentially being some quirk for an empty actions list.
self.__parse_actions(actions_matches) | ||
|
||
# If there are no actions to take but an answer is provided, set the answer as the output. | ||
if len(self.actions) == 0 and self.output is None and len(answer_matches) > 0: | ||
self.output = TextArtifact(answer_matches[-1]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're currently testing which branch to take based on whether or not actions_matches
has elements or not. If an empty list is provided, there's a match and we try to parse the actions. If, while parsing, we decide there actually aren't any actions, we continue execution without any actions or output. This is interpreted by ToolkitTask
as a deviation from the ReAct prompt and the Subtask
's raw input is set as its output. We don't want that, we want to extract the answer just like we would if there were no actions_matches
.
This change updates the logic to first try parsing actions. If there are no actions_matches
, or if an empty set is provided by the model, this results in an empty self.actions
list. If the model chooses to take no actions, we set the output to the answer as we would expect, standardizing behavior.
if len(self.actions) == 0 and self.output is None and len(answer_matches) > 0: | ||
self.output = TextArtifact(answer_matches[-1]) | ||
|
||
def __parse_actions(self, actions_matches: list[str]) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for refactoring this out. Long overdue!
This PR addresses a reported issue when parsing a list of Actions while initializing an ActionsSubtask.
In some cases, Models include an empty list of Actions in their responses. We currently raise a SchemaError this happens:
If the model does not return an Actions list at all, we do not attempt to parse actions and do not raise an exception. In both cases the model is electing to take no actions. If "no actions" is a valid response, allowing one case and disallowing another is inconsistent behavior.