Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support case when model returns empty Actions list #779

Merged
merged 6 commits into from
May 15, 2024

Conversation

andrewfrench
Copy link
Member

@andrewfrench andrewfrench commented May 14, 2024

This PR addresses a reported issue when parsing a list of Actions while initializing an ActionsSubtask.

In some cases, Models include an empty list of Actions in their responses. We currently raise a SchemaError this happens:

[05/11/24 12:58:44] INFO     ToolkitTask 9f94e60e40d04516b9cce029eab11a25                                                                                     
                             Input: hi my name is jason                                                                                                       
[05/11/24 12:58:47] ERROR    Subtask 9f94e60e40d04516b9cce029eab11a25                                                                                         
                             Invalid action JSON: Array item count 0 is less than minimum count of 1.                                                         
                    INFO     Subtask 3352ba0de32041969bbb7d9781a8c236                                                                                         
                             Thought: The user has introduced himself as Jason. There is no specific question or request, so I will acknowledge the           
                             introduction and ask how I can assist him.                                                                                       
                             Actions: []                                                                                                                      
                             Answer: Hi Jason! How can I assist you today?                                                                                    
                    INFO     Subtask 3352ba0de32041969bbb7d9781a8c236                                                                                         
                             Response: Action JSON validation error: Array item count 0 is less than minimum count of 1.                                      

If the model does not return an Actions list at all, we do not attempt to parse actions and do not raise an exception. In both cases the model is electing to take no actions. If "no actions" is a valid response, allowing one case and disallowing another is inconsistent behavior.

@andrewfrench andrewfrench force-pushed the french/240514/empty-actions-list branch from 585b903 to ad5d50c Compare May 14, 2024 16:50
@andrewfrench andrewfrench marked this pull request as draft May 14, 2024 16:57
@andrewfrench andrewfrench marked this pull request as ready for review May 14, 2024 18:14
@andrewfrench andrewfrench requested a review from vasinov May 14, 2024 18:14
Copy link
Member

@vasinov vasinov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you test it outside of unit tests with a real LLM? I remember there potentially being some quirk for an empty actions list.

vasinov
vasinov previously approved these changes May 14, 2024
Comment on lines +201 to +205
self.__parse_actions(actions_matches)

# If there are no actions to take but an answer is provided, set the answer as the output.
if len(self.actions) == 0 and self.output is None and len(answer_matches) > 0:
self.output = TextArtifact(answer_matches[-1])
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're currently testing which branch to take based on whether or not actions_matches has elements or not. If an empty list is provided, there's a match and we try to parse the actions. If, while parsing, we decide there actually aren't any actions, we continue execution without any actions or output. This is interpreted by ToolkitTask as a deviation from the ReAct prompt and the Subtask's raw input is set as its output. We don't want that, we want to extract the answer just like we would if there were no actions_matches.

This change updates the logic to first try parsing actions. If there are no actions_matches, or if an empty set is provided by the model, this results in an empty self.actions list. If the model chooses to take no actions, we set the output to the answer as we would expect, standardizing behavior.

if len(self.actions) == 0 and self.output is None and len(answer_matches) > 0:
self.output = TextArtifact(answer_matches[-1])

def __parse_actions(self, actions_matches: list[str]) -> None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for refactoring this out. Long overdue!

@andrewfrench andrewfrench merged commit 72e3022 into dev May 15, 2024
7 checks passed
@andrewfrench andrewfrench deleted the french/240514/empty-actions-list branch May 15, 2024 17:42
hkhajgiwale pushed a commit to hkhajgiwale/griptape that referenced this pull request May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants