Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluation Dataset mentioned in Hugging GPT paper is not available #208

Open
ssdasgupta opened this issue Jun 8, 2023 · 1 comment
Open

Comments

@ssdasgupta
Copy link

As mentioned in the paper - "Furthermore, we also invite some expert annotators to label task planning for some complex requests (46 examples) as a high-quality human annotated dataset. We also plan to further improve the quality and quantity of this dataset to better help us to evaluate the LLM capability in planning, which leaves as future work.", are you planning to release the evaluation dataset? Or if it is there already in the repository, could you send me the folder location?

Thanks.

@StillKeepTry
Copy link
Collaborator

@ssdasgupta We are currently working with our labeling teams to iteratively improve the quality of this dataset and our legal team to ensure compliance of the dataset release. We will release a work about this dataset in the future. Please be patient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants