Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset v2 discussion & feedback #88

Open
natolambert opened this issue Mar 26, 2024 · 1 comment
Open

Dataset v2 discussion & feedback #88

natolambert opened this issue Mar 26, 2024 · 1 comment
Labels
question Further information is requested

Comments

@natolambert
Copy link
Collaborator

Hey! Post any questions or complaints on the dataset. We'll log our internal goals and limitations here too.

  1. It was pointed out by Rishabh Agarwal that the PRM Math subset has two structural issues. 1) we added newlines to the human reference answers (debatably could be called a bug). 2) with GPT4 always as rejected, some models may be biased there.
@natolambert natolambert added the question Further information is requested label Mar 26, 2024
@natolambert natolambert pinned this issue Mar 26, 2024
@natolambert
Copy link
Collaborator Author

Idea: Now that we have a bunch of RMs, we can see if there are any datapoints that the models all think are wrong and double check our labels for future releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@natolambert