Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[R-249] metrics reproducibility: better statement generation #925

Closed
shahules786 opened this issue Apr 30, 2024 · 0 comments
Closed

[R-249] metrics reproducibility: better statement generation #925

shahules786 opened this issue Apr 30, 2024 · 0 comments
Assignees
@shahules786
Labels
High priority Created by Linear-GitHub Sync

Comments

@shahules786
Copy link
Member

From SyncLinear.com | R-249
@shahules786 shahules786 self-assigned this Apr 30, 2024
@shahules786 shahules786 added the High priority Created by Linear-GitHub Sync label Apr 30, 2024
@shahules786 shahules786 mentioned this issue Apr 30, 2024
Merged
3 tasks
jjmachan pushed a commit that referenced this issue May 3, 2024
@shahules786
reproducibility PR1 : introduce ensembler (#924)
bb97763 
Changes

- [x] An ensemble to combine multiple outputs for the same query
(binary)
- [x] introduce the concept of reproducibility to 3 metrics (self
consistency)
- [x] Modify and break answer-correctness into 2 step process to improve
stability and decrease errors.


#925
@shahules786 shahules786 modified the milestones: v.2, v.3 May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shahules786 shahules786

Labels
High priority Created by Linear-GitHub Sync
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shahules786