Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require disclosure of Accuracy when quoting Open division results #24

Open
DilipSequeira opened this issue Apr 3, 2020 · 2 comments
Open
Assignees
Labels
Next Meeting Item to be discussed in the next Working Group

Comments

@DilipSequeira
Copy link
Contributor

Propose replacing

When comparing Open and Closed division results any ways in which the Open result would not qualify as a Closed result must be identified.

with

When comparing Open and Closed division results any ways in which the Open result would not qualify as a Closed result must be identified. When quoting an Open division result, any reduction in accuracy vs the requirements for a Closed result must be identified even if not making a comparison vs other submissions.

@bitfort bitfort added the Backlog An issue to be discussed in a future Working Group, but not the immediate next one. label Jun 25, 2020
@bitfort
Copy link
Contributor

bitfort commented Jun 25, 2020

SWG:

This is an interesting discuss, for now the rules say the accuracy must be same. We would love to discuss other options here during the WG.

@TheKanter
Copy link
Contributor

TheKanter commented Sep 1, 2020

Does not apply to training
Applies to inference

Must be resolved prior to results, but can be handled post submission.

@TheKanter TheKanter added Next Meeting Item to be discussed in the next Working Group and removed Backlog An issue to be discussed in a future Working Group, but not the immediate next one. labels Sep 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Next Meeting Item to be discussed in the next Working Group
Projects
None yet
Development

No branches or pull requests

4 participants