Make the improvements suggested in improvements.pdf

[philipcmonk]

The improvements suggested here seem like they would be very good. We're also interested in other improvements to the checks.

There is a 100 NMR bounty on this issue.

[isaykatsman]

Hi Philip! The improvements doc was a fairly interesting read. Sounds like there are some fairly interesting things there I would like to get into. However I have a clarification - is the 100 NMR bounty for implementing/solving all of the described issues? Or would it be possible to submit a PR for solving only some of the issues (e.g. focusing on improving Concordance specifically)?

[philipcmonk]

@isaykatsman The 100 NMR bounty is for all of the issues, however I'm open to splitting that up into specific parts of it. Can you suggest a reasonable division of tasks/bounty?

Alternatively, if you work together with someone to solve all the tasks, you could split the bounty between yourselves.

[zoso95]

We need a quality benchmark before I'd feel comfortable pushing a large algorithmic change. To do this initially I used actual user submissions to calibrate things, but that would be hard to do.

[cpurta]

For the third proposed concordance solution (use of the targets in the Test Set), how exactly would comparing the log loss of the validation and the test sets incur a data leak? Is it that users could modify only the test set predictions and based on the concordance outcome determine if their test set predictions are getting a better log loss?