New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Standard metrics #658
Standard metrics #658
Conversation
79f30cf
to
d877d7c
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #658 +/- ##
==========================================
- Coverage 89.83% 89.05% -0.78%
==========================================
Files 96 97 +1
Lines 9118 9350 +232
==========================================
+ Hits 8191 8327 +136
- Misses 927 1023 +96 ☔ View full report in Codecov by Sentry. |
0f69426
to
9271fcc
Compare
9271fcc
to
ef62ff5
Compare
@dafnapension @elronbandel - Can you explain the motivation for this PR? What are standard metrics and how do they relate to the existing metrics? |
46ecaef
to
a90e3aa
Compare
Current evaluation of a global metric starts by laying the whole stream in main memory, adding "next to it" a couple of hundreds copies thereof (for the re_samplings). The resampling is somewhat more trickier: |
@elronbandel @dafnapension - I'm sure you discussed it between you alot, but I want to provide a different perspective. I think stream in unitxt may be useful if unitxt used for large scale training - however, it also has significant cost in terms of code and API complexity. In evaluation , where typically only hundres of samples are tested, streaming will have no significant value. We need metrics API that are Our direction should be of simplification and not making things more complex. Therefore, I think it's worth to have a discussion if this direction will have a net gain in terms of unitxt acceptance. (@eladven - will be glad your input as well). |
3f79003
to
bc6a4e7
Compare
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
…t. as in global classification metrics in metrics.py Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
Signed-off-by: dafnapension <dafnashein@yahoo.com>
bc6a4e7
to
91805ce
Compare
Leave for now. If at all, continue via #845 |
No description provided.