Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add accuracy binary classification objective #584

Closed
wants to merge 4 commits into from

Conversation

angela97lin
Copy link
Contributor

Closes #294

(Updating base branch from #559)

@angela97lin angela97lin changed the base branch from master to improved_objectives April 6, 2020 19:11
@codecov
Copy link

codecov bot commented Apr 6, 2020

Codecov Report

Merging #584 into improved_objectives will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@                 Coverage Diff                  @@
##           improved_objectives     #584   +/-   ##
====================================================
  Coverage                98.95%   98.96%           
====================================================
  Files                      132      133    +1     
  Lines                     4504     4532   +28     
====================================================
+ Hits                      4457     4485   +28     
  Misses                      47       47           
Impacted Files Coverage Δ
evalml/objectives/__init__.py 100.00% <ø> (ø)
evalml/objectives/utils.py 94.44% <ø> (ø)
evalml/exceptions/__init__.py 100.00% <100.00%> (ø)
evalml/exceptions/exceptions.py 100.00% <100.00%> (ø)
evalml/objectives/standard_metrics.py 99.51% <100.00%> (+0.02%) ⬆️
evalml/tests/objective_tests/test_objectives.py 96.29% <100.00%> (ø)
...lml/tests/objective_tests/test_standard_metrics.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a88e69c...d8be7c8. Read the comment docs.

@angela97lin angela97lin requested a review from dsherry April 6, 2020 19:31
Copy link
Contributor

@dsherry dsherry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Please add unit test coverage. Actually, looking now, I see on master we don't have direct coverage of our objectives from standard_metrics.py. If that's correct, let's create a new unit test file or something and just cover this new metric, because it's important to have direct coverage for all our objectives.

Also let's freeze this until we've merged improved_objectives to master, which I hope will be today!

@angela97lin
Copy link
Contributor Author

@dsherry Yeah, the way that the objectives have been covered via tests thus far is in their usage in AutoML. We might have talked about this before but I'm not 100% sure what unit test coverage would look like that's helpful / not just "testing that sklearn is doing what it claims to be doing," with the objective function?

@dsherry
Copy link
Contributor

dsherry commented Apr 8, 2020

Yeah, pretty much! We need coverage of each objective so that a) if we alter the implementation of an objective to do something more complicated than call sklearn, we know it conforms to the same standard as before, and b) in the unlikely but not impossible event that sklearn ships a bug, or our usage of their code somehow produces a bug, we catch it!

A great example is #588 : we should have unit test coverage for precision and f1 which asserts that the edge case described there has the behavior we expect, and that the warning suppression works.

@angela97lin angela97lin changed the base branch from improved_objectives to master April 13, 2020 17:25
@angela97lin angela97lin changed the base branch from master to improved_objectives April 13, 2020 17:26
@angela97lin
Copy link
Contributor Author

Closing in favor of #624

@angela97lin angela97lin deleted the 294_accuracy_obj branch April 17, 2020 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a classification accuracy objective
2 participants