Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FEA add TunedThresholdClassifier meta-estimator to post-tune the cut-off threshold #26120
FEA add TunedThresholdClassifier meta-estimator to post-tune the cut-off threshold #26120
Changes from 180 commits
b44dd9d
516f62f
d2fbee0
29e5e87
b645ade
3397c56
092689a
200ec31
e871558
31aa1c0
74614e8
27713af
ed1d9b3
8410317
c7d1fe4
c9d7a22
9981f3a
b9c9d5e
588f1c4
243d173
883e929
69333ed
8616da1
99a10b3
0f6dce2
d6fb9f7
7ff3d0d
6985ae9
239793a
92083ed
55d0844
7dfc4a6
729c9a8
787be21
146b170
03b1f7f
8a09a5f
d56f57f
cf164c5
c943f5e
bf1462b
fa89431
862519d
aa520da
5403cf6
cd37743
e7d07af
bc20a47
bba2f97
f925503
d539235
c0acd44
f87baa7
e4dac09
4da7cef
bd86595
45e6e5a
402a1a7
6745afc
4d557cc
2c6ee7e
91c8222
9a96ae1
3d4ce81
d7d8dac
aa3e83d
ab97d63
acb6af8
486a2bd
6d4c4aa
1d12e1f
21e20e0
7952cce
66ad513
7e8b824
d7e50a6
3844706
0866c42
43f971b
db63769
a9b984f
97105a4
52f5921
637c18e
314bc83
9a8ef4e
5b723a0
9ce463d
bba8f55
1c5487d
d302678
2ade221
dca5770
8897533
75bd7ac
c07a980
cc5ba48
66c4c7f
378930e
c88ed94
4715e67
b3bb39f
915624a
b72a72a
5108e43
95150b0
5b66ab8
b4e67fb
005126a
767a05f
080ba5c
05ec85d
63c32bd
1584c5b
1a5a247
4cc61b9
8f36235
9e6b384
5490ce4
8dad0a4
b918708
5e23523
d5578f9
f3f844e
44ad195
e489eab
1cf5528
26dc94e
6a1a6c7
b17b59e
43c1da8
41a6d07
ca06717
ab8b466
235abf5
d9ec528
69f60a6
45a8504
8c4c88d
b97ebf4
e37f831
383937f
d4ce3fb
b6b3548
759d680
23e65e6
6904817
bee1ebe
4d86a36
48fd7cd
0854cd4
ac75300
2df616e
b14225c
98dcefd
7e3d7aa
b958bb0
e7722f6
98a1db8
c28a3e1
076fd29
e728f1d
c73b205
a4890df
f8a5a79
5dfa435
d45a71b
a32c151
7592437
dc5346b
843ca04
51ed9a8
3c89ab3
8cd5582
a48487c
dd18549
27515ca
8a87b26
811dec9
c83b4e1
d4e232f
92f6e05
1c5c3f4
94160ba
85c8484
4f86e9d
d747098
6d0f418
5671dd6
f04085d
d179b5f
2c375f8
66ba8da
a6b19c1
b3b99ff
dda0d2c
48e7829
17839e8
3f02bc3
553cfce
6ae6d27
d010096
8bb8ca6
fd971c7
bf57dac
0409932
66ea575
1c97dd4
c8c1d0c
8a52bc6
ef668cf
fdbf68e
9c0c13d
f419371
42eafe5
581133f
0f803d9
eb0defc
ffd5669
18abafe
ce9464c
89d67cf
db3360b
1789cc0
e7c31b9
0fd667c
07e4387
9bd68e6
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I mentioned in the other example, I would introduce the roc curve earlier in the explanation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need justify using "internal cross-validation" somewhere rather than tuning this like any other hyperparameter? Indeed, ignoring
constraint_value
is this tool just a workaround for a design flaw wherein a parameter toBaseClassifier
is not exposed to*SearchCV
with warm start?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Today, I would indeed consider it as a design and methodological flaw. I don't know what level of justification do you think we should add-on here:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I follow. Isn't the problem that we can't freeze the estimator? Maybe I need to look at the code first...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I realized there's two sort-of independent long-term API issues going on here and you were talking about the other one lol. So we need to have internal cross-validation because we still don't have an API for path-like algorithms and efficient CV and that's what you were talking about. But there's also the
"prefit"
part which I was thinking about, which is now actually fixed via the__clone__
protocol.Basically if we're not fitting the underlying estimator, then we should use the
__clone__
protocol to not clone it if theTunedThresholdClassifier
is being cloned, right?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@amueller What do you mean by that?
and
@glemaitre @jnothman Shouldn't we fix this design flaw? Do you have suggestions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is something that I would introduce in 2.0 because we will surely break some API. In this case, we could the freedom to think about a good API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess what I am missing is a roadmap or a plan/vision.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we note the contrast with what
CalibratedClassifierCV
is for?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes for sure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would maybe mention that a common tuning is picking the top right point on the ROC curve which is the same as picking f2 score (I think?) here. Or maybe mention that that has a nice geometric explanation but doesn't really consider the application.