Add Covariate Shift Conformal Regressor #151

RudrakshTuwani · 2022-03-30T10:17:12Z

Description

Add covariate shift conformal method (https://arxiv.org/abs/1904.06019) to MAPIE for regression.

Fixes #72

Type of change

Please remove options that are irrelevant.

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

Reproducing the results of the original paper.

Checklist

I have read the contributing guidelines
I have updated the HISTORY.rst and AUTHORS.rst files
Linting passes successfully : make lint
Typing passes successfully : make type-check
Unit tests pass successfully : make tests
Coverage is 100% : make coverage
Documentation builds successfully : make doc

RudrakshTuwani · 2022-03-30T12:18:33Z

Hey @gmartinonQM, please let me know how to prioritize tasks and would really appreciate feedback. Thanks! :)

gmartinonQM · 2022-03-30T12:24:37Z

Thanks for the PR @RudrakshTuwani , for sure. Maybe we will take several days to discuss it internally. Stay tuned !

vtaquet · 2022-05-06T19:26:25Z

Hi @RudrakshTuwani , thank you very much for the PR ! The implementation and the notebook look really great. I am sorry for the delay of our feedback, we have been quite busy during the last past weeks. Before giving you a more thorough feedback by next week, just a few general comments :

please move the class MapieCovShiftRegressor in a dedicated module (like covariate_shift_regression.py)
please move your notebook in the notebooks/regression folder. An alternative is to create a notebook-like example from a .py script like in this example.

RudrakshTuwani · 2022-05-09T22:27:50Z

Hello @vtaquet , thank you for getting back to me! No worries about the delay! :)

…scikit-learn-contrib/MAPIE

RudrakshTuwani · 2022-08-29T15:15:49Z

Couldn't push to @vtaquet 's branch, so pulled the changes to the fork and re-opened PR. Still a WIP!

thibaultcordier

Your contribution is relevant and appreciated for the evolution of MAPIE for conformal prediction with covariate shift. Moreover, it provides the basis for future developments towards conditional conformal prediction. In this perspective, I give you a list of suggestions to improve your code.

I suggest you implement a new WeightedConformalScore class (in a new file weighted_conformity_scores.py), which inherits from ConformalScore with all your current changes.
Thus the CovariateShiftConformityScore class will inherit from WeightedConformalScore and will be the only class that benefits from your changes without impacting the others.
In MapieRegressor class, I suggest to adapt the if condition with a test like isinstance(self.conformity_score_function_, WeightedConformityScore). Is this possible in your opinion? Or am I missing something?

I remain available if you have any questions and thank you in advance for your feedback.

thibaultcordier · 2023-03-14T09:04:01Z

mapie/conformity_scores.py

@@ -30,6 +32,7 @@ def __init__(
            - get_estimation_distribution and
            - get_signed_conformity_scores
            by default True.
+        compute_score_weights : TODO


Need to define what is compute_score_weights

thibaultcordier · 2023-03-14T09:05:58Z

mapie/conformity_scores.py


    @abstractmethod
    def get_signed_conformity_scores(
-        self, y: ArrayLike, y_pred: ArrayLike,
+        self, y: ArrayLike, y_pred: ArrayLike, conformity_scores: ArrayLike


I think this is a mistake: conformity_scores should not be an argument to get_signed_conformity_scores.

thibaultcordier · 2023-03-14T09:07:54Z

mapie/conformity_scores.py

+        self,
+        y: NDArray,
+        y_pred: NDArray,
+        conformity_scores: NDArray,


There is no reason to change the style (for consistency).

thibaultcordier · 2023-03-14T09:10:15Z

mapie/conformity_scores.py

-        self, y: ArrayLike, y_pred: ArrayLike,
+        self,
+        y: ArrayLike,
+        y_pred: ArrayLike,


There is no reason to change the style (for consistency).

thibaultcordier · 2023-03-14T09:10:29Z

mapie/conformity_scores.py

-        self, y: ArrayLike, y_pred: ArrayLike,
+        self,
+        y: ArrayLike,
+        y_pred: ArrayLike,


There is no reason to change the style (for consistency).

thibaultcordier · 2023-03-14T09:33:59Z

mapie/regression.py

@@ -700,25 +724,41 @@ def predict(
            )
        )

+        if self.conformity_score_function_.compute_score_weights:


I understand the motivation to modify the ConformityScore class by adding a compute_score_weights attribute and a get_weights method.

I suggest to adapt the if condition with a test like isinstance(self.conformity_score_function_, WeightedConformityScore). Is this possible in your opinion? Or am I missing something?

thibaultcordier · 2023-03-14T09:35:07Z

mapie/regression.py

@@ -484,6 +485,29 @@ def _pred_multi(self, X: ArrayLike) -> NDArray:
        y_pred_multi = self._aggregate_with_mask(y_pred_multi, self.k_)
        return y_pred_multi

+    def _quantile(


A description would be appreciated (even if the method name is explicit).

thibaultcordier · 2023-03-14T09:39:47Z

mapie/utils.py

+    """
+    Returns the indices that would sort the array.
+    In case of ties, breaks them randomly.
+    """


A detailed description would be appreciated (with Parameters, Returns, Raises and Examples sections if relevant).

thibaultcordier · 2023-03-14T09:40:00Z

mapie/utils.py

+    """
+    Calculates the weighted empirical quantile of `values`.
+    Converted to Python from https://github.com/ryantibs/conformal/
+    """


A detailed description would be appreciated (with Parameters, Returns, Raises and Examples sections if relevant).

mapie/utils.py

RudrakshTuwani · 2023-03-14T10:11:58Z

Thanks for the feedback! I will try to address the comments over the weekend. :)

vtaquet self-requested a review May 6, 2022 19:26

vtaquet added the enhancement New feature or request label May 6, 2022

Vianney Taquet added 2 commits July 11, 2022 21:29

Adapt Covariate Shift as conformity score

4d3a3ac

Merge master branch and fix conflicts

e07ced5

RudrakshTuwani closed this Aug 27, 2022

RudrakshTuwani force-pushed the master branch from 11940ae to 04a4555 Compare August 27, 2022 15:01

Rudraksh Tuwani added 2 commits August 27, 2022 11:10

Merge branch 'add-covshift-in-conformityscore' of https://github.com/…

4a5270c

…scikit-learn-contrib/MAPIE

get master updates

1a30e3f

RudrakshTuwani reopened this Aug 27, 2022

LacombeLouis requested a review from vincentblot28 January 17, 2023 14:15

vincentblot28 mentioned this pull request Mar 2, 2023

A question about sample_weights #276

Closed

thibaultcordier added this to In progress in Developments via automation Mar 3, 2023

thibaultcordier moved this from In progress to Review in progress in Developments Mar 3, 2023

thibaultcordier reviewed Mar 14, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/master'

d30c33e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Covariate Shift Conformal Regressor #151

Add Covariate Shift Conformal Regressor #151

RudrakshTuwani commented Mar 30, 2022 •

edited

RudrakshTuwani commented Mar 30, 2022

gmartinonQM commented Mar 30, 2022

vtaquet commented May 6, 2022

RudrakshTuwani commented May 9, 2022

RudrakshTuwani commented Aug 29, 2022

thibaultcordier left a comment •

edited

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

thibaultcordier Mar 14, 2023

RudrakshTuwani commented Mar 14, 2023 •

edited

Add Covariate Shift Conformal Regressor #151

Are you sure you want to change the base?

Add Covariate Shift Conformal Regressor #151

Conversation

RudrakshTuwani commented Mar 30, 2022 • edited

Description

Type of change

How Has This Been Tested?

Checklist

RudrakshTuwani commented Mar 30, 2022

gmartinonQM commented Mar 30, 2022

vtaquet commented May 6, 2022

RudrakshTuwani commented May 9, 2022

RudrakshTuwani commented Aug 29, 2022

thibaultcordier left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RudrakshTuwani commented Mar 14, 2023 • edited

RudrakshTuwani commented Mar 30, 2022 •

edited

thibaultcordier left a comment •

edited

RudrakshTuwani commented Mar 14, 2023 •

edited