Add support for linear-time mmd estimator. #475

Srceh · 2022-04-01T13:02:04Z

This PR implements the linear-time estimator in (Lemma14 in paper), as asked in #288.

…linear_time_mmd

ojcobb · 2022-04-01T13:51:46Z

Do we think users are ever going to have equal reference and test batch sizes in practice? I'd guess almost always the reference set is going to be much larger. I wonder if we'd be better off using the B-stat estimator by default for the linear case rather than Gretton's estimator for equal sample sizes. This additionally has the advantage of a tunable parameter that allows for interpolation between a linear and quadratic time estimator.

@arnaudvl @Srceh

Edit: It's actually not quite this simple. However I think we should put some thought into how best to address the n!=m case.

Srceh · 2022-04-01T15:03:56Z

Do we think users are ever going to have equal reference and test batch sizes in practice? I'd guess almost always the reference set is going to be much larger. I wonder if we'd be better off using the B-stat estimator by default for the linear case rather than Gretton's estimator for equal sample sizes. This additionally has the advantage of a tunable parameter that allows for interpolation between a linear and quadratic time estimator.

@arnaudvl @Srceh

Edit: It's actually not quite this simple. However I think we should put some thought into how best to address the n!=m case.

Agree, maybe we can leave the current PR as it is for the linear-time one, and do a separate one for the additional B-stat implementation.

arnaudvl · 2022-04-01T15:29:24Z

@Srceh @ojcobb Thinking if it wouldn't be cleaner to have a separate LinearMMDDrift and BMMDDrift (for lack of a better name) instead of grouping everything in the existing MMD implementation. It would be a bit easier to debug as well and can just share the MMD base class.

…linear_time_mmd # Conflicts: # alibi_detect/cd/pytorch/mmd.py

…eshold with the linear-time estimator, instead of permutation.

Srceh · 2022-04-06T13:06:10Z

Now the linear-time estimator also uses Gaussian under null for the test threshold, so no permutation is required. It should be the fastest at the cost of lower test power and some unused samples.

ojcobb · 2022-04-14T09:12:25Z

alibi_detect/utils/pytorch/kernels.py

+    def forward(self, x: Union[np.ndarray, torch.Tensor],
+                y: Union[np.ndarray, torch.Tensor],
+                infer_sigma: bool = False,
+                diag: bool = False) -> torch.Tensor:


Given that they refer to the same thing, perhaps we could keep consistency between this kwarg name and the naming convention adopted for the squared distance functions? So perhaps pairwise: bool = True?

Indeed, now uses pairwise as suggested.

ojcobb · 2022-04-14T09:19:20Z

alibi_detect/utils/pytorch/kernels.py


        if infer_sigma or self.init_required:
            if self.trainable and infer_sigma:
                raise ValueError("Gradients cannot be computed w.r.t. an inferred sigma value")
-            sigma = self.init_sigma_fn(x, y, dist)
+            if not diag:


Is this a good default behaviour to have? Could we end up with O(n^2) costs in places where the linear time estimator is being used specifically because such a cost would be infeasible?

Now directly use the median of the non-pairwise distance.

ojcobb · 2022-04-14T09:20:36Z

alibi_detect/utils/tensorflow/kernels.py

@@ -69,15 +69,24 @@ def __init__(
    def sigma(self) -> tf.Tensor:
        return tf.math.exp(self.log_sigma)

-    def call(self, x: tf.Tensor, y: tf.Tensor, infer_sigma: bool = False) -> tf.Tensor:
+    def call(self, x: tf.Tensor, y: tf.Tensor,


See comments on pytorch version

ojcobb · 2022-04-14T09:22:14Z

alibi_detect/utils/pytorch/distance.py

@@ -93,7 +115,43 @@ def batch_compute_kernel_matrix(
    return k_mat


-def mmd2_from_kernel_matrix(kernel_mat: torch.Tensor, m: int, permute: bool = False,
+def linear_mmd2(x: torch.Tensor,


Nitpick but probs worth keeping indentation within function definitions consistent with all of the other functions.

Fixed, should be consistent all across now.

ojcobb · 2022-04-14T09:43:10Z

alibi_detect/cd/mmd.py

@@ -18,6 +18,7 @@ def __init__(
            x_ref: Union[np.ndarray, list],
            backend: str = 'tensorflow',
            p_val: float = .05,
+            estimator: str = 'quad',


Would estimator_complexity be more descriptive? (Or at least make clear in the docstring)

Added extra description in the docstring.

ojcobb · 2022-04-14T10:01:41Z

alibi_detect/utils/pytorch/distance.py

+    k_yz = kernel(x=y[0::2, :], y=x[1::2, :], diag=True)
+
+    h = k_xx + k_yy - k_xy - k_yz
+    mmd2 = h.sum() / (n / 2.)


Is there a reason we don't just use h.mean() and h.var()?

Now uses h.mean(), and torch.var(, unbiased=True) in the torch version. TF version uses tf.reduce_mean and manual correction.

ojcobb · 2022-04-14T10:07:30Z

alibi_detect/utils/tensorflow/distance.py

+def linear_mmd2(x: tf.Tensor,
+                y: tf.Tensor,
+                kernel: Callable,
+                permute: bool = False) -> Tuple[tf.Tensor, tf.Tensor]:


Is there a reason we offer permute option for tensorflow and not torch?

Legacy issue, now removed for the tensorflow version.

ojcobb · 2022-04-14T10:07:58Z

alibi_detect/utils/tensorflow/distance.py

+        k_xx = kernel(x_hat[0::2, :], x_hat[1::2, :], diag=True)
+        k_yy = kernel(y_hat[0::2, :], y_hat[1::2, :], diag=True)
+        k_xy = kernel(x_hat[0::2, :], y_hat[1::2, :], diag=True)
+        k_yz = kernel(y_hat[0::2, :], x_hat[1::2, :], diag=True)


Seems like unnecessary duplication

ojcobb · 2022-04-14T10:09:03Z

alibi_detect/cd/pytorch/mmd.py

+        mmd2 = mmd2.numpy().item()
+        var_mmd2 = var_mmd2.numpy().item()
+        std_mmd2 = np.sqrt(var_mmd2)
+        p_val = 1 - stats.norm.cdf(mmd2 * np.sqrt(n_hat), loc=0., scale=std_mmd2*np.sqrt(2))


Nitpick but should this be a t-test?

Nice spot, now fixed with t-test for both versions.

arnaudvl · 2022-04-27T15:34:43Z

alibi_detect/cd/pytorch/mmd.py

+            mmd2 = mmd2.cpu()
+        mmd2 = mmd2.numpy().item()
+        var_mmd2 = var_mmd2.numpy().item()
+        std_mmd2 = np.sqrt(var_mmd2)


Can directly use torch.std(...) in linear_mmd2? This would remove the few additional lines of code here.

The new version uses np.sqrt(np.clip(var_mmd2, 1e-8, 1e-8)) for numeric stability.

arnaudvl · 2022-04-27T15:40:06Z

alibi_detect/utils/pytorch/distance.py

@@ -30,6 +30,28 @@ def squared_pairwise_distance(x: torch.Tensor, y: torch.Tensor, a_min: float = 1
    return dist.clamp_min_(a_min)


+def squared_distance(x: torch.Tensor, y: torch.Tensor, a_min: float = 1e-30) -> torch.Tensor:


Can we just apply a reduction to the squared_pairwise_distance instead of using an extra function?

Now implemented as a single function.

arnaudvl · 2022-04-27T15:41:26Z

alibi_detect/utils/pytorch/distance.py

+    m = np.shape(y)[0]
+    if n != m:
+        raise RuntimeError("Linear-time estimator requires equal size samples")
+    k_xx = kernel(x=x[0::2, :], y=x[1::2, :], pairwise=False)


We should be able to do this at init time (so self.k_xx becomes useful again), saving compute at prediction time.

Fixed, now the kernel matrix is reused for prediction.

arnaudvl · 2022-04-27T15:42:05Z

alibi_detect/utils/pytorch/distance.py

+    """
+    n = np.shape(x)[0]
+    m = np.shape(y)[0]
+    if n != m:


This behaviour should in my opinion already be checked beforehand (see comment in the method itself).

arnaudvl · 2022-04-27T15:43:17Z

alibi_detect/utils/pytorch/distance.py

+    k_xx = kernel(x=x[0::2, :], y=x[1::2, :], pairwise=False)
+    k_yy = kernel(x=y[0::2, :], y=y[1::2, :], pairwise=False)
+    k_xy = kernel(x=x[0::2, :], y=y[1::2, :], pairwise=False)
+    k_yz = kernel(x=y[0::2, :], y=x[1::2, :], pairwise=False)


Is k_yz the paper notation? B/c it might be easier to follow by just calling it k_yx.

Typo, thanks for noticing, fixed.

arnaudvl · 2022-04-27T15:45:57Z

alibi_detect/utils/pytorch/kernels.py

@@ -68,16 +68,24 @@ def __init__(
    def sigma(self) -> torch.Tensor:
        return self.log_sigma.exp()

-    def forward(self, x: Union[np.ndarray, torch.Tensor], y: Union[np.ndarray, torch.Tensor],
-                infer_sigma: bool = False) -> torch.Tensor:
+    def forward(self, x: Union[np.ndarray, torch.Tensor],


Nitpicking big time here, but let's try to keep same type of indentation as e.g. in the DeepKernel below.

arnaudvl · 2022-04-27T15:46:57Z

alibi_detect/utils/pytorch/kernels.py


        x, y = torch.as_tensor(x), torch.as_tensor(y)
-        dist = distance.squared_pairwise_distance(x.flatten(1), y.flatten(1))  # [Nx, Ny]
+        if pairwise:


Check my comment in distance.py which might make this if else redundant and reduce it to a kwarg of the distance function.

Fixed, now as part of the squared_pairwise_distance function argument.

arnaudvl · 2022-04-27T15:48:35Z

alibi_detect/utils/pytorch/kernels.py

+            if pairwise:
+                sigma = self.init_sigma_fn(x, y, dist)
+            else:
+                sigma = (.5 * dist.flatten().sort().values[dist.shape[0] // 2 - 1].unsqueeze(dim=-1)) ** .5


Again I think we can avoid the hard-coding of this behaviour and fall back on self.init_sigma_fn but with the desired linear detector behaviour.

Slightly tricky as the default init_sigma_fn is used by other detectors. Might be easier to keep the additional line here?

arnaudvl · 2022-04-27T15:51:06Z

Left a number of comments related to the PyTorch implementation. Let's work through those first and then we can apply the desired changes to TensorFlow as well.

ascillitoe

"Requesting changes" to ensure we do not merge until #489 has been merged and predict updated in this PR.

…ments.

review-notebook-app · 2022-05-08T22:46:50Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

ascillitoe · 2022-05-09T16:26:36Z

alibi_detect/utils/tensorflow/distance.py

@@ -20,14 +26,44 @@ def squared_pairwise_distance(x: tf.Tensor, y: tf.Tensor, a_min: float = 1e-30,
        Lower bound to clip distance values.
    a_max
        Upper bound to clip distance values.
-
+    pairwise


Isn't it a bit unclear to have a function named squared_pairwise_distance that optionally computes non-pairwise distances? Perhaps squared_pairwise_distance should be renamed squared_distance? Or the pairwise=False functionality separated out into a separate distance function?

I was thinking the same. It was previously a separate function and @arnaudvl was suggesting making the repeated parts minimal. Guess changing the function name across all related methods would be preferable.

Srceh · 2022-05-18T12:02:41Z

Left a number of comments related to the PyTorch implementation. Let's work through those first and then we can apply the desired changes to TensorFlow as well.

TF version is also fixed for the above ones replied with "fixed".

ascillitoe · 2022-05-18T13:46:39Z

@Srceh I've just merged the score/predict refactoring (#489). This will have introduced a few conflicts you need to resolve. It should simplify your life wrt to the implementation in this PR though!

Srceh · 2022-05-18T13:47:44Z

@Srceh I've just merged the score/predict refactoring (#489). This will have introduced a few conflicts you need to resolve. It should simplify your life wrt to the implementation in this PR though!

Nice! will start working on that!

…IO#489). Merge branch 'master' into linear_time_mmd # Conflicts: # alibi_detect/cd/base.py # alibi_detect/cd/mmd.py # alibi_detect/cd/pytorch/mmd.py # alibi_detect/cd/tensorflow/mmd.py

ascillitoe · 2022-07-14T09:21:31Z

@arnaudvl @Srceh I will resolve the conflicts for you once #537 has been merged.

ascillitoe · 2022-07-26T14:47:19Z

@Srceh I have now merged in the v0.10.0 related changes from master. This primarily involved changes to the kwargs related to preprocessing, tweaking some tests, and adding your estimator kwarg to the MMDDrift pydantic models (see saving/schemas.py).

ascillitoe · 2022-07-26T14:49:40Z

alibi_detect/cd/pytorch/mmd.py

        if self.device.type == 'cuda':
            mmd2, mmd2_permuted = mmd2.cpu(), mmd2_permuted.cpu()
        p_val = (mmd2 <= mmd2_permuted).float().mean()
        # compute distance threshold
        idx_threshold = int(self.p_val * len(mmd2_permuted))
        distance_threshold = torch.sort(mmd2_permuted, descending=True).values[idx_threshold]
        return p_val.numpy().item(), mmd2.numpy().item(), distance_threshold.numpy()
+
+
+class LinearTimeMMDDriftTorch(BaseMMDDrift):


Since these new subclasses don't make use of self.n_permutations (set in BaseMMDDrift), shall we set this to None? I had a moment of confusion when updating the tests since self.n_permuations == 100 when estimator == 'linear'.

Good point. The default number of permutations then can be initialised in /cd/mmd.py when estimator is 'quad'.

ascillitoe · 2022-07-26T15:33:18Z

alibi_detect/cd/mmd.py

+                self._detector = MMDDriftTF(*args, **kwargs)  # type: ignore
+            elif estimator == 'linear':
+                kwargs.pop('n_permutations', None)
+                self._detector = LinearTimeMMDDriftTF(*args, **kwargs)  # type: ignore


Since the logic to set self._detector is located here, we should add additional tests to alibi_detect/cd/tests/test_mmd.py to check that the correct subclass is selected conditional on backend and estimator.

Indeed, will modify the tests.

Simply rewrite the test to go through different backend and estimator options, should do the job.

… base MMD class.

CLAassistant · 2024-05-07T14:13:51Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ Srceh
❌ ascillitoe
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Srceh added 3 commits April 1, 2022 13:32

Add support for linear-time MMD estimator.

f4d2692

Add support for linear-time MMD estimator.

8ef820d

Merge branch 'linear_time_mmd' of github.com:Srceh/alibi-detect into …

ba29712

…linear_time_mmd

Srceh requested a review from arnaudvl April 1, 2022 13:02

Srceh marked this pull request as ready for review April 1, 2022 13:05

Minor fixes.

eaa2e45

Srceh removed the request for review from arnaudvl April 1, 2022 13:37

Srceh marked this pull request as draft April 1, 2022 13:37

minor fixes.

ea97f52

Srceh marked this pull request as ready for review April 1, 2022 15:35

Srceh added 4 commits April 1, 2022 17:01

minor fixes.

f6b93d6

Merge branch 'linear_time_mmd' of github.com:Srceh/alibi-detect into …

59110ec

…linear_time_mmd # Conflicts: # alibi_detect/cd/pytorch/mmd.py

mnior fixes

7bacbec

Created separated class for linear-time estimator

7507276

Srceh requested review from arnaudvl and ojcobb April 4, 2022 15:15

Use Gaussian asymptotic distribution under null to calculate test thr…

29cd155

…eshold with the linear-time estimator, instead of permutation.

ojcobb reviewed Apr 14, 2022

View reviewed changes

arnaudvl reviewed Apr 27, 2022

View reviewed changes

ascillitoe suggested changes Apr 27, 2022

View reviewed changes

Added tests and doc for the new estimator, fixes following review com…

7952ab3

…ments.

fixes on the distance function.

016f23f

ascillitoe reviewed May 9, 2022

View reviewed changes

Following the changes of the score function for mmd detectors (Seldon…

05626ec

…IO#489). Merge branch 'master' into linear_time_mmd # Conflicts: # alibi_detect/cd/base.py # alibi_detect/cd/mmd.py # alibi_detect/cd/pytorch/mmd.py # alibi_detect/cd/tensorflow/mmd.py

Srceh requested a review from ascillitoe May 21, 2022 19:08

ascillitoe modified the milestones: v0.10.0, v0.10.1 Jul 12, 2022

ascillitoe added 2 commits July 26, 2022 15:27

Merge master and resolve conflicts

20e442c

Update tests and pydantic models

3b96ad4

ascillitoe reviewed Jul 26, 2022

View reviewed changes

Added estimator to mmd tests, changed default permutation to none for…

95f3de4

… base MMD class.

		@@ -30,6 +30,28 @@ def squared_pairwise_distance(x: torch.Tensor, y: torch.Tensor, a_min: float = 1
		return dist.clamp_min_(a_min)


		def squared_distance(x: torch.Tensor, y: torch.Tensor, a_min: float = 1e-30) -> torch.Tensor:

Add support for linear-time mmd estimator. #475

Are you sure you want to change the base?

Add support for linear-time mmd estimator. #475

Conversation

Srceh commented Apr 1, 2022

ojcobb commented Apr 1, 2022 • edited

Srceh commented Apr 1, 2022

arnaudvl commented Apr 1, 2022

Srceh commented Apr 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnaudvl Apr 27, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnaudvl commented Apr 27, 2022

ascillitoe left a comment

Choose a reason for hiding this comment

review-notebook-app bot commented May 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Srceh commented May 18, 2022

ascillitoe commented May 18, 2022

Srceh commented May 18, 2022

ascillitoe commented Jul 14, 2022

ascillitoe commented Jul 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented May 7, 2024 • edited

ojcobb commented Apr 1, 2022 •

edited

arnaudvl Apr 27, 2022 •

edited

CLAassistant commented May 7, 2024 •

edited