The tf.keras.Model.compile metrics do not respect masking since TF 2.2

[foxik]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Debian Stable
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: --
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): TF 2.2, TF-nightly 2.3.0-dev20200529
• Python version: 3.7

Describe the current behavior
The metrics passed to tf.keras.Model.compile as metrics do not respect model masks.

Describe the expected behavior
Until TF 2.1, these metrics did respect model masks.

Standalone code to reproduce the issue
Consider the following code which masks the input element.

import numpy as np
import tensorflow as tf
print(tf.__version__)

model = tf.keras.Sequential([
    tf.keras.layers.Masking(1.),
    tf.keras.layers.Dense(1)
])
model.compile(optimizer=tf.optimizers.Adam(),
              loss=tf.losses.MeanSquaredError(),
              metrics=[tf.metrics.MeanSquaredError()],
              weighted_metrics=[tf.metrics.MeanSquaredError()])

print(model.train_on_batch(np.ones([1, 1]), np.ones([1, 1])))

TensorFlow until 2.1 masks also the metric in metrics, while TensorFlow 2.2 and later do not.

• TensorFlow 2.1 (colab) prints [0.0, 0.0, 0.0]
• TensorFlow 2.2 (colab) prints [0.0, 1.0, 0.0]
• TensorFlow-Nightly 2.3.0-dev20200529 (colab) prints [0.0, 1.0, 0.0]

Other info / logs
The logic of applying the mask in master is here:

tensorflow/tensorflow/python/keras/engine/compile_utils.py

Lines 404 to 414 in a1ae008

	sw = apply_mask(y_p, sw)
	for metric_obj in metric_objs:
	if metric_obj is None:
	continue
	metric_obj.update_state(y_t, y_p)
	for weighted_metric_obj in weighted_metric_objs:
	if weighted_metric_obj is None:
	continue
	weighted_metric_obj.update_state(y_t, y_p, sample_weight=sw)

The metrics do not get called with sample_weight, but that is the place where the masks are applied (in apply_mask).

On the other hand, in TF 2.1

tensorflow/tensorflow/python/keras/engine/training.py

Lines 2000 to 2012 in 3ffdb91

	if (return_weighted_and_unweighted_metrics or
	not return_weighted_metrics):
	metric_results.extend(
	self._handle_per_output_metrics(self._per_output_metrics[i],
	target, output, output_mask))
	if return_weighted_and_unweighted_metrics or return_weighted_metrics:
	metric_results.extend(
	self._handle_per_output_metrics(
	self._per_output_weighted_metrics[i],
	target,
	output,
	output_mask,
	weights=sample_weights[i] if sample_weights else None))

the output_mask was passed even for the unweighted metrics.

[Saduf2019]

I am able to replicate this issue, masking occurring in 2.1 and works as not in tf 2.2

[goldiegadde]

@foxik this issue has been fixed in f66d36cc.
Please reopen if thats not the case.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No