[GPU] Add new flag xla_gpu_exclude_nondeterministic_ops.

[sergachev]

It's more granular than the existing --xla_gpu_deterministic_ops because it allows doing an autotuning compilation with non-deterministic ops disabled.

--xla_gpu_deterministic_ops is a superset of --xla_gpu_exclude_nondeterministic_ops, so --xla_gpu_deterministic_ops=true will be setting --xla_gpu_exclude_nondeterministic_ops=true too.

[dimitar-asenov]

This breaks one of TensorFlow's deterministic ops tests. I'm looking into it.

[dimitar-asenov]

@sergachev Could you please rebase this PR, as it has conflicting changes at the moment.

[sergachev]

Done

[akuegel]

Approving the rebase

[dimitar-asenov]

This change fails this OSS test (running on Linux): https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/kernel_tests/nn_ops/cudnn_deterministic_ops_test.py

The failure is miscomparison in numerics, I guess something in the way the determinism is being controlled doesn't work out.

Interestingly, running the same test in our internal build environment passes. I'll have a look at this, but if you have any ideas in the meantime, please let me know.

[sergachev]

Maybe I know what the problem is - after reading it again I think this line

xla/xla/service/gpu/stream_executor_util.cc

Line 627 in b17918d

config.debug_options().xla_gpu_deterministic_ops();

should stay unchanged (it's expected that RequireDeterminism() here disables autotuning).

[sergachev]

Removed that change and updated the comments.

[dimitar-asenov]

Thanks. I'm not sure if that was the issue. In any case there is another which I'm fixing internally:

• Simply using the special setter setter_for_xla_gpu_deterministic_ops to set the new flag whenever --xla_gpu_deterministic_ops is also set, doesn't work. The reason is that it's possible to manually construct debug options without calling into this logic at all. So the correct implementation is to change to code everywhere where needed to check both flags.