Error when training with my own dataset using train.py: Failed to get convolution algorithm.

[Guanzhao-hub]

Has any one encountered this issue when training with your own dataset?

Traceback (most recent call last):
File "train.py", line 93, in
train_step(image_data, target)
File "train.py", line 52, in train_step
pred_result = model(image_data, training=True)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py", line 386, in call
inputs, training=training, mask=mask)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 247, in call
outputs = self._convolution_op(inputs, self.kernel)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1018, in convolution_v2
name=name)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1148, in convolution_internal
name=name)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 2592, in _conv2d_expanded_batch
name=name)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 938, in conv2d
_ops.raise_from_not_ok_status(e, name)
File "/home/ubuntu/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [Op:Conv2D]