DNN library is not found.

[ShangWeiKuo]

Hello. When I ran the function model.predict_instances_big(img, axes='YXC', block_size=4096, min_overlap=128, context=128, normalizer=normalizer, n_tiles=(4,4,1)), I got the error message showing that DNN library is not found.

How should I do to deal with the problem ?

Thank you for your help in advance.

• Ubuntu version is 20.04
• Tensorflow version is 2.12.0
• Python version is 3.8.13
• Cuda version is V11.6.124

This is the error message:

2023-04-06 12:44:35.570076: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.4.0 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2023-04-06 12:44:35.571376: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at conv_ops_fused_impl.h:625 : UNIMPLEMENTED: DNN library is not found.
2023-04-06 12:44:35.571420: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNIMPLEMENTED: DNN library is not found.
	 [[{{node model/conv2d/Relu}}]]
---------------------------------------------------------------------------
UnimplementedError                        Traceback (most recent call last)
Input In [23], in <cell line: 2>()
      1 #Slow - takes time to segment the large image
----> 2 labels, polys = model.predict_instances_big(img, axes='YXC', block_size=4096, min_overlap=128, context=128,
      3                                                 normalizer=normalizer, n_tiles=(4,4,1))

File /opt/conda/lib/python3.8/site-packages/stardist/models/base.py:912, in StarDistBase.predict_instances_big(self, img, axes, block_size, min_overlap, context, labels_out, labels_out_dtype, show_progress, **kwargs)
    909 # print(f"input: shape {img.shape} with axes {axes}")
    910 print(f'effective: block_size={block_size}, min_overlap={min_overlap}, context={context}', flush=True)
--> 912 for a,c,o in zip(axes,context,self._axes_tile_overlap(axes)):
    913     if c < o:
    914         print(f"{a}: context of {c} is small, recommended to use at least {o}", flush=True)

File /opt/conda/lib/python3.8/site-packages/stardist/models/base.py:1084, in StarDistBase._axes_tile_overlap(self, query_axes)
   1082     self._tile_overlap
   1083 except AttributeError:
-> 1084     self._tile_overlap = self._compute_receptive_field()
   1085 overlap = dict(zip(
   1086     self.config.axes.replace('C',''),
   1087     tuple(max(rf) for rf in self._tile_overlap)
   1088 ))
   1089 return tuple(overlap.get(a,0) for a in query_axes)

File /opt/conda/lib/python3.8/site-packages/stardist/models/base.py:1069, in StarDistBase._compute_receptive_field(self, img_size)
   1067 z = np.zeros_like(x)
   1068 x[(0,)+mid+(slice(None),)] = 1
-> 1069 y  = self.keras_model.predict(x)[0][0,...,0]
   1070 y0 = self.keras_model.predict(z)[0][0,...,0]
   1071 grid = tuple((np.array(x.shape[1:-1])/np.array(y.shape)).astype(int))

File /opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File /opt/conda/lib/python3.8/site-packages/tensorflow/python/eager/execute.py:52, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50 try:
     51   ctx.ensure_initialized()
---> 52   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                       inputs, attrs, num_outputs)
     54 except core._NotOkStatusException as e:
     55   if name is not None:

UnimplementedError: Graph execution error:

Detected at node 'model/conv2d/Relu' defined at (most recent call last):
    File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/opt/conda/lib/python3.8/site-packages/ipykernel_launcher.py", line 17, in <module>
      app.launch_new_instance()
    File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 846, in launch_instance
      app.start()
    File "/opt/conda/lib/python3.8/site-packages/ipykernel/kernelapp.py", line 712, in start
      self.io_loop.start()
    File "/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
      self.asyncio_loop.run_forever()
    File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
      self._run_once()
    File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
      handle._run()
    File "/opt/conda/lib/python3.8/asyncio/events.py", line 81, in _run
      self._context.run(self._callback, *self._args)
    File "/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py", line 504, in dispatch_queue
      await self.process_one()
    File "/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py", line 493, in process_one
      await dispatch(*args)
    File "/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
      await result
    File "/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py", line 724, in execute_request
      reply_content = await reply_content
    File "/opt/conda/lib/python3.8/site-packages/ipykernel/ipkernel.py", line 390, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/opt/conda/lib/python3.8/site-packages/ipykernel/zmqshell.py", line 528, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 2863, in run_cell
      result = self._run_cell(
    File "/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 2909, in _run_cell
      return runner(coro)
    File "/opt/conda/lib/python3.8/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3106, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3309, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3369, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_13402/2241451282.py", line 2, in <cell line: 2>
      labels, polys = model.predict_instances_big(img, axes='YXC', block_size=4096, min_overlap=0, context=128,
    File "/opt/conda/lib/python3.8/site-packages/stardist/models/base.py", line 912, in predict_instances_big
      for a,c,o in zip(axes,context,self._axes_tile_overlap(axes)):
    File "/opt/conda/lib/python3.8/site-packages/stardist/models/base.py", line 1084, in _axes_tile_overlap
      self._tile_overlap = self._compute_receptive_field()
    File "/opt/conda/lib/python3.8/site-packages/stardist/models/base.py", line 1069, in _compute_receptive_field
      y  = self.keras_model.predict(x)[0][0,...,0]
    File "/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      except Exception as e:  # pylint: disable=broad-except
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 2382, in predict
      raise NotImplementedError(
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 2169, in predict_function
      saving to SavedModel.
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 2155, in step_function
      Args:
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 2143, in run_step
      include_optimizer=True,
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 2111, in predict_step
      non_trainable_variables = []
    File "/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      except Exception as e:  # pylint: disable=broad-except
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 558, in __call__
      At most, one full epoch will be run each
    File "/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      except Exception as e:  # pylint: disable=broad-except
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      # Priority 4: trace layer with the default training argument specified
    File "/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      raise e
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/functional.py", line 512, in call
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph
    File "/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      except Exception as e:  # pylint: disable=broad-except
    File "/opt/conda/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      # Priority 4: trace layer with the default training argument specified
    File "/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      raise e
    File "/opt/conda/lib/python3.8/site-packages/keras/layers/convolutional/base_conv.py", line 321, in call
    File "/opt/conda/lib/python3.8/site-packages/keras/activations.py", line 317, in relu
      """Applies the Gaussian error linear unit (GELU) activation function.
    File "/opt/conda/lib/python3.8/site-packages/keras/backend.py", line 5396, in relu
      Raises:
Node: 'model/conv2d/Relu'
DNN library is not found.
	 [[{{node model/conv2d/Relu}}]] [Op:__inference_predict_function_844]

[Deepskyhunter]

Try follow GPU setup at
www.tensorflow.org/install/pip

[maweigert]

Yes, this seems to be an issue with the CuDNN library version installed, so upgrading it might help (see first error message).