Tensorflow model conversion for tensorrt not working

[andrewwong09]

Hi!

I'm trying to use DeepLabCut-live with "tensorrt" as the model_type. Running the below minimum example works when model_type="base", but not "tensorrt". I exported the dlc model using deeplabcut.export_model(). Exporting the model runs to completion and is successfully loaded into the DLCLive class. I suspect the issue is when tensorflow is trying to convert the model in trtconvert, the model is not in the expected format. Is there a specific version of tensorflow that I should be using for exporting and importing the model that works with trtconvert? Or is there a trick to updating the exported tensorflow model to the right SavedModel format? I've included my tensorflow and tensorrt package version below.

Thanks,
Andrew

Code snippet

import cv2
from dlclive import DLCLive, Processor

cap = cv2.VideoCapture("my_video.mp4")
dlc_proc = Processor()
dlc_live = DLCLive("/home/andrew/dlc_app/andrew-2023-08-15/exported-models/DLC_resnet_50_iteration-6_shuffle-1", model_type='tensorrt', processor=dlc_proc)
ret, frame = cap.read()
dlc_live.init_inference(frame)
dlc_live.get_pose(frame)

Error Stack Track Below

python3 qs.py 
2024-02-06 13:44:28.260459: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-02-06 13:44:28.386822: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-06 13:44:30.759005: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-02-06 13:44:31.253483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21951 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4090, pci bus id: 0000:65:00.0, compute capability: 8.9
Traceback (most recent call last):
  File "/home/andrew/src/analysis/andrew/dlc-live/qs.py", line 13, in <module>
    dlc_live.init_inference(frame)
  File "/home/andrew/.local/lib/python3.10/site-packages/dlclive/dlclive.py", line 353, in init_inference
    converter.convert()
  File "/home/andrew/.local/lib/python3.10/site-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 1296, in convert
    self._saved_model = load.load(self._input_saved_model_dir,
  File "/home/andrew/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 800, in load
    result = load_partial(export_dir, None, tags, options)["root"]
  File "/home/andrew/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 949, in load_partial
    root = load_v1_in_v2.load(export_dir, tags)
  File "/home/andrew/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load_v1_in_v2.py", line 284, in load
    result = loader.load(tags=tags)
  File "/home/andrew/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load_v1_in_v2.py", line 209, in load
    meta_graph_def = self.get_meta_graph_def_from_tags(tags)
  File "/home/andrew/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/load_v1_in_v2.py", line 88, in get_meta_graph_def_from_tags
    return super(_EagerSavedModelLoader, self).get_meta_graph_def_from_tags(
  File "/home/andrew/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/loader_impl.py", line 394, in get_meta_graph_def_from_tags
    raise RuntimeError(
RuntimeError: MetaGraphDef associated with tags 'serve' could not be found in SavedModel, with available tags '[set()]'. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: `saved_model_cli`.

Below are my library versions running in python 3.10.12 environment.

tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.0
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.35.0
tensorpack==0.11
tensorrt==8.6.1
tensorrt-dispatch==8.6.1
tensorrt-lean==8.6.1
torch-tensorrt==1.4.0

Thanks

[MMathisLab]

Definitely the TF version for export model needs to be the same as in dlc-live env; and below 2.10.

I have not used tensorrt actually; is it clear in docs we support this? Cc @n-poulsen

[n-poulsen]

Yes the docs specify tensorrt as a valid model type (see here) - I'm working on replicating the issue

[n-poulsen]

Hi @andrewwong09! Did you make any changes to the DLCLive codebase to load the model? With your script and tensorflow==2.10.0 installed, I get another error than you do:

RuntimeError: Please use tf.experimental.tensorrt.Converter in TF 2.0.

[andrewwong09]

Hi @n-poulsen, thank you for taking the time! You're right, I changed by code base in dlclive.py to load the model and get the previously posted Traceback; sorry for the confusion. I assume to create your error, you had to modify the try/except block here, to enable loading the trt_convert with:

from tensorflow.python.compiler.tensorrt import trt_convert as trt

and updated trt.TrtGraphConverter to accept a nodes_denylist instead of a nodes_blacklist?

In any case, I've arrived at the same error message you have doing the above. Do you have any recommendations at this point? Have you had much luck using tensorrt within dlclive?

For what it's worth, I tried using

converter = tf.experimental.tensorrt.Converter(input_saved_model_dir="path_to_my_model_dir")

And got the below error:

ERROR:tensorflow:Tensorflow needs to be built with TensorRT support enabled to allow TF-TRT to operate.
Traceback (most recent call last):
  File "/home/andrew/scratch/qs.py", line 13, in <module>
    dlc_live.init_inference(frame)
  File "/home/andrew/venv310_tf2.10/lib/python3.10/site-packages/dlclive/dlclive.py", line 345, in init_inference
    converter = tf.experimental.tensorrt.Converter(
  File "/home/andrew/venv310_tf2.10/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py", line 561, in new_func
    return func(*args, **kwargs)
  File "/home/andrew/venv310_tf2.10/lib/python3.10/site-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 1156, in __init__
    _check_trt_version_compatibility()
  File "/home/andrew/venv310_tf2.10/lib/python3.10/site-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 221, in _check_trt_version_compatibility
    raise RuntimeError("Tensorflow has not been built with TensorRT support.")
RuntimeError: Tensorflow has not been built with TensorRT support.