Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Tails' architecture to use TF >=2.6 #11

Open
dmitryduev opened this issue Sep 17, 2021 · 0 comments
Open

Update Tails' architecture to use TF >=2.6 #11

dmitryduev opened this issue Sep 17, 2021 · 0 comments

Comments

@dmitryduev
Copy link
Owner

Works with tf<2.5. tf>=2.5 requires some minor fixes.

With tf==2.6, watcher.py throws the following exception:

➜ [private]  tails git:(master) ✗ docker exec -it tails_sentinel_1 bash
(base) root@31f5ac54e255:/app# python watcher.py
2021-09-17 02:52:19.755559: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-09-17 02:52:19.755861: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "watcher.py", line 925, in <module>
    fire.Fire(sentinel)
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "watcher.py", line 811, in sentinel
    dask_client.register_worker_plugin(worker_initializer, name="worker-init")
  File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 4113, in register_worker_plugin
    return self.sync(
  File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 840, in sync
    return sync(
  File "/opt/conda/lib/python3.8/site-packages/distributed/utils.py", line 326, in sync
    raise exc.with_traceback(tb)
  File "/opt/conda/lib/python3.8/site-packages/distributed/utils.py", line 309, in f
    result[0] = yield future
  File "/opt/conda/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 4034, in _register_worker_plugin
    raise exc.with_traceback(tb)
  File "watcher.py", line 708, in setup
    self.tails_worker = TailsWorker()
  File "watcher.py", line 121, in __init__
    self.model = Tails()
  File "/app/tails/efficientdet.py", line 1473, in Tails
    features = EfficientNetB0(input_tensor=x_input, weights=None)
  File "/app/tails/efficientdet.py", line 560, in EfficientNetB0
    return EfficientNet(
  File "/app/tails/efficientdet.py", line 475, in EfficientNet
    if not is_keras_tensor(input_tensor):
  File "/opt/conda/lib/python3.8/site-packages/tensorflow-2.6.0-py3.8-linux-x86_64.egg/tensorflow/python/keras/backend.py", line 1281, in is_keras_tensor
    raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) +
ValueError: Unexpectedly found an instance of type `<class 'keras.engine.keras_tensor.KerasTensor'>`. Expected a symbolic tensor instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant