At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost" #28287

haitham-b · 2019-04-30T14:18:54Z

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes, found here (https://github.com/viaboxxsystems/deeplearning-showcase/blob/tensorflow_2.0/flaskApp.py)
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MAC OSX 10.14.4
TensorFlow version (use command below): 2.0.0-alpha0
Python version: Python 3.6.5

You can collect some of this information using our environment capture
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
v1.12.0-9492-g2c319fb415 2.0.0-alpha0

Describe the current behavior
when running "flaskApp.py", After loading the model and trying to classify an image using "predict", it fails with the error:

Describe the expected behavior
a result of image classification should be returned.

Code to reproduce the issue
Steps to reproduce:

git clone https://github.com/viaboxxsystems/deeplearning-showcase.git
git checkout tensorflow_2.0
(if needed) pip3 install -r requirements.txt
export FLASK_APP=flaskApp.py
start the app with flask run
using Postman or curl send any image of a dog or cat to the app

OR

curl -X POST \
  http://localhost:5000/net/MobileNet \
  -H 'Postman-Token: ea35b79b-b34d-4be1-a80c-505c104050ec' \
  -H 'cache-control: no-cache' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F image=@/Users/haitham.b/Projects/ResearchProjects/CNNs/deeplearning-showcase/data/sample/valid/dogs/dog.1008.jpg

Other info / logs

E0430 13:36:10.374372 123145501933568 app.py:1761] Exception on /net/MobileNet [POST]
Traceback (most recent call last):
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/haitham.b/Projects/Virtualenvs/deeplearning-showcase/flaskApp.py", line 97, in use_net_to_classify_image
    prediction, prob = predict(net_name, image)
  File "/Users/haitham.b/Projects/Virtualenvs/deeplearning-showcase/flaskApp.py", line 59, in predict
    output_probability = net_models[cnn_name].predict(post_processed_input_images)
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1167, in predict
    callbacks=callbacks)
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 352, in model_iteration
    batch_outs = f(ins_batch)
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3096, in __call__
    run_metadata=self.run_metadata)
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1440, in __call__
    run_metadata_ptr)
  File "/Users/haitham.b/venv/tensorflow2.0alpha/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 548, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable softmax/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/softmax/kernel/N10tensorflow3VarE does not exist.
	 [[{{node softmax/MatMul/ReadVariableOp}}]]

The text was updated successfully, but these errors were encountered:

haitham-b · 2019-05-15T07:59:56Z

Any updates on this?

jvishnuvardhan · 2019-05-22T21:05:02Z

This is not Build/Installation or Bug/Performance issue. Please post this kind of support questions at Stackoverflow. There is a big community to support and learn from your questions. GitHub is mainly for addressing bugs in installation and performance. Thanks!

MischaPanch · 2019-05-22T22:59:11Z

I had the same issue in tensorflow 1.13.1 which I have resolved by creating a reference to the session that is used for loading the models and then to set it to be used by keras in each request. I.e. I have done the following:

from tensorflow.python.keras.backend import set_session
from tensorflow.python.keras.models import load_model

tf_config = some_custom_config
sess = tf.Session(config=tf_config)
graph = tf.get_default_graph()

# IMPORTANT: models have to be loaded AFTER SETTING THE SESSION for keras! 
# Otherwise, their weights will be unavailable in the threads after the session there has been set
set_session(sess)
model = load_model(...)

and then in each request (i.e. in each thread):

global sess
global graph
with graph.as_default():
    set_session(sess)
    model.predict(...)

dsaichand3 · 2019-05-23T10:23:43Z

You are amazing!!!!! This is the best solution. Can you tell me why did it just work after adding session?

MischaPanch · 2019-05-23T10:46:19Z

Thank you and you are very welcome :).

As far as I understand, the problem is that tensorflow graphs and sessions are not thread safe. So by default a new session (which does not contain any previously loaded weights, models a.s.o.) is created for each thread, i.e. for each request. By saving the global session that contains all your models and setting it to be used by keras in each thread the problem is solved.

jvishnuvardhan · 2019-05-24T16:33:01Z

Closing this out since I understand it to be resolved, but please let me know if I'm mistaken. Thanks!

wangcj05 · 2019-07-11T22:06:29Z

I have the same issue with tensor flow version 1.13.1, the above solution works for me.

aachoo · 2019-07-18T22:32:47Z

I am having the same issues and was wondering what the value of some_custom_config was?

MischaPanch · 2019-07-19T08:23:51Z

In case you want to configure your session (which I had to do), you can pass the config in this parameter. Else just leave it out.

aachoo · 2019-07-19T16:08:34Z

Thank you so much! Everything is running perfectly now.

sangurocactus · 2019-07-29T14:20:09Z

Thanks for providing the codes. I ran into similar error message while running BERT on Kera. I tried your solution but can't seem to get it to work. Any guidance is most appreciated!

<ipython-input-42-dc70f6f3f83d> in <module>()
     11     validation_data=([test_input_ids, test_input_masks, test_segment_ids], test_labels),
     12     epochs=1,
---> 13     batch_size=32
     14 )

3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
   1456         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1457                                                self._handle, args,
-> 1458                                                run_metadata_ptr)
   1459         if run_metadata:
   1460           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

FailedPreconditionError: Error while reading resource variable bert_layer_9_module/bert/encoder/layer_3/output/LayerNorm/gamma from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/bert_layer_9_module/bert/encoder/layer_3/output/LayerNorm/gamma/N10tensorflow3VarE does not exist.
	 [[{{node bert_layer_9/bert_layer_9_module_apply_tokens/bert/encoder/layer_3/output/LayerNorm/batchnorm/mul/ReadVariableOp}}]]```

georgealexandruvlad · 2019-07-29T20:25:42Z

I have a similar error when using Elmo embeddings from tf-hub inside a custom keras layer.

eliadl · 2019-08-07T14:47:02Z

Thank you so much for this.

In my case I did it a bit differently, in case it helps anyone:

# on thread 1
session = tf.Session(graph=tf.Graph())
with session.graph.as_default():
    k.backend.set_session(session)
    model = k.models.load_model(filepath)

# on thread 2
with session.graph.as_default():
    k.backend.set_session(session)
    model.predict(x, **kwargs)

The novelty here is allowing for multiple models to be loaded (once) and used in multiple threads.
By default, the "default" Session and the "default" Graph are used while loading a model.
But here you create new ones.
Also note the Graph is stored in the Session object, which is a bit more convenient.

eliadl · 2020-04-17T18:34:44Z

@SungmanHong instead of tf.Session try to use tf.compat.v1.Session.

akash-harijan · 2020-05-05T19:00:34Z

import session in TF 2.X
tf.compat.v1.Session()
import keras.backend.get_session in TF 2.X
tf.compat.v1.keras.backend.get_session()

AllanNyasha · 2020-05-10T10:17:47Z

This worked for me , thanks

vamsikri1234 · 2020-05-13T07:18:47Z

Man you are genius and awesome , you just saved my project , Thank you so muuch

I have applied the suggestion from tensorflow/tensorflow#28287 (comment) I'm not familiar with tensorflow, so I don't know if I've fixed the problem correctly. Now the trainer is no longer stuck in the waiting state.

amalmathews · 2020-06-02T02:42:59Z

I have used the given solution but didn't work for me still, I'm getting the error

/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
  UserWarning('Using a generator with `use_multiprocessing=True`'
Epoch 1/4
---------------------------------------------------------------------------
FailedPreconditionError                   Traceback (most recent call last)
<ipython-input-13-05f25cba3043> in <module>()
      4             learning_rate=config.LEARNING_RATE,
      5             epochs=4,
----> 6             layers='heads')

6 frames
/usr/local/lib/python3.6/dist-packages/mrcnn/model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers, augmentation)
   2350             max_queue_size=100,
   2351             workers=workers,
-> 2352             use_multiprocessing=True,
   2353         )
   2354         self.epoch = max(self.epoch, epochs)

/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     89                 warnings.warn('Update your `' + object_name + '` call to the ' +
     90                               'Keras 2 API: ' + signature, stacklevel=2)
---> 91             return func(*args, **kwargs)
     92         wrapper._original_function = func
     93         return wrapper

/usr/local/lib/python3.6/dist-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   1730             use_multiprocessing=use_multiprocessing,
   1731             shuffle=shuffle,
-> 1732             initial_epoch=initial_epoch)
   1733 
   1734     @interfaces.legacy_generator_methods_support

/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
    218                                             sample_weight=sample_weight,
    219                                             class_weight=class_weight,
--> 220                                             reset_metrics=False)
    221 
    222                 outs = to_list(outs)

/usr/local/lib/python3.6/dist-packages/keras/engine/training.py in train_on_batch(self, x, y, sample_weight, class_weight, reset_metrics)
   1512             ins = x + y + sample_weights
   1513         self._make_train_function()
-> 1514         outputs = self.train_function(ins)
   1515 
   1516         if reset_metrics:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py in __call__(self, inputs)
   3630 
   3631     fetched = self._callable_fn(*array_vals,
-> 3632                                 run_metadata=self.run_metadata)
   3633     self._call_fetch_callbacks(fetched[-len(self._fetches):])
   3634     output_structure = nest.pack_sequence_as(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
   1470         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1471                                                self._handle, args,
-> 1472                                                run_metadata_ptr)
   1473         if run_metadata:
   1474           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

FailedPreconditionError: Error while reading resource variable anchors/Variable from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/anchors/Variable/N10tensorflow3VarE does not exist.
	 [[{{node ROI/ReadVariableOp}}]] ```


**PLEASE HELP!!!**

tochile · 2020-06-25T17:48:46Z

Pls it's showing some_custom_config not defined

tochile · 2020-06-25T17:49:26Z

Pls it's showing some_custom_config not defined

juan-csv · 2020-07-02T05:48:23Z

work's for me, thanks!!

lminer · 2021-01-11T20:37:13Z

Am I correct that the only way to solve this issue in TF2 is to disable eager execution?

minniekabra · 2021-05-17T18:44:34Z

Hi, I am using tensorflow2.4.1 and getting similar error.. Below is my code & error. Can you help?

Code pasted on stack overflow:

https://stackoverflow.com/questions/67575144/error-while-reading-resource-variable-encoder-conv2-1-b-from-container-localhos

omarabdullalghafoor · 2021-05-24T07:22:47Z

from tensorflow.python.keras.backend import set_session
from tensorflow.python.keras.models import load_model

Hi brother , I have same problem in this : can be solve it ?
https://stackoverflow.com/questions/67659336/tensor-tensorflatten-reshape0-shape-2622-dtype-float32-is-not-an-ele

Yessmin · 2021-06-09T14:31:43Z

tf 2.5 please any solution ?
I've disabled the eager mode but am still getting the same error

dorbittonn · 2021-12-05T09:12:11Z

thanks for your answer, Im having problems particularly, using tf 1.14.
still getting this error when trying to run 2 different models one after the other.
I get the error in model.fit. I paid attention to the fact that if Im not passing tensorboard callback the error is gone.

Anyone has an idea what it could be?

NeighborhoodCoding · 2021-12-07T06:45:19Z

from tensorflow.python.keras.backend import set_session
from tensorflow.python.keras.models import load_model

tf_config = some_custom_config
sess = tf.Session(config=tf_config)
graph = tf.get_default_graph()

IMPORTANT: models have to be loaded AFTER SETTING THE SESSION for keras!

Otherwise, their weights will be unavailable in the threads after the session there has been set

set_session(sess)
model = load_model(...)
and then in each request (i.e. in each thread):

global sess
global graph
with graph.as_default():
set_session(sess)
model.predict(...)

haitham-b changed the title ~~Error while reading resource variable softmax/kernel from Container: localhost~~ At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost" Apr 30, 2019

achandraa self-assigned this May 2, 2019

achandraa added 2.0.0-alpha0 type:support Support issues comp:ops OPs related issues labels May 3, 2019

achandraa assigned jvishnuvardhan and unassigned achandraa May 8, 2019

jvishnuvardhan added the stat:awaiting response Status - Awaiting response from author label May 22, 2019

jvishnuvardhan closed this as completed May 24, 2019

justusschock mentioned this issue Jun 4, 2019

Trainer refactoring delira-dev/delira#66

Merged

gowthamkpr mentioned this issue Jul 30, 2019

FailedPreconditionError when training BERT on Colab #31148

Closed

eliadl mentioned this issue Aug 7, 2019

Tensorflow backend - bug in model._make_predict_function(...) keras-team/keras#2397

Closed

2 tasks

eliadl mentioned this issue Aug 7, 2019

'The Session graph is empty' when run behind Flask API keras-team/keras#10585

Closed

BraveDistribution mentioned this issue Sep 10, 2019

FailedPreconditionError PrimozGodec/keras-explain#5

Open

amitnjha mentioned this issue Oct 12, 2019

Error while reading resource variable dense_2/bias from Container: localhost. This could mean that the variable was uninitialized. cleverhans-lab/cleverhans#1117

Closed

WisTiCeJEnT mentioned this issue Oct 20, 2019

save session for running with webbase app(Flask) rkcosmos/deepcut#59

Merged

loretoparisi mentioned this issue Oct 26, 2019

Tensor zero_padding2d_1_input:0, specified in either feed_devices or fetch_devices was not found in the Graph #33744

Closed

lvenugopalan added the TF 2.0 Issues relating to TensorFlow 2.0 label Apr 29, 2020

FHof mentioned this issue May 14, 2020

Fix the self.model.predict crash Sentdex/Carla-RL#9

Open

dizwe mentioned this issue Jun 10, 2020

ML Flask 사용해서 속도 빠르게 하기 b-chae/VibrationFromWarningForHearingImpaired#55

Closed

JesperDramsch mentioned this issue Jun 11, 2020

Error in ComplexBatchNormalization JesperDramsch/keras-complex#6

Closed

Saduf2019 mentioned this issue Jul 10, 2020

Error when running inference on CPU using C++ bindings: Not found: Container localhost does not exist #41226

Closed

kabirwalia8300 mentioned this issue Jul 24, 2020

Aspect-classification for different data UKPLab/controlled-argument-generation#3

Closed

amahendrakar mentioned this issue Aug 13, 2020

saving a large kersa model through exception #42142

Closed

bhack mentioned this issue Sep 22, 2020

failed precondition: error while reading resource variable block8_sepconv3_bn/gamma #43452

Closed

lminer mentioned this issue Jan 11, 2021

Resource exhausted error cortexlabs/cortex#1740

Closed

UsharaniPagadala mentioned this issue Jul 7, 2021

Container __per_step_0 does not exist #50631

Closed

MeghnaNatraj mentioned this issue Jul 28, 2021

Adding some Boosted tree ops to the 'allowed' list #50801

Closed

tilakrayal mentioned this issue Aug 19, 2021

How can I make multiple sessions share a graph? Not found: Container localhost does not exist #51568

Closed

SennriSyunnga mentioned this issue Aug 24, 2021

【JavaAPI】IllegalStateException happended while running a model loading from SavedModel and the graph instance cant close itself #51648

Closed

sanatmpa1 mentioned this issue Nov 23, 2021

Try in tensorflow2.6 by myself. But had some problems. Hope get some help! keras-team/keras#15676

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost" #28287

At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost" #28287

haitham-b commented Apr 30, 2019

haitham-b commented May 15, 2019

jvishnuvardhan commented May 22, 2019

MischaPanch commented May 22, 2019 •

edited

dsaichand3 commented May 23, 2019 •

edited

MischaPanch commented May 23, 2019

jvishnuvardhan commented May 24, 2019

wangcj05 commented Jul 11, 2019 •

edited

aachoo commented Jul 18, 2019

MischaPanch commented Jul 19, 2019

aachoo commented Jul 19, 2019

sangurocactus commented Jul 29, 2019 •

edited

georgealexandruvlad commented Jul 29, 2019

eliadl commented Aug 7, 2019

eliadl commented Apr 17, 2020

akash-harijan commented May 5, 2020

AllanNyasha commented May 10, 2020

vamsikri1234 commented May 13, 2020

amalmathews commented Jun 2, 2020 •

edited

tochile commented Jun 25, 2020

tochile commented Jun 25, 2020

juan-csv commented Jul 2, 2020

lminer commented Jan 11, 2021

minniekabra commented May 17, 2021 •

edited

omarabdullalghafoor commented May 24, 2021

Yessmin commented Jun 9, 2021 •

edited

dorbittonn commented Dec 5, 2021

NeighborhoodCoding commented Dec 7, 2021

At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost" #28287

At Runtime : "Error while reading resource variable softmax/kernel from Container: localhost" #28287

Comments

haitham-b commented Apr 30, 2019

haitham-b commented May 15, 2019

jvishnuvardhan commented May 22, 2019

MischaPanch commented May 22, 2019 • edited

dsaichand3 commented May 23, 2019 • edited

MischaPanch commented May 23, 2019

jvishnuvardhan commented May 24, 2019

wangcj05 commented Jul 11, 2019 • edited

aachoo commented Jul 18, 2019

MischaPanch commented Jul 19, 2019

aachoo commented Jul 19, 2019

sangurocactus commented Jul 29, 2019 • edited

georgealexandruvlad commented Jul 29, 2019

eliadl commented Aug 7, 2019

eliadl commented Apr 17, 2020

akash-harijan commented May 5, 2020

AllanNyasha commented May 10, 2020

vamsikri1234 commented May 13, 2020

amalmathews commented Jun 2, 2020 • edited

tochile commented Jun 25, 2020

tochile commented Jun 25, 2020

juan-csv commented Jul 2, 2020

lminer commented Jan 11, 2021

minniekabra commented May 17, 2021 • edited

omarabdullalghafoor commented May 24, 2021

Yessmin commented Jun 9, 2021 • edited

dorbittonn commented Dec 5, 2021

NeighborhoodCoding commented Dec 7, 2021

IMPORTANT: models have to be loaded AFTER SETTING THE SESSION for keras!

Otherwise, their weights will be unavailable in the threads after the session there has been set

MischaPanch commented May 22, 2019 •

edited

dsaichand3 commented May 23, 2019 •

edited

wangcj05 commented Jul 11, 2019 •

edited

sangurocactus commented Jul 29, 2019 •

edited

amalmathews commented Jun 2, 2020 •

edited

minniekabra commented May 17, 2021 •

edited

Yessmin commented Jun 9, 2021 •

edited