Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed precondition: error while reading resource variable block8_sepconv3_bn/gamma #43452

Closed
8rax opened this issue Sep 22, 2020 · 5 comments
Closed
Assignees
Labels
comp:gpu GPU related issues TF 2.3 Issues related to TF 2.3 type:support Support issues

Comments

@8rax
Copy link

8rax commented Sep 22, 2020

Hello,

I have found very few documentation concerning the error : failed precondition: error while reading resource variable block8_sepconv3_bn/gamma from Container: localhost. This could mean that the variable was uninitialized. Not found : Resource localhost/block8_speconv3_bn/gamma/class tensorflow::Var does not exist

I guess the problem must be in the train function before the model.predict and has something to do with the sessions and threads but I would be happy to get any help I can get.

I use :
Python 3.7.8
Cuda 10.1
CUDNN v7.6.5
Tensorflow 2.3.0
keras 2.4.3

You can find my code in attachment or here : https://github.com/8rax/Carla/blob/master/JDO_Tutorial_5.py
Carla_Env_DQN.zip

@8rax 8rax added the type:build/install Build and install issues label Sep 22, 2020
@bhack
Copy link
Contributor

bhack commented Sep 22, 2020

Can you check #28287 (comment)

@8rax
Copy link
Author

8rax commented Sep 22, 2020

Hello, thank you very much for the suggestion. I indeed had a look at that thread as well and tried the solution. But I might have done it wrong. My understanding about sessions, graphs and threads might be a little bit lackluster as well, apologies for that.

I created on line 70 my gpu_options and my sess at the begining of my script after the regular imports:

gpu_options=tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=MEMORY_FRACTION) #TENSORFLOW DON'T USE ALL MY GPU PLZ
sess=set_session(tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(gpu_options=gpu_options)))

I understand I should also declare here the graph globally instead of declaring it in the init function of my DQNAgent and should therefore also put :

graph = tf.compat.v1.get_default_graph()

then, if my understanding of the problem is correct, I should then set the session every time I am calling the .predict method in the train function of my DQNAgent class.

I suppose that I should then use the following syntax:

global sess
global graph
with self.graph.as_default():
  set_session(sess)
  current_qs_list = self.model.predict(current_states, PREDICTION_BATCH_SIZE)
  new_current_states = np.array([transition[3] for transition in minibatch])/255 
global sess
global graph
with self.graph.as_default():
  set_session(sess)
  future_qs_list = self.target_model.predict(new_current_states, PREDICTION_BATCH_SIZE)

Not sure I should add it here though as there is no graph involved :

def get_qs(self, state):
        return self.model.predict(np.array(state).reshape(-1, * state.shape)/255)[0]

would that make sense?

@bhack
Copy link
Contributor

bhack commented Sep 22, 2020

I think that it is not a bug so you can close this.
But if you need a specific support on your use case I suggest you to use our channel at:
https://stackoverflow.com/questions/tagged/tensorflow

@8rax
Copy link
Author

8rax commented Sep 22, 2020

Hi,
Alright thank you for the suggestion, posted it there : https://stackoverflow.com/q/64004994/14319427

@8rax 8rax closed this as completed Sep 22, 2020
@google-ml-butler
Copy link

Are you satisfied with the resolution of your issue?
Yes
No

@amahendrakar amahendrakar added type:others issues not falling in bug, perfromance, support, build and install or feature comp:gpu GPU related issues type:support Support issues TF 2.3 Issues related to TF 2.3 and removed type:build/install Build and install issues type:others issues not falling in bug, perfromance, support, build and install or feature labels Sep 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:gpu GPU related issues TF 2.3 Issues related to TF 2.3 type:support Support issues
Projects
None yet
Development

No branches or pull requests

3 participants