How can I make multiple sessions share a graph? Not found: Container localhost does not exist

[hankewei]

How can I make multiple sessions share a graph by c_api?
My usage is as follows:
The first session is created using TF_LoadSessionFromSavedModel.
The second session is created using TF_NewSession.

It is normal to call the TF_SessionRun of the first session. However, when calling the TF_SessionRun of the second session from another thread, the following error is reported:

2021-08-19 14:56:00.772594: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at resource_variable_ops.cc:633 : Not found: Container localhost does not exist. (Could not find resource: localhost/embeddings/charactor_embeddings/weight)
{{function_node __inference__inference_5828_specialized_for_StatefulPartitionedCall_StatefulPartitionedCall_at_tf_graph}} {{function_node _inference__inference_5828_specialized_for_StatefulPartitionedCall_StatefulPartitionedCall_at_tf_graph}} Error while reading resource variable encoder/layer.0/attention/self/key/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/encoder/layer.0/attention/self/key/bias)
[[{{node encoder/layer._0/attention/self/key/BiasAdd/ReadVariableOp}}]]
terminate called after throwing an instance of 'std::runtime_error'
what(): {{function_node __inference__inference_5828_specialized_for_StatefulPartitionedCall_StatefulPartitionedCall_at_tf_graph}} {{function_node _inference__inference_5828_specialized_for_StatefulPartitionedCall_StatefulPartitionedCall_at_tf_graph}} Error while reading resource variable encoder/layer.0/attention/self/key/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/encoder/layer.0/attention/self/key/bias)
[[{{node encoder/layer._0/attention/self/key/BiasAdd/ReadVariableOp}}]]
[[StatefulPartitionedCall/StatefulPartitionedCall]]
[[StatefulPartitionedCall/StatefulPartitionedCall]]Aborted (core dumped)

Why? Usage problem or bug? We look forward to your reply.

System information
tensorflow version(2.3.0)
Linux Ubuntu 16.04

code snippet:

static pthread_mutex_t sSpeakerClassToGraphMutex = PTHREAD_MUTEX_INITIALIZER;
static std::map<const void*, const void*> sSpeakerClassToGraph;

Model::Model(const void* classID, const std::string& model_filename, const std::vector<uint8_t>& config_options) {

const char* v = TF_Version();
printf("Model********************************************************************** v(%s)\n", v);
    this->status = TF_NewStatus();
// Create the session.
TF_SessionOptions* sess_opts = TF_NewSessionOptions();

if (!config_options.empty())
{
	TF_SetConfig(sess_opts, static_cast<const void*>(config_options.data()), config_options.size(), this->status);
	this->status_check(true);
}


pthread_mutex_lock(&sSpeakerClassToGraphMutex);

 std::map<const void*, const void*>::const_iterator iter = sSpeakerClassToGraph.find(classID);
if (iter != sSpeakerClassToGraph.cend())
{
	this->graph = (TF_Graph* ) iter->second;
	printf("(%p, %p)\n", classID, this->graph);
	this->session =TF_NewSession(this->graph,sess_opts,status);
}
else
{
	this->graph = TF_NewGraph();
	TF_Buffer* RunOpts = NULL;
	const char* tags = "serve";
	int ntags = 1;
	this->session = TF_LoadSessionFromSavedModel(sess_opts, RunOpts, model_filename.c_str(), &tags, ntags, this->graph, NULL, this->status);
	sSpeakerClassToGraph[classID] = graph;
	printf("(%p, %p) create\n", classID, graph);
}
pthread_mutex_unlock(&sSpeakerClassToGraphMutex);
if (TF_GetCode(this->status) == TF_OK)
{
	printf("TF_LoadSessionFromSavedModel OK\n");
}
else
{
	printf("%s", TF_Message(this->status));
}
TF_DeleteSessionOptions(sess_opts);

// Check the status
this->status_check(true);

}

[tilakrayal]

@hankewei ,
In order to expedite the trouble-shooting process, could you please provide the complete code and dataset to reproduce the issue reported here.Thanks!

[tilakrayal]

Also please take a look at this issue with similiar error.It helps.Thanks

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No