Tensorflow freezes on iOS during Session::Run

[vahidk]

Tensorflow hangs on iOS during Session::Run. I have a deep LSTM model that requires running session.run many times. The program occasionally hangs after running a few sessions without consuming any cpu. Tensorflow seems to get stuck at DirectSession::WaitForNotification.

What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?

#2121
#2788

Environment info

Operating System: iOS

git rev-parse HEAD: e60e724

Build label: 0.2.3
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue May 17 14:22:21 2016 (1463494941)
Build timestamp: 1463494941
Build timestamp as int: 1463494941

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

std::vector<tensorflow::Tensor> outputs;
for (int t = 0; t < count; ++t) {        
    std::vector<std::pair<std::string, tensorflow::Tensor>> feed =....
    auto status = g_session->Run(feed, out_layer_names, {}, &outputs);
    if (!status.ok()) {
        LOG(ERROR) << status.ToString();
        return @"Internal Error!";
    }
    ....
}

Logs or other output that would be helpful

This is a stack trace of all of the threads when the program freezes:

• thread Add support for Python 3.x #1: tid = 0x206350, 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP frame #0: 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8
  frame Add support for Python 3.x #1: 0x000000018331c9c0 libsystem_pthread.dylib_pthread_cond_wait + 640 frame #2: 0x0000000182c453ec libc++.1.dylibstd::__1::condition_variable::wait(std::__1::unique_lockstd::__1::mutex&) + 56
  frame JVM, .NET Language Support #3: 0x00000001000ef6fc Apptensorflow::DirectSession::WaitForNotification(tensorflow::Notification*, long long) + 176 frame #4: 0x00000001000eb1cc Apptensorflow::DirectSession::WaitForNotification(tensorflow::DirectSession::RunState*, tensorflow::CancellationManager*, long long) + 48
  frame Java interface #5: 0x00000001000e91b8 App`tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::__1::vector<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, tensorflow::Tensor>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, tensorflow::Tensor> > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, std::__1::vector<tensorflow::Tensor, std::__1::allocatortensorflow::Tensor >, tensorflow::RunMetadata) + 1868

  • frame Pre-trained models #6: 0x00000001000e8a40 Apptensorflow::DirectSession::Run(std::__1::vector<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, tensorflow::Tensor>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, tensorflow::Tensor> > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::vector<tensorflow::Tensor, std::__1::allocator<tensorflow::Tensor> >*) + 112 frame #7: 0x000000010052e01c Apptensorflow::internal::AppendProtoDebugString(tensorflow::strings::ProtoTextOutput*, tensorflow::Feature const&) + 6028
    frame Setting lower gcc version for cuda #8: 0x000000010053c138 Appmain + 24316 frame #9: 0x0000000100535f00 Apptensorflow::internal::AppendProtoDebugString(tensorflow::strings::ProtoTextOutput*, tensorflow::Feature const&) + 38512
    frame Go API #10: 0x0000000100535794 Apptensorflow::internal::AppendProtoDebugString(tensorflow::strings::ProtoTextOutput*, tensorflow::Feature const&) + 36612 frame #11: 0x000000018a173d30 UIKit-[UIApplication sendAction:to:from:forEvent:] + 96
    frame Remote worker configuration #12: 0x000000018a2e7880 UIKit-[UIBarButtonItem(UIInternal) _sendAction:withEvent:] + 168 frame #13: 0x000000018a173d30 UIKit-[UIApplication sendAction:to:from:forEvent:] + 96
    frame g3doc format #14: 0x000000018a173cb0 UIKit-[UIControl sendAction:to:forEvent:] + 80 frame #15: 0x000000018a15e128 UIKit-[UIControl _sendActionsForEvents:withEvent:] + 452
    frame iOS Support and Example #16: 0x000000018a15e290 UIKit-[UIControl _sendActionsForEvents:withEvent:] + 812 frame #17: 0x000000018a17359c UIKit-[UIControl touchesEnded:withEvent:] + 584
    frame C# api #18: 0x000000018a1730c4 UIKit-[UIWindow _sendTouchesForEvent:] + 2484 frame #19: 0x000000018a16e328 UIKit-[UIWindow sendEvent:] + 2988
    frame CUDA 7.5 fails with pip install and docker (Ubuntu 14.04) #20: 0x000000018a13eda0 UIKit-[UIApplication sendEvent:] + 340 frame #21: 0x000000018a92875c UIKit__dispatchPreprocessedEventFromEventQueue + 2736
    frame OpenCL support #22: 0x000000018a922130 UIKit__handleEventQueue + 784 frame #23: 0x0000000184236b5c CoreFoundationCFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION + 24
    frame Problems running the image example (Python 2.7.10, PyEnv, Xubuntu 14.04 64bit) #24: 0x00000001842364a4 CoreFoundation__CFRunLoopDoSources0 + 524 frame #25: 0x00000001842340a4 CoreFoundation__CFRunLoopRun + 804
    frame simplify contributing process #26: 0x00000001841622b8 CoreFoundationCFRunLoopRunSpecific + 444 frame #27: 0x0000000185c16198 GraphicsServicesGSEventRunModal + 180
    frame Could port to OpenCL? #28: 0x000000018a1a97fc UIKit-[UIApplication _run] + 684 frame #29: 0x000000018a1a4534 UIKitUIApplicationMain + 208
    frame Go API #30: 0x00000001005362b4 Appmain + 120 frame #31: 0x00000001831455b8 libdyld.dylibstart + 4

  thread Installation over pip fails to import with protobuf 2.6.1 #4: tid = 0x206397, 0x000000018331ad88 libsystem_pthread.dylibstart_wqthread frame #0: 0x000000018331ad88 libsystem_pthread.dylibstart_wqthread

  thread Setting lower gcc version for cuda #8: tid = 0x20639b, 0x0000000183239188 libsystem_kernel.dylibmach_msg_trap + 8, name = 'com.apple.uikit.eventfetch-thread' frame #0: 0x0000000183239188 libsystem_kernel.dylibmach_msg_trap + 8
  frame Add support for Python 3.x #1: 0x0000000183238ff8 libsystem_kernel.dylibmach_msg + 72 frame #2: 0x00000001842365d0 CoreFoundation__CFRunLoopServiceMachPort + 192
  frame JVM, .NET Language Support #3: 0x00000001842341ec CoreFoundation__CFRunLoopRun + 1132 frame #4: 0x00000001841622b8 CoreFoundationCFRunLoopRunSpecific + 444
  frame Java interface #5: 0x0000000184c9f26c Foundation-[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 304 frame #6: 0x0000000184cbfdd0 Foundation-[NSRunLoop(NSRunLoop) runUntilDate:] + 96
  frame API docs does not list RNNs #7: 0x000000018ab1dc38 UIKit-[UIEventFetcher threadMain] + 136 frame #8: 0x0000000184d9ce68 FoundationNSThread__start + 1024
  frame Typo in getting started guide #9: 0x000000018331d850 libsystem_pthread.dylib_pthread_body + 240 frame #10: 0x000000018331d760 libsystem_pthread.dylib_pthread_start + 284
  frame 0.5.0 wheel install on Mac OS X using Homebrew python broken #11: 0x000000018331ad94 libsystem_pthread.dylib`thread_start + 4

  thread Typo in getting started guide #9: tid = 0x2063c0, 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8 frame #0: 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8
  frame Add support for Python 3.x #1: 0x000000018331c9c0 libsystem_pthread.dylib_pthread_cond_wait + 640 frame #2: 0x0000000182c453ec libc++.1.dylibstd::__1::condition_variable::wait(std::__1::unique_lockstd::__1::mutex&) + 56
  frame JVM, .NET Language Support #3: 0x000000010019535c Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 6296 frame #4: 0x0000000100194e40 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 4988
  frame Java interface #5: 0x00000001001949dc Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3864 frame #6: 0x00000001001946e4 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3104
  frame API docs does not list RNNs #7: 0x00000001001a5828 Appvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::function<void ()> > >(void*) + 100 frame #8: 0x000000018331d850 libsystem_pthread.dylib_pthread_body + 240
  frame Typo in getting started guide #9: 0x000000018331d760 libsystem_pthread.dylib_pthread_start + 284 frame #10: 0x000000018331ad94 libsystem_pthread.dylibthread_start + 4

  thread Go API #10: tid = 0x2063c1, 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8 frame #0: 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8
  frame Add support for Python 3.x #1: 0x000000018331c9c0 libsystem_pthread.dylib_pthread_cond_wait + 640 frame #2: 0x0000000182c453ec libc++.1.dylibstd::__1::condition_variable::wait(std::__1::unique_lockstd::__1::mutex&) + 56
  frame JVM, .NET Language Support #3: 0x000000010019535c Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 6296 frame #4: 0x0000000100194e40 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 4988
  frame Java interface #5: 0x00000001001949dc Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3864 frame #6: 0x00000001001946e4 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3104
  frame API docs does not list RNNs #7: 0x00000001001a5828 Appvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::function<void ()> > >(void*) + 100 frame #8: 0x000000018331d850 libsystem_pthread.dylib_pthread_body + 240
  frame Typo in getting started guide #9: 0x000000018331d760 libsystem_pthread.dylib_pthread_start + 284 frame #10: 0x000000018331ad94 libsystem_pthread.dylibthread_start + 4

  thread 0.5.0 wheel install on Mac OS X using Homebrew python broken #11: tid = 0x2063c2, 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8 frame #0: 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8
  frame Add support for Python 3.x #1: 0x000000018331c9c0 libsystem_pthread.dylib_pthread_cond_wait + 640 frame #2: 0x0000000182c453ec libc++.1.dylibstd::__1::condition_variable::wait(std::__1::unique_lockstd::__1::mutex&) + 56
  frame JVM, .NET Language Support #3: 0x000000010019535c Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 6296 frame #4: 0x0000000100194e40 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 4988
  frame Java interface #5: 0x00000001001949dc Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3864 frame #6: 0x00000001001946e4 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3104
  frame API docs does not list RNNs #7: 0x00000001001a5828 Appvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::function<void ()> > >(void*) + 100 frame #8: 0x000000018331d850 libsystem_pthread.dylib_pthread_body + 240
  frame Typo in getting started guide #9: 0x000000018331d760 libsystem_pthread.dylib_pthread_start + 284 frame #10: 0x000000018331ad94 libsystem_pthread.dylibthread_start + 4

  thread Remote worker configuration #12: tid = 0x2063c3, 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8 frame #0: 0x0000000183256e1c libsystem_kernel.dylib__psynch_cvwait + 8
  frame Add support for Python 3.x #1: 0x000000018331c9c0 libsystem_pthread.dylib_pthread_cond_wait + 640 frame #2: 0x0000000182c453ec libc++.1.dylibstd::__1::condition_variable::wait(std::__1::unique_lockstd::__1::mutex&) + 56
  frame JVM, .NET Language Support #3: 0x000000010019535c Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 6296 frame #4: 0x0000000100194e40 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 4988
  frame Java interface #5: 0x00000001001949dc Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3864 frame #6: 0x00000001001946e4 Apptensorflow::thread::ThreadPool::CurrentThreadId() const + 3104
  frame API docs does not list RNNs #7: 0x00000001001a5828 Appvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::function<void ()> > >(void*) + 100 frame #8: 0x000000018331d850 libsystem_pthread.dylib_pthread_body + 240
  frame Typo in getting started guide #9: 0x000000018331d760 libsystem_pthread.dylib_pthread_start + 284 frame #10: 0x000000018331ad94 libsystem_pthread.dylibthread_start + 4

[petewarden]

I got some advice from @mrry and he explained that DirectSession::WaitForNotification blocking means that the run call is never signalling that it's finishing. On iOS we should only ever have one thread executing ops (inter-op parallelism in the arguments) so it should be tough to get into this condition. From Derek: "my guess is (i) some op that we don't really support on that platform is running and failing in a bad way (perhaps an AsyncOpKernel?), or (ii) some Session functionality that we haven't really tested on that platform is being used and failing in a bad way (perhaps a run timeout?)"

To help understand this, can you run tensorflow/tools/graph_transforms:summarize_graph on your model file, so we can see which ops are being used?

[vahidk]

This is the result of running summarize_graph:

Found 7 possible inputs: (name=token, type=int32(3), shape=[1]) (name=lstm_state/0/c/in, type=float(1), shape=[1,512]) (name=lstm_state/0/h/in, type=float(1), shape=[1,512]) (name=lstm_state/1/c/in, type=float(1), shape=[1,512]) (name=lstm_state/1/h/in, type=float(1), shape=[1,512]) (name=lstm_state/2/c/in, type=float(1), shape=[1,512]) (name=lstm_state/2/h/in, type=float(1), shape=[1,512])
Found 7 possible outputs: (name=lstm_state/0/c/out, op=Identity) (name=lstm_state/0/h/out, op=Identity) (name=lstm_state/1/c/out, op=Identity) (name=lstm_state/1/h/out, op=Identity) (name=lstm_state/2/c/out, op=Identity) (name=lstm_state/2/h/out, op=Identity) (name=logits, op=Identity)
Found 19 consts, 0 variables, 14 identities, and 0 control_edges
Op types used: 19 Const, 14 Identity, 9 Mul, 9 Sigmoid, 7 Placeholder, 7 Tanh, 6 Add, 4 MatMul, 3 BiasAdd, 3 Concat, 3 Split, 2 Reshape, 1 Gather, 1 Pack, 1 Unpack

[vahidk]

Any progress on this?

[jam-world]

Same problem, Is there any progress? Thanks.

[ratzinho87]

I am having the same issue running the inceptionV3 network (the one offered by tensorflow/models).

[andreapiso]

Problem still persists, bad random freezes on iOS.

[ratzinho87]

@AndreaPisoni I have solved my problem by setting inter_op_parallelism_threads and intra_op_parallelism_threads to 1. This would make Tensorflow not use parallelism at all. My session creation code looks like this:

    tensorflow::SessionOptions options;
    options.config.mutable_graph_options()
        ->mutable_optimizer_options()
        ->set_opt_level(::tensorflow::OptimizerOptions::L0);
    options.config.set_inter_op_parallelism_threads(1);
    options.config.set_intra_op_parallelism_threads(1);
    options.env = memmapped_env->get();
    tensorflow::Session* session_pointer = nullptr;
    tensorflow::Status session_status = tensorflow::NewSession(options, &session_pointer);
    if (!session_status.ok()) {
        LOG(ERROR) << "Could not create TensorFlow Session: " << session_status;
        return session_status;
    }

Setting just inter_op_parallelism_threads to 1, as suggested by @petewarden was not enough for me.

[andreapiso]

Brilliant, I will give it a try, thanks! Just wondering, does this make the inference any slower?

[h4x3rotab]

Thank you! It works for me but it's still far from a solution. I'm wondering why the official demo "camera" can work smoothly...

[guilt]

Is someone still looking into the issue? Do we have logs on what the notifier thread is doing?

[gazzterran]

I met the same problem. use LSTM, and fixed by the upper method "setting inter_op_parallelism_threads and intra_op_parallelism_threads to 1".
Is there any better methods to fix it ? Or tensorflow on IOS not support LSTM?

[cancan101]

We are also seeing this issue on iOS. We believe the issue is caused by a where op which does in fact seem to have a fair bit of threading related code.

[gazzterran]

@cancan101 use the lastest version still freeze. Has someone fixed?

[liamnaka]

@gazzterran did you try what @ratzinho87 suggested?

options.config.set_inter_op_parallelism_threads(1); options.config.set_intra_op_parallelism_threads(1);

[gazzterran]

@liamnaks I've tried the "inter_op_parallelism_threads" method and it works in most of time ( but it still occurs during testing, in a very small probability, like 1/100)

From @cancan101 said, is there some bugs in code?

[cancan101]

Why close the issue? I believe it still exists on new builds of TF.

[petewarden]

A fix for general threading issues on iOS went in last month, in #12573

Could you let me know if this solves your case also? It appears to fix #12298.

[cancan101]

Did it make it into 1.3?

[aselle]

It does not look like it is in 1.3

[adennie]

@cancan101 did the fix from #12573 solve this issue for you?

[sadlerjw]

I reported #12298 and I'm using a version including that commit, but I am also having this issue. For me, it seems to only happen when in some other place in my iOS app I have an NSOperationQueue with a ton of operations and myQueue.maxConcurrentOperationCount = NSOperationQueueDefaultMaxConcurrentOperationCount. Maybe that indicates the there's some thread contention between my operation queue and whatever thread Tensorflow is trying to use for its ops?
In any case, setting maxConcurrentOperationCount to a smaller number (for example, 4) fixed the problem for me, so that's a potential workaround.

[eigen2017]

same issue. ios froze occasionally on LSTM,
and bad luck, things worse after i did:

void InitGraph()
{
    Session * session;
    GraphDef graph_def;
    SessionOptions opts;


    TF_CHECK_OK(ReadBinaryProto(Env::Default(), heartPrintPbPathFile, &graph_def));
    opts.config.set_inter_op_parallelism_threads(1);
    opts.config.set_intra_op_parallelism_threads(1);

    TF_CHECK_OK(NewSession(opts, &session));

    TF_CHECK_OK(session->Create(graph_def));

    return session;
}

ios freezes every time i do inferece. -_-#

[eigen2017]

greate news for me ~~ i resolved this issue,
@ratzinho87 :not only set_inter_op_parallelism_threads and set_intra_op_parallelism_threads
but also add_session_inter_op_thread_pool and set_use_per_session_threads !!!!!

here:~

Session * HeartPrint::HpInitGraph()
{
    Session * session;
    GraphDef graph_def;
    SessionOptions opts;

    TF_CHECK_OK(ReadBinaryProto(Env::Default(), heartPrintPbPathFile, &graph_def));

    opts.config.set_intra_op_parallelism_threads(1);
    opts.config.set_inter_op_parallelism_threads(1);
    opts.config.add_session_inter_op_thread_pool()->set_num_threads(1);
    opts.config.set_use_per_session_threads(false);

    TF_CHECK_OK(NewSession(opts, &session));

    TF_CHECK_OK(session->Create(graph_def));

    return session;
}

[aselle]

@eigen2017, is there a PR you need to make, or should this go int he docs somewhere?

[cancan101]

That doesn't seem like a solution as much as a workaround.

[tensorflowbutler]

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

[tensorflowbutler]

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

[tensorflowbutler]

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

[michaelisard]

Closing due to lack of activity, but please reopen if there's some followup needed.