Not able to port a 6-layered mobilenet tflite model to mobile

[ychen404]

Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.
3. It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

---

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  No.
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Ubuntu 16.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  Pixel 2
• TensorFlow installed from (source or binary):
  Source
• TensorFlow version (use command below):
  1.8
• Python version:
  2.7
• Bazel version (if compiling from source):
  0.15.2
• GCC/Compiler version (if compiling from source):
  5.4.0
• CUDA/cuDNN version:
  N/A
• GPU model and memory:
  N/A
• Exact command to reproduce:

1. Set the mobileNet model endpoint to conv6-depthwise
2. Re-train the model from scratch with using cifar10 dataset
3. Freeze the graph with the checkpoints
   import tensorflow as tf
   from tensorflow.python.framework import graph_util
   import os,sys

output_node_names = "MobilenetV1/Predictions/Reshape"
saver = tf.train.import_meta_graph('/home/users/saman/yitao/tensorflow_android/models/research/slim/batch_32/model.ckpt-156300.meta', clear_devices=True)
graph = tf.get_default_graph()
input_graph_def = graph.as_graph_def()
sess = tf.Session()
saver.restore(sess, "/home/users/saman/yitao/tensorflow_android/models/research/slim/batch_32/model.ckpt-156300")
output_graph_def = graph_util.convert_variables_to_constants(
sess, # The session is used to retrieve the weights
input_graph_def, # The graph_def is used to retrieve the nodes
output_node_names.split(",") # The output node names are used to select the usefull nodes
)
output_graph="frozen-model-conv6-bat-32.pb"
with tf.gfile.GFile(output_graph, "wb") as f:
f.write(output_graph_def.SerializeToString())
sess.close()

(4) Optimize the model
bazel-bin/tensorflow/tools/graph_transforms/transform_graph
--in_graph=/home/yitao/TF_1.8/tensorflow/my_frozen_pb/frozen-model-conv6-bat-32.pb
--out_graph=/home/yitao/TF_1.8/tensorflow/my_frozen_pb/frozen-model-conv6-bat-32-optimized.pb
--inputs='input'
--outputs='MobilenetV1/Predictions/Reshape'
--transforms='
strip_unused_nodes(type=float, shape="1,32,32,3")
remove_nodes(op=Identity, op=CheckNumerics)
fold_constants(ignore_errors=true)
fold_batch_norms
fold_old_batch_norms

(5) Convert the model to tflite
bazel-bin/tensorflow/contrib/lite/toco/toco
--input_file=/home/yitao/TF_1.8/tensorflow/my_frozen_pb/frozen-model-conv6-bat-32-optimized.pb
--input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE
--output_file=/home/yitao/TF_1.8/tensorflow/my_frozen_pb/frozen-model-conv6-bat-32-optimized.tflite --inference_type=FLOAT
--input_type=FLOAT --input_arrays=input
--seed2
--output_arrays=MobilenetV1/Predictions/Reshape --input_shapes=1,32,32,3
--allow_custom_ops

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Not able to train a model from scratch and port it Android to utilize the Android Nerual Network api through TFLite. After training a model and following the steps to convert the graph to tflite model, there are still some ops that are not supported by the TFLite runtime in my graph. What should I do?
Any help is apreciated!

Logcat is throwing the following errors. It seems that those ops are not stripped from the model during the optimization step.

Source code / logs

08-03 15:14:52.183 10271-10271/android.example.com.tflitecamerademo E/AndroidRuntime: FATAL EXCEPTION: main
Process: android.example.com.tflitecamerademo, PID: 10271
java.lang.RuntimeException: Unable to start activity ComponentInfo{android.example.com.tflitecamerademo/com.example.android.tflitecamerademo.CameraActivity}: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Didn't find custom op for name 'RandomUniform' with version 1
Didn't find custom op for name 'FLOOR' with version 1
Didn't find custom op for name 'RSQRT' with version 1
Didn't find custom op for name 'FIFOQueueV2' with version 1
Didn't find custom op for name 'QueueDequeueV2' with version 1
Didn't find custom op for name 'SquaredDifference' with version 1
Registration failed.

[ychen404]

Following is the log of the TOCO converter.

tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 224 operators, 311 arrays (0 quantized)
2018-08-03 14:39:47.544660: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 224 operators, 311 arrays (0 quantized)
2018-08-03 14:39:47.544705: W tensorflow/contrib/lite/toco/graph_transformations/resolve_constant_random_uniform.cc:85] RandomUniform op outputting "MobilenetV1/Logits/Dropout_1b/dropout/random_uniform/RandomUniform" is truly random (using /dev/random system entropy). Therefore, cannot resolve as constant. Set "seed" or "seed2" attr non-zero to fix this
2018-08-03 14:39:47.547793: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 179 operators, 277 arrays (0 quantized)
2018-08-03 14:39:47.548771: W tensorflow/contrib/lite/toco/graph_transformations/resolve_constant_random_uniform.cc:85] RandomUniform op outputting "MobilenetV1/Logits/Dropout_1b/dropout/random_uniform/RandomUniform" is truly random (using /dev/random system entropy). Therefore, cannot resolve as constant. Set "seed" or "seed2" attr non-zero to fix this
2018-08-03 14:39:47.550304: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 179 operators, 277 arrays (0 quantized)
2018-08-03 14:39:47.552372: I tensorflow/contrib/lite/toco/allocate_transient_arrays.cc:329] Total transient array allocated size: 131072 bytes, theoretical optimal value: 131072 bytes.

I noticed that the RandomUniform is complaining about the seed. Is there a way I can set the seed for the /dev/random from the system?
Thanks.

[gragundier]

Correct me if I'm wrong, but those ops seem to be part of your model. If it's part of the model, it won't get stripped and you'll need custom ops for them. I'm curious about your model can you post it?

[ychen404]

Hi gragundier,

Thanks for your reply.
It is a simplified version of mobilenet v1. I changed the end point of the mobilenet model.
You can find the end point in models/research/slim/nets/mobilenet_v1.py
I changed that to Conv2d_6_pointwise layer.

def mobilenet_v1_base(inputs,
                      final_endpoint='Conv2d_6_pointwise',
                       min_depth=8,
                       depth_multiplier=1.0,
                       conv_defs=None,
                       output_stride=None,
                       use_explicit_padding=False,
                       scope=None):

[aselle]

It could be that you are now feeding in a placeholder input. You seem to have a queue and some loss functions. Could you provide the frozen graphdef (or even a screenshot of the graph) so that we can see what else it could be. @gargn, could you comment.

[ychen404]

Hi,

Attached is the frozen model.
frozen-model-conv6-bat-32.zip

[ychen404]

I am still not able to understand this fully, although I was able to find a way to work around this problem.
I need to provide an eval model to the freeze_graph script instead of using the one I save while training my model.
If I train a model and save the pbtxt file, which contains the graph, should TensoFlow freeze_graph be able to remove all the ops that are not related to inference? Because we only freeze a model when we do not want to update the weights anymore.
I found this quite inconvenient to write up another graph only for inference.
Please correct me if I have any misunderstanding on this, thanks!

[gargn]

I ran the following command on the TensorFlow nightly build (installed using the command pip install tf-nightly). The command resulted in the error ValueError: Invalid tensors 'input' were found.:

tflite_convert \
--graph_def_file=$TENSORFLOW_FILE \
--output_file=$TFLITE_FILE \
--input_arrays=input \
--output_arrays=MobilenetV1/Predictions/Reshape \
--input_shapes=1,32,32,3
--allow_custom_ops

I add this point because this error seems different than the one that you noted. I looked into the model using TensorBoard and it appears that your model is a MobileNet training graph containing the ops FIFOQueueV2, QueueDequeueV2, SquaredDifference. TensorFlow Lite only works with eval graphs, not training graphs.

In order to create a MobileNet eval graph:

1. Create a separate eval graph. You can use this MobileNet eval script to generate it from the checkpoints: https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1_eval.py
2. Pass the correct input and output nodes to the freeze_graph.py script. I suggest using the command line tool.

After you do this, try using the tflite_convert command above from the tf-nightly build if possible. You might need different inputs and outputs. If you run into any issues during this process, please provide any intermediate files including the meta graph and checkpoint files as well as any updated commands.

[ychen404]

@gargn hi,

Thanks for your reply!
My workaround is below, which I believe is similar to what you pointed out.
It seems that the construct of the eval graph is based on the slim and arg_scope.
But what should I do if I am trying to train/deploy a custom model which does not slim in the model definition? I do not want to rewrite my model using slim.....
Can you tell me what exactly does the eval model contain?

import tensorflow as tf
slim = tf.contrib.slim
from nets import mobilenet_v1

NUM_CLASSES = 10

def export_eval_pbtxt():
  """Export eval.pbtxt."""
  with tf.Graph().as_default() as g:
    images = tf.placeholder(dtype=tf.float32,shape=[None,32,32,3])
    # using one of the following methods to create graph, depends on you
    #_, _ = mobilenet_v1.mobilenet_v1(inputs=images,num_classes=NUM_CLASSES, is_training=False)
    with slim.arg_scope(mobilenet_v1.mobilenet_v1_arg_scope(is_training=False,regularize_depthwise=True)):
      _, _ = mobilenet_v1.mobilenet_v1(inputs=images, is_training=False, depth_multiplier=1.0, num_classes=NUM_CLASSES)
    eval_graph_file = '/home/users/saman/yitao/tensorflow_android/models/research/slim/mobilenet_v1_eval.pbtxt'
    with tf.Session() as sess:
        with open(eval_graph_file, 'w') as f:
            f.write(str(g.as_graph_def()))

def main():
    print("python main function")
    export_eval_pbtxt()

if __name__ == '__main__':
    main()

[ychen404]

I trained a straightforward model which contains only two convolutional layers. The freeze and tflite conversion when smoothly, but when I deploy to mobile, the application through a segmentation fault.
Thanks.

[suharshs]

Since you are able to convert and the segfault is a new issue, can you please provide the resulting segfault stack trace/core dump?

[ychen404]

Hi following is the error message. Is that the stack trace you referred to?

08-19 13:39:32.244 1583-1663/? A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 1663 (CameraBackgroun), pid 1583 (flitecamerademo)
08-19 13:39:32.244 1134-1161/? I/ActivityManager: Displayed android.example.com.tflitecamerademo/com.example.android.tflitecamerademo.CameraActivity: +401ms
08-19 13:39:32.272 1724-1724/? I/crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstone
08-19 13:39:32.272 873-873/? I//system/bin/tombstoned: received crash request for pid 1583
08-19 13:39:32.272 1724-1724/? I/crash_dump64: performing dump of process 1583 (target tid = 1663)
08-19 13:39:32.273 1724-1724/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/walleye/walleye:8.1.0/OPM2.171019.029/4657601:user/release-keys'
Revision: 'MP1'
ABI: 'arm64'
pid: 1583, tid: 1663, name: CameraBackgroun >>> android.example.com.tflitecamerademo <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
Cause: null pointer dereference
x0 00000072d8403000 x1 0000000000000000 x2 0000000000060000 x3 00000072d8403000
x4 0000000000060000 x5 00000072d8463000 x6 0000000000000000 x7 0000000000430001
x8 00000072ea9538a0 x9 0000000000000000 x10 00000072ea953800 x11 0000000000000050
x12 0000007378abe250 x13 4d4630dd011c21f3 x14 000000737a2d0000 x15 ffffffffffffffff
x16 00000072dea53008 x17 00000073792b02f0 x18 000000000000000a x19 00000072f5842900
x20 000000000000002a x21 00000072ea9d6500 x22 0000000000000050 x23 0000000000000000
x24 00000072ea9d6500 x25 00000072ea953800 x26 00000072ea9d6538 x27 00000072f5842920
x28 0000000000000000 x29 00000072d82f8110 x30 00000072de995a94
sp 00000072d82f8110 pc 00000073792b03d8 pstate 0000000020000000
08-19 13:39:32.279 1724-1724/? A/DEBUG: backtrace:
#00 pc 000000000001c3d8 /system/lib64/libc.so (memcpy+232)
#1 pc 00000000000bba90 /data/app/android.example.com.tflitecamerademo-KoNQ6lWiyX75U9zVntbapA==/lib/arm64/libtensorflowlite_jni.so
#2 pc 00000000000d9aa8 /data/app/android.example.com.tflitecamerademo-KoNQ6lWiyX75U9zVntbapA==/lib/arm64/libtensorflowlite_jni.so
#3 pc 00000000000122e4 /data/app/android.example.com.tflitecamerademo-KoNQ6lWiyX75U9zVntbapA==/lib/arm64/libtensorflowlite_jni.so (Java_org_tensorflow_lite_NativeInterpreterWrapper_run+32)
#4 pc 0000000000553bf0 /system/lib64/libart.so (art_quick_generic_jni_trampoline+144)
#5 pc 000000000054ae4c /system/lib64/libart.so (art_quick_invoke_static_stub+604)
#6 pc 00000000000dc5d0 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+264)
#7 pc 000000000029b49c /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
#8 pc 0000000000295a90 /system/lib64/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+700)
#9 pc 0000000000533f50 /system/lib64/libart.so (MterpInvokeStatic+264)
#10 pc 000000000053ca94 /system/lib64/libart.so (ExecuteMterpImpl+14612)
#11 pc 0000000000275c00 /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+444)
#12 pc 000000000027b7cc /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+216)
#13 pc 0000000000295a70 /system/lib64/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+668)
#14 pc 0000000000532ad8 /system/lib64/libart.so (MterpInvokeVirtual+652)
#15 pc 000000000053c914 /system/lib64/libart.so (ExecuteMterpImpl+14228)
#16 pc 0000000000275c00 /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+444)
#17 pc 000000000027b7cc /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+216)
#18 pc 0000000000295a70 /system/lib64/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+668)
#19 pc 0000000000532ad8 /system/lib64/libart.so (MterpInvokeVirtual+652)
#20 pc 000000000053c914 /system/lib64/libart.so (ExecuteMterpImpl+14228)
#21 pc 0000000000275c00 /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+444)
#22 pc 000000000027b7cc /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+216)
#23 pc 0000000000295a70 /system/lib64/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+668)
#24 pc 0000000000532ad8 /system/lib64/libart.so (MterpInvokeVirtual+652)
#25 pc 000000000053c914 /system/lib64/libart.so (ExecuteMterpImpl+14228)
#26 pc 0000000000275c00 /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+444)
#27 pc 000000000027b7cc /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame*, art::JValue*)+216)
#28 pc 0000000000295a70 /system/lib64/libart.so (_ZN3art11interpreter6DoCallILb0ELb0EEEbPNS_9ArtMethodEPNS_6ThreadERNS_11ShadowFrameEPKNS_11InstructionEtPNS_6JValueE+668)
#29 pc 0000000000532ad8 /system/lib64/libart.so (MterpInvokeVirtual+652)
#30 pc 000000000053c914 /system/lib64/libart.so (ExecuteMterpImpl+14228)
#31 pc 0000000000275c00 /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::DexFile::CodeItem const*, art::ShadowFrame&, art::JValue, bool)+444)
#32 pc 0000000000525450 /system/lib64/libart.so (artQuickToInterpreterBridge+1052)
#33 pc 0000000000553d0c /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
08-19 13:39:32.280 1724-1724/? A/DEBUG: #34 pc 00000000000070f8 /dev/ashmem/dalvik-jit-code-cache (deleted)
08-19 13:39:32.558 764-1668/? I/EaselControlClient: easelConnThread: Opening easel_conn

[tensorflowbutler]

Nagging Assignee @gargn: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[gargn]

I was able to get the following code working with last night's tf-nightly. It is based off of the Python code that you provided. The main difference is that it freezes the graph and converts the Flatbuffer to a TFLite model within the Python code itself. Can you clarify if this is what you are looking for:

import tensorflow as tf
import numpy as np
slim = tf.contrib.slim
from nets import mobilenet_v1

NUM_CLASSES = 10
MOBILENET_FILENAME = 'PATH-TO-DATA/mobilenet_v1_eval.pbtxt'

INPUT_ARRAYS = ['input']
OUTPUT_ARRAYS = ['MobilenetV1/Predictions/Reshape']

def export_eval_pbtxt():
  """Export eval.pbtxt."""
  with tf.Graph().as_default() as g:
    # Need to provide the name in order to have the name of the input arrays for conversion.
    images = tf.placeholder(dtype=tf.float32,shape=[None,32,32,3], name=INPUT_ARRAYS[0])
    # using one of the following methods to create graph, depends on you
    # _, _ = mobilenet_v1.mobilenet_v1(inputs=images,num_classes=NUM_CLASSES, is_training=False)
    with slim.arg_scope(mobilenet_v1.mobilenet_v1_arg_scope(is_training=False,regularize_depthwise=True)):
     _, _ = mobilenet_v1.mobilenet_v1(inputs=images, is_training=False, depth_multiplier=1.0, num_classes=NUM_CLASSES)

    with tf.Session().as_default() as sess:
        sess.run(tf.global_variables_initializer())
        # Freeze the graph so that you can convert to TFLite it later.
        frozen_graph = tf.graph_util.convert_variables_to_constants(
            sess, sess.graph_def, OUTPUT_ARRAYS)
        with open(MOBILENET_FILENAME, 'w') as f:
            f.write(str(frozen_graph))

def main():
    print("python main function")
    export_eval_pbtxt()

    # Convert the graph.
    converter = tf.contrib.lite.TocoConverter.from_frozen_graph(
            MOBILENET_FILENAME, INPUT_ARRAYS, OUTPUT_ARRAYS)
    tflite_model = converter.convert()

    # Load TFLite model and allocate tensors.
    interpreter = tf.contrib.lite.Interpreter(model_content=tflite_model)
    interpreter.allocate_tensors()

    # Get input and output tensors.
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()

    # Test model on random input data.
    input_shape = input_details[0]['shape']
    input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
    interpreter.set_tensor(input_details[0]['index'], input_data)

    interpreter.invoke()
    output_data = interpreter.get_tensor(output_details[0]['index'])
    print(output_data)

if __name__ == '__main__':
    main()

In order to get the from models import mobilenet_v1 working, I had to download the models repository and update the PYTHONPATH via the command: export PYTHONPATH=PATH-TO-MODELS/models/research/slim:$PYTHONPATH

[gargn]

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!