TFLite Interpreter fails to load quantized model on Android (stock ssd_mobilenet_v2)

[dkashkin]

System information
Android 5.1.1 on LGL52VL, also tested on Android 9 Simulator (Nexus 5)
Latest Tensorflow in build.gradle:
compile 'org.tensorflow:tensorflow-lite:+'

Also, please include a link to a GraphDef or the model if possible.
http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_quantized_300x300_coco_2019_01_03.tar.gz

Any other info / logs
`
// I convert the stock quantized mobilenet_v2 to TFLite using the following code:
converter = tf.lite.TFLiteConverter.from_frozen_graph(graphDefPath, input_arrays, output_arrays, input_shapes)
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.allow_custom_ops=True
converter.convert()

//The resulting .tflite file works fine in Python (including inference):
interpreter = tf.lite.Interpreter(model_path=modelFilePath)
interpreter.allocate_tensors()

// When I try to load the same tflite model file on Android the Interpreter constructor gives me error "Didn't find op for builtin opcode 'CONV_2D' version '2'":

MappedByteBuffer buffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
Interpreter.Options options = new Interpreter.Options();
tfLite = new Interpreter(buffer, options);

E/flutter ( 7796): [ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: PlatformException(error, Unsupported value: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Didn't find op for builtin opcode 'CONV_2D' version '2'
E/flutter ( 7796): Registration failed.
E/flutter ( 7796): , null)
E/flutter ( 7796): #0 StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:564:7)
E/flutter ( 7796): #1 MethodChannel.invokeMethod (package:flutter/src/services/platform_channel.dart:302:33)
E/flutter ( 7796):
E/flutter ( 7796): #2 Tflutter.loadModelAndLabels (package:tflutter/tflutter.dart:129:20)
E/flutter ( 7796):
E/flutter ( 7796): #3 Detector.initialize (package:AIM/services/detector.dart:51:26)
E/flutter ( 7796):
E/flutter ( 7796): #4 _MonitorState._initStateAsync (package:AIM/screens/Monitor.dart:47:20)
E/flutter ( 7796): #5 _AsyncAwaitCompleter.start (dart:async-patch/async_patch.dart:49:6)
E/flutter ( 7796): #6 _MonitorState._initStateAsync (package:AIM/screens/Monitor.dart:45:25)
E/flutter ( 7796): #7 _MonitorState.initState (package:AIM/screens/Monitor.dart:42:5)
E/flutter ( 7796): #8 StatefulElement._firstBuild (package:flutter/src/widgets/framework.dart:3853:58)
E/flutter ( 7796): #9 ComponentElement.mount (package:flutter/src/widgets/framework.dart:3724:5)
E/flutter ( 7796): #10 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #11 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #12 SingleChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:4883:14)
E/flutter ( 7796): #13 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #14 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #15 ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3757:16)
E/flutter ( 7796): #16 Element.rebuild (package:flutter/src/widgets/framework.dart:3572:5)
E/flutter ( 7796): #17 ComponentElement._firstBuild (package:flutter/src/widgets/framework.dart:3729:5)
E/flutter ( 7796): #18 ComponentElement.mount (package:flutter/src/widgets/framework.dart:3724:5)
E/flutter ( 7796): #19 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #20 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #21 SingleChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:4883:14)
E/flutter ( 7796): #22 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #23 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #24 SingleChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:4883:14)
E/flutter ( 7796): #25 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #26 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #27 SingleChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:4883:14)
E/flutter ( 7796): #28 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #29 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #30 SingleChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:4883:14)
E/flutter ( 7796): #31 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #32 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #33 ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3757:16)
E/flutter ( 7796): #34 Element.rebuild (package:flutter/src/widgets/framework.dart:3572:5)
E/flutter ( 7796): #35 ComponentElement._firstBuild (package:flutter/src/widgets/framework.dart:3729:5)
E/flutter ( 7796): #36 StatefulElement._firstBuild (package:flutter/src/widgets/framework.dart:3871:11)
E/flutter ( 7796): #37 ComponentElement.mount (package:flutter/src/widgets/framework.dart:3724:5)
E/flutter ( 7796): #38 Element.inflateWidget (package:flutter/src/widgets/framework.dart:2968:14)
E/flutter ( 7796): #39 Element.updateChild (package:flutter/src/widgets/framework.dart:2771:12)
E/flutter ( 7796): #40 ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:3757:16)
E/flutter ( 7796): #41 Element.rebuild (package:flutter/src/widgets/framework.dart:3572:5)
E/flutter ( 7796): #42 ComponentElement._firstBuild (package:flutter/src/widgets/framework.dart:3729:5)
E/flutter ( 7796): #43 ComponentElement.mount (package:flutter/src/widgets/fra
`
PS. The same Android code works fine for non-quantized tflite models that I converted, so I am confident that there should not be any silly bugs in my code. The problem seems to be triggered by Quantized + Android TFlite

[sinaasadiyan]

i am facing the same error !

[ele33a]

yes, i'm also having problems with this the exact same problem. i'm using the next lines for convertion:

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
tflite_model = converter.convert()

Also.. ¿What is the correct way to preprocess an image in order to make a non-quantized model get the correct prediction in android?

[prateethvnayak]

Use thepip install -U tf-nightly and it should solve the Op problem. Even while converting.

The tf nightly has the most latest build of tensorflow. And the most recent is two days ago from today.

[MichalMisiaszek]

I have installed tf-nightly and converted model but it still would not load:

E/CustomCompatChecker(19664): The model is INCOMPATIBLE. It may contain unrecognized custom ops, or not FlatBuffer format: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Didn't find op for builtin opcode 'CONV_2D' version '2'
E/CustomCompatChecker(19664): Registration failed.
W/System.err(19664): com.google.firebase.ml.common.FirebaseMLException: Remote model load failed with the model options: Local model name: unspecified. Remote model name: detector.
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzon.zza(Unknown Source:33)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzpr.zza(Unknown Source:109)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzpr.zzln(Unknown Source:105)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzoa.zzf(Unknown Source:56)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zznt.call(Unknown Source:3)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zznn.zza(Unknown Source:30)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zznq.run(Unknown Source:2)
W/System.err(19664): 	at android.os.Handler.handleCallback(Handler.java:873)
W/System.err(19664): 	at android.os.Handler.dispatchMessage(Handler.java:99)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzf.dispatchMessage(Unknown Source:6)
W/System.err(19664): 	at android.os.Looper.loop(Looper.java:214)
W/System.err(19664): 	at android.os.HandlerThread.run(HandlerThread.java:65)
W/System.err(19664): Caused by: com.google.firebase.ml.common.FirebaseMLException: Model is not compatible with TFLite run time
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzpd.zza(Unknown Source:61)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzpk.zzah(Unknown Source:52)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzpk.load(Unknown Source:18)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzon.zza(Unknown Source:41)
W/System.err(19664): 	at com.google.android.gms.internal.firebase_ml.zzon.zza(Unknown Source:14)
W/System.err(19664): 	... 11 more
E/FirebaseModelInterpreter(19664): Remote model load failed with the model options: Local model name: unspecified. Remote model name: detector.
W/System.err(19664): com.google.android.gms.tasks.RuntimeExecutionException: com.google.firebase.ml.common.FirebaseMLException: Remote model load failed with the model options: Local model name: unspecified. Remote model name: detector.
W/System.err(19664): 	at com.google.android.gms.tasks.zzu.getResult(Unknown Source:15)
W/System.err(19664): 	at com.azihsoyn.flutter.mlkit.MlkitPlugin$10.then(MlkitPlugin.java:358)
W/System.err(19664): 	at com.azihsoyn.flutter.mlkit.MlkitPlugin$10.then(MlkitPlugin.java:350)
W/System.err(19664): 	at com.google.android.gms.tasks.zzd.run(Unknown Source:5)
W/System.err(19664): 	at android.os.Handler.handleCallback(Handler.java:873)
W/System.err(19664): 	at android.os.Handler.dispatchMessage(Handler.java:99)
W/System.err(19664): 	at android.os.Looper.loop(Looper.java:214)
W/System.err(19664): 	at android.app.ActivityThread.main(ActivityThread.java:7032)
W/System.err(19664): 	at java.lang.reflect.Method.invoke(Native Method)
W/System.err(19664): 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:494)
W/System.err(19664): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:965)
W/System.err(19664): Caused by: com.google.firebase.ml.common.FirebaseMLException: Remote model load failed with the model options: Local model name: unspecified. Remote model name: detector.

[aayushthakur]

I am trying to load a custom trained quantized model and the android sample code of TensorFlow gives me the same error as above. But if I use the float model instead, it works perfectly. Does the android dependency implementation 'org.tensorflow:tensorflow-lite:1.13.1' not yet supports the CONV_2D conversion parameter?

[jdduke]

You'll need org.tensorflow:tensorflow-lite:1.14.0, which was just released this past week, or try the nightly.

[tensorflow-bot]

Are you satisfied with the resolution of your issue?
Yes
No

[aayushthakur]

I no longer face the above error but now if I use my custom quantized model and I get this error
java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite buffer with 602112 bytes and a ByteBuffer with 150528 bytes.

If I use a custom float model then there are no errors.

My code is as follows:

@Override
 public List<Recognition> recognizeImage(Bitmap bitmap) {
        ByteBuffer byteBuffer = convertBitmapToByteBuffer(bitmap);
        if(quant){
            byte[][] result = new byte[1][labelList.size()]; //labelList.size() = 10;
            interpreter.run(byteBuffer, result);
            return getSortedResultByte(result);
        } else {
            float [][] result = new float[1][labelList.size()];
            interpreter.run(byteBuffer, result);
            return getSortedResultFloat(result);
        }
}

  private ByteBuffer convertBitmapToByteBuffer(Bitmap bitmap) {
        ByteBuffer byteBuffer;

        if(quant) {
            byteBuffer = ByteBuffer.allocateDirect(1 * 224 * 224 * 3);
        } else {
            byteBuffer = ByteBuffer.allocateDirect(4 * 1 * 224 * 224 * 3);
        }

        byteBuffer.order(ByteOrder.nativeOrder());
        int[] intValues = new int[224 * 224];
        bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());
        int pixel = 0;
        for (int i = 0; i < 224; ++i) {
            for (int j = 0; j < 224; ++j) {
                final int val = intValues[pixel++];
                if(quant){
                    byteBuffer.put((byte) ((val >> 16) & 0xFF));
                    byteBuffer.put((byte) ((val >> 8) & 0xFF));
                    byteBuffer.put((byte) (val & 0xFF));
                } else {
                    byteBuffer.putFloat((((val >> 16) & 0xFF))/255.0f);
                    byteBuffer.putFloat((((val >> 8) & 0xFF))/255.0f);
                    byteBuffer.putFloat((((val) & 0xFF))/255.0f);
                }

            }
        }
        return byteBuffer;
    }

 @SuppressLint("DefaultLocale")
    private List<Recognition> getSortedResultByte(byte[][] labelProbArray) {

        PriorityQueue<Recognition> pq =
                new PriorityQueue<>(
                        MAX_RESULTS,
                        new Comparator<Recognition>() {
                            @Override
                            public int compare(Recognition lhs, Recognition rhs) {
                                return Float.compare(rhs.getConfidence(), lhs.getConfidence());
                            }
                        });

        for (int i = 0; i < labelList.size(); ++i) {
            float confidence = (labelProbArray[0][i] & 0xff) / 255.0f;
            if (confidence > THRESHOLD) {
                pq.add(new Recognition("" + i,
                        labelList.size() > i ? labelList.size() : "unknown",
                        confidence, quant));
            }
        }

        final ArrayList<Recognition> recognitions = new ArrayList<>();
        int recognitionsSize = Math.min(pq.size(), MAX_RESULTS);
        for (int i = 0; i < recognitionsSize; ++i) {
            recognitions.add(pq.poll());
        }

        return recognitions;
    }

[jdduke]

How did you quantize your model? Note that, with post-training quantization, the default approach is to preserve float inputs/outputs, implicitly quantizing/dequantizing as necessary. You can check the input tensor type using interpreter.getInputTensor(0).dataType(), or by using the visualization script.

[aayushthakur]

My Model takes FLOAT32 as input, but when I use the float code mentioned above the code works fine but the time taken increases dramatically (2.3 seconds per image) as compared to my float model (0.8 seconds per image). Float Model file is which is of larger size ~80MB as compared to post-training Quantized Model size ~20MB.

[jdduke]

Just to confirm, you're saying that the quantized model (which is 4x smaller), takes 3x longer during inference? One thing to try is to explicitly set the number of threads when executing your model (e.g., compare single-threaded performance). We're still working on multi-threading support for the "hybrid" quantization path for post-training quantization. 
With the new post-training integer quantization path, if you can provide a representative dataset at conversion time, you should get both the size reduction and a substantial inference latency reduction (this includes multi-threading support).

[aayushthakur]

Yes, you understood it correctly. Currently, I am running the classifier class from a DB class as soon as media path gets inserted into my DB, I pass the media path to classifier methods which then runs respective code to get the confidence probabilities. So this whole thing is running in a background thread synchronously. One thing which I don't get is why the smaller model takes more time than the larger one. The quantized model is supposed to be faster. Is there any other specific way to train the model for quantization so that I can use the byte code instead of the float as used in the sample application?

[jdduke]

One thing which I don't get is why the smaller model takes more time than the larger one.

In this case, it's because you're using a "hybrid" quantized model, which doesn't yet have multi-threading support. I would try using the new post-training, integer quantization path, which should get you both faster execution and a smaller model.

If you don't mind, could you attach the command you used to convert your model? And the generated .tflite model? Feel free to PM me directly if you prefer not to share openly. Thanks.