Tensorflow lite GPU delegate can't work well and NNAPI can't make the model faster

[Raphaelzrf]

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

tensorflow-lite-task-text:0.3.0 tensorflow-lite-gpu-delegate-plugin:0.4.4 tensorflow-lite-gpu:2.13.0 tensorflow-lite:2.14.0 tensorflow-lite-support:0.4.4

Custom code

Yes

OS platform and distribution

Linux Ubuntu 22.04

Mobile device

Vivo x100: cpu: MediaTek Dimensity 9300 GPU:Immortalis-G720

Python version

3.10

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

I used TensorFlow Lite Model Maker to fine-tune BERT to get a text classification model. I want to use the GPU to speed up my model. I tried two methods. One is to use NNAPI and the other is to use GPU delegate. But it turned out that NNAPI didn't make my model faster, it was much slower. Before using GPU delegate for acceleration, I used compatList.isDelegateSupportedOnThisDevice() to detect if there is a supported GPU on the device, and the result is no. But in fact there is.

Standalone code to reproduce the issue

fun classifyGpu(text: String){
        val baseOptionsBuilder = BaseOptions.builder()
        lateinit var bertClassifierGpu: BertNLClassifier
        
        baseOptionsBuilder.useNnapi()
   
        val baseOptions = baseOptionsBuilder.build()
        
        val options = BertNLClassifier.BertNLClassifierOptions
                .builder()
                .setBaseOptions(baseOptions)
                .build()

        bertClassifierGpu = BertNLClassifier.createFromFileAndOptions(
                context,
                MOBILEBERT,
                options)

        results = bertClassifierGpu.classify(text)
 
    }

Relevant log output

No response

[sawantkumar]

Hi @Raphaelzrf ,

I don't have a vivo x100 to replicate this issue , but i have vivo x80 and the GPU delegate works fine on that device. What i can suggest you is to use is tflite benchmark tool tflite benchmark tool and see if there is a difference between the timings on NNAPI and GPU delegate. This will confirm the GPU support on your device. Let me know once you are able to do it.

[Raphaelzrf]

@sawantkumar Thank you for your reply. I have tried to use tflite benchmark tool to run Mobilenet and there was a difference between the timings on NNAPI and GPU delegate. So I 'm sure that there is the GPU supported on my device. But compatList.isDelegateSupportedOnThisDevice() return false. How did you use the GPU delegate works on vivo x80? tflite benchmark tool? Have you tried the GPU delegate on an App?If you have, can you share your code?

[sawantkumar]

Hi @Raphaelzrf ,

Yes i was able to access the GPU delegate in an android app , also i was able to access the neuron delegate which is an AI accelerator for your int8 models . The reason behind no speedup could be model specific also. Maybe you can try out any of examples and see for yourself if you are able to access the GPU delegate in these examples.

[Raphaelzrf]

@sawantkumar As I said above, compatList.isDelegateSupportedOnThisDevice() return false. So it has nothing to do with the model. Neuron delegate? I don't konw that. Do you mean using NNAPI to access NPU？

[sawantkumar]

@Raphaelzrf Yes, i meant using NNAPI to access NPU . Looking at all the sources x100 has a relatively powerful gpu as you can see from here . Can you please check if you have any updates available on your phone ?

[Raphaelzrf]

@sawantkumar My device is already up to date. Can you tell me the version of tensorflow-lite and tensorflow-lite-gpu that you use?

[sawantkumar]

@Raphaelzrf i can't recall very clearly what version of tflite it was, but i think it was 2.4.0 . I am looking into your issue btw and will let you know if i come up with something .