Initial Implementation of MediaTek Backend for Executorch #3571

neuropilot-captain · 2024-05-10T10:43:53Z

This pull request introduces the initial implementation of the MediaTek backend for Executorch. Below are the key points and instructions related to this contribution:

The "Ahead of Time" compilation feature is not included in this pull request due to technical limitations. We plan to address this and include it in a subsequent update.
The build process and execution phase rely on libraries provided by the MediaTek NeuroPilot SDK. For the purpose of verification, these libraries will be made available offline.

Build Instructions:

To build the MediaTek backend for Executorch, please adhere to the guidelines provided in the backends/mediatek/scripts/README.md file within this repository.

On-device Execution:

For instructions on how to run an example on MediaTek devices, refer to the documentation in examples/mediatek/README.md.

Llama Example:

A sample Llama runner is built together with MediaTek backend.

- Add backends/mediatek & examples/mediatek - Add runtime: Neuron ExecuTorch Backend - Add example: mtk_executor_runner

pytorch-bot · 2024-05-10T10:43:55Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/3571

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kimishpatel · 2024-05-10T16:00:09Z

backends/mediatek/scripts/README.md

+Download Android NDK and make sure that $ANDROID_NDK is specified as the NDK Path.
+
+3. Download **MediaTek ExercuTorch libraries**
+Download related libraries.


What are these and where do you get it from? I would like to repro

kimishpatel · 2024-05-10T16:01:31Z

examples/mediatek/README.md

@@ -0,0 +1,29 @@
+# ExecuTorch Neuron Backend examples
+
+## Directory structure


Can you add pre-requisite here as to which phones support MTK chipset? ALso do add "validated on" to indicate which phone or chipset this is validated on

Thanks for your suggestion! We have added the list of supported chips that the examples are validated on. Also, we have refactor this README for better readability.

kimishpatel · 2024-05-10T16:02:17Z

examples/mediatek/README.md

+Please follow the build step in backends/mediatek/scripts.
+The mtk_executor_runner will be automatically generated.
+
+## Execute


How do you generate the pte file?

kimishpatel · 2024-05-10T16:04:56Z

I dont see any instructions/code on the ahead of time component

pytorch-bot · 2024-05-10T17:22:25Z

Please seek CI approval before scheduling CIFlow labels

mergennachin · 2024-05-10T17:27:11Z

Thanks for your contribution and PR.

Will take a look at it deeply soon. I just kicked off OSS CI jobs. One thing, we have a lint rule. Here's how you can invoke:

https://github.com/pytorch/executorch/blob/main/CONTRIBUTING.md#coding-style

mergennachin · 2024-05-10T17:34:17Z

The build process and execution phase rely on libraries provided by the MediaTek NeuroPilot SDK. For the purpose of verification, these libraries will be made available offline.

Is there a public link that we can download from MediaTek website? If not, is there a plan to publish it publicly?

swolchok

just dropping in with a bunch of small C++ suggestions; feel free to apply them or not

backends/mediatek/runtime/include/NeuronBackend.h

backends/mediatek/runtime/NeuronExecutor.cpp

backends/mediatek/runtime/include/NeuronLog.h

swolchok · 2024-05-10T17:05:12Z

examples/mediatek/executor_runner/llama_runner/MultiModelLoader.cpp

+    ET_CHECK_MSG(
+        GetModelInstance() == nullptr,
+        "Model is already initialized before calling LoadModels.");
+    void* instance = CreateModelInstance(mModelPathMap[id]);


s/mModelPathMap[id]/modelPath/ , no need to look it up again

swolchok · 2024-05-10T17:05:52Z

examples/mediatek/executor_runner/llama_runner/MultiModelLoader.cpp

+}
+
+template <typename IdType>
+std::string MultiModelLoader<IdType>::GetModelPath() const {


shouldn't this return a const reference to avoid copying?

swolchok · 2024-05-10T17:08:11Z

examples/mediatek/executor_runner/llama_runner/llm_helper/include/rotary_embedding.h

+    size_t getRotEmbedLength() const;
+
+private:
+    char* mMasterLut; // byte flatten array


why not use std::unique_ptr<char[]>?

swolchok · 2024-05-10T17:10:22Z

examples/mediatek/executor_runner/llama_runner/llm_helper/token_embedding.cpp

+    if (mLutBuffer != nullptr) {
+        delete mLutBuffer;
+    }


deleting nullptr is fine

this should be delete[]

but why not use std::unique_ptr<uint8_t[]> and avoid the whole thing?

swolchok · 2024-05-10T17:11:43Z

examples/mediatek/executor_runner/mtk_executor_runner.cpp

+  size_t num_memory_planned_buffers = method_meta->num_memory_planned_buffers();
+  for (size_t id = 0; id < num_memory_planned_buffers; ++id) {


nit: it's probably not performance-critical, but reserving these vectors before filling them wouldn't hurt

As answered in the conversation below.
The current implementation of mtk_executor_runner is based on examples/portable/executor_runner, and we'll look into executorch/extension/module to determine if we need to modify the code.

examples/mediatek/executor_runner/mtk_executor_runner.cpp

cccclai

Thank you pushing the first PR! Looks great :)

Overall I think NeuronBackend and mtk_executor_runner are reasonable. It will be great if we can share the expected flow for users to generate the .pte file and run it.

Regarding the llama runner logic, I can see why we need the multimodel loader and the caching logic, but it's not quite clear to me what logic from .sois needed here. Maybe we can walk through it.

cccclai · 2024-05-14T15:25:37Z

backends/mediatek/runtime/NeuronBackend.cpp

+        return Error::InvalidState;
+    };
+
+    auto& allocator = GET_NEURON_ALLOCATOR;


We can set up temp allocator pointing to neuron allocator, which will be passed to NeuronExecuTorchDelegate::execute

cccclai · 2024-05-14T15:27:56Z

backends/mediatek/runtime/include/NeuronBufferAllocator.h

+    size_t mSize = 0;
+};
+
+class BufferAllocator {


It looks to me that this allocator is for allocating memory in Neuron, but not CPU, is it correct?

Yes, you are right! The BufferAllocator is mainly used for allocating memory within the Neuron. However, the allocated buffers are also accessible to the CPU, allowing for shared use between the Neuron and the CPU.

cccclai · 2024-05-14T15:30:37Z

backends/mediatek/runtime/NeuronBackend.cpp

+                                            ArrayRef<CompileSpec> compile_specs) const {
+    NeuronDelegateSetting setting;
+    for (auto& compile_spec : compile_specs) {
+        if (std::strcmp(compile_spec.key, kHighAddrKey) == 0) {


Glad that you're leveraging the compile spec! I'm trying to follow what these keys mean, maybe we can discuss about it.

cccclai · 2024-05-14T15:35:24Z

backends/mediatek/runtime/NeuronExecutor.cpp

+
+    // ---------------------------Execution------------------------------------
+    // Run the compiled model against a set of inputs
+    err = NeuronExecution_create(compilation, &execution);


Any specific reason we need to execute the model during compilation stage? Is it for verification purpose only?

No, this function only creates an execution instance for execution, not executing it.

The comment said "Run the compiled model against a set of inputs", and I thought it was executing the model. If not, can we update the comment to make it clear?

cccclai · 2024-05-14T15:37:37Z

backends/mediatek/runtime/NeuronBackend.cpp

+    LogInfo("NeuronBackend", "version %u, input %u, output %u, length %u, payload size: %zu",
+                 Payload.Header.Version, Payload.Header.InputCount, Payload.Header.OutputCount, Payload.Header.DataLen, processed->size());
+
+    auto delegate = std::unique_ptr<NeuronExecuTorchDelegate>(


Any specific reason we have a NeuronExecuTorchDelegate wrapper but not applying the logic inside NeuronBackend directly?

cccclai · 2024-05-14T15:38:29Z

backends/mediatek/runtime/NeuronBackend.cpp

+    return mExecutor.Compute() == NEURON_NO_ERROR ? Error::Ok : Error::InvalidState;
+};
+
+int NeuronExecuTorchDelegate::HintNeuronBackend(EValue** args) const {


I'm trying to understand what HintNeuronBackend mean - is there some import forever issue?

'import forever' is an optimization hint designed for specific models including Llama. We prefer not to automatically apply it to every model because it could leads unknow effects.

cccclai · 2024-05-14T19:51:21Z

examples/mediatek/executor_runner/llama_runner/LlamaConfig.h

+  size_t prompt_token_batch_size = 1;
+  size_t cache_size        = 1024;
+  size_t hidden_size       = 4096;
+  size_t num_head          = 32;
+  size_t num_layer         = 32;
+  size_t max_token_length  = 2048;
+  double rot_emb_base      = 10000.0f;


this data ideally don't need to be hard-coded but retrieved from the model. If they're hardcoded, something might go wrong if users export a llama-based model but with different model arch

cccclai · 2024-05-14T19:51:48Z

examples/mediatek/executor_runner/llama_runner/LlamaConfig.h

+  double rot_emb_base      = 10000.0f;
+
+  // Types
+  LLMType model_input_type  = LLMType::INT16;


Hmm how do you quantize the model?

cccclai · 2024-05-14T19:52:41Z

examples/mediatek/executor_runner/llama_runner/LlamaModelChunk.cpp

+  return indexes;
+}
+
+LlamaModelChunk::LlamaModelChunk(


Does user export 4 .pte file or 1 .pte file?

- Fix typos in NeuronExecutor.cpp. - Remove duplicate APUWareUtilsLib.h in CMakeLists.txt. - Update NEURON_BUFFER_ALLOCATOR to NEURON_BUFFER_ALLOCATOR_LIB. - Clean up mtk_build.sh and set BUCK_PATH. - Modify ReadSystemProperty signature to take string reference.

- Add MediaTek libraries introduction - Improve readability

- Add the list of supported devices - Improve readability

backends/mediatek/scripts/README.md

- minor wording change

- Fix Buck2 download link

- Add command example

Changed kHighAddrKey and kImportForeverKey from constexpr to extern in NeuronBackend.h. Defined kHighAddrKey and kImportForeverKey in NeuronBackend.cpp. Moved backend registration logic to an anonymous namespace in NeuronBackend.cpp.

neuropilot-captain added 3 commits May 8, 2024 17:49

MediaTek Neuron ExecuTorch Backend

cce2ef1

- Add backends/mediatek & examples/mediatek - Add runtime: Neuron ExecuTorch Backend - Add example: mtk_executor_runner

Set NeuronDelegateSetting from compile specs

c2fefd0

Add MediaTek Llama runner

eac37ac

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 10, 2024

mergennachin requested review from mergennachin and kimishpatel May 10, 2024 15:56

kimishpatel reviewed May 10, 2024

View reviewed changes

mergennachin added the ciflow/trunk label May 10, 2024

pytorch-bot bot removed the ciflow/trunk label May 10, 2024

swolchok reviewed May 10, 2024

View reviewed changes

shoumikhin reviewed May 11, 2024

View reviewed changes

examples/mediatek/executor_runner/mtk_executor_runner.cpp Show resolved Hide resolved

cccclai reviewed May 14, 2024

View reviewed changes

neuropilot-captain and others added 4 commits May 17, 2024 15:08

Merge branch 'pytorch:main' into main

345181c

Update README.md

a2d505c

- Add MediaTek libraries introduction - Improve readability

Update README.md

04f14b9

- Add the list of supported devices - Improve readability

cccclai reviewed May 22, 2024

View reviewed changes

backends/mediatek/scripts/README.md Outdated Show resolved Hide resolved

neuropilot-captain and others added 6 commits May 23, 2024 09:38

Update README.md

d671511

- minor wording change

Update README.md

dc59f82

- Fix Buck2 download link

Update README.md

3fb9461

- Add command example

Merge branch 'pytorch:main' into main

2524c11

Merge branch 'pytorch:main' into main

22d0e16

Refactor NeuronBackend constants and registration

151e6fc

Changed kHighAddrKey and kImportForeverKey from constexpr to extern in NeuronBackend.h. Defined kHighAddrKey and kImportForeverKey in NeuronBackend.cpp. Moved backend registration logic to an anonymous namespace in NeuronBackend.cpp.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial Implementation of MediaTek Backend for Executorch #3571

Initial Implementation of MediaTek Backend for Executorch #3571

neuropilot-captain commented May 10, 2024

pytorch-bot bot commented May 10, 2024 •

edited

kimishpatel May 10, 2024

kimishpatel May 10, 2024

neuropilot-captain May 22, 2024

kimishpatel May 10, 2024

kimishpatel commented May 10, 2024

pytorch-bot bot commented May 10, 2024

mergennachin commented May 10, 2024

mergennachin commented May 10, 2024

swolchok left a comment

swolchok May 10, 2024

swolchok May 10, 2024

swolchok May 10, 2024

swolchok May 10, 2024

swolchok May 10, 2024

neuropilot-captain May 17, 2024

cccclai left a comment

cccclai May 14, 2024

cccclai May 14, 2024

neuropilot-captain May 15, 2024

cccclai May 14, 2024

cccclai May 14, 2024

neuropilot-captain May 15, 2024

cccclai May 29, 2024

cccclai May 14, 2024

cccclai May 14, 2024

neuropilot-captain May 15, 2024

cccclai May 14, 2024

cccclai May 14, 2024

cccclai May 14, 2024

		@@ -0,0 +1,29 @@
		# ExecuTorch Neuron Backend examples

		## Directory structure

		size_t num_memory_planned_buffers = method_meta->num_memory_planned_buffers();
		for (size_t id = 0; id < num_memory_planned_buffers; ++id) {

Initial Implementation of MediaTek Backend for Executorch #3571

Are you sure you want to change the base?

Initial Implementation of MediaTek Backend for Executorch #3571

Conversation

neuropilot-captain commented May 10, 2024

Build Instructions:

On-device Execution:

Llama Example:

pytorch-bot bot commented May 10, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/3571

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kimishpatel commented May 10, 2024

pytorch-bot bot commented May 10, 2024

mergennachin commented May 10, 2024

mergennachin commented May 10, 2024

swolchok left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cccclai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pytorch-bot bot commented May 10, 2024 •

edited