.Net: Slow performance while using Native Functions Plugins with AutoInvokeKernelFunctions?

[ankiga-MSFT]

Discussed in #5912

^{Originally posted by ankiga-MSFT April 17, 2024}
We are building an AI assistant using Semantic Kernel. We have written a set of Native Functions in a Plugin, and we are using the AutoInvokeKernelFunctions for AI assistant to identify the right function(s) and chain the calls to finally produce the result.

We are observing that while doing so, Semantic Kernel is giving very slow performance.

To rule out issues with the network calls inside our functions & client logic, we tried to fiddle with the example as shown here

Tried a sample question - "Take the square root of 17 and 11 and add the results. Finally, round off the value to 1 decimal place."

This is resulting in 4 function calls (Sqrt + Sqrt + Add + Round) which is perfectly fine. But in order to do the entire operation, it took a whopping 48 seconds. The model gpt-4 was used for this Test deployed in an Azure Open AI instance.

Adding a Stepwise Planner slightly improved the performance (35 seconds) but still it is slow, considering the fact that hitting the same query using the prompt (without any Math Plugins) yields the result in 6 seconds.

Any idea on what is going wrong here, or how we can improve performance for similar use cases involving multiple function calls.

[matthewbolanos]

This sounds like how the OpenAI model works "by design" (and isn't necessarily behavior that specific to Semantic Kernel). For something like "Take the square root of 17 and 11 and add the results" each step needs to happen sequentially so that the AI can use the results of the previous calculation. This means multiple AI requests need to be made (one for each step).

For operations like this, we're in the process of building a Python plugin that will allow the LLM to author python code to complete multiple steps at the same time. This functionality will allow you to mimic the code interpreter feature in the Assistant API. This should be available around Microsoft Build at the end of this month.