.Net: Invoking kernel function directly with auto invoke enabled results in multiple nested calls to the function

[TaoChenOSU]

Describe the bug
When a kernel function is invoked directly via

public async Task<TResult?> InvokeAsync<TResult>(
        string? pluginName,
        string functionName,
        KernelArguments? arguments = null,
        CancellationToken cancellationToken = default)

, and with auto invoke enabled, it may result in multiple calls to the function, which means multiple calls to the model, causing unnecessary round trips. 

To Reproduce
Steps to reproduce the behavior:

1. Create a simple console app.
2. Copy the WriterPlugin to the project root.
3. Do the following:

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using System;
using System.IO;
using System.Threading.Tasks;

public sealed class Program
{
    /// <summary>
    /// The main entry point for the application.
    /// </summary>
    /// <returns>A <see cref="Task"/> representing the asynchronous operation.</returns>
    public static async Task Main()
    {
        var loggerFactory = new NullLoggerFactory();
        var kernel = GetKernel(loggerFactory);

        OpenAIPromptExecutionSettings settings = new() { ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions };
        KernelArguments arguments = new(settings);
        arguments["input"] = "Write a poem about John Doe.";
        var poem = await kernel.InvokeAsync<string>(
            "WriterPlugin",
            "ShortPoem",
            arguments
        );
        Console.WriteLine($"Poem:\n{poem}\n");
    }

    private static Kernel GetKernel(ILoggerFactory loggerFactory)
    {
        IKernelBuilder builder = Kernel.CreateBuilder();

        builder.Services.AddSingleton(loggerFactory);
#pragma warning disable SKEXP0001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
        builder.Services.AddSingleton<IFunctionInvocationFilter, FirstFunctionFilter>();
#pragma warning restore SKEXP0001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
        builder.AddAzureOpenAIChatCompletion(
            deploymentName: "...",
            modelId: "...",
            endpoint: "...",
            apiKey: "...");

        builder.Plugins.AddFromPromptDirectory(Path.Combine("./", "WriterPlugin"));

        return builder.Build();
    }

#pragma warning disable SKEXP0001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
    private sealed class FirstFunctionFilter() : IFunctionInvocationFilter
    {
        public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)
        {
            Console.WriteLine($"FunctionInvoking - {context.Function.PluginName}.{context.Function.Name}");
            await next(context);
            Console.WriteLine($"FunctionInvoked - {context.Function.PluginName}.{context.Function.Name}");
        }
    }
#pragma warning restore SKEXP0001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
}

4. Run the app and you will see more than one outputs of FunctionInvoking - WriterPlugin.ShortPoem in the terminal.

Expected behavior
A single invocation of the ShortPoem function, regardless if AutoInvokeKernelFunctions is enabled.

With auto invoke disabled, you will see the filter being run only once:

Screenshots
Multiple invocations:

Platform

• OS: Windows
• IDE: VS Code
• Language: C#, Python
• Source: main branch

Additional context
Tested on GPT-4 and GPT-4 turbo

[markwallace-microsoft]

In the above code WriterPlugin.ShortPoem is called directly but also providing it as a function to the model for it to invoke and expecting it to call it. The ShortPoem prompt starts with Generate a short funny poem and the input also has Write a poem about John Doe. When the model is called it doesn't know it's already being called from WriterPlugin.ShortPoem, it sees that function advertised and it's a good fit for the ask so it cals it again.

There are a few things that could help here:

1. Don't use AutoInvokeKernelFunctions as it's not needed because the required function is being called directly
2. Call var poem = await kernel.InvokePromptAsync with the input and allow the model decide what function(s) to call

// Copyright (c) Microsoft. All rights reserved.

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using Resources;

namespace GettingStarted;

/// <summary>
/// This example shows how to create a prompt <see cref="KernelFunction"/> from a YAML resource.
/// </summary>
public sealed class Generate_Poem(ITestOutputHelper output) : BaseTest(output)
{
    /// <summary>
    /// Show how to create a prompt <see cref="KernelFunction"/> from a YAML resource.
    /// </summary>
    [Fact]
    public async Task RunAsync()
    {
        // Create a kernel with OpenAI chat completion
        Kernel kernel = Kernel.CreateBuilder()
            .AddOpenAIChatCompletion(
                modelId: TestConfiguration.OpenAI.ChatModelId,
                apiKey: TestConfiguration.OpenAI.ApiKey)
            .Build();
        kernel.FunctionInvocationFilters.Add(new FunctionInvocationFilter(Output));

        // Load prompt from resource
        var generatePoemYaml = EmbeddedResource.Read("GeneratePoem.yaml");
        var function = kernel.CreateFunctionFromPromptYaml(generatePoemYaml);

        // Add to the Kernel
        kernel.Plugins.AddFromFunctions("WriterPlugin", [function]);

        // Invoke the prompt function and display the result
        OpenAIPromptExecutionSettings settings = new() { ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions };

        // Don't use AutoInvokeKernelFunctions
        Console.WriteLine(await kernel.InvokeAsync(function, arguments: new()
            {
                { "input", "Write a poem about John Doe" },
            }));

        Console.WriteLine();

        // Call prompt with AutoInvokeKernelFunctions
        Console.WriteLine(await kernel.InvokePromptAsync("Write a poem about John Doe", arguments: new(settings)));
    }

    private sealed class FunctionInvocationFilter(ITestOutputHelper output) : IFunctionInvocationFilter
    {
        private readonly ITestOutputHelper _output = output;

        public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)
        {
            this._output.WriteLine($"FunctionInvoking - {context.Function.PluginName}.{context.Function.Name}");
            await next(context);
            this._output.WriteLine($"FunctionInvoked - {context.Function.PluginName}.{context.Function.Name}");
        }
    }
}

name: GeneratePoem
template: |
  Generate a short funny poem or limerick to explain the given event. Be creative and be funny. Let your imagination run wild.
  Event:{{$input}}
template_format: semantic-kernel
description: Turn a scenario into a short and entertaining poem.
input_variables:
  - name: input
    description: The scenario to turn into a poem.
    is_required: true
output_variable:
  description: The generated poem.
execution_settings:
  default:
    max_tokens: 60
    temperature: 0.5
    top_p: 0.0
    presence_penalty: 0.0
    frequency_penalty: 0.0

[TaoChenOSU]

The sample just shows how to reproduce the issue. In reality, the execution settings with auto invoke enabled could come from somewhere else, such as a service selector. It's fairly easy for someone to have auto invoke enabled by default in their apps and still invoke registered functions using InvokeAsync, which could silently result in multiple round trips to the model.

I suggest we have a list of anti-patterns somewhere if we don't suggest users to have auto invoke on and still plan to invoke registered functions directly.

[TaoChenOSU]

Say someone has set up a service selector that returns some default settings with auto invoke enabled.

bool TrySelectAIService<T>(
        Kernel kernel,
        KernelFunction function,
        KernelArguments arguments,
        [NotNullWhen(true)] out T? service,
        out PromptExecutionSettings? serviceSettings) where T : class, IAIService;

And in their apps, they invoke functions (because nothing suggests they should not). They wouldn't notice anything different but in reality, they could be consuming more tokens they need, and their apps will be slower.

[markwallace-microsoft]

Agreed. Possibly we shouldn't advertise the current function to the model, I'm not sure if that would ever be useful.

We have a task to provide more control over what kernel functions are advertised to the model, I can include this use case as part of that.