What's the approach to autocall function using two different kernels?

[MithrilMan]

Ok let me picture the scenario:
I'm creating a web app section where the user can automate some operations:
I get the user request, pass it through a kernel with several plugins that fetch data from our internal API and I use ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions

Now the kernel doing this job (named) has plugin registered at startup
I use this code to trigger it

  var kernelArguments = new KernelArguments(
     new Dictionary<string, object?>(),
     new Dictionary<string, PromptExecutionSettings>(){
        { SERVICE_ID, _promptExecutionSettings } // named service
     });

  var response = await kernel.InvokePromptAsync(
     prompt,
     kernelArguments,
     cancellationToken: cancellationToken
     ).ConfigureAwait(false);

and, beside the model limitations, it's doing it's job and I see it fetches data and use the data to chain function calls as I expected.

Now... this part generates a textual output but I want also to generate a set of encoded actions to perform on the UI.
So I've created another service that uses its own kernel where I register the plugins that should convert the data collected from previous steps of the first kernel and generate required actions

The problem is that I don't know how natively pass the various function results of the first kernel, to the second
I thought they were stored into KernelArguments but it's empty, what's the current approach to solve the current problem?

[matthewbolanos]

Is there a reason you are using two different kernels? Based on my understanding of your scenario, I would have expected that you would use a single kernel with both plugins loaded into it. If you then used automatic function calling, the AI should use the first plugin to retrieve data and the second one to perform the action requested by the user.

[MithrilMan]

I've lot of plugins and they serve 2 disting purpose:

• one kind of plugins are for extracting data from our internal servers based on user request (Autocalling functions decide which one to call)
• the other serves to elaborate further the raw data and based on user request, return back a list of "UI Actions" that I use to perform UI automation on our web app.

I managed to get it to work by using IChatCompletionService manually taken from 2 different keyed kernels, passing a new
ChatHistory to the first one, and then reusing it (with slightly modification) to the second to obtain a list of UI-Actions (think it as POCO as

 public record CenterMap {
    public Coordinate Location {get; set;}
 }

That I convert to plugin functions using Reflection.Emit (yeah.. I know... but it's working well) into Plugins

to put more context, the UI is blazor WASM, I can share the UIActions POCOs in a shared project and then just deserialize in the UI my POCOs and execute them

It's not really clear to me how much overhead does creating a Kernel and injecting plugins have, anyway I didn't wanted to inject in a single kernel all plugins that belongs to two different problem spaces.

I could give a try anyway to see how it behave and also I could try to use the planner (another thing I haven't tried yet and I'm not sure which one is better)

To summarize better what I did (still simplifying the scenario):
Imagine having trucks tracked by GPS in a web app, as a user you ask to find which vehicles are closer to a specific address, so you ask something as
"Tell me which vehicles are close to 123 Main Street Apartment 4B Cityville, State 54321 United States"
The request is sent to the server, that using the first kernel that have "data extraction" plugins use the Plugin GeocodeAddress that has a "string Address" parameter

something like

[KernelFunction, Description("Perform geocoding to find a coordinage giving an address")]
public Coordinate Geocoding(
   [Description("The address to geocode to obtain a coordinate")]
   string Address) {
}

Then I have another plugin that find the closest vehicles to a specific coordinate and return a list of N vehicles that match the criteria, and the autofunction is supposed to call them both, in order
This part is working quite often (gpt 3.5-turbo)

During the process I store everything in a chathistory, then when I have a textual description that respond to user request, I call another service that uses its own kernel and only knows UI actions like CenterMap, ZoomMap, DrawVehicle, etc... each one with its own parameters and the plugin is dynamically generated at startup by converting POCOs (decorated with DescriptionAttribute and RequiredAttribute where needed) into proper plugin methods at runtime with Reflection.Emit and in their body, what I did is transposing the POCO properties in Plugin parameters, then when the method is called it create a new instance of the POCO populating it with the values in the plugin parameters and then add the created instance into a List of uiactions using a service that I inject in each one of the generated method using [FromKernelService] attribute.

at the end of everything I return to the UI

• the textual response
• the list of UIActions to perform
• a log of the UIActions (as last step of the last service I want back a log of which function it called)

this is still in early POC, I just spend few days this week, here a video

Video.mp4

As you can see it "somewhat" works but I've to work on prompts and description to improve it, also the zoom bug is because I hardcoded some location on geocoding/reverse geocoding but I think you'll get the point about what I'm doing here

[MithrilMan]

this is an example of what my service returned to the client (just showing UIActions)

    "UIActions": [
        {
            "Location": {
                "Latitude": 45.0703,
                "Longitude": 7.6869
            },
            "ActionName": "CenterMap"
        },
        {
            "ZoomLevel": 15,
            "ActionName": "ZoomMap"
        },
        {
            "Vehicles": [
                "Rocinante"
            ],
            "ActionName": "DrawVehicles"
        }
    ],
    "UIActionsLog": "I have centered the map on the location of the vehicle Rocinante, zoomed in close to it, and drawn the vehicle on the map. Now you should see Rocinante marked on the map."
}