Retry using a different model for failures

[prajwalpyakurelMS]

Hi, do we have a guidance or tips on how we can perform retry operations using a different openai model by specifiying error codes and/or status codes, and a fallback model? If not, is it something you can add to your plans for the future?

[sophialagerkranspandey]

Hey @dmytrostruk is this something you can take a look at and add to your research of filters? Thanks.

[dmytrostruk]

@prajwalpyakurelMS I created a PR with example how to achieve your scenario: #6152

Please let me know if it's helpful. Thanks!

[dmytrostruk]

@prajwalpyakurelMS Here is a direct link to retry example:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Filtering/RetryWithFilters.cs

[prajwalpyakurelMS]

Hi Dmytro, Thanks a lot! This looks helpful. I have a few questions: 1. I see we changed the model id in your example but I think we can also change the endpoint, deploymentName, and credentials, am I correct? The way I see it is `modelId` is like a custom key that points to the settings used when using the `AddOpenAIChatCompletion` method in the builder. Reason I'm asking is we have multiple instances behind afd endpoints. 2. We can potentially have multiple fallback endpoints, will we be able to retry in an order? For example: Try endpoint/model 1, if exception endpoint 2, then endpoint 3, etc. 3. I see a warning saying "IFunctionInvocationFilter" is subject to change or removal. How reliable is this method? Is there any way to make it safer? Thanks, Prajwal

…

________________________________ From: Dmytro Struk ***@***.***> Sent: Wednesday, May 8, 2024 7:26 AM To: microsoft/semantic-kernel ***@***.***> Cc: Prajwal Pyakurel ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/semantic-kernel] Retry using a different model for failures (Discussion #6105) @prajwalpyakurelMS<https://github.com/prajwalpyakurelMS> Here is a direct link to retry example: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Filtering/RetryWithFilters.cs — Reply to this email directly, view it on GitHub<#6105 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BFI26YVXKNSAIGF4TODVDFLZBIY3DAVCNFSM6AAAAABHEFVXUSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNJVGUYDI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[prajwalpyakurelMS]

Update: I tried this out, and I saw a couple of issues:

1. The exception didn't end up getting caught, and the retry didn't happen. Would you be able to reproduce and see if you could help fix this? (I tried catching "Exception ex" so that any exception would get caught).
2. It looks like the order of execution of the models (deployment + endpoint) is the opposite of the order in which they are added?

[dmytrostruk]

@prajwalpyakurelMS

1. I see we changed the model id in your example but I think we can also change the endpoint, deploymentName, and credentials, am I correct?

Yes, but I will need to prepare another example for that, since it will contain more code. Thanks for providing more details!

The way I see it is modelId is like a custom key that points to the settings used when using the AddOpenAIChatCompletion method in the builder.

Actually, modelId is the same as described on OpenAI side (e.g. gpt-4, gpt-3.5-turbo-0301). But there is another parameter serviceId, which is a custom key that points to the settings used. It's available in AddOpenAIChatCompletion method and I will use it to implement the example with scenario you shared above. Meanwhile, you can try to experiment with this parameter on your side.

2. We can potentially have multiple fallback endpoints, will we be able to retry in an order?
   For example: Try endpoint/model 1, if exception endpoint 2, then endpoint 3, etc.

Yes, but you will need to define this retry logic manually. You can also avoid the example with Filters I shared above and try to use another approach by specifying retry policy on HttpClient level, since OpenAI chat completion service accepts custom HttpClient. Here is an example:
https://github.com/microsoft/semantic-kernel/blob/5249aed73292123fe0ef80515fe6b6973ae0dd50/dotnet/samples/Concepts/DependencyInjection/HttpClient_Resiliency.cs

3. I see a warning saying "IFunctionInvocationFilter" is subject to change or removal. How reliable is this method? Is there any way to make it safer?

At the moment, Filters are temporarily marked as Experimental, since it's new functionality. We do this with every new functionality to give it some time for experimentation and bug fixing. This functionality won't be removed, although minor changes in API could be added in case we see any issues. After some period of time, we will remove Experimental flag, and after this we won't add any breaking changes.

The exception didn't end up getting caught, and the retry didn't happen. Would you be able to reproduce and see if you could help fix this? (I tried catching "Exception ex" so that any exception would get caught).

I'm trying it in main branch and I catch an exception as expected. I will try to investigate it in more detail and will get back to you with results soon.

It looks like the order of execution of the models (deployment + endpoint) is the opposite of the order in which they are added?

That could be an issue why you don't see that exception didn't end up getting caught. We have IAIServiceSelector abstraction, which you can implement for custom service selection logic. By default, there is OrderedAIServiceSelector, but since OpenAI services are registered through DI, it doesn't guarantee that the execution order will be the same as registration order:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Abstractions/Services/OrderedAIServiceSelector.cs

In all cases, I have more information about your case now, and I will try to add new example that covers your scenario soon.

[prajwalpyakurelMS]

Thank you so much for the detailed response! For more context, I am using the 10.0.0 release in my dotnet project. We really appreciate your help on this.