Duende IdentityServer6 upgrade - backward compatibility - token introspect issue- token created with IdentityServer4 is not work after upgrade.

[susan12-web]

Which version of Duende IdentityServer are you using? 6

Which version of .NET are you using? .net 6

Describe the bug

A clear and concise description of what the bug is. --- Token created with IdentityServer4 was supplied to relying party this was having 1 year time. So after the code upgrade from idp 4 to duende idp 6 all these grant data inside refresh tokens are failing to deserialize.

To Reproduce

https://localhost:5001/connect/external/introspect during the api call to external introspect. below lines for code will internal call persistent grants which is failing to deserialze :

public async Task GetRefreshTokenAsync(string refreshTokenHandle)
{
var obRefreshToken = await GetItemAsync(refreshTokenHandle);
return obRefreshToken;
}

protected virtual async Task GetItemAsync(string key)
{
var hashedKey = GetHashedKey(key);
var item = await GetItemByHashedKeyAsync(hashedKey);
if (item == null)
{
Logger.LogDebug("{grantType} grant with value: {key} not found in store.", GrantType, key);
}
return item;
}

var grant = await Store.GetAsync(hashedKey);
if (grant != null && grant.Type == GrantType)
{
try
{
var data = grant.Data;
return Serializer.Deserialize(output); ---- > failing here with error Message=The JSON value could not be converted to Duende.IdentityServer.Models.AccessTokenType. Path: $.AccessToken.AccessTokenType | LineNumber: 140 | BytePositionInLine: 34.

}
catch (Exception ex)
{
Logger.LogError(ex, "Failed to deserialize JSON from grant store.");
}
Expected behavior

Expecting some way for duende to support old tokens generated along with new.

Log output/exception with stacktrace

System.Text.Json.JsonException
HResult=0x80131500
Message=The JSON value could not be converted to Duende.IdentityServer.Models.AccessTokenType. Path: $.AccessToken.AccessTokenType | LineNumber: 140 | BytePositionInLine: 34.
Source=System.Text.Json
StackTrace:
at System.Text.Json.ThrowHelper.ThrowJsonException(String message)
at System.Text.Json.Serialization.Converters.EnumConverter1.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) at System.Text.Json.Serialization.Metadata.JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.Metadata.JsonPropertyInfo1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
at System.Text.Json.Serialization.JsonConverter1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable1 actualByteCount)
at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
at Duende.IdentityServer.Stores.Serialization.PersistentGrantSerializer.Deserialize[T](String json)
at IdentityProvider.Module.Stores.OpenBankingRefreshTokenStore.d__8.MoveNext() in C:\dev\projects\openbanking\digitalidentityprovider\src\app\IdentityProvider.Module\src\library\IdentityProvider.Module\Stores\OpenBankingRefreshTokenStore.cs:line 116

This exception was originally thrown at this call stack:
[External Code]
IdentityProvider.Module.Stores.OpenBankingRefreshTokenStore.GetItemByHashedKeyAsync(string) in OpenBankingRefreshTokenStore.cs

data--- refresh token

Additional context

Add any other context about the problem here.

Another issue is the below one :

Failed to deserialize JSON from grant store.","Exception":"System.Text.Json.JsonException: The JSON value could not be converted to Duende.IdentityServer.Models.AccessTokenType

Any help on the backward compatibility - token introspect issue ?

[RolandGuijt]

Please tell me from which IdentityServer 4 version you're upgrading so I can investigate.

[susan12-web]

Identity server4 version -4.1.2

[susan12-web]

Duende.IdentityServer version 6.3.3

[susan12-web]

Please tell me from which IdentityServer 4 version you're upgrading so I can investigate.

Identity server4 version -4.1.2

[RolandGuijt]

It complains that AccessTokenType (which is an enum) can't be deserialized from the JSON. But that type didn't change at all in the transition to v6, it just contains 2 possible values.
Can you show me an example of the JSON you're trying to deserialize?

[susan12-web]

It complains that AccessTokenType (which is an enum) can't be deserialized from the JSON. But that type didn't change at all in the transition to v6, it just contains 2 possible values. Can you show me an example of the JSON you're trying to deserialize?

yeah json contains tokentype as below :

"AccessTokenType": "Reference",

[AndersAbel]

The persisted grant storage format was changed substantially between IdentityServer4 and Duende IdentityServer. The high level API was kept the same and our code for reading persisted grants can read both the old and new format to provide a seamless migration.

The stack trace indicates that you have some custom code that interacts directly with the stores. That code will probably be affected by the changes to the behaviour and need to be adjusted.

It is also a bit confusing if you are really working with refresh or reference tokens. In the stack trace there is a method called OpenBankingRefreshTokenStore. The Json however has a token type of "Reference". Do you know which one it actually is that you are trying to use?

[RolandGuijt]

@susan12-web Did things work out for you or do you want to continue this by answering Anders' question? If all is fine, I'd like to close this issue.

[RolandGuijt]

Closing for now but please re-open if there are further questions.

[susan12-web]

Sorry for the late reply.

It is also a bit confusing if you are really working with refresh or reference tokens. In the stack trace there is a method called OpenBankingRefreshTokenStore. The Json however has a token type of "Reference". Do you know which one it actually is that you are trying to use?

Am working with both refresh and reference token grant types. The accesstokentype is creating the problem. For each type for older version I have to do the below by overriding DefaultGrantStore GetItemByHashedKeyAsync method as below :

protected override async Task GetItemByHashedKeyAsync(string hashedKey)
{
var grant = await Store.GetAsync(hashedKey);
if (grant != null && grant.Type == GrantType)
{
try
{
var data = grant.Data;
dynamic jsonObj = Newtonsoft.Json.JsonConvert.DeserializeObject(data);
if (jsonObj != null
&& GrantType == PersistedGrantTypes.RefreshToken
&& jsonObj["OriginalSubject"]["AuthenticationType"]
== Common.Constants.AuthenticationTypeidpserv4)
{
jsonObj["AccessToken"]["AccessTokenType"] = Convert.ToInt32(AccessTokenType.Reference);
}
else if (jsonObj != null
&& GrantType == PersistedGrantTypes.ReferenceToken
&& jsonObj["Type"] == TokenTypes.AccessToken)
{
if (jsonObj["Version"] == Common.Constants.Version)
jsonObj["AccessTokenType"] = Convert.ToInt32(AccessTokenType.Reference);
}
string output = Newtonsoft.Json.JsonConvert.SerializeObject(jsonObj,
Newtonsoft.Json.Formatting.Indented);
return Serializer.Deserialize(output);
}
catch (Exception ex)
{
Logger.LogError(ex, "Failed to deserialize JSON from grant store.");
}
}

return default;

}

Is there any other way to handle this?

[RolandGuijt]

Duplicate of:
#1261

[susan12-web]

resolved for now. Will reopen when needed any help.