UsePeriodicPasswordProvider causes intermittent access denied errors

[andreas-antoniou-cko]

Software versions
MySqlConnector version: 2.3.5
Server type (MySQL, MariaDB, Aurora, etc.) and version: MySQL
.NET version: net6.0
(Optional) ORM NuGet packages and versions: n/a

Describe the bug
We have an app that connects to a Aurora MySQL on AWS. We recently switched from using a fixed password to IAM authentication. After we realized we hit this issue we are testing a version using UsePeriodicPasswordProvider. The solution is working (connecting fine to the DB and no memory utilisation increase) but we do periodically get a spike of errors accessing the database with the following error: Access denied for user 'db_user'@'redacted ip address' (using password: YES)

Exception

MySqlConnector.MySqlException (0x80004005): Access denied for user 'db_user'@'redacted ip' (using password: YES)
   at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 897
   at MySqlConnector.Core.ServerSession.SwitchAuthenticationAsync(ConnectionSettings cs, String password, PayloadData payload, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 733
   at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 537
   at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 493
   at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 493
   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 146
   at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 146
   at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 964
   at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 453
   at StackExchange.Profiling.Data.ProfiledDbConnection.OpenAsync(CancellationToken cancellationToken) in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbConnection.cs:line 142
   at Dapper.SqlMapper.MultiMapAsync[TFirst,TSecond,TThird,TFourth,TFifth,TSixth,TSeventh,TReturn](IDbConnection cnn, CommandDefinition command, Delegate map, String splitOn) in C:\projects\dapper\Dapper\SqlMapper.Async.cs:line 956
...redacted application code

Code sample
The following snippet is used to setup the MySqlDataSourceBuilder instance:

var dataSourceBuilder = new MySqlDataSourceBuilder(GetToken());

_mySqlDataSource = dataSourceBuilder.UsePeriodicPasswordProvider(
                (_, __) => new ValueTask<string>(GetConnectionStringWithToken()),
                TimeSpan.FromMinutes(10), 
                TimeSpan.FromSeconds(10));

and then when creating a connection:

_mySqlDataSource.CreateConnection()

/* A concise code sample to reproduce the bug */

Expected behavior
No access denied errors to occur.

Additional context
Rolling back to the version where we update the connection string every 10min we do not observe these errors so I do not think this is an issue from an infrastructure perspective.

[bgrainger]

we do periodically get a spike of errors accessing the database with the following error

How frequently do these spikes occur? Is it every 10 minutes, when UsePeriodicPasswordProvider calls your callback? Every 15 minutes when the IAM token expires?

Does changing TimeSpan.FromMinutes(10) to a shorter (or longer) period change the frequency of authentication error spikes?