DataApiBuilderException: Cannot obtain Schema for entity with underlying database object source: due to: Connection Timeout Expired.

[wrutkowski-xebia]

What happened?

Single Web Aapp - hosted Blazor WASM
DAB - hosted on Container App - one container, one single revision.

1. Everything connected and setup, scaled to 1 revision running.

2. Wait some time until scaled back to 0 - no traffic.

3. Make call from SWA - revision failed to scale to 1 - timeout error.

4. After next few calls from SWA, revision scaled to 1 and response received.

Version

0.10.21

What database are you using?

Azure SQL

What hosting model are you using?

Container Apps

Which API approach are you accessing DAB through?

REST

Relevant log output

[41m[30mfail[39m[22m[49m: Azure.DataApiBuilder.Service.Startup[0]
12.04.2024, 12:49:51.964
Unable to complete runtime initialization. Refer to exception for error details.
12.04.2024, 12:49:51.964
Azure.DataApiBuilder.Service.Exceptions.DataApiBuilderException: Cannot obtain Schema for entity Customer with underlying database object source: SalesLT.Customer due to: Connection Timeout Expired. The timeout period elapsed during the post-login phase. The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=114; handshake=575; [Login] initialization=1; authentication=6; [Post-Login] complete=14231;
12.04.2024, 12:49:51.964
at Azure.DataApiBuilder.Core.Services.SqlMetadataProvider`3.HandleOrRecordException(Exception e) in /src/src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:line 90
12.04.2024, 12:49:51.964
at Azure.DataApiBuilder.Core.Services.SqlMetadataProvider`3.PopulateObjectDefinitionForEntities() in /src/src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:line 919
12.04.2024, 12:49:51.964
at Azure.DataApiBuilder.Core.Services.SqlMetadataProvider`3.InitializeAsync() in /src/src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:line 273
12.04.2024, 12:49:51.964
at Azure.DataApiBuilder.Core.Services.MetadataProviders.MetadataProviderFactory.InitializeAsync() in /src/src/Core/Services/MetadataProviders/MetadataProviderFactory.cs:line 64
12.04.2024, 12:49:51.964
at Azure.DataApiBuilder.Service.Startup.PerformOnConfigChangeAsync(IApplicationBuilder app) in /src/src/Service/Startup.cs:line 599
12.04.2024, 12:49:51.964
[41m[30mfail[39m[22m[49m: Azure.DataApiBuilder.Service.Startup[0]
12.04.2024, 12:49:51.964
Could not initialize the engine with the runtime config file: ./swa-db-connections/dab.config.json
12.04.2024, 12:49:51.964
[40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]
12.04.2024, 12:49:51.964
Application is shutting down...

Code of Conduct

• I agree to follow this project's Code of Conduct

[wrutkowski-xebia]

Some other timeout.

fail: Azure.DataApiBuilder.Core.Resolvers.IQueryExecutor[0]
      2b3adbf5-2e11-4a9e-9d05-226968ad4824 Query execution error due to:
      Connection Timeout Expired.  The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=282; handshake=193; [Login] initialization=0; authentication=0; [Post-Login] complete=29432;
      Microsoft.Data.SqlClient.SqlException (0x80131904): Connection Timeout Expired.  The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=282; handshake=193; [Login] initialization=0; authentication=0; [Post-Login] complete=29432;
       ---> System.ComponentModel.Win32Exception (258): Unknown error 258
         at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
         at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
         at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
         at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
         at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
         at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
         at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
         at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
         at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
         at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
         at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
         at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
         at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
      --- End of stack trace from previous location ---
         at Azure.DataApiBuilder.Core.Resolvers.QueryExecutor`1.ExecuteQueryAgainstDbAsync[TResult](TConnection conn, String sqltext, IDictionary`2 parameters, Func`3 dataReaderHandler, HttpContext httpContext, String dataSourceName, List`1 args) in /src/src/Core/Resolvers/QueryExecutor.cs:line 155
         at Azure.DataApiBuilder.Core.Resolvers.QueryExecutor`1.<>c__DisplayClass19_0`1.<<ExecuteQueryAsync>b__0>d.MoveNext() in /src/src/Core/Resolvers/QueryExecutor.cs:line 102
      ClientConnectionId:0a714fa6-f6f3-4c4d-996d-b9f023ce5a0c
      Error Number:-2,State:0,Class:11
fail: Azure.DataApiBuilder.Service.Controllers.RestController[0]
      2b3adbf5-2e11-4a9e-9d05-226968ad4824 Error handling REST request.
      Azure.DataApiBuilder.Service.Exceptions.DataApiBuilderException: Connection Timeout Expired.  The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=282; handshake=193; [Login] initialization=0; authentication=0; [Post-Login] complete=29432;
       ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Connection Timeout Expired.  The timeout period elapsed during the post-login phase.  The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.  The duration spent while attempting to connect to this server was - [Pre-Login] initialization=282; handshake=193; [Login] initialization=0; authentication=0; [Post-Login] complete=29432;
       ---> System.ComponentModel.Win32Exception (258): Unknown error 258
         at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
         at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
         at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
         at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
         at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
         at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
         at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
         at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
         at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
         at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
         at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
         at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
         at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
      --- End of stack trace from previous location ---
         at Azure.DataApiBuilder.Core.Resolvers.QueryExecutor`1.ExecuteQueryAgainstDbAsync[TResult](TConnection conn, String sqltext, IDictionary`2 parameters, Func`3 dataReaderHandler, HttpContext httpContext, String dataSourceName, List`1 args) in /src/src/Core/Resolvers/QueryExecutor.cs:line 155
         at Azure.DataApiBuilder.Core.Resolvers.QueryExecutor`1.<>c__DisplayClass19_0`1.<<ExecuteQueryAsync>b__0>d.MoveNext() in /src/src/Core/Resolvers/QueryExecutor.cs:line 102
      ClientConnectionId:0a714fa6-f6f3-4c4d-996d-b9f023ce5a0c
      Error Number:-2,State:0,Class:11
         --- End of inner exception stack trace ---
         at Azure.DataApiBuilder.Core.Resolvers.QueryExecutor`1.<>c__DisplayClass19_0`1.<<ExecuteQueryAsync>b__0>d.MoveNext() in /src/src/Core/Resolvers/QueryExecutor.cs:line 129
      --- End of stack trace from previous location ---
         at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
         at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
         at Azure.DataApiBuilder.Core.Resolvers.QueryExecutor`1.ExecuteQueryAsync[TResult](String sqltext, IDictionary`2 parameters, Func`3 dataReaderHandler, HttpContext httpContext, List`1 args, String dataSourceName) in /src/src/Core/Resolvers/QueryExecutor.cs:line 90
         at Azure.DataApiBuilder.Core.Resolvers.SqlQueryEngine.ExecuteAsync(SqlQueryStructure structure, String dataSourceName) in /src/src/Core/Resolvers/SqlQueryEngine.cs:line 228
         at Azure.DataApiBuilder.Core.Resolvers.SqlQueryEngine.ExecuteAsync(FindRequestContext context) in /src/src/Core/Resolvers/SqlQueryEngine.cs:line 150
         at Azure.DataApiBuilder.Core.Services.RestService.DispatchQuery(RestRequestContext context, DatabaseType databaseType) in /src/src/Core/Services/RestService.cs:line 219
         at Azure.DataApiBuilder.Core.Services.RestService.ExecuteAsync(String entityName, EntityActionOperation operationType, String primaryKeyRoute) in /src/src/Core/Services/RestService.cs:line 195
         at Azure.DataApiBuilder.Service.Controllers.RestController.HandleOperation(String route, EntityActionOperation operationType) in /src/src/Service/Controllers/RestController.cs:line 214