Unable to read API metadata since version 24 in containers

[ivandjordjevic]

Since the switch to new BC 24.x containers we're unable to read API metadata that results in a simple timeout error. Same thing works perfectly in older versions of BC. It could be that it's also related to using hyperv now, as with 24 I had to change that.

The script I'm using:

$artifactUrl = Get-BCArtifactUrl -storageAccount 'bcartifacts' -select Latest -country "us"
$auth = "NavUserPassword"
$username = 'ivan'
$credential = New-Object pscredential $username, (ConvertTo-SecureString -String $username -AsPlainText -Force)
$licenseFile = "C:\myfolder\s\mylicense.bclicense"
$containerName = 'dev1'

# Create Business Central container
New-BCContainer -accept_eula `
                 -artifactUrl $artifactUrl `
                 -containerName $containerName `
                 -shortcuts None `
                 -isolation hyperv `
                 -licenseFile $licenseFile `
                 -auth $auth `
                 -restart always `
                 -dns '8.8.8.8' `
                 -memoryLimit 12G `
                 -Credential $Credential `
                 -updateHosts

The Output:

BcContainerHelper is version 6.0.12
BcContainerHelper is running as administrator
HyperV is Enabled
UsePsSession is True
UsePwshForBc24 is True
Host is Microsoft Windows 10 Pro - 10.0.19045.4170
Docker Client Version is 25.0.3
Docker Server Version is 25.0.3
Removing entries from hosts
Removing dev1 from container hosts file
Removing dev1-* from container hosts file
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\dev1
Fetching all docker images
Fetching all docker volumes
Using image mcr.microsoft.com/businesscentral:ltsc2019
Creating Container dev1
Style: sandbox
Multitenant: Yes
Version: 24.0.16410.18241
Platform: 24.0.18227.0
Generic Tag: 1.0.2.17
Container OS Version: 10.0.17763.5576 (ltsc2019)
Host OS Version: 10.0.19045.4170 (22H2)
Using hyperv isolation
Using locale en-US
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file C:\myfolder\s\mylicense.bclicense
Files in C:\ProgramData\BcContainerHelper\Extensions\dev1\my:
- AdditionalOutput.ps1
- HelperFunctions.ps1
- license.bclicense
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container dev1 from image mcr.microsoft.com/businesscentral:ltsc2019
df667b20fb8d809eb4f6b850ce3d8a055a751b30ab41f29d72935cfff6a8e100
Waiting for container dev1 to be ready
Using artifactUrl https://bcartifacts.azureedge.net/sandbox/24.0.16410.18241/us
Using installer from C:\Run\240
Installing Business Central: multitenant=True, installOnly=False, filesOnly=False, includeTestToolkit=False, includeTestLibrariesOnly=False, includeTestFrameworkOnly=False, includePerformanceTo
olkit=False, appArtifactPath=c:\dl\sandbox\24.0.16410.18241\us, platformArtifactPath=c:\dl\sandbox\24.0.16410.18241\platform, databasePath=c:\dl\sandbox\24.0.16410.18241\us\BusinessCentral-US.b
ak, licenseFilePath=c:\dl\sandbox\24.0.16410.18241\us\Cronus.bclicense, rebootContainer=True
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\24.0.16410.18241\platform\ServiceTier\Program Files
c:\dl\sandbox\24.0.16410.18241\platform\ServiceTier\System64Folder
Copying Web Client Files
c:\dl\sandbox\24.0.16410.18241\platform\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
c:\dl\sandbox\24.0.16410.18241\platform
c:\dl\sandbox\24.0.16410.18241\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\24.0.16410.18241\us\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\24.0.16410.18241\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\24.0.16410.18241\us\Extensions
Copying Applications
C:\dl\sandbox\24.0.16410.18241\platform\Applications
Copying Applications.US
C:\dl\sandbox\24.0.16410.18241\us\Applications.US
Copying dependencies
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing license file
Copying Database on localhost\SQLEXPRESS from tenant to default
Taking database tenant offline
Copying database files
Attaching files as new Database default
Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Stopping Business Central Service Tier
Installation took 259 seconds
Installation complete
Initializing...
Setting host.docker.internal to 192.168.0.5 in container hosts file (copy from host hosts file)
Setting gateway.docker.internal to 192.168.0.5 in container hosts file (copy from host hosts file)
Setting kubernetes.docker.internal to 127.0.0.1 in container hosts file (copy from host hosts file)
Setting host.containerhelper.internal to 172.25.160.1 in container hosts file
Starting Container
Hostname is dev1
PublicDnsName is dev1
Using NavUserPassword Authentication
Creating Self Signed Certificate
Self Signed Certificate Thumbprint 465FBE052DB467D5C5A76CA2373837B2A0736412
DNS identity dev1
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: BC
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\BC
Create the application pool BC
Create website: NavWebApplicationContainer without SSL
Update configuration: navsettings.json
Done Configuring Web Client
Enabling Financials User Experience
Using license file 'c:\run\my\license.bclicense'
Import License
Dismounting Tenant
Mounting Tenant
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Creating http download site
Setting SA Password and enabling SA
Creating ivan as SQL User and add to sysadmin
WARNING: This license is not compatible with this version of Business Central.
Creating SUPER user
WARNING: The password that you entered does not meet the minimum requirements. 
It should be at least 8 characters long and contain at least one uppercase 
letter, one lowercase letter, and one number.
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Container IP Address: 172.25.161.211
Container Hostname  : dev1
Container Dns Name  : dev1
Web Client          : http://dev1/BC/?tenant=default
Dev. Server         : http://dev1
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting dev1 to 172.25.161.211 in host hosts file
Setting dev1-default to 172.25.161.211 in host hosts file
Setting dev1-default to 172.25.161.211 in container hosts file

Files:
http://dev1:8080/ALLanguage.vsix

Container Total Physical Memory is 12.5Gb
Container Free Physical Memory is 8.8Gb

Initialization took 43 seconds
Ready for connections!
Installing SqlServer module
Reading CustomSettings.config from dev1
Cleanup old dotnet core assemblies
Container dev1 successfully created

Use:
Get-BcContainerEventLog -containerName dev1 to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName dev1 to get debug information about the container
Enter-BcContainer -containerName dev1 to open a PowerShell prompt inside the container
Remove-BcContainer -containerName dev1 to remove the container again
docker logs dev1 to retrieve information about URL's again

I've created the Web Service Access Key:

But the $metadata is giving me either Timeout error or ECONNRESET:

http://dev1:7048/BC/api/v2.0/$metadata?tenant=default

[freddydk]

And if you spin up a 23.5 container (using hyperv) with the same username password etc - does it work then?

[ivandjordjevic]

Hi @freddydk,

The 23.5 works like a charm:

GET http://dev1:7048/BC/api/v2.0/$metadata?tenant=default

I used the same script as above, with updated artifact URL:

$artifactUrl = Get-BCArtifactUrl -storageAccount 'bcartifacts' -select Closest -country "us" -version "23.5.0.0"

The Output:

BcContainerHelper is version 6.0.12
BcContainerHelper is running as administrator
HyperV is Enabled
UsePsSession is True
UsePwshForBc24 is True
Host is Microsoft Windows 10 Pro - 10.0.19045.4170
Docker Client Version is 25.0.3
Docker Server Version is 25.0.3
Removing Session dev1
Removing container dev1
Removing entries from hosts
Removing dev1 from container hosts file
Removing dev1-* from container hosts file
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\dev1
Downloading artifact /sandbox/23.5.16502.16887/us
Downloading C:\Users\Ivan\AppData\Local\Temp\f49efd31-3bb6-45c8-bb6f-199f3c5d144b.zip
Downloading using HttpClient
Unpacking artifact to tmp folder using 7zip
Downloading platform artifact /sandbox/23.5.16502.16887/platform
Downloading C:\Users\Ivan\AppData\Local\Temp\b349f83b-9481-4c40-9d4b-f8534362b4d4.zip
Downloading using HttpClient
Unpacking artifact to tmp folder using 7zip
Downloading Prerequisite Components
Downloading D:\bcartifacts.cache\sandbox\23.5.16502.16887\platform\Prerequisite Components\IIS URL Rewrite Module\rewrite_2.0_rtw_x64.msi
Downloading using HttpClient
Downloading D:\bcartifacts.cache\sandbox\23.5.16502.16887\platform\Prerequisite Components\DotNetCore\DotNetCore.1.0.4_1.1.1-WindowsHosting.exe
Downloading using HttpClient
Fetching all docker images
Fetching all docker volumes
Using image mcr.microsoft.com/businesscentral:ltsc2019
Creating Container dev1
Style: sandbox
Multitenant: Yes
Version: 23.5.16502.16887
Platform: 23.0.16766.0
Generic Tag: 1.0.2.17
Container OS Version: 10.0.17763.5576 (ltsc2019)
Host OS Version: 10.0.19045.4170 (22H2)
Using hyperv isolation
Using locale en-US
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file C:\myfolder\s\mylicense.bclicense
Files in C:\ProgramData\BcContainerHelper\Extensions\dev1\my:
- AdditionalOutput.ps1
- HelperFunctions.ps1
- license.bclicense
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container dev1 from image mcr.microsoft.com/businesscentral:ltsc2019
494bcefbdaad0f0cee7917276cadd3d4300b6b2f543c2d7fc5cac0660055e6a8
Waiting for container dev1 to be ready
Using artifactUrl https://bcartifacts.azureedge.net/sandbox/23.5.16502.16887/us
Using installer from C:\Run\210-new
Installing Business Central: multitenant=True, installOnly=False, filesOnly=False, includeTestToolkit=False, includeTestLibrariesOnly=False, includeTestFrameworkOnly=False, includePerf
ormanceToolkit=False, appArtifactPath=c:\dl\sandbox\23.5.16502.16887\us, platformArtifactPath=c:\dl\sandbox\23.5.16502.16887\platform, databasePath=c:\dl\sandbox\23.5.16502.16887\us\Bu
sinessCentral-US.bak, licenseFilePath=c:\dl\sandbox\23.5.16502.16887\us\Cronus.bclicense, rebootContainer=True
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\23.5.16502.16887\platform\ServiceTier\Program Files
c:\dl\sandbox\23.5.16502.16887\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\sandbox\23.5.16502.16887\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\sandbox\23.5.16502.16887\platform\WindowsPowerShellScripts\WebSearch
Copying Web Client Files
c:\dl\sandbox\23.5.16502.16887\platform\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
c:\dl\sandbox\23.5.16502.16887\platform
c:\dl\sandbox\23.5.16502.16887\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\23.5.16502.16887\us\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\23.5.16502.16887\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\23.5.16502.16887\us\Extensions
Copying Applications
C:\dl\sandbox\23.5.16502.16887\platform\Applications
Copying Applications.US
C:\dl\sandbox\23.5.16502.16887\us\Applications.US
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing license file
Copying Database on localhost\SQLEXPRESS from tenant to default
Taking database tenant offline
Copying database files
Attaching files as new Database default
Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Stopping Business Central Service Tier
Installation took 207 seconds
Installation complete
Initializing...
Setting host.docker.internal to 192.168.0.5 in container hosts file (copy from host hosts file)
Setting gateway.docker.internal to 192.168.0.5 in container hosts file (copy from host hosts file)
Setting kubernetes.docker.internal to 127.0.0.1 in container hosts file (copy from host hosts file)
Setting host.containerhelper.internal to 172.25.160.1 in container hosts file
Starting Container
Hostname is dev1
PublicDnsName is dev1
Using NavUserPassword Authentication
Creating Self Signed Certificate
Self Signed Certificate Thumbprint D57D7BDF0DC4241DF5D418B9CDFF91894915C292
DNS identity dev1
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: BC
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\BC
Create the application pool BC
Create website: NavWebApplicationContainer without SSL
Update configuration: navsettings.json
Done Configuring Web Client
Enabling Financials User Experience
Using license file 'c:\run\my\license.bclicense'
Import License
Dismounting Tenant
Mounting Tenant
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Creating http download site
Setting SA Password and enabling SA
Creating ivan as SQL User and add to sysadmin
Creating SUPER user
WARNING: The password that you entered does not meet the minimum requirements. 
It should be at least 8 characters long and contain at least one uppercase 
letter, one lowercase letter, and one number.
Container IP Address: 172.25.166.215
Container Hostname  : dev1
Container Dns Name  : dev1
Web Client          : http://dev1/BC/?tenant=default
Dev. Server         : http://dev1
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting dev1 to 172.25.166.215 in host hosts file
Setting dev1-default to 172.25.166.215 in host hosts file
Setting dev1-default to 172.25.166.215 in container hosts file

Files:
http://dev1:8080/ALLanguage.vsix

Container Total Physical Memory is 12.5Gb
Container Free Physical Memory is 9.0Gb

Initialization took 30 seconds
Ready for connections!
Reading CustomSettings.config from dev1
Cleanup old dotnet core assemblies
Container dev1 successfully created

Use:
Get-BcContainerEventLog -containerName dev1 to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName dev1 to get debug information about the container
Enter-BcContainer -containerName dev1 to open a PowerShell prompt inside the container
Remove-BcContainer -containerName dev1 to remove the container again
docker logs dev1 to retrieve information about URL's again

[ivandjordjevic]

Any workarounds are welcome also until we get to root cause

[freddydk]

You should upgrade to BcContainerHelper 6.0.15 - even though it probably doesn't fix the issue.
I cannot repro the problem - trying to open the metadata url in a briowser:

Could you try a browser too when using 6.0.15

Thanks

[ivandjordjevic]

Same result after upgrading to 6.0.15. We're able to reproduce this in multiple environments, since it happens across our team.

Browser just keeps loading with blank screen when trying to access metadata.

[freddydk]

Does the browser ask for authentication?
Did you try another browser?
Did you try a different containername (like bc24us - something you haven't used before)

[ivandjordjevic]

Yes, it gets to the auth prompt, but after that white screen just keeps on loading. I've tried entering both username + password and username + webservice access key, in different browsers with private sessions.

The new container name gives the same results. I don't see anything special in the container event log also, perhaps only this:

[ivandjordjevic]

Also this one:

[Djordje-Arsic]

I can also replicate this issue since moving to 24.0 version and using HyperV.

Tried Chrome and Edge, on both I only get white screen that is loading forever without any output.

Base app version is v. 24.0.16410.18056.

[freddydk]

Did you try without specifying a license file?

[ivandjordjevic]

Yes, same thing.

[freddydk]

I can repro this when using HyperV isolation.
Using Process isolation - it works fine.
Unfortunately, Windows 10 cannot run process isolation:-(
Will have to investigate, but it might take some time...

[ivandjordjevic]

Thanks for the update Freddy! And we should try getting everyone to Win11 in the meantime. Glad that we came to the Root Cause.

[christianbraeunlich]

Just came across this issue as I wanted to create an issue myself right now for the problem that: if you create a bc v24 container in isolation mode hyperv and do not use SSL, you will not receive a response to API calls that return more than 50-60 result sets (tested with a custom table/api and 3 fields).
You can avoid this issue by:

• using SSL (maybe a nice workaround for you @ivandjordjevic as you can still use isolation mode hyperv)
• using isolation mode process

@freddydk could you update the label on this issue as it says Cannot repro. Perhaps this threshold of 50-60 records will guide us further in the investigation.

Choose a table, maybe even a custom table and a custom API, and create 50 records. Send several GET requests to them => success. Fast. Without any problem.
Now increase the number of records by incrementally inserting new records and sending a GET request in between. At some point in the range of 50-60 records (maybe it also depends on the number of fields?) you will no longer receive a response. It seems like the server is stuck in a loop trying to process this request, causing a bottleneck for all other requests, as @ivandjordjevic pointed out.

Some additional thoughts, remains to be seen if related to this issue:

• the operational limits are now per-user (https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/administration/operational-limits-online#ODataServicesUser)
• findset is used for >50 records (https://www.youtube.com/watch?v=VpNQ-CYNrkM&list=PL1FESh9FqyhSjW63Ouft5vbc2NSxXKV2k&index=40&t=263s)

EDIT: Perhaps it was just an unfortunate coincidence that the number of requests was in this range.

1. I can reproduce this even with only 30 vendors (note that this api shows more fields (22) per record). The size of the previous response with 29 vendors is 20.1 KB, while the custom table/api with 58 records and 3 fields is 8.24 KB.
2. If the vendors are created without any information (empty fields), the error would occur with at a number of 32 vendors. The size of the previous response with 31 vendors is 18.96 KB. So it's not just the number of records, but also the value of the fields / size of the response.
3. If we add $select=displayName, we can add a lot more vendors (did not test how far I could go)
4. I attached the vscode debugger to the WebServiceClient, set a breakpoint in the OnAfterGetCurrRecord trigger and sent a request that will timeout after +5 minutes. It iterates through all the records and exits the loop correctly, as far as I can tell. I couldn't see the content of the following files, but it looks to me like the code also closes the company at the end: CompanyTriggers --> LogInManagement --> Client Type Management --> Client Type Management Impl.

That should be it for now. Really curious what the cause of this strange problem is.

[freddydk]

Thanks @christianbraeunlich - that is indeed a strange bug...
That API responses was depending on HyperV isolation was strange - that -useSSL can fix the issue is even stranger...
Will discuss this with the server team - not sure this is a docker problem.