Download Symbols: Network latency when using -updateHosts

[dominicstarkl]

Problem description

When I create a BC Docker Container on my local machine with -updateHosts, downloading symbols lasts extremely long (about 40 seconds).
Downloading symbols in a BC Docker Container not created with -updateHosts lasts only about 1 second.
Furthermore, the debugging experience is very slow: Stepping through code sometimes lasts about 5-10 seconds.
I am observing this behaviour since 1 year.

Downloading the same symbols in a BC Docker Container created on other machines in our network environment lasts 5 times faster (about 0.2 sec.).

I am looking for ways to limit this network latency and find out possfible bottlenecks.
Could you please advise me in this issue? Many thanks in advance!

System information

BcContainerHelper

• V6.0.12, 2024/03/27, https://www.powershellgallery.com/packages/BcContainerHelper/6.0.12

OS

- Windows 11 Enterprise
- Version 23H2 (OS build 22631)
- Build: 22631.3374, KB5035942, 2024-03-26

Hardware

- CPU: ntel(R) Core(TM) i9-10885H CPU @ 2.40GHz   2.40 GHz
- RAM: 32.0 GB (31.8 GB verwendbar)
- System: 64-Bit-Betriebssystem, x64-basierter Prozessor

Docker

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:44 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.24)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:37 2023
  OS/Arch:          windows/amd64
  Experimental:     false

BC Docker Container created with -updateHosts

Script:
New-BcContainer bc-dev -accept_eula -artifactUrl (Get-BCArtifactUrl Sandbox -country ch -select Latest) -auth NavUserPassword -updateHosts

Output:

C:\WINDOWS\system32> C:\Users\d.starkl\Desktop\_todo\_issues\Docker-Network-Latency\Create-BcContainer.ps1
BcContainerHelper version 6.0.12
BC.HelperFunctions emits usage statistics telemetry to Microsoft
Running on Windows, PowerShell 5.1.22621.2506`
BcContainerHelper is version 6.0.12
BcContainerHelper is running as administrator
HyperV is Enabled
UsePsSession is True
UsePwshForBc24 is True
Host is Microsoft Windows 11 Enterprise - 10.0.22631.3374
Docker Client Version is 24.0.7
Docker Server Version is 24.0.7
Removing Desktop shortcuts
Fetching all docker images
Fetching all docker volumes
Using image mcr.microsoft.com/businesscentral:ltsc2022
Creating Container bc-dev
Style: sandbox
Multitenant: Yes
Version: 23.5.16502.17894
Platform: 23.0.17870.0
Generic Tag: 1.0.2.17
Container OS Version: 10.0.20348.2340 (ltsc2022)
Host OS Version: 10.0.22631.3374 (23H2)
WARNING: Container and host OS build is 20348 or above, defaulting to process isolation. If you encounter issues, you could try to install HyperV.
Using process isolation
Using locale de-CH
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Files in C:\ProgramData\BcContainerHelper\Extensions\bc-dev\my:
- AdditionalOutput.ps1
- HelperFunctions.ps1
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container bc-dev from image mcr.microsoft.com/businesscentral:ltsc2022
93ee4277a75a8b1ae0584443d9b58dc91e686aea520763afa2732855578f5299
Waiting for container bc-dev to be ready
Using artifactUrl https://bcartifacts.azureedge.net/sandbox/23.5.16502.17894/ch
Using installer from C:\Run\210-new
Installing Business Central: multitenant=True, installOnly=False, filesOnly=False, includeTestToolkit=False, includeTestLibrariesOnly=False, includeTestFrameworkOnly=False, includePerformanceToolkit=False, appArtifactPath=c:\dl\sandbox\23.5.16502.17894\ch, platformArtifac
tPath=c:\dl\sandbox\23.5.16502.17894\platform, databasePath=c:\dl\sandbox\23.5.16502.17894\ch\BusinessCentral-CH.bak, licenseFilePath=c:\dl\sandbox\23.5.16502.17894\ch\Cronus.bclicense, rebootContainer=True
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\23.5.16502.17894\platform\ServiceTier\Program Files
c:\dl\sandbox\23.5.16502.17894\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\sandbox\23.5.16502.17894\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\sandbox\23.5.16502.17894\platform\WindowsPowerShellScripts\WebSearch
Copying Web Client Files
c:\dl\sandbox\23.5.16502.17894\platform\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
c:\dl\sandbox\23.5.16502.17894\platform
c:\dl\sandbox\23.5.16502.17894\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\23.5.16502.17894\ch\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\23.5.16502.17894\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\23.5.16502.17894\ch\Extensions
Copying Applications
C:\dl\sandbox\23.5.16502.17894\platform\Applications
Copying Applications.CH
C:\dl\sandbox\23.5.16502.17894\ch\Applications.CH
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing license file
Copying Database on localhost\SQLEXPRESS from tenant to default
Taking database tenant offline
Copying database files
Attaching files as new Database default
Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Stopping Business Central Service Tier
Installation took 103 seconds
Installation complete
Initializing...
Setting host.docker.internal to 10.0.8.109 in container hosts file (copy from host hosts file)
Setting gateway.docker.internal to 10.0.8.109 in container hosts file (copy from host hosts file)
Setting kubernetes.docker.internal to 127.0.0.1 in container hosts file (copy from host hosts file)
Setting host.containerhelper.internal to 172.25.0.1 in container hosts file
Starting Container
Hostname is bc-dev
PublicDnsName is bc-dev
Using NavUserPassword Authentication
Creating Self Signed Certificate
Self Signed Certificate Thumbprint B32E61202F2BC1265B281D246D06C01F4A120E69
DNS identity bc-dev
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: BC
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\BC
Create the application pool BC
Create website: NavWebApplicationContainer without SSL
Update configuration: navsettings.json
Done Configuring Web Client
Enabling Financials User Experience
Dismounting Tenant
Mounting Tenant
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Creating http download site
Setting SA Password and enabling SA
Creating logico as SQL User and add to sysadmin
Creating SUPER user
WARNING: The password that you entered does not meet the minimum requirements. 
It should be at least 8 characters long and contain at least one uppercase 
letter, one lowercase letter, and one number.
Container IP Address: 172.25.9.97
Container Hostname  : bc-dev
Container Dns Name  : bc-dev
Web Client          : http://bc-dev/BC/?tenant=default
Dev. Server         : http://bc-dev
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting bc-dev to 172.25.9.97 in host hosts file
Setting bc-dev-default to 172.25.9.97 in host hosts file
Setting bc-dev-default to 172.25.9.97 in container hosts file

Files:
http://bc-dev:8080/ALLanguage.vsix

Container Total Physical Memory is 31.8Gb
Container Free Physical Memory is 17.4Gb

Initialization took 21 seconds
Ready for connections!
Reading CustomSettings.config from bc-dev
Creating Desktop Shortcuts for bc-dev
Cleanup old dotnet core assemblies
Container bc-dev successfully created

Use:
Get-BcContainerEventLog -containerName bc-dev to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName bc-dev to get debug information about the container
Enter-BcContainer -containerName bc-dev to open a PowerShell prompt inside the container
Remove-BcContainer -containerName bc-dev to remove the container again
docker logs bc-dev to retrieve information about URL's again

C:\WINDOWS\system32> 

Docker inspect:

{
	"Id": "93ee4277a75a8b1ae0584443d9b58dc91e686aea520763afa2732855578f5299",
	"Created": "2024-03-27T12:58:54.6695161Z",
	"Path": "powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; .\\Run\\start.ps1",
	"Args": [],
	"State": {
		"Status": "running",
		"Running": true,
		"Paused": false,
		"Restarting": false,
		"OOMKilled": false,
		"Dead": false,
		"Pid": 41632,
		"ExitCode": 0,
		"Error": "",
		"StartedAt": "2024-03-27T12:58:56.1184821Z",
		"FinishedAt": "0001-01-01T00:00:00Z",
		"Health": {
			"Status": "healthy",
			"FailingStreak": 0,
			"Log": [
				{
					"Start": "2024-03-27T14:54:06.1279575+01:00",
					"End": "2024-03-27T14:54:06.4839847+01:00",
					"ExitCode": 0,
					"Output": ""
				},
				{
					"Start": "2024-03-27T14:54:36.4912023+01:00",
					"End": "2024-03-27T14:54:36.8207198+01:00",
					"ExitCode": 0,
					"Output": ""
				},
				{
					"Start": "2024-03-27T14:55:06.8304332+01:00",
					"End": "2024-03-27T14:55:07.1679948+01:00",
					"ExitCode": 0,
					"Output": ""
				},
				{
					"Start": "2024-03-27T14:55:37.1755098+01:00",
					"End": "2024-03-27T14:55:37.5250258+01:00",
					"ExitCode": 0,
					"Output": ""
				},
				{
					"Start": "2024-03-27T14:56:07.5363547+01:00",
					"End": "2024-03-27T14:56:07.8553491+01:00",
					"ExitCode": 0,
					"Output": ""
				}
			]
		}
	},
	"Image": "sha256:c4cd601a3c1b2d716b7eb8270a90a1355e58a2b9a18e10ed2e2e84a5f8108ed0",
	"ResolvConfPath": "",
	"HostnamePath": "",
	"HostsPath": "",
	"LogPath": "C:\\ProgramData\\Docker\\containers\\93ee4277a75a8b1ae0584443d9b58dc91e686aea520763afa2732855578f5299\\93ee4277a75a8b1ae0584443d9b58dc91e686aea520763afa2732855578f5299-json.log",
	"Name": "/bc-dev",
	"RestartCount": 0,
	"Driver": "windowsfilter",
	"Platform": "windows",
	"MountLabel": "",
	"ProcessLabel": "",
	"AppArmorProfile": "",
	"ExecIDs": null,
	"HostConfig": {
		"Binds": [
			"c:\\bcartifacts.cache:c:\\dl",
			"C:\\ProgramData\\BcContainerHelper:C:\\ProgramData\\BcContainerHelper",
			"C:\\ProgramData\\BcContainerHelper\\Extensions\\bc-dev\\my:C:\\Run\\my",
			"c:\\windows\\system32\\drivers\\etc:C:\\driversetc"
		],
		"ContainerIDFile": "",
		"LogConfig": {
			"Type": "json-file",
			"Config": {}
		},
		"NetworkMode": "default",
		"PortBindings": {},
		"RestartPolicy": {
			"Name": "unless-stopped",
			"MaximumRetryCount": 0
		},
		"AutoRemove": false,
		"VolumeDriver": "",
		"VolumesFrom": null,
		"ConsoleSize": [
			0,
			0
		],
		"CapAdd": null,
		"CapDrop": null,
		"CgroupnsMode": "",
		"Dns": [],
		"DnsOptions": [],
		"DnsSearch": [],
		"ExtraHosts": null,
		"GroupAdd": null,
		"IpcMode": "",
		"Cgroup": "",
		"Links": null,
		"OomScoreAdj": 0,
		"PidMode": "",
		"Privileged": false,
		"PublishAllPorts": false,
		"ReadonlyRootfs": false,
		"SecurityOpt": null,
		"UTSMode": "",
		"UsernsMode": "",
		"ShmSize": 0,
		"Isolation": "process",
		"CpuShares": 0,
		"Memory": 0,
		"NanoCpus": 0,
		"CgroupParent": "",
		"BlkioWeight": 0,
		"BlkioWeightDevice": [],
		"BlkioDeviceReadBps": [],
		"BlkioDeviceWriteBps": [],
		"BlkioDeviceReadIOps": [],
		"BlkioDeviceWriteIOps": [],
		"CpuPeriod": 0,
		"CpuQuota": 0,
		"CpuRealtimePeriod": 0,
		"CpuRealtimeRuntime": 0,
		"CpusetCpus": "",
		"CpusetMems": "",
		"Devices": [],
		"DeviceCgroupRules": null,
		"DeviceRequests": null,
		"MemoryReservation": 0,
		"MemorySwap": 0,
		"MemorySwappiness": null,
		"OomKillDisable": false,
		"PidsLimit": null,
		"Ulimits": null,
		"CpuCount": 0,
		"CpuPercent": 0,
		"IOMaximumIOps": 0,
		"IOMaximumBandwidth": 0,
		"MaskedPaths": null,
		"ReadonlyPaths": null
	},
	"GraphDriver": {
		"Data": {
			"dir": "C:\\ProgramData\\Docker\\windowsfilter\\93ee4277a75a8b1ae0584443d9b58dc91e686aea520763afa2732855578f5299"
		},
		"Name": "windowsfilter"
	},
	"Mounts": [
		{
			"Type": "bind",
			"Source": "c:\\programdata\\bccontainerhelper\\extensions\\bc-dev\\my",
			"Destination": "c:\\run\\my",
			"Mode": "",
			"RW": true,
			"Propagation": ""
		},
		{
			"Type": "bind",
			"Source": "c:\\windows\\system32\\drivers\\etc",
			"Destination": "c:\\driversetc",
			"Mode": "",
			"RW": true,
			"Propagation": ""
		},
		{
			"Type": "bind",
			"Source": "c:\\bcartifacts.cache",
			"Destination": "c:\\dl",
			"Mode": "",
			"RW": true,
			"Propagation": ""
		},
		{
			"Type": "bind",
			"Source": "c:\\programdata\\bccontainerhelper",
			"Destination": "c:\\programdata\\bccontainerhelper",
			"Mode": "",
			"RW": true,
			"Propagation": ""
		}
	],
	"Config": {
		"Hostname": "bc-dev",
		"Domainname": "",
		"User": "",
		"AttachStdin": false,
		"AttachStdout": false,
		"AttachStderr": false,
		"ExposedPorts": {
			"1433/tcp": {},
			"443/tcp": {},
			"7045/tcp": {},
			"7046/tcp": {},
			"7047/tcp": {},
			"7048/tcp": {},
			"7049/tcp": {},
			"7083/tcp": {},
			"80/tcp": {},
			"8080/tcp": {}
		},
		"Tty": false,
		"OpenStdin": false,
		"StdinOnce": false,
		"Env": [
			"licenseFile=",
			"auth=NavUserPassword",
			"enableApiServices=Y",
			"multitenant=Y",
			"username=logico",
			"ExitOnError=N",
			"locale=de-CH",
			"filesOnly=False",
			"securePassword=76492d1116743f0423413b16050a5345MgB8ADIAUABNADcAMgAyAHcARwBkAGgAbwBtAE8AWABLAE8AawBjAHEAagBHAEEAPQA9AHwAZABmAGEANQA5AGUAOABlAGYAZABkAGQAZgBkADkANAA2AGEAMwBiADAAZABlADcAMwBkADMAZABlAGMANAA0AA==",
			"removePasswordKeyFile=Y",
			"isBcSandbox=Y",
			"artifactUrl=https://bcartifacts.azureedge.net/sandbox/23.5.16502.17894/ch",
			"databaseServer=",
			"databaseInstance=",
			"accept_eula=Y",
			"useSSL=N",
			"passwordKeyFile=c:\\run\\my\\aes.key",
			"accept_outdated=Y",
			"DOTNET_RUNNING_IN_CONTAINER=true",
			"COMPLUS_RUNNING_IN_CONTAINER=1",
			"COMPLUS_NGenProtectedProcess_FeatureEnabled=0",
			"only24=False"
		],
		"Cmd": [
			"powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; .\\Run\\start.ps1"
		],
		"Healthcheck": {
			"Test": [
				"CMD",
				"powershell",
				".\\Run\\HealthCheck.ps1"
			],
			"Interval": 30000000000,
			"Timeout": 10000000000
		},
		"ArgsEscaped": true,
		"Image": "mcr.microsoft.com/businesscentral:ltsc2022",
		"Volumes": null,
		"WorkingDir": "",
		"Entrypoint": null,
		"OnBuild": null,
		"Labels": {
			"country": "CH",
			"created": "202403131353",
			"eula": "https://go.microsoft.com/fwlink/?linkid=861843",
			"maintainer": "Dynamics SMB",
			"nav": "",
			"osversion": "10.0.20348.2340",
			"platform": "23.0.17870.0",
			"tag": "1.0.2.17",
			"version": "23.5.16502.17894"
		}
	},
	"NetworkSettings": {
		"Bridge": "",
		"SandboxID": "93ee4277a75a8b1ae0584443d9b58dc91e686aea520763afa2732855578f5299",
		"HairpinMode": false,
		"LinkLocalIPv6Address": "",
		"LinkLocalIPv6PrefixLen": 0,
		"Ports": {
			"1433/tcp": null,
			"443/tcp": null,
			"7045/tcp": null,
			"7046/tcp": null,
			"7047/tcp": null,
			"7048/tcp": null,
			"7049/tcp": null,
			"7083/tcp": null,
			"80/tcp": null,
			"8080/tcp": null
		},
		"SandboxKey": "93ee4277a75a8b1ae0584443d9b58dc91e686aea520763afa2732855578f5299",
		"SecondaryIPAddresses": null,
		"SecondaryIPv6Addresses": null,
		"EndpointID": "",
		"Gateway": "",
		"GlobalIPv6Address": "",
		"GlobalIPv6PrefixLen": 0,
		"IPAddress": "",
		"IPPrefixLen": 0,
		"IPv6Gateway": "",
		"MacAddress": "",
		"Networks": {
			"nat": {
				"IPAMConfig": null,
				"Links": null,
				"Aliases": null,
				"MacAddress": "00:15:5d:26:d9:ed",
				"NetworkID": "e437330e39469583b90503c91b7b8b630c590c1a460f4ca89a0baf141c9f0e6e",
				"EndpointID": "2f1d55290ef641ced63e40ee745a5fe15a7a7214ed009246d755a4612261c112",
				"Gateway": "172.25.0.1",
				"IPAddress": "172.25.9.97",
				"IPPrefixLen": 16,
				"IPv6Gateway": "",
				"GlobalIPv6Address": "",
				"GlobalIPv6PrefixLen": 0,
				"DriverOpts": null
			}
		}
	}
}

Test script

Input:

$SymbolsPath = "http://bc-dev:7049/BC/dev/packages?publisher=Microsoft&appName=Base Application&tenant=default"

$pair = "logico:xxx"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($pair))
$basicAuthValue = "Basic $encodedCreds"
$Headers = @{
    Authorization = $basicAuthValue
}

(Measure-Command { 
    $ProgressPreference = 'SilentlyContinue'     
    iwr $SymbolsPath -Headers $Headers -OutFile "C:\Temp\app.zip"
    $ProgressPreference = 'Continue' 
}).TotalMilliseconds

Output: 37347.6224

[freddydk]

Tried the same - download is 923 milliseconds - so definitely not something that always happens.

Tried adding -dns '8.8.8.8' to see if DNS was involved - time 743 ms.

Removing -updateHosts returned 636ms

But... - all of them changes between 414ms and 923ms

So... - it seems like this is something on your machine?

The functionality of updateHosts is the following.

UpdateHosts was created because in a lot of situations I saw that DNS resolution on container names didn't work.
UpdateHosts will (upon restart of the container) update the c:\windows\system32\drivers\etc\hosts file on the host with IP numbers and host names for the container. Looks like:

<containerIP> bcserver

Now you can try a few things to see what makes the difference.
You can try installing the latest docker (I run 25.0.4) and that actually has a DNSNames section in the network of docker inspect:

You could try to create the container with -doNotCheckHealth to see whether health checking is the issue.

You could create a container with updateHosts - and then remove the entry from the hosts hosts file manually and see if that makes things speed up? If that is the case, we know that the host entry is the problem - next issue would then be to find out why.

Let me know the outcome of these tests.

[dominicstarkl]

Thanks Freddy. I will have a look at it beside my holidays 😏