Performance degree between BC23 and BC24

[RonKoppelaar]

I noticed a performance degradation in the build pipeline moving to BC24.
If I look at the compile and publish steps these are the time used in minutes and seconds.

BC24 - https://bcartifacts.azureedge.net/sandbox/24.0.16410.18040/nl
Compile: 12:13 (mm:ss)
Publish: 8:34 (mm:ss)

BC23.5 - https://bcartifacts.azureedge.net/sandbox/23.5.16502.16887/nl
Compile 8:18 (mm:ss)
Publish: 3:58 (mm:ss)

Both pipelines compile and publish the same amount of apps using same build scripts

Buildservers are running with W2019 and uses Process Isolation. Do you know there are known issues?
According to things I read new platform is based on .Net8 which should actually be faster then .Net6

For Compile and publish I uses the std. cmdlets from BCContainer:
Compile-AppInNavContainer
Publish-NavContainerApp

BcContainerHelper is version 6.0.15

[freddydk]

Have you set usePwshForBC24 to true or false in settings? (or are you using the default value for this)

[RonKoppelaar]

No not in those scripts.. Later in the pipeline I do. Basically using the default

[freddydk]

Do you have detailed logs of before and after with timestamps?

[RonKoppelaar]

Please find attached logfiles
LogFiles.zip

It contains the raw logfile output from Azure devops for prepare and compile step.
There is a slide difference in prepare step between BC23 and BC24
For BC23

• Start with empty DB and publish the MS apps we need

For BC23

• Start with Cronus container
• Remove apps we dont need

Beside this all should be the same.

[RonKoppelaar]

I'll also add the comparison for PostBuild. Also there huge difference, But I expect I used UsePwsBC24 there.

PostBuildLogs.zip

PostBuild is for creating bacpacs
PostbuildOnprem recreates a container by publishing the compiled apps and creating runtime packages.

[MattTraxinger]

Just confirming that I have also noticed this over the past two weeks. Our pipelines that used compiler folders were clocking in around 2 minutes, now they are closer to 4 and 5. Full container builds are seeing roughly the same increase is time. It's not just you, but I don't know if it was a BCContainerHelper change or a BC change.

[marknitek]

The whole -usePwsh thing new to me, so maybe this is unrelated but when running simple commands with invoke-scriptinbccontainer it will generally take 2x as long with PS7 compared to using PS5 (-usePwsh:$false)

Measure-Command { Invoke-ScriptInBcContainer $containerName -scriptblock { $config = Get-Navserverconfiguration $ServerInstance} -usePwsh:$false }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 833
Ticks             : 8335212
TotalDays         : 9.64723611111111E-06
TotalHours        : 0.000231533666666667
TotalMinutes      : 0.01389202
TotalSeconds      : 0.8335212
TotalMilliseconds : 833.5212


Measure-Command { Invoke-ScriptInBcContainer $containerName -scriptblock { $config = Get-Navserverconfiguration $ServerInstance} -usePwsh:$true } 


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 639
Ticks             : 16392312
TotalDays         : 1.89725833333333E-05
TotalHours        : 0.000455342
TotalMinutes      : 0.02732052
TotalSeconds      : 1.6392312
TotalMilliseconds : 1639.2312

These were measurements on a BC23 container.

In BC24 i'am unable to get these results (800ms) :

Measure-Command { Invoke-ScriptInBcContainer test24 -scriptblock { $config = Get-Navserverconfiguration $ServerInstance} -usePwsh:$false }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 361
Ticks             : 13618158
TotalDays         : 1.57617569444444E-05
TotalHours        : 0.000378282166666667
TotalMinutes      : 0.02269693
TotalSeconds      : 1.3618158
TotalMilliseconds : 1361.8158 

[marknitek]

This is true for any container by the way because of the version check which adds the 800ms to any command executed by invoke-scriptinbccontainer:

navcontainerhelper/ContainerHandling/Invoke-ScriptInNavContainer.ps1

Line 37 in f157a40

[System.Version]$platformVersion = Get-BcContainerPlatformVersion -containerOrImageName $containerName

[freddydk]

and if you set $bcContainerHelperConfig.usePwshForBC24 = $false - then everything is back to normal?
The reason for this was that all PowerShell cmdlets in BC24 are PowerShell 7 and the bridge also added quite a big overhead.
Therefore I created this and defaulted that to true.
Let me hear what the results are for the above and then figure out what the best approach is

[RonKoppelaar]

Freddy according to your question, is it related to my Initial problem? The build uses the default cmdlet to create the container,, without setting the $bcContainerHelperConfig.usePwshForBC24. Next it will compile and publish using the bccontainerhelper cmdlets. Basicly both steps are significant slower. But mainly the publish step.

[marknitek]

and if you set $bcContainerHelperConfig.usePwshForBC24 = $false - then everything is back to normal? The reason for this was that all PowerShell cmdlets in BC24 are PowerShell 7 and the bridge also added quite a big overhead. Therefore I created this and defaulted that to true. Let me hear what the results are for the above and then figure out what the best approach is

If that was question was related to my comments: setting it to $false does the same effect as using -usepwsh:$false (as you would expect) and lowers execution times for Versions. But BC24 is still slower then BC23 (~800ms compared to ~1400ms). I have no idea where these differences come from...

@RonKoppelaar interesting findings, i checked by entering two containers (BC23/BC24) and running a simple Get-NAVServerInstance:

BC23

 Measure-Command { Get-NAVServerConfiguration $ServerInstance }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 72
Ticks             : 720100
TotalDays         : 8.33449074074074E-07
TotalHours        : 2.00027777777778E-05
TotalMinutes      : 0.00120016666666667
TotalSeconds      : 0.07201
TotalMilliseconds : 72.01

BC24

 Measure-Command { Get-NAVServerConfiguration $ServerInstance }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 666
Ticks             : 6660190
TotalDays         : 7.70855324074074E-06
TotalHours        : 0.000185005277777778
TotalMinutes      : 0.0111003166666667
TotalSeconds      : 0.666019
TotalMilliseconds : 666.019

Huge difference! is this comming from the ps5 -> ps7 wrapper?

[freddydk]

@marknitek Yes, this is the reason why I default containers to run PowerShell 7 by default.
But that comes with another cost - PowerShell 7 cannot (as far as I know) do PowerShell remoting - I cannot create a session - I need to use docker exec, which then has a similar cost - but maybe we can do something about that...
Will have to investigate that.

[marknitek]

@freddydk i tested with both now:

$bcContainerHelperConfig.usePwshForBC24 = $true
Write-Host "Total Execution: "(Measure-Command { 
    Invoke-ScriptInBcContainer test24 -scriptblock { 
        Write-Host ($PSVersionTable | Out-String);
        Write-Host "Get-NavServerConfiguration: " (Measure-Command { Get-Navserverconfiguration $ServerInstance}).TotalSeconds "seconds"
    } 
}).TotalSeconds "seconds"

Name                           Value
----                           -----
PSVersion                      7.4.1
PSEdition                      Core
GitCommitId                    7.4.1
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Get-NavServerConfiguration:  0.1663978 seconds
Total Execution:  2.6827118 seconds

$bcContainerHelperConfig.usePwshForBC24 = $false
Write-Host "Total Execution: "(Measure-Command { 
    Invoke-ScriptInBcContainer test24 -scriptblock { 
        Write-Host ($PSVersionTable | Out-String);
        Write-Host "Get-NavServerConfiguration: " (Measure-Command { Get-Navserverconfiguration $ServerInstance}).TotalSeconds "seconds"
    } 
}).TotalSeconds "seconds"

Name                           Value
----                           -----
PSVersion                      5.1.20348.2227
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.20348.2227
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1


Get-NavServerConfiguration:  0.7761208 seconds
Total Execution:  1.5641623 seconds

[marknitek]

@freddydk regarding psremoting in ps7. I tinkered around a bit and i think it works quite well but must be enabled with Enable-PSRemoting in the pwsh first.

I had a look at Get-NavContainerSession and used these commands to get a session but for Powershell7:

$session = New-PSSession -Credential $credential -ComputerName $containerName -Authentication Basic -useSSL -SessionOption (New-PSSessionOption -SkipCACheck -SkipCNCheck) -ConfigurationName "Powershell.7" 

But that failed at first because there was no configuraiton for "Powershell.7". Then i enabled PSRemoting with:

Invoke-ScriptInBcContainer $containerName -scriptblock { Enable-PSRemoting;Get-PSSessionConfiguration }

Name          : PowerShell.7
PSVersion     : 7.4
StartupScript :
RunAsUser     :
Permission    : NT AUTHORITY\INTERACTIVE AccessAllowed, BUILTIN\Administrators AccessAllowed, BUILTIN\Remote
                Management Users AccessAllowed

Name          : PowerShell.7.4.1
PSVersion     : 7.4
StartupScript :
RunAsUser     :
Permission    : NT AUTHORITY\INTERACTIVE AccessAllowed, BUILTIN\Administrators AccessAllowed, BUILTIN\Remote
                Management Users AccessAllowed

Then i was able to use the session:

$session = New-PSSession -Credential $credential -ComputerName $containerName -Authentication Basic -useSSL -SessionOption (New-PSSessionOption -SkipCACheck -SkipCNCheck) -ConfigurationName "Powershell.7"
Invoke-Command -Session $session -ScriptBlock { $PSVersionTable | Out-String }

Name                           Value
----                           -----
PSVersion                      7.4.1
PSEdition                      Core
GitCommitId                    7.4.1
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion
SerializationVersion           1.1.0.1
WSManStackVersion              3.0 

[freddydk]

That is great news, thanks for investigating.
I will add this to containerhelper and assume that we will see performance increases after this

[freddydk]

@marknitek what did you use as credential? Did you create a windows user inside the container?

[freddydk]

It looks like you are using Windows Authentication, right?

[marknitek]

@freddydk i did not create anything, i just used the code that was present in Get-NavContainerSession to get the credentials which previously created all this when using remoting with regular powershell.
Imo it was just a matter of Enabling remoting in the pwsh instance.

But the code you have in place for generating all this still remains valid and works across powershell and pwsh:

$UUID = (Get-CimInstance win32_ComputerSystemProduct).UUID
                $credential = New-Object PSCredential -ArgumentList 'winrm', (ConvertTo-SecureString -string $UUID -AsPlainText -force)
                Invoke-ScriptInBcContainer -containerName $containerName -useSession:$false -scriptblock { Param([PSCredential] $credential)
                    $winrmuser = get-localuser -name $credential.UserName -ErrorAction SilentlyContinue
                    if (!$winrmuser) {
                        $cert = New-SelfSignedCertificate -DnsName "dontcare" -CertStoreLocation Cert:\LocalMachine\My
                        winrm create winrm/config/Listener?Address=*+Transport=HTTPS ('@{Hostname="dontcare"; CertificateThumbprint="' + $cert.Thumbprint + '"}') | Out-Null
                        winrm set winrm/config/service/Auth '@{Basic="true"}' | Out-Null
                        Write-Host "`nCreating Container user $($credential.UserName)"
                        New-LocalUser -AccountNeverExpires -PasswordNeverExpires -FullName $credential.UserName -Name $credential.UserName -Password $credential.Password | Out-Null
                        Add-LocalGroupMember -Group administrators -Member $credential.UserName | Out-Null
                    }
                } -argumentList $credential
                $session = New-PSSession -Credential $credential -ComputerName $containerName -Authentication Basic -useSSL -SessionOption (New-PSSessionOption -SkipCACheck -SkipCNCheck)

[freddydk]

Got it - that was what I was after - it creates a local administrator in the container which it can use for this.
I thought you found a different way:-)

[RonKoppelaar]

@freddydk is there anything I can do to further test/validate performance issues which could help for a resolution?

[freddydk]

I will modify the Invoke-ScriptInBcContainer to use sessions in all combinations (ps5 -> ps5, ps5 -> ps7, ps7 -> ps5 and ps7 -> ps7 - when that is done - I would like some tests and specific issues on which things might still be slow.

[freddydk]

@marknitek what OS are you running?
I am having issues with this on my Windows 11 machine - the winrm stuff doesn't work at all...:-(
WIll need to check some more

[freddydk]

Please try the ContainerHelper changes from this PR:
#3496

and give me some feedback on which changes you see.
With this, I can create PS5 and PS7 sessions in the container from both PS5 and PS7 (admin or non-admin) on the host.

If a Session is created from PS5 in admin mode, the New-PSSession with the ContainerId parameter is used - no credentials are needed.
If a session is created from PS7 or PS5 in non-admin mode, then winrm is used. For winrm, BcContainerHelper automatically creates a user inside the container with username = winrm and the password is the UUID of the host computer.
If the config setting useSslForWinRmSession is true (default), it uses HTTPS for the container connection (port 5986).
If the config setting useSslForWinRmSession is false, it uses HTTP for the container connection (port 5985).

Invoke-scriptInBcContainer -containerName $containerName -scriptblock { $PSVersionTable | out-host }

should show the PS version used in the container.
It takes a little time to create the session - but subsequent calls should be fast as the session is cached.

On my machine, the first invoke to create the winrm session takes 1-2 seconds (unless I am running PS5 as admin) and after that - every call takes ~300ms

[freddydk]

Here are some examples of running the new session mechanisms in the different modes, which differ.
The script I am running is basically showing the PowerShell version inside the container, the username inside the container and the number of milliseconds it took to run the script

[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc23 -useSession $true -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc23 -useSession $true -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc23 -useSession $true -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc24 -useSession $true -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc24 -useSession $true -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc24 -useSession $true -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds

Running in PS5 in admin mode

PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
2193
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
5456
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
796
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
4906
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
809
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
794

Running in PS5 in non-admin mode

PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
1383
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
5404
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
760
PowerShell Version: 7.4.1
Username: bc24\winrm
2965
PowerShell Version: 7.4.1
Username: bc24\winrm
513
PowerShell Version: 7.4.1
Username: bc24\winrm
500

Running in PS7 in admin mode

PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
1433
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
5420
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
750
PowerShell Version: 7.4.1
Username: bc24\winrm
4277
PowerShell Version: 7.4.1
Username: bc24\winrm
699
PowerShell Version: 7.4.1
Username: bc24\winrm
659

Running in PS7 in non-admin mode

PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
1595
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
5836
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
707
PowerShell Version: 7.4.1
Username: bc24\winrm
4093
PowerShell Version: 7.4.1
Username: bc24\winrm
772
PowerShell Version: 7.4.1
Username: bc24\winrm
777

In interesting observation is, that the second time we run a PS5 session inside the container (where the session is cached), it for some reasons takes much longer than the first time - I have no idea why this happens. Subsequent invokes are all faster - around 800ms.

Note that when running any combination beside PS5 in admin mode, the user is a newly created local admin inside the container called winrm with a hardcoded password set to the UUID of the host computer.

On my machine - SSL connection to winrm doesn't work at all - I have to set:

$bcContainerHelperConfig.useSslForWinRmSession = $false

If you want to always use WinRm (also on PS5 in admin mode) you can set:

$bcContainerHelperConfig.alwaysUseWinRmSession = $true

This isn't set by default as this would force everybody to always use new behavior - and with the number of people using BcContainerHelper, I know that any change causes disruption.

Setting useSession to false will cause Invoke-ScriptInBcContainer to use docker exec when running scripts inside the container.
The script looks then like this:

[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc23 -useSession $false -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc23 -useSession $false -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc23 -useSession $false -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc24 -useSession $false -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc24 -useSession $false -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds
[int](Measure-Command {Invoke-ScriptInBcContainer -containerName bc24 -useSession $false -scriptblock { Write-Host "PowerShell Version: $($PSVersionTable.PSVersion)"; Write-Host "Username: $(whoami)" }}).TotalMilliseconds

Running in PS5 in admin mode

PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1986
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1455
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1451
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1503
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1505
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1505

Running in PS5 in non-admin mode

PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1938
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1342
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1351
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1403
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1402
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1430

Running in PS7 in admin mode

PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1621
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1083
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1092
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1156
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1149
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1161

Running in PS7 in non-admin mode

PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1076
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1055
PowerShell Version: 5.1.20348.2400
Username: user manager\containeradministrator
1073
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1141
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1110
PowerShell Version: 7.4.1
Username: user manager\containeradministrator
1135

Using docker exec for script invoke inside the container are (on my machine) fairly consistent and the first call is faster then creating a winrm session.

Things to investigate:

1. Why is the second script invoke into a PS5 session inside the container waaaaay slower than the first and the third
2. Are there any things that the winrm user cannot do inside the container (or stuff that works differently)
3. Why can my machine not run winrm over SSL using a self-signed certificate inside the container (it works in GitHub hosted runners just fine...). Please let me know if other people are having this issue as well.

[freddydk]

Wanting to investigate no. 1 from above, and the behavior disappeared without any changes...
Now the time is fairly consistent like this:

PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
1151
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
412
PowerShell Version: 5.1.20348.2400
Username: bc23\winrm
412
PowerShell Version: 7.4.1
Username: bc24\winrm
1267
PowerShell Version: 7.4.1
Username: bc24\winrm
401
PowerShell Version: 7.4.1
Username: bc24\winrm
393

Will not investigate further unless other people are reporting stranger things...

[freddydk]

Found out that the whoami for some reason some times takes a lot of time (5 seconds) - replaced this with $env:username

[freddydk]

Created new containers and running without using whoami

Running in PS5 in admin mode (with alwaysUseWinRmSession = false)

PowerShell Version: 5.1.20348.2400
Username: ContainerAdministrator
2654
PowerShell Version: 5.1.20348.2400
Username: ContainerAdministrator
524
PowerShell Version: 5.1.20348.2400
Username: ContainerAdministrator
522
PowerShell Version: 7.4.1
Username: ContainerAdministrator
3523
PowerShell Version: 7.4.1
Username: ContainerAdministrator
530
PowerShell Version: 7.4.1
Username: ContainerAdministrator
505

Running in PS5 in admin mode (with alwaysUseWinRmSession = true)

PowerShell Version: 5.1.20348.2400
Username: winrm
1980
PowerShell Version: 5.1.20348.2400
Username: winrm
535
PowerShell Version: 5.1.20348.2400
Username: winrm
529
PowerShell Version: 7.4.1
Username: winrm
2135
PowerShell Version: 7.4.1
Username: winrm
538
PowerShell Version: 7.4.1
Username: winrm
534

Running in PS5 in non-admin mode

PowerShell Version: 5.1.20348.2400
Username: winrm
1331
PowerShell Version: 5.1.20348.2400
Username: winrm
486
PowerShell Version: 5.1.20348.2400
Username: winrm
469
PowerShell Version: 7.4.1
Username: winrm
2041
PowerShell Version: 7.4.1
Username: winrm
489
PowerShell Version: 7.4.1
Username: winrm
476

Running in PS7 in admin mode

PowerShell Version: 5.1.20348.2400
Username: winrm
1095
PowerShell Version: 5.1.20348.2400
Username: winrm
362
PowerShell Version: 5.1.20348.2400
Username: winrm
367
PowerShell Version: 7.4.1
Username: winrm
1967
PowerShell Version: 7.4.1
Username: winrm
376
PowerShell Version: 7.4.1
Username: winrm
356

Running in PS7 in non-admin mode

PowerShell Version: 5.1.20348.2400
Username: winrm
1057
PowerShell Version: 5.1.20348.2400
Username: winrm
365
PowerShell Version: 5.1.20348.2400
Username: winrm
351
PowerShell Version: 7.4.1
Username: winrm
1748
PowerShell Version: 7.4.1
Username: winrm
369
PowerShell Version: 7.4.1
Username: winrm
360

Things still to investigate:

1. Why is the second script invoke into a PS5 session inside the container waaaaay slower than the first and the third
   Are there any things that the winrm user cannot do inside the container (or stuff that works differently)
2. Why can my machine not run winrm over SSL using a self-signed certificate inside the container (it works in GitHub hosted runners just fine...). Please let me know if other people are having this issue as well.

[RonKoppelaar]

@freddydk lot of investigations regarding session management. Is there anything I can already test in my build containers regarding the compile and publish step to see any improvements?

[freddydk]

@marknitek in the latest containerHelper I have removed all -usepwsh:$false - and there is a pscoreoverrides.ps1 in c:\run with these lines:

function Invoke-SqlCmd { SqlServer\Invoke-Sqlcmd @args -Encrypt Optional }
function Backup-SqlDatabase { SqlServer\Backup-SqlDatabase @args -Encrypt Optional }
function Restore-SqlDatabase { SqlServer\Restore-SqlDatabase @args -Encrypt Optional }

You can override this file by adding one with the same name in the my folder if you don't want these or if you want to add more.
Remember to invoke c:\run\pscoreoverrides.ps1 if you want to include MS overrides.

These pscoreoverrides are loaded in the prompt.ps1 together with the ps7 BC modules.

[MattTraxinger]

@freddydk Once I get everything together I'll post a new issue. I don't think it's anything for BCContainerHelper so far. The size of the compiler folder seems to have increased by +/-30% to over 1GB and ALTool is significantly slower when enumerating through the artifacts compared the code analysis that was being run before. I think Microsoft has dumped every app in there as well.

[freddydk]

Thanks @MattTraxinger - I saw that myself.
The codeanalysis cannot be used anymore though will check whether we can add the cache to the artifacts in the first place maybe

[MattTraxinger]

@freddydk Fair enough. Those are really the two big differences I saw. Compiler Folder is now 1.34GB instead of 984MB. The increase in copy time is proportional to the increase in size. So be it.

The enumerating apps part is really bad, though. It's gone from 3s to 38s in my tests.

It's all just a bunch of small things. 30s here. 30s there. 15s over there. Not that these were long builds to begin with, but suddenly they take twice as long. I won't put it in a separate issue unless you want me to since there's not a lot to be done.

[RonKoppelaar]

@freddydk Below script to create my container
License file to be changed in your own...

$BCArtifactURL = Get-BCArtifactUrl -type OnPrem -country nl -version "24.0" -Verbose 
$BCArtifactURL 
$OnpremContainer="demo"
$DockerImageName="local"
$UseNewDatabase=$True
$ContainerName="demo"
$BuildCredential = New-Object System.Management.Automation.PSCredential ('devadmin', (ConvertTo-SecureString 'Welkom01!' -AsPlainText -Force))
$MultiTenant=$True           
$LicenseFileName="C:\agent2\_work\1\s\ERP AL\licenses\dev.bclicense"

New-NavContainer `
    -imageName $DockerImageName `
    -artifactUrl $BCArtifactURL `
    -containerName $ContainerName `
    -auth 'NavUserPassword' `
    -Credential $BuildCredential `
    -memoryLimit '25G' `
    -licenseFile $LicenseFileName `
    -accept_eula `
    -accept_outdated `
    -updateHosts `
    -alwaysPull `
    -multitenant:$MultiTenant `
    -useNewDatabase:$UseNewDatabase `
    -dns 8.8.8.8 `
    -doNotCheckHealth `
    -isolation process 

Which results in error as earlier mentioned.
Server W2019.

[freddydk]

My guess is that the ImageName parameter causes the issue - that you have an old image, which it is reusing.
If you add -alwaysPull - does it work then?

[RonKoppelaar]

Allways pull is part of the parameter list. But I can remove the old local image to be sure

[RonKoppelaar]

Remaining local images...

PS C:\Windows\system32> docker images
REPOSITORY                          TAG                                   IMAGE ID       CREATED        SIZE
local                               onprem-23.5.16502.16757-nl            b4df7925a1a0   22 hours ago   14.8GB
local                               sandbox-23.5.16502.16887-nl-nodb-mt   9c6596361f0c   4 days ago     14.2GB
local                               sandbox-23.5.16502.16887-nl-mt        9c199594d95a   5 days ago     17.3GB
local                               sandbox-23.5.16502.16887-nl           d5a51ba03403   5 days ago     15.8GB
local                               sandbox-23.5.16502.16887-nl-nodb      65a6ab305c02   5 days ago     14.2GB
mcr.microsoft.com/businesscentral   ltsc2019                              6cb4e8ccc602   5 days ago     10.2GB

Recreating container using BcContainerHelper is version 6.0.16-preview1184
Also installed .Net 8 SDK on the host (Not sure if relevant)
Unfortune still same error.

After creating container these are the images

PS C:\Windows\system32> docker images
REPOSITORY                          TAG                                   IMAGE ID       CREATED          SIZE
local                               onprem-24.0.16410.18056-nl            fe1d3e4c3afb   6 minutes ago    15.6GB
<none>                              <none>                                823fb407215b   16 minutes ago   12.7GB
mcr.microsoft.com/businesscentral   ltsc2019-dev                          d65d38f7fb9f   22 hours ago     10.5GB
local                               onprem-23.5.16502.16757-nl            b4df7925a1a0   23 hours ago     14.8GB
local                               sandbox-23.5.16502.16887-nl-nodb-mt   9c6596361f0c   4 days ago       14.2GB
local                               sandbox-23.5.16502.16887-nl-mt        9c199594d95a   5 days ago       17.3GB
local                               sandbox-23.5.16502.16887-nl           d5a51ba03403   5 days ago       15.8GB
local                               sandbox-23.5.16502.16887-nl-nodb      65a6ab305c02   5 days ago       14.2GB
mcr.microsoft.com/businesscentral   ltsc2019                              6cb4e8ccc602   5 days ago       10.2GB

[RonKoppelaar]

On my laptop (w11) the above script just works... :-( But not on buildserver W2019...

[freddydk]

Did you try without the image name on the build server?

[freddydk]

Are you using container helper preview on the build machine? It looks like it doesn’t pull the lots 2019-dev image???

[RonKoppelaar]

BcContainerHelper is version 6.0.16-preview1184
Is being used. I'm now trying without the image.

[RonKoppelaar]

Without local Docker image it passed. Shoudl I try to remove all images?

[freddydk]

Worth a try -

[RonKoppelaar]

FYI we are running multiple agents on our buildmachine. Each agent with each own local (admin) user. We install the PS modules in the user document folder. Meaning each agent can have its own PS module versions.
Could there be impact on the created images? Because those are shared cross the agents.

For now I've removed ALL local images on the machine.

[RonKoppelaar]

After cleaning up all images and using -DockerImage "local" it failed with same error.

[freddydk]

So, on windows server 2019, it cannot build an image on the fly.
Will test that

[RonKoppelaar]

For now I put a workaround in my build to not use image if using NewDatabase and artifacturl like /24. Just to bypass this error. To at least see if other improvements do work.

[freddydk]

@RonKoppelaar I have a repro of the problem and know what is happening - will see if I can figure out why?

[RonKoppelaar]

I was able to run my build end-to-end. Performance wise I still see two diffs:

• Publishing all apps into the Empty container takes 13min compared to bc23.5 5,5 min.
• Get-BcContainerAppRuntimePackage for BC24: 23 min. compared to bc23.5: 2,5 mins..

There still seems to be an issue there. If you want I can create a local testset with all the apps to publish and to export.

[freddydk]

The reason why things failed when creating an image was due to session caching - which probably also means that your build was slow due to this.
A new prerelease has just been shipped, which should fix this - please retry (also with -imagename)

[RonKoppelaar]

Can confirm the "local" image is working again. Performance is a bit harder to confirm. As both builds are running on the same machine. So all in all is a bit slower. Will run a new build when these are ready.

[RonKoppelaar]

Finished testing with latest pre-release 6.0.16-preview1186:

• Publishing apps still very slow.
• Exporting to runtime apps dropped to 7mins. compared to previous build: 23min. (bc23.5 --> 2,5 mins.)

I'm setting up a local test case with BC23.5 and BC24. Keep you posted.

[RonKoppelaar]

I Created testcase local. Based on the results (using HyperV) all is kind of comparable. No big performance diff.
Testing on buildserver is kind of unexpected due to other influences (Other builds / Azure in general).

Local 23.5
Create container: 245.6628468 4 min.
Publish time: 501.6631784 8 min.
Export runtime pkg: 125.6097803 2 min.

Local 24.0
Create container: 299.1621385 5 min.
Publish time: 764.543689 12,5 min.
Export runtime pkg: 112.3128834 1,5 min.

Not sure if process isolation make big difference but I'll check.

[freddydk]

I would like to investigate the publish time - that one is the only thing that feels wrong.
If you can provide me with your local test scripts and I will have a look where this is off.

Thanks for this investigation @RonKoppelaar

[freddydk]

Latest preview of ContainerHelper shuold fix the last issues...
Also found that session caching didn't really work for PS7 - fixed now.

[RonKoppelaar]

Builds are running on my end again. Will also do the local check with the testcase I provided yesetrday

[RonKoppelaar]

Builds are kind of comparable with 23.5 again. Stil a bit slower but as you mentioned the publish cmdlet itself in BC24 is a bit slower. Also ran the local test case again compared to yesterday preview build I can see good improvements:

Local 24.0 - HyperV - with preview build 24/04
Create container: 299.1621385
Publish time: 764.543689
Export time: 112.3128834

Local 24.0 - HyperV - with preview build 25/04
Create container: 282.792769
Publish time: 575.700574
Export time: 75.2510381

For me its case closed. Thx freddy and all other contributors on solving this topic!

[freddydk]

BcContainerHelper 6.0.16 has shipped together with generic images 1.0.2.20