Run-ALPipeline stops creating container with hcs::System::Start...

[horstoeko]

Hi

Since a few days it is no longer possible to get Run-ALPipeline to build a container. The process aborts with the following error:

DockerDo : docker: Error response from daemon: container 
ceaf390d9b1fee1b9f516b9434f2c9c096adbbe42e52429b9544fdf83c34ff25 encountered an error during hcs::System::Start: 
failure in a Windows system call: The virtual computer or container was terminated unexpectedly. (0xc0370106).
ExitCode: 125

Full output of scripts

2024-02-23T08:20:57.7851216Z   _____                               _
2024-02-23T08:20:57.7851631Z  |  __ \                             | |
2024-02-23T08:20:57.7853018Z  | |__) |_ _ _ __ __ _ _ __ ___   ___| |_ ___ _ __ ___
2024-02-23T08:20:57.7853618Z  |  ___/ _` | '__/ _` | '_ ` _ \ / _ \ __/ _ \ '__/ __|
2024-02-23T08:20:57.7853919Z  | |  | (_| | | | (_| | | | | | |  __/ |_  __/ |  \__ \
2024-02-23T08:20:57.7854203Z  |_|   \__,_|_|  \__,_|_| |_| |_|\___|\__\___|_|  |___/
2024-02-23T08:20:57.7854928Z 
2024-02-23T08:20:57.7882874Z Pipeline name                   
2024-02-23T08:20:57.7908314Z Container name                  bcserver
2024-02-23T08:20:57.7929496Z Image name                      my
2024-02-23T08:20:57.7966072Z ArtifactUrl                     https://bcartifacts.azureedge.net/sandbox/23.4.15643.15786/de
2024-02-23T08:20:57.8004814Z SasToken                        Not Specified
2024-02-23T08:20:57.8025319Z BcAuthContext                   Not Specified
2024-02-23T08:20:57.8043187Z Environment                     
2024-02-23T08:20:57.8061875Z ReUseContainer                  False
2024-02-23T08:20:57.8074894Z KeepContainer                   True
2024-02-23T08:20:57.8087812Z useCompilerFolder               False
2024-02-23T08:20:57.8100422Z artifactCachePath               
2024-02-23T08:20:57.8116772Z useDevEndpoint                  False
2024-02-23T08:20:57.8139036Z Auth                            UserPassword
2024-02-23T08:20:57.8185121Z Credential                      Specified
2024-02-23T08:20:57.8212499Z CompanyName                     My Company
2024-02-23T08:20:57.8233112Z MemoryLimit                     8G
2024-02-23T08:20:57.8251896Z FailOn                          none
2024-02-23T08:20:57.8278693Z TreatTestFailuresAsWarnings     False
2024-02-23T08:20:57.8298616Z Enable Task Scheduler           False
2024-02-23T08:20:57.8316209Z Assign Premium Plan             False
2024-02-23T08:20:57.8332280Z Install Test Runner             False
2024-02-23T08:20:57.8347769Z Install Test Framework          True
2024-02-23T08:20:57.8363301Z Install Test Libraries          True
2024-02-23T08:20:57.8379437Z Install Perf. Toolkit           False
2024-02-23T08:20:57.8394878Z InstallOnlyReferencedApps       False
2024-02-23T08:20:57.8411796Z generateDependencyArtifact      False
2024-02-23T08:20:57.8428962Z CopySymbolsFromContainer        False
2024-02-23T08:20:57.8446466Z enableCodeCop                   False
2024-02-23T08:20:57.8461489Z enableAppSourceCop              False
2024-02-23T08:20:57.8479864Z enableUICop                     False
2024-02-23T08:20:57.8505338Z enablePerTenantExtensionCop     True
2024-02-23T08:20:57.8540149Z enableCodeAnalyzersOnTestApps   False
2024-02-23T08:20:57.8564073Z doNotPerformUpgrade             False
2024-02-23T08:20:57.8578650Z doNotPublishApps                False
2024-02-23T08:20:57.8592193Z uninstallRemovedApps            False
2024-02-23T08:20:57.8606786Z escapeFromCops                  False
2024-02-23T08:20:57.8620075Z doNotBuildTests                 False
2024-02-23T08:20:57.8633162Z doNotRunTests                   False
2024-02-23T08:20:57.8646646Z doNotRunBcptTests               False
2024-02-23T08:20:57.8661425Z useDefaultAppSourceRuleSet      False
2024-02-23T08:20:57.8672082Z rulesetFile                     
2024-02-23T08:20:57.8684472Z generateErrorLog                False
2024-02-23T08:20:57.8697987Z enableExternalRulesets          False
2024-02-23T08:20:57.8713728Z azureDevOps                     True
2024-02-23T08:20:57.8733527Z gitLab                          False
2024-02-23T08:20:57.8746021Z gitHubActions                   False
2024-02-23T08:20:57.8758012Z vsixFile                        
2024-02-23T08:20:57.8767775Z License file                    Not specified
2024-02-23T08:20:57.8776436Z CodeSignCertPfxFile             Not specified
2024-02-23T08:20:57.8797064Z CodeSignCertPfxPassword         Not specified
2024-02-23T08:20:57.8829138Z CodeSignCertIsSelfSigned        False
2024-02-23T08:20:57.8842426Z KeyVaultCertPfxFile             Not specified
2024-02-23T08:20:57.8853731Z KeyVaultCertPfxPassword         Not specified
2024-02-23T08:20:57.8865096Z KeyVaultClientId                
2024-02-23T08:20:57.8881572Z BuildOutputFile                 
2024-02-23T08:20:57.8899403Z ContainerEventLogFile           
2024-02-23T08:20:57.8913735Z TestResultsFile                 C:\Agent\_work\31\s\testresults
2024-02-23T08:20:57.8928501Z BcptTestResultsFile             C:\Agent\_work\31\s\bcptTestResults.json
2024-02-23T08:20:57.8942137Z TestResultsFormat               JUnit
2024-02-23T08:20:57.8985308Z AdditionalCountries             
2024-02-23T08:20:57.8993398Z PackagesFolder                  C:\Agent\_work\31\s\.packages
2024-02-23T08:20:57.9005734Z OutputFolder                    C:\Agent\_work\31\s\.output
2024-02-23T08:20:57.9020086Z BuildArtifactFolder             C:\Agent\_work\31\s\dist
2024-02-23T08:20:57.9036057Z CreateRuntimePackages           False
2024-02-23T08:20:57.9052128Z AppVersion                      23.3
2024-02-23T08:20:57.9071571Z AppBuild                        20240223
2024-02-23T08:20:57.9084887Z AppRevision                     2
2024-02-23T08:20:57.9096434Z SourceRepositoryUrl             
2024-02-23T08:20:57.9107445Z SourceCommit                    
2024-02-23T08:20:57.9120220Z BuildBy                         BcContainerHelper,6.0.5
2024-02-23T08:20:57.9131453Z BuildUrl                        
2024-02-23T08:20:57.9139579Z Install Apps
2024-02-23T08:20:57.9139579Z   .......
2024-02-23T08:20:57.9139579Z   .......
2024-02-23T08:20:57.9168327Z Install Test Apps
2024-02-23T08:20:57.9174689Z - None
2024-02-23T08:20:57.9180928Z Previous Apps
2024-02-23T08:20:57.9187064Z - None
2024-02-23T08:20:57.9193442Z Application folders
2024-02-23T08:20:57.9193442Z   .......
2024-02-23T08:20:57.9302378Z Test application folders
2024-02-23T08:20:57.9316634Z   .......
2024-02-23T08:20:57.9323778Z BCPT Test application folders
2024-02-23T08:20:57.9330238Z - None
2024-02-23T08:20:57.9335565Z BCPT Test suites
2024-02-23T08:20:57.9341736Z - None
2024-02-23T08:20:57.9347588Z Custom CodeCops
2024-02-23T08:20:57.9353133Z - None
2024-02-23T08:20:57.9617746Z   _____       _ _ _                                          _        _
2024-02-23T08:20:57.9617940Z  |  __ \     | | (_)                                        (_)      (_)
2024-02-23T08:20:57.9618432Z  | |__) |   _| | |_ _ __   __ _    __ _  ___ _ __   ___ _ __ _  ___   _ _ __ ___   __ _  __ _  ___
2024-02-23T08:20:57.9618855Z  |  ___/ | | | | | | '_ \ / _` |  / _` |/ _ \ '_ \ / _ \ '__| |/ __| | | '_ ` _ \ / _` |/ _` |/ _ \
2024-02-23T08:20:57.9618973Z  | |   | |_| | | | | | | | (_| | | (_| |  __/ | | |  __/ |  | | (__  | | | | | | | (_| | (_| |  __/
2024-02-23T08:20:57.9619090Z  |_|    \__,_|_|_|_|_| |_|\__, |  \__, |\___|_| |_|\___|_|  |_|\___| |_|_| |_| |_|\__,_|\__, |\___|
2024-02-23T08:20:57.9619203Z                            __/ |   __/ |                                                 __/ |
2024-02-23T08:20:57.9619309Z                           |___/   |___/                                                 |___/
2024-02-23T08:20:57.9619353Z 
2024-02-23T08:20:58.7175449Z INFO: Windows 10 21H1/21H2 images are not yet available, using 2004 as these are found to work better than 20H2 on 21H1/21H2
2024-02-23T08:20:58.7359129Z Pulling mcr.microsoft.com/businesscentral:10.0.19041.1415
2024-02-23T08:20:59.6467079Z 
2024-02-23T08:20:59.6470258Z Pulling generic image took 2 seconds
2024-02-23T08:20:59.6682556Z 
2024-02-23T08:20:59.6682933Z    _____                _   _                               _        _
2024-02-23T08:20:59.6683092Z   / ____|              | | (_)                             | |      (_)
2024-02-23T08:20:59.6683275Z  | |     _ __ ___  __ _| |_ _ _ __   __ _    ___ ___  _ __ | |_ __ _ _ _ __   ___ _ __
2024-02-23T08:20:59.6683891Z  | |    | '__/ _ \/ _` | __| | '_ \ / _` |  / __/ _ \| '_ \| __/ _` | | '_ \ / _ \ '__|
2024-02-23T08:20:59.6684191Z  | |____| | |  __/ (_| | |_| | | | | (_| | | (__ (_) | | | | |_ (_| | | | | |  __/ |
2024-02-23T08:20:59.6684654Z   \_____|_|  \___|\__,_|\__|_|_| |_|\__, |  \___\___/|_| |_|\__\__,_|_|_| |_|\___|_|
2024-02-23T08:20:59.6684760Z                                      __/ |
2024-02-23T08:20:59.6684849Z                                     |___/
2024-02-23T08:20:59.6685215Z 
2024-02-23T08:20:59.9255904Z Creaing docker container
2024-02-23T08:21:01.4400157Z BcContainerHelper is version 6.0.5
2024-02-23T08:21:01.4416552Z BcContainerHelper is running as administrator
2024-02-23T08:21:02.0176636Z HyperV is Enabled
2024-02-23T08:21:02.0213430Z UsePsSession is True
2024-02-23T08:21:02.0251167Z Host is Microsoft Windows 10 Enterprise - 10.0.19045.4046
2024-02-23T08:21:02.2981525Z Docker Client Version is 25.0.3
2024-02-23T08:21:02.2999752Z Docker Server Version is 25.0.3
2024-02-23T08:21:02.7335685Z Removing entries from hosts
2024-02-23T08:21:02.8663256Z Removing bcserver from container hosts file
2024-02-23T08:21:03.0565837Z Removing bcserver-* from container hosts file
2024-02-23T08:21:03.0630826Z Removing Desktop shortcuts
2024-02-23T08:21:03.1358470Z Removing C:\ProgramData\BcContainerHelper\Extensions\bcserver
2024-02-23T08:21:04.1573509Z Fetching all docker images
2024-02-23T08:21:04.2966157Z Fetching all docker volumes
2024-02-23T08:21:04.4341692Z ArtifactUrl and ImageName specified
2024-02-23T08:21:05.3484013Z Image my:sandbox-23.4.15643.15786-de-mt already exists
2024-02-23T08:21:06.2505223Z Using image my:sandbox-23.4.15643.15786-de-mt
2024-02-23T08:21:06.4991798Z Creating Container bcserver
2024-02-23T08:21:06.5073428Z Style: sandbox
2024-02-23T08:21:06.5080972Z Multitenant: Yes
2024-02-23T08:21:06.5089578Z Version: 23.4.15643.15786
2024-02-23T08:21:06.5109962Z Platform: 23.0.15712.0
2024-02-23T08:21:06.5126606Z Generic Tag: 1.0.2.15
2024-02-23T08:21:06.5165930Z Container OS Version: 10.0.19041.1415 (2004)
2024-02-23T08:21:06.5178605Z Host OS Version: 10.0.19045.4046 (22H2)
2024-02-23T08:21:06.5282742Z Using hyperv isolation
2024-02-23T08:21:06.5354109Z Using locale de-DE
2024-02-23T08:21:06.5416864Z Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
2024-02-23T08:21:06.5851913Z Additional Parameters:
2024-02-23T08:21:06.5882408Z --volume "C:\Agent\_work\31\s:c:\sources"
2024-02-23T08:21:06.5892813Z --env customNavSettings=EnableTaskScheduler=False
2024-02-23T08:21:06.5901921Z Files in C:\ProgramData\BcContainerHelper\Extensions\bcserver\my:
2024-02-23T08:21:06.5954792Z - AdditionalOutput.ps1
2024-02-23T08:21:06.5970528Z - MainLoop.ps1
2024-02-23T08:21:06.5986271Z - SetupVariables.ps1
2024-02-23T08:21:06.6004003Z - updatehosts.ps1
2024-02-23T08:21:06.6014512Z Creating container bcserver from image my:sandbox-23.4.15643.15786-de-mt
2024-02-23T08:21:06.9278822Z ceaf390d9b1fee1b9f516b9434f2c9c096adbbe42e52429b9544fdf83c34ff25
2024-02-23T08:21:09.4485600Z New-BcContainer Telemetry Correlation Id: af9df726-e1f1-4fd8-b53c-e4e51f2cee72
2024-02-23T08:21:09.9434322Z Run-AlPipeline Telemetry Correlation Id: b8ed64f4-423b-4f86-a21a-cdc712effbe7
2024-02-23T08:21:09.9920510Z DockerDo : docker: Error response from daemon: container 
2024-02-23T08:21:09.9921512Z ceaf390d9b1fee1b9f516b9434f2c9c096adbbe42e52429b9544fdf83c34ff25 encountered an error during hcs::System::Start: 
2024-02-23T08:21:09.9921782Z failure in a Windows system call: Der virtuelle Computer oder Container wurde unerwartet beendet. (0xc0370106).
2024-02-23T08:21:09.9921937Z ExitCode: 125
2024-02-23T08:21:09.9922335Z Commandline: docker run --env multitenant=Y --env licenseFile="" --name bcserver --hostname bcserver --env 
2024-02-23T08:21:09.9922845Z auth=NavUserPassword --env username="admin" --env ExitOnError=N --env locale=de-DE --env databaseServer="" --env 
2024-02-23T08:21:09.9923326Z databaseInstance="" --volume "C:\ProgramData\BcContainerHelper:C:\ProgramData\BcContainerHelper" --volume 
2024-02-23T08:21:09.9923789Z "C:\ProgramData\BcContainerHelper\Extensions\bcserver\my:C:\Run\my" --isolation hyperv --restart unless-stopped --env 
2024-02-23T08:21:09.9924154Z filesOnly=False --memory 8G --env enableApiServices=Y --env useSSL=N --volume 
2024-02-23T08:21:09.9924521Z "c:\windows\system32\drivers\etc:C:\driversetc" --env securePassword=XXXXXXXX --env 
2024-02-23T08:21:09.9925674Z passwordKeyFile="c:\run\my\aes.key" --env removePasswordKeyFile=Y --volume "C:\Agent\_work\31\s:c:\sources" --env 
2024-02-23T08:21:09.9926014Z customNavSettings=EnableTaskScheduler=False --env accept_eula=Y --env accept_outdated=Y --detach 
2024-02-23T08:21:09.9926247Z my:sandbox-23.4.15643.15786-de-mt
2024-02-23T08:21:09.9926664Z In C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\6.0.5\ContainerHandling\New-NavContainer.ps1:1982 
2024-02-23T08:21:09.9926848Z Zeichen:15
2024-02-23T08:21:09.9927104Z + ...       if (!(DockerDo -accept_eula -accept_outdated:$accept_outdated - ...
2024-02-23T08:21:09.9927313Z +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-02-23T08:21:09.9927569Z     + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
2024-02-23T08:21:09.9927742Z     + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,DockerDo
2024-02-23T08:21:09.9927861Z  
2024-02-23T08:21:10.0926112Z ##[error]PowerShell exited with code '1'.
2024-02-23T08:21:10.1285357Z ##[section]Finishing: Run Pipeline

We have tried various things, but unfortunately without success. However, the containers can be created manually via Powershell. So (in our opinion) this does not seem to be a direct problem with Docker.

Many thanks in advance for your support and a possible solution to the problem.

[freddydk]

This command tries to use a pre-build image to create the container.
If you specify -imageName "" - then you should avoid this step and you can see whether that makes a difference

[horstoeko]

Hi @freddydk,

Thanks for the tip. We have temporarily adjusted our build infrastructure and added the -imageName "" parameter (as you mentioned). Now a container is created. However, the runtime is of course much longer. From our point of view, this can of course only be a temporary solution. ;-)

[dannoe]

Try to recreate the image, maybe docker was updated and the image is no longer valid.

[horstoeko]

Hi!

Deleting all images seems to have solved the problem. We will continue to monitor this.

[horstoeko]

Hi!

A small update: After deleting the my-images yesterday and setting up a new one, everything worked to our satisfaction. This morning the same problem occurred again (after our infrastructure was restarted). Apart from a reboot, nothing was changed in the infrastructure - nor was Docker updated in any way.

[freddydk]

You could try to run the cleanup tool from this old blog post: https://freddysblog.com/2018/12/11/clean-up-after-yourself-docker-your-mom-isnt-here/ - the docker file system contains a lot of symbolic links and if any of this goes bad - then you might have stuff in there, which fails everytime somebody touches it.