Download-BcNuGetPackageToFolder: The remote server returned an error: (503) Server Unavailable. Operations per second is over the account limit.

[Arthurvdv]

Describe the issue
In our Azure DevOps pipelines we've encountered an error on the Download-BcNuGetPackageToFolder step of the BcContainerHelper. The error message was The remote server returned an error: (503) Server Unavailable. Operations per second is over the account limit..

This seems to be an issue of Azure DevOps and and not the Download-BcNuGetPackageToFolder, I was hoping if there could be an automatically retry possibility on this, kinda like the retry methodes on the Get-BcArtifactUrl.

Scripts used to create container and cause the issue

Download-BcNuGetPackageToFolder -nuGetServerUrl  https://pkgs.dev.azure.com/vanroeybe/_packaging/BusinessCentral/nuget/v3/index.json 
                                -nuGetToken *** 
                                -packageName 18364d41-a124-4b93-98c8-73b700228c06 
                                -select Latest 
                                -folder C:\vsts-agent-alops-warp10\_work\2019\b\.previous 
                                -downloadDependencies none

Full output of scripts

2024-03-19T09:32:07.3210654Z ##[section]Starting: Download Current artifact (Breaking Changes Check)
2024-03-19T09:32:07.3293569Z ==============================================================================
2024-03-19T09:32:07.3293685Z Task         : PowerShell
2024-03-19T09:32:07.3293744Z Description  : Run a PowerShell script on Linux, macOS, or Windows
2024-03-19T09:32:07.3293833Z Version      : 2.236.2
2024-03-19T09:32:07.3293890Z Author       : Microsoft Corporation
2024-03-19T09:32:07.3293959Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2024-03-19T09:32:07.3294050Z ==============================================================================
2024-03-19T09:32:08.0826392Z Generating script.
2024-03-19T09:32:08.1131954Z ========================== Starting Command Output ===========================
2024-03-19T09:32:08.1490079Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\vsts-agent-alops-warp10\_work\_temp\8b82f00b-5df2-4dba-8946-6b2020cefa91.ps1'"
2024-03-19T09:32:08.3321205Z ##[command]Get-AppIdFromAppJson -appFilePath C:\vsts-agent-alops-warp10\_work\2019\s\App\app.json
2024-03-19T09:32:08.3578912Z ##[command]Retrieved AppId '18364d41-a124-4b93-98c8-73b700228c06' from 'C:\vsts-agent-alops-warp10\_work\2019\s\App\app.json'.
2024-03-19T09:32:08.3627566Z ##[command]Import-Module BcContainerHelper -DisableNameChecking
2024-03-19T09:32:08.4538761Z BcContainerHelper version 6.0.11
2024-03-19T09:32:08.6481823Z BC.HelperFunctions emits usage statistics telemetry to Microsoft
2024-03-19T09:32:08.7672228Z Running on Windows, PowerShell 5.1.17763.3770
2024-03-19T09:32:10.5025694Z ##[command]
2024-03-19T09:32:10.5026062Z Download-BcNuGetPackageToFolder -nuGetServerUrl  https://pkgs.dev.azure.com/vanroeybe/_packaging/BusinessCentral/nuget/v3/index.json 
2024-03-19T09:32:10.5027719Z                                 -nuGetToken *** 
2024-03-19T09:32:10.5027909Z                                 -packageName 18364d41-a124-4b93-98c8-73b700228c06 
2024-03-19T09:32:10.5028060Z                                 -select Latest 
2024-03-19T09:32:10.5028201Z                                 -folder C:\vsts-agent-alops-warp10\_work\2019\b\.previous 
2024-03-19T09:32:10.5028344Z                                 -downloadDependencies none
2024-03-19T09:32:10.5279615Z Looking for NuGet package 18364d41-a124-4b93-98c8-73b700228c06 version 0.0.0.0 (Latest match)
2024-03-19T09:32:10.5458827Z Search NuGetFeed https://pkgs.dev.azure.com/vanroeybe/_packaging/BusinessCentral/nuget/v3/index.json
2024-03-19T09:32:10.8475717Z Search package using https://pkgs.dev.azure.com/vanroeybe/_packaging/82099a1a-fe04-4c03-9594-3ab7937dc1df/nuget/v3/query2/?q=18364d41-a124-4b93-98c8-73b700228c06&take=50
2024-03-19T09:32:12.6768756Z 1 matching packages found
2024-03-19T09:32:12.6788620Z - VanRoey.<redacted>.18364d41-a124-4b93-98c8-73b700228c06
2024-03-19T09:32:12.6856629Z PackageId: VanRoey.<redacted>.18364d41-a124-4b93-98c8-73b700228c06
2024-03-19T09:32:12.7111100Z 5 versions found
2024-03-19T09:32:12.7193376Z First version is 23.0.134131
2024-03-19T09:32:12.7231834Z Last version is 23.0.133303
2024-03-19T09:32:12.7396954Z Latest version is 23.0.134131
2024-03-19T09:32:12.7483423Z Best match for package name 18364d41-a124-4b93-98c8-73b700228c06 Version 0.0.0.0: VanRoey.<redacted>.18364d41-a124-4b93-98c8-73b700228c06 Version 23.0.134131 from https://pkgs.dev.azure.com/vanroeybe/_packaging/BusinessCentral/nuget/v3/index.json
2024-03-19T09:32:12.7603120Z Download package using https://pkgs.dev.azure.com/vanroeybe/_packaging/82099a1a-fe04-4c03-9594-3ab7937dc1df/nuget/v3/flat2/vanroey.<redacted>.18364d41-a124-4b93-98c8-73b700228c06/23.0.134131/vanroey.<redacted>.18364d41-a124-4b93-98c8-73b700228c06.23.0.134131.nupkg
2024-03-19T09:32:12.9476218Z Error Message: The remote server returned an error: (503) Server Unavailable. Operations per second is over the account limit. 
2024-03-19T09:32:12.9477677Z StackTrace: at DownloadPackage, C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\6.0.11\NuGet\NuGetFeedClass.ps1: line 327
2024-03-19T09:32:12.9477985Z at Download-BcNuGetPackageToFolder, C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\6.0.11\NuGet\Download-BcNuGetPackageToFolder.ps1: line 143
2024-03-19T09:32:12.9478429Z at <ScriptBlock>, C:\vsts-agent-alops-warp10\_work\_temp\8b82f00b-5df2-4dba-8946-6b2020cefa91.ps1: line 45
2024-03-19T09:32:12.9480821Z at <ScriptBlock>, <No file>: line 1
2024-03-19T09:32:12.9919809Z The remote server returned an error: (503) Server Unavailable.
2024-03-19T09:32:12.9920469Z Operations per second is over the account limit.
2024-03-19T09:32:12.9922986Z At C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\6.0.11\NuGet\NuGetFeedClass.ps1:327 char:13
2024-03-19T09:32:12.9923230Z +             throw (GetExtendedErrorMessage $_)
2024-03-19T09:32:12.9924878Z +             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-03-19T09:32:12.9926650Z     + CategoryInfo          : OperationStopped: (The remote serv...ccount limit.
2024-03-19T09:32:12.9928402Z :String) [], RuntimeException
2024-03-19T09:32:12.9928961Z     + FullyQualifiedErrorId : The remote server returned an error: (503) Server Unavailable.
2024-03-19T09:32:12.9929489Z Operations per second is over the account limit.
2024-03-19T09:32:12.9929731Z  
2024-03-19T09:32:13.0740335Z ##[error]PowerShell exited with code '1'.
2024-03-19T09:32:13.1046237Z ##[section]Finishing: Download Current artifact (Breaking Changes Check)```

[Arthurvdv]

Some of our Azure DevOps pipelines failed this night on the Download-BcNuGetPackageToFolder step.

Download-BcNuGetPackageToFolder -packageName $($appJsonObj.id) `
    -select Latest `
    -folder $tempArtifactFolder `
    -downloadDependencies none

The error messages seems to be related to an temporary unavailability of Azure DevOps Services. Not all pipelines have failed and some before and some after has been successful. So probably with an retry mechanism this could be auto resolved?

2024-03-30T04:45:40.4833400Z ##[section]Starting: Download current version
2024-03-30T04:45:40.5045736Z ==============================================================================
2024-03-30T04:45:40.5045829Z Task         : PowerShell
2024-03-30T04:45:40.5045873Z Description  : Run a PowerShell script on Linux, macOS, or Windows
2024-03-30T04:45:40.5045937Z Version      : 2.236.2
2024-03-30T04:45:40.5045978Z Author       : Microsoft Corporation
2024-03-30T04:45:40.5046030Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2024-03-30T04:45:40.5046096Z ==============================================================================
2024-03-30T04:45:41.5315058Z Generating script.
2024-03-30T04:45:41.5589328Z ========================== Starting Command Output ===========================
2024-03-30T04:45:41.5795057Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\azure-vsts-agent-build\_work\_temp\daaa2631-71df-4d68-9455-bb4b0e183b28.ps1'"
2024-03-30T04:45:42.1882454Z BcContainerHelper version 6.0.12
2024-03-30T04:45:42.5059211Z BC.HelperFunctions emits usage statistics telemetry to Microsoft
2024-03-30T04:45:42.6631394Z Running on Windows, PowerShell 5.1.20348.2227
2024-03-30T04:45:45.2892588Z Looking for NuGet package e920e831-71f3-45ea-bc7b-f0f2c9ba2534 version 0.0.0.0 (Latest match)
2024-03-30T04:45:45.3037655Z Search NuGetFeed https://vanroeybe.pkgs.visualstudio.com/_packaging/BusinessCentral/nuget/v3/index.json
2024-03-30T04:45:45.5886424Z Search package using https://vanroeybe.pkgs.visualstudio.com/_packaging/82099a1a-fe04-4c03-9594-3ab7937dc1df/nuget/v3/query2/?q=e920e831-71f3-45ea-bc7b-f0f2c9ba2534&take=50
2024-03-30T04:45:47.7841693Z 1 matching packages found
2024-03-30T04:45:47.7851129Z - VanRoey.<redacted>.e920e831-71f3-45ea-bc7b-f0f2c9ba2534
2024-03-30T04:45:47.7894100Z PackageId: VanRoey.<redacted>.e920e831-71f3-45ea-bc7b-f0f2c9ba2534
2024-03-30T04:45:47.8080580Z 5 versions found
2024-03-30T04:45:47.8142713Z First version is 23.0.134739
2024-03-30T04:45:47.8166057Z Last version is 23.0.132071
2024-03-30T04:45:47.8292900Z Latest version is 23.0.134739
2024-03-30T04:45:47.8299317Z Search NuGetFeed https://vanroeybe.pkgs.visualstudio.com/_packaging/BusinessCentralLegacyApps/nuget/v3/index.json
2024-03-30T04:45:48.0874790Z Search package using https://vanroeybe.pkgs.visualstudio.com/_packaging/dbb1cbd4-c83e-443f-a0ea-debcb32b6828/nuget/v3/query2/?q=e920e831-71f3-45ea-bc7b-f0f2c9ba2534&take=50
2024-03-30T04:45:48.1365508Z 0 matching packages found
2024-03-30T04:45:48.1430609Z Best match for package name e920e831-71f3-45ea-bc7b-f0f2c9ba2534 Version 0.0.0.0: VanRoey.<redacted>.e920e831-71f3-45ea-bc7b-f0f2c9ba2534 Version 23.0.134739 from https://vanroeybe.pkgs.visualstudio.com/_packaging/BusinessCentral/nuget/v3/index.json
2024-03-30T04:45:48.1525367Z Download package using https://vanroeybe.pkgs.visualstudio.com/_packaging/82099a1a-fe04-4c03-9594-3ab7937dc1df/nuget/v3/flat2/vanroey.<redacted>.e920e831-71f3-45ea-bc7b-f0f2c9ba2534/23.0.134739/vanroey.<redacted>.e920e831-71f3-45ea-bc7b-f0f2c9ba2534.23.0.134739.nupkg
2024-03-30T04:45:48.7145838Z Package successfully downloaded
2024-03-30T04:45:48.7477328Z 
2024-03-30T04:45:51.7672677Z Invoke-WebRequest : 
2024-03-30T04:45:51.7673028Z     
2024-03-30T04:45:51.7673288Z         
2024-03-30T04:45:51.7673462Z         
2024-03-30T04:45:51.7674183Z             body {
2024-03-30T04:45:51.7674497Z                 margin: 0;
2024-03-30T04:45:51.7675268Z                 color: #000;
2024-03-30T04:45:51.7677183Z                 font-family: "-apple-system",BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Ubuntu,Ari
2024-03-30T04:45:51.7677882Z al,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";
2024-03-30T04:45:51.7677992Z             }
2024-03-30T04:45:51.7678080Z             header {
2024-03-30T04:45:51.7679727Z                 height: 40px;
2024-03-30T04:45:51.7680255Z                 line-height: 40px;
2024-03-30T04:45:51.7680386Z                 font-size: 14px;
2024-03-30T04:45:51.7680991Z                 margin-bottom: 20px;
2024-03-30T04:45:51.7681182Z                 background-color: #0078d4;
2024-03-30T04:45:51.7681431Z                 color: white;
2024-03-30T04:45:51.7681712Z             }
2024-03-30T04:45:51.7681794Z             img {
2024-03-30T04:45:51.7681920Z                 border: 0 none;
2024-03-30T04:45:51.7682224Z             }
2024-03-30T04:45:51.7682375Z             header img.logo {
2024-03-30T04:45:51.7682518Z                 height: 20px;
2024-03-30T04:45:51.7682662Z                 margin: 9px 5px 0 9px;
2024-03-30T04:45:51.7682801Z             }
2024-03-30T04:45:51.7682969Z             header .title {
2024-03-30T04:45:51.7683169Z                 vertical-align: top;
2024-03-30T04:45:51.7683470Z             }
2024-03-30T04:45:51.7683670Z             #content { text-align: center; }
2024-03-30T04:45:51.7684292Z             #content img { width: 320px; height: 320px; }
2024-03-30T04:45:51.7684490Z             #message h2 { font-size: 28px; font-weight: 200; margin: 40px 0 10px; }
2024-03-30T04:45:51.7684683Z             #message p  { font-size: 17px; font-weight: 300; margin: 0; }
2024-03-30T04:45:51.7684888Z             #message p.subheading { font-size: 14px; color: #212121; font-weight: 300; margin: 0; }
2024-03-30T04:45:51.7685012Z             a {color:#0778d4;}
2024-03-30T04:45:51.7685186Z             #errorref { font-size: 13px; color: #737373; margin-top: 24px }
2024-03-30T04:45:51.7685286Z         
2024-03-30T04:45:51.7685383Z         Azure DevOps Services Unavailable
2024-03-30T04:45:51.7685471Z     
2024-03-30T04:45:51.7685546Z     
2024-03-30T04:45:51.7685620Z         
2024-03-30T04:45:51.7685764Z             
2024-03-30T04:45:51.7685919Z             Azure DevOps Services
2024-03-30T04:45:51.7686061Z         
2024-03-30T04:45:51.7686222Z         
2024-03-30T04:45:51.7686375Z             
2024-03-30T04:45:51.7686523Z             
2024-03-30T04:45:51.7686751Z                 Sorry! Our services aren't available right now.
2024-03-30T04:45:51.7686954Z                 We're working to restore all services as quickly as possible. Please check back soon.
2024-03-30T04:45:51.7687106Z                 To see the latest status on our services, please visit our support page.
2024-03-30T04:45:51.7687215Z             
2024-03-30T04:45:51.7687292Z             
2024-03-30T04:45:51.7687425Z                 
2024-03-30T04:45:51.7687708Z                     Ref A: 2E7316B1F3CA42A09D93D835A88578B0 Ref B: BRU30EDGE0915 Ref C: 2024-03-30T04:45:48Z
2024-03-30T04:45:51.7687834Z                 
2024-03-30T04:45:51.7687914Z             
2024-03-30T04:45:51.7688034Z         
2024-03-30T04:45:51.7688179Z     
2024-03-30T04:45:51.7688655Z At C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\6.0.12\HelperFunctions.ps1:1251 char:16
2024-03-30T04:45:51.7689224Z +     $listing = Invoke-WebRequest -Method POST -UseBasicParsing `
2024-03-30T04:45:51.7689365Z +                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-03-30T04:45:51.7689922Z     + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc 
2024-03-30T04:45:51.7690060Z    eption
2024-03-30T04:45:51.7690186Z     + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
2024-03-30T04:45:51.8371665Z ##[error]PowerShell exited with code '1'.
2024-03-30T04:45:51.8615619Z ##[section]Finishing: Download current version