Pester test script cross Azure DevOps git repo in Azure pipeline fails

[Marc013]

When running my Pester test pwsh script cross repo in an Azure pipeline it fails.

Question:
How can I run a Pester test pwsh script cross Azure git repo within an Azure pipeline?

In repo GeneralPurpose I have a pwsh script that runs a Pester test (Invoke-Pester) using a Pester configuration defined by that script based on script input.
This Pester test scripts is triggered in an Azure Pipeline YAML using a PowerShell@2 task. This pipeline task is defined in a so called pipeline template YAML file.

Directory structure

 GeneralPurpose/CodeValidation/Pipelines
 │   buildValidation.yml
 │
 └───Templates
         codeValidation.yml

Running this pipeline (as a build validation) within repo GeneralPurpose works like a charm.
However, when I run the pipeline from another Azure repo it fails with an error message that I can't make heads or tails of.

In the other repo I have created an Azure pipeline YAML file in where I do a checkout of self and repo GeneralPurpose so I can use the code validation pipeline template.

When I move the Pester test code from the pwsh file to an Azure pipeline PowerShell@2 inline solution it does work 🤯
I'm most definitely doing something wrong, please tell me what!?

The error message:

Pester v5.3.3

Starting discovery in 1 files.
##[error] [-] Discovery in D:\a\1\GeneralPurpose\CodeValidation\Tests\PwshGuidelineValidation\Assert-PwshGuidelineValidation.tests.ps1 failed with:
##[error] System.Exception: Exception of type 'System.Exception' was thrown.
##[error]    at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)
##[error] at <ScriptBlock>, D:\a\1\GeneralPurpose\CodeValidation\Tests\PwshGuidelineValidation\Assert-PwshGuidelineValidation.tests.ps1: line 76
##[error] at New-Block, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 799
##[error] at Describe, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 9654
##[error] at <ScriptBlock>, D:\a\1\GeneralPurpose\CodeValidation\Tests\PwshGuidelineValidation\Assert-PwshGuidelineValidation.tests.ps1: line 41
##[error] at <ScriptBlock>, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 2985
##[error] at Invoke-File, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 2994
##[error] at Invoke-BlockContainer, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 2907
##[error] at Discover-Test, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 1454
##[error] at Invoke-Test, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 2427
##[error] at Invoke-Pester<End>, C:\Users\VssAdministrator\Documents\PowerShell\Modules\Pester\5.3.3\Pester.psm1: line 5225
##[error] at <ScriptBlock>, D:\a\1\GeneralPurpose\CodeValidation\Src\Scripts\Invoke-PwshGuidelineValidation.ps1: line 221
##[error] at <ScriptBlock>, D:\a\_temp\f5c6bc4f-2fef-4da2-b829-884c3ac703f8.ps1: line 13
##[error] at <ScriptBlock>, D:\a\_temp\f5c6bc4f-2fef-4da2-b829-884c3ac703f8.ps1: line 10
##[error] at <ScriptBlock>, <No file>: line 1
Discovery found 0 tests in 2.75s.
Running tests.
Tests completed in 2.72s

[fflaten]

Hard to say without seeing the code that throws @ Assert-PwshGuidelineValidation.tests.ps1: line 76.

[Marc013]

@fflaten,
That's absolutely true.
However, I'm not in the position to share the complete code, sorry.

The issue occurs whenever a Pester test is run in an Azure pipeline by calling a pwsh script. In the scenario that the pipeline template is located in another Azure git repo.
Running the exact same code in an Azure pipeline within the originating Azure git repo works without any issue.

When the code is placed inline it works without a problem.

As the Pester code runs perfectly when:

• run locally
• run in an Azure pipeline located in the repo that contains the Pester code
• run as in inline script (in the local repo and cross repo)

Line 76 of the code is:

[fflaten]

Understandable.

Tried printing $CustomRulePath to console? Maybe it's relative or using a variable like $PSScriptRoot that doesn't point the correct place? Can't really help without a repro as it looks to be a user-code exception.

[Marc013]

The issue is caused by PSScriptAnalyzer, not Pester.

In the Pester test I validate if a pwsh script is compliant with the default rules of PSScriptAnalyzer. I also added an option to provide a path to custom PSSA rules. This is where it goes wrong.

To obtain the custom rules I run cmdlet Get-ScriptAnalyzerRule

$rules += Get-ScriptAnalyzerRule -CustomRulePath $CustomRulePath -RecurseCustomRulePath

When a path is specified that does not contain custom rules error Exception of type 'System.Exception' was thrown. is thrown.

I've fixed the issue by first validating if custom rules are present at the provided path. If not, the path is ignored.