Scripts using PNP.PowerShell do not work in VSCode

[Tom-CK]

Prerequisites

• I have written a descriptive issue title.
• I have searched all issues to ensure it has not already been reported.
• I have read the troubleshooting guide.
• I have verified that I am using the latest version of Visual Studio Code and the PowerShell extension.

Summary

I've installed PNP.PowerShell-Module Version 1.7.0.
When working within a normal PowerShell, all CMDLets work fine - for example running "Get-PBPListItem" returns the expected items.

When I work in VSCode with installed PowerShell Extension (v 2021.8.0), ALL PNP-PowerShell CMDLets throw an exception (TargetInvocationException).

This used to work perfect earlier ....

PowerShell Version

Name                           Value
----                           -----
PSVersion                      5.1.19041.1023
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.1023
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Visual Studio Code Version

1.59.0
379476f0e13988d90fab105c5c19e7abc8b1dea8
x64

Extension Version

ms-vscode.powershell@2021.5.1
ms-vscode.powershell@2021.8.0
ms-vscode.powershell-preview@2021.8.0

Steps to Reproduce

• Install PNP.PowerShell module
• run a CMDLet like Get-PNListItem in a regular powershell window -> working !
• run a CMDLet like Get-PNListItem in a vscode powershell window -> TargetInvocationException

Visuals

Logs

No response

[rjmholt]

Can you provide the output of $error[0] | fl * -force after the error occurs?

[Tom-CK]

 $error[0] | fl * -force



writeErrorStream      : True
PSMessageDetails      : 
Exception             : System.Reflection.TargetInvocationException: Ein Aufrufziel hat einen Ausnahmefehler verursacht. ---> System.MissingMethodException:  
                        Methode nicht gefunden: "Int32 System.Text.Encodings.Web.TextEncoder.FindFirstCharacterToEncodeUtf8(System.ReadOnlySpan`1<Byte>)".    
                           bei System.Text.Json.JsonWriterHelper.NeedsEscaping(ReadOnlySpan`1 value, JavaScriptEncoder encoder)
                           bei System.Text.Json.JsonEncodedText.EncodeHelper(ReadOnlySpan`1 utf8Value, JavaScriptEncoder encoder)
                           bei System.Text.Json.JsonPropertyInfo.DeterminePropertyName()
                           bei System.Text.Json.JsonPropertyInfo.GetPolicies()
                           bei System.Text.Json.JsonClassInfo.CreateProperty(Type declaredPropertyType, Type runtimePropertyType, Type 
                        implementedPropertyType, PropertyInfo propertyInfo, Type parentClassType, JsonConverter converter, JsonSerializerOptions options)     
                           bei System.Text.Json.JsonClassInfo.AddProperty(Type propertyType, PropertyInfo propertyInfo, Type classType, JsonSerializerOptions 
                        options)
                           bei System.Text.Json.JsonClassInfo..ctor(Type type, JsonSerializerOptions options)
                           bei System.Text.Json.JsonSerializerOptions.GetOrAddClass(Type classType)
                           bei System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader)
                           bei System.Text.Json.JsonSerializer.Deserialize(String json, Type returnType, JsonSerializerOptions options)
                           bei System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
                           bei PnP.Framework.Utilities.TokenHelper.GetMetadataDocument(String realm)
                           bei PnP.Framework.Utilities.TokenHelper.GetStsUrl(String realm)
                           bei PnP.Framework.Utilities.TokenHelper.GetAppOnlyAccessToken(String targetPrincipalName, String targetHost, String targetRealm)    
                           bei PnP.Framework.Utilities.ACSTokenGenerator.GetToken(Uri siteUrl)
                           bei PnP.Framework.AuthenticationManager.<>c__DisplayClass75_0.<GetAccessTokenContext>b__0(Object sender, WebRequestEventArgs args)  
                           bei Microsoft.SharePoint.Client.ClientRuntimeContext.OnExecutingWebRequest(WebRequestEventArgs args)
                           --- Ende der internen Ausnahmestapelüberwachung ---
                           bei System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
                           bei System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
                           bei System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, 
                        CultureInfo culture)
                           bei Microsoft.SharePoint.Client.ClientContextExtensions.<>c__DisplayClass10_0.<Clone>b__0(Object oSender, WebRequestEventArgs       
                        webRequestEventArgs)
                           bei System.EventHandler`1.Invoke(Object sender, TEventArgs e)
                           bei Microsoft.SharePoint.Client.ClientRuntimeContext.OnExecutingWebRequest(WebRequestEventArgs args)
                           bei Microsoft.SharePoint.Client.ClientContext.GetWebRequestExecutor()
                           bei Microsoft.SharePoint.Client.ClientContext.<GetFormDigestInfoPrivateAsync>d__37.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                           bei Microsoft.SharePoint.Client.ClientContext.<EnsureFormDigestAsync>d__36.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                           bei Microsoft.SharePoint.Client.ClientContext.<ExecuteQueryAsync>d__28.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei Microsoft.SharePoint.Client.ClientContextExtensions.<ExecuteQueryImplementation>d__6.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                           bei Microsoft.SharePoint.Client.ClientContextExtensions.ExecuteQueryRetry(ClientRuntimeContext clientContext, Int32 retryCount,     
                        String userAgent)
                           bei PnP.PowerShell.Commands.PnPWebCmdlet.GetWeb()
                           bei PnP.PowerShell.Commands.Lists.GetListItem.ExecuteCmdlet()
                           bei PnP.PowerShell.Commands.PnPSharePointCmdlet.ProcessRecord()
TargetObject          :
CategoryInfo          : WriteError: (:) [Get-PnPListItem], TargetInvocationException
FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.Lists.GetListItem
ErrorDetails          : Ein Aufrufziel hat einen Ausnahmefehler verursacht.
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : bei <ScriptBlock>, <Keine Datei>: Zeile 1
PipelineIterationInfo : {0, 1}

[Tom-CK]

Maybe a typo in a configuration ?
I did not change anything in config.

[andyleejordan]

Can you confirm that $PSVersion from the integrated console session itself? I'm wondering if maybe you're in PowerShell 7 there...(I mean, it should still work, but it would help narrow things down).

@rjmholt Look at the accepted answer in https://docs.microsoft.com/en-us/answers/questions/302548/jsonserializer-throws-exception-in-xamarinios.html, very curious.

[Tom-CK]

This is the result of $PSVersionTable in PowerShell Integrated Console of vscode:

Name Value

---

PSVersion 5.1.19041.1023
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.19041.1023
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

[Tom-CK]

Maybe same problem here:
pnp/powershell#727

[Tom-CK]

I have removed module "PNP.PowerShell" in Version 1.7.0 and installed this module in version 1.5.0.
The error went away.
Then I updated module "PNP.PowerShell" back to Version 1.7.0 - the error was there again.
This can be reproduced.

Conclusion:
The problem must have something to do with module PNP.PowerShell Version 1.7.0.

But strange anyway: The exception is only thrown when running CMDLets in VSCode-PowerShell.

I will post an issue in PNP.PowerShell.

[rjmholt]

Going to reopen the issue here since it's quite possible this is at least partially an issue in PSES

[andyleejordan]

Just for reference, the module sources: https://github.com/pnp/powershell

[andyleejordan]

@Tom-CK Can you check if this works with PowerShell 7? We have a theory right now...

[andyleejordan]

Hey @erwinvanhunen, this bug appears between 1.5.0 and 1.7.0, and we think it was the introduction of your own ALC that may have caused this unfortunate interaction. PSES's assembly load logic when running with Windows PowerShell (so full .NET) does not check for the assembly version, it only resolves by name, and we think that's the issue. Since PSES has its own resolve assembly event, the one in pnp/powershell@fd65170#diff-05a628297ec69a1e93f2c84d148c6fac638d6499eda9b04d32a914b628fbe660 isn't being fired. We fear that a lot might break if we change our assembly loader to start checking for name + version. 🧐

[rjmholt]

It looks like PnP also does some dependency wrangling and registers their own assembly resolve event, but neither of us checks the required assembly version.

I've had a go implementing that. Give the extension here a try and let me know what happens.

pses.zip

[TobyLeduc]

I loaded the zip, disabled the original powershell extension and it did not fix the issue for me. My issue might be different because so far I am only having issues using the "Teams" commands. The SharePoint commands I have used so far have not caused me any issue.

PS> code --list-extensions --show-versions | Select-String powershell
ms-vscode.powershell@2021.8.2
ms-vscode.powershell-preview@2021.8.2

PS> connect-pnponline -Url "https://m365x167780.sharepoint.com/sites/<site>" 
-interactive
PS> Get-PnPTeamsTeam
PS>

The return should have been a list of 20-30 sites, I get nothing in the integrated console.

[andyleejordan]

My issue might be different because so far I am only having issues using the "Teams" commands. The SharePoint commands I have used so far have not caused me any issue.

Well, that's good to know. 🤔

[rjmholt]

The return should have been a list of 20-30 sites, I get nothing in the integrated console.

Ok that's much stranger than getting an exception, and harder to debug unfortunately

[TobyLeduc]

The return should have been a list of 20-30 sites, I get nothing in the integrated console.

Ok that's much stranger than getting an exception, and harder to debug unfortunately

Here is the error when I try to target a specific site as this actually gives an error.

PS> Connect-PnPOnline -url "https://<tenant>.sharepoint.com/teams/TLTestTeam" -Interactive
PS> Get-PnPTeamsTeam -Identity "TLTestTeam"

Get-PnPTeamsTeam : Team not found
At C:\Users\tleduc\OneDrive - City of Kelowna\SharePoint Online\scripts\test2.ps1:2 char:1
+ Get-PnPTeamsTeam -Identity "TLTestTeam"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (:) [Get-PnPTeamsTeam], PSArgumentException
    + FullyQualifiedErrorId : PnP.PowerShell.Commands.Graph.GetTeamsTeam

PS> $error[0] | fl * -force
writeErrorStream      : True
PSMessageDetails      :
Exception             : System.Management.Automation.PSArgumentException: Team not found
TargetObject          :
CategoryInfo          : ObjectNotFound: (:) [Get-PnPTeamsTeam], PSArgumentException
FullyQualifiedErrorId : PnP.PowerShell.Commands.Graph.GetTeamsTeam
ErrorDetails          :
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, C:\Users\tleduc\OneDrive - City of Kelowna\SharePoint Online\scripts\test2.ps1: line 2
PipelineIterationInfo : {0, 1}

[Tom-CK]

Sorry for my late reply, I was on holidays for some days:

Please remind:

• the error is thrown only when working VSCode-PowerShell. It it not thrown when using regular PowerShell.
• after installing package PNP.PowerShell in Version 1.5.0 everything worked fine. Went back to 1.7.0 the error was thrown
  in VSCode PowerShell-Extension only.

[TobyLeduc]

Sorry for my late reply, I was on holidays for some days:

Please remind:

• the error is thrown only when working VSCode-PowerShell. It it not thrown when using regular PowerShell.
• after installing package PNP.PowerShell in Version 1.5.0 everything worked fine. Went back to 1.7.0 the error was thrown
  in VSCode PowerShell-Extension only.

For me the error only exists in the "Powershell Integrated Console" terminal, switching terminals to "Powershell" I can run the commands and they work.

Yes 1.5.0 is the last working version for this issue.

[andyleejordan]

I wonder if we can "fix" this by having PnP.PowerShell update their dependency on Microsoft.Bcl.AsyncInterfaces.../cc @SeeminglyScience

[dietrichmeyer]

Using Windows, PowerShell 7.3.2 and Visual Studio Code 1.74.3 and PowerShell for Visual Studio Code v2022.11.0 and also tried v2023.1.0. PnP.PowerShell 1.12.0.
Running code in Visual Studio Code, sometimes my script works (posted below), but most often I get this error:
Connect-PnPOnline: Could not load file or assembly 'Microsoft.Identity.Client, Version=4.36.1.0, Culture=neutral, PublicKeyToken=0a613f4dd989e8ae'. Could not
find or load a specific file. (0x80131621).
In above posts it sounds like this cannot be fixed. I'm new to PowerShell 7 and Visual Code. Is there a workaround? I'm starting to get used to Visual Studio Code, but should I go back to the Windows PowerShell ISE?
##code is below
Install-Module -Name Microsoft.PowerShell.SecretManagement, Microsoft.PowerShell.SecretStore
Get-Secret -Name 'PS PnP'
Get-SecretInfo -Name 'PS PnP' | Select-Object -ExpandProperty Metadata
$certificatePassword = Get-Secret -Name 'PS PnP'
$siteurl = "https://XXXX.sharepoint.com/sites/yyyy"
$listname= "User Setup"
Connect-PnPOnline -Url $siteurl
-Tenant XXXX.onmicrosoft.com -ClientId "xxxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb"
-CertificatePath "C:\Data\PS\creds\BBBB PnP.pfx" `
-CertificatePassword $certificatePassword

[andyleejordan]

I will try to spend the next community day (Monday) looking at this again, though I'm not hopeful. I think it's on PnP.PowerShell to either implement or fix their own Assembly Load Context.

[gautamdsheth]

@andschwa - thank you taking the time to investigate it. We have launched a new 2.x nightly which has dropped support PS5 and is based completely on .NET 6 , maybe it can help ?

The nightly builds are available via PSGallery and using -Allowprerelease parameter to install the module

[andyleejordan]

@gautamdsheth that sounds super promising!

[andyleejordan]

@gautamdsheth have you seen pnp/powershell#1814 (reply in thread)? I'm going to see if we load Microsoft.Identity.Client.dll but that comment sure seems to be making a lot of sense.

[andyleejordan]

Just dumping some info here, I can confirm that PnP.PowerShell (both 2.x and 1.x) depend on older versions of Microsoft.Identity.Client than requested by ExchangeOnlineManagement and MicrosoftTeams modules. And so far as I've been able to dig, none of PowerShell Editor Services, Secret Management, nor Secret Store depend on this module.

@gautamdsheth is there a way to get PnP.PowerShell to accept 4.0 and up? I think you might have some "must be this version" logic that's causing the 4.29 from Teams and the 4.44 from Exchange to be rejected, but I'm willing to bet that if it did allow it to be used, it might just work fine. I think that's something to do with bindingRedirects right?

[andyleejordan]

Considering people are having issues with PowerShell 7 and only in the Extension Terminal, where we have an ALC, I'm thinking Rob may have been onto something when he surmised that it's because we don't check assembly versions in PSES. But this is far outside my expertise, I'll chat with @SeeminglyScience.

[gautamdsheth]

@andschwa - Yes, we are going to bump the MSAL.NET to the latest version , mostly 4.49 as part of the update to v2. Hopefully, will fix things.

[PowerBugi]

@gautamdsheth have you seen pnp/powershell#1814 (reply in thread)? I'm going to see if we load Microsoft.Identity.Client.dll but that comment sure seems to be making a lot of sense.

Hello @andschwa & @gautamdsheth
I posted a similar message 4 days ago here, I think it went unnoticed because it is not an issue :

pnp/powershell#1814 (comment)

I hope version 2 will be able solve this problem... It's really annoying to have module cohabitation problems because of this DLL.

[JustinGrote]

Since you now have a .net 6 only version, you really should look into using assemblyLoadContext to abstract your dependencies from the PowerShell shared assembly space to avoid this problem.
https://github.com/rjmholt/ModuleDependencyIsolationExample

EDIT: Sorry this was directed at PnP development, I forgot what repo I was in, oops.

[andyleejordan]

@JustinGrote thing is, they are using an ALC. Somehow the combination of being loaded in the extension terminal where PSES is using an ALC breaks their ALC. Yes, it should "just be working" because of module dependency isolation...but for reasons I don't know where to begin to find out, it's not.

[andyleejordan]

Oh, ohhh, oh! @gautamdsheth I was finding PnP.PowerShell's ALC to link here again, which is nearly an exact copy of the example from Rob that @JustinGrote just shared...and with a fresh set of eyes and a cup of coffee in me, I think I spied the issue:

        protected override Assembly Load(AssemblyName assemblyName)
        {
            string assemblyFileName = $"{assemblyName.Name}.dll";

            // Make sure we allow other common PowerShell dependencies to be loaded by PowerShell
            // But specifically exclude Microsoft.ApplicationInsightssince we want to use a different version here
            if (!assemblyName.Name.Equals("Microsoft.ApplicationInsights", StringComparison.OrdinalIgnoreCase))
            {
                string psHomeAsmPath = Path.Join(s_psHome, assemblyFileName);
                if (File.Exists(psHomeAsmPath))
                {
                    // With this API, returning null means nothing is loaded
                    return null;
                }
            }

            // Now try to load the assembly from the dependency directory
            string dependencyAsmPath = Path.Join(_dependencyDirPath, assemblyFileName);
            if (File.Exists(dependencyAsmPath))
            {
                return LoadFromAssemblyPath(dependencyAsmPath);
            }

            return null;
        }

My understanding of this is that Microsoft.ApplicationInsights is the only dependency being loaded from the module's dependency directory path, everything else (such as Microsoft.Identity.Client, the problem) is being loaded from the shared path of PowerShell (or from a previous resolution).

[andyleejordan]

@lucahost and @gautamdsheth check out what we have here: https://github.com/PowerShell/PowerShellEditorServices/blob/main/src/PowerShellEditorServices.Hosting/Internal/PsesLoadContext.cs

[lucahost]

I think you are quite close to the issue @andschwa, the problem occurs, if shared dependencies exist in s_psHome which are requested. Then they are returned / loaded from PnP's ALC even if the requested Version does not match.
I'm not so sure, but a Version-Check as in PsesLoadContext could do the trick.
For dependencies not existing in s_psHome the code is partially working, they are loaded from the modules-directory but could have the wrong version as well.

[ghost]

This issue has been marked as external. It has been automatically closed for housekeeping purposes.

[andyleejordan]

@lucahost I think you are right and that it's the version check that's missing, presumably causing an the older version of Microsoft.Identity.Client shipped with PnP.PowerShell to be returned as resolved even when the requestor requires a newer version.

It's actually worse than this. The AssemblyLoadContext as implemented in PnP.PowerShell is only isolating Microsoft.ApplicationInsights, and PnP.PowerShell has a lot more dependencies (namely Microsoft.Identity.Client) that conflict and need to be isolated in the same manner.

pnp/powershell#488 (comment)

[all2001]

any suggestions or workarounds for this?
This issue seems to be going for a years and I stucked with my development with this
I can do nothing in VS Code which is the one IDE I can use for Powershell 7.x

[andyleejordan]

You can use a regular pwsh terminal in VS Code, just not the "Extension Terminal." Click the + button to spawn a new terminal in the pane.

[all2001]

You can use a regular pwsh terminal in VS Code, just not the "Extension Terminal." Click the + button to spawn a new terminal in the pane.

Thank you, this workaround works for me