SDK: Closing a Runspace does not clean up resources

[arknu]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

We have a .NET Windows Service that needs to interact with Exchange Online. For various reasons we can't use the Graphs APIs, so we are using the Exchange Powershell Module via the Powershell SDK (version 7.4.1). This works fine, but I have noticed a memory leak.

Now, the Exchange Online Poweshell module is not exactly known for good memory use, but that shouldn't really matter. Once I close the Runspace and dispose both the Runspace and the Powershell objects, I would expect Powershell to clean up all resources allocated, no matter if the code run cleaned up properly or not.

We have previously used an out-of-process runspace on .NET Framework to work around this issue, but with the Powershell SDK on .NET 8 the Powershell .exe file is not distributed in the NuGet package, making this more difficult. Thus, I'd like to be able to run in-process, but a memory leak makes this difficult.

I have followed the steps here (https://techcommunity.microsoft.com/t5/exchange-team-blog/reducing-memory-consumption-of-the-exchange-online-powershell-v3/ba-p/3970086) to reduce memory usage significantly, but this is merely a workaround. The core issue here is that Powershell does not correctly free resources when asked to do so.

Expected behavior

Disposing and closing the Powershell and Runspace objects should release all resources.

Actual behavior

Some allocated memory stays behind, leading to a memory leak.

Error details

No response

Environment data

Powershell SDK verison 7.4.1 running on .NET 8 using Exchange Powershell Module version 3.4.0

Visuals

No response

[MatejKafka]

Is this issue specific to the Exchange PowerShell module, or are you able to reproduce it without any external modules?

[arknu]

@MatejKafka I will have a go, but I suspect it is the Exchange Module causing it.

However, that shouldn't really matter. A PowerShell runspace should be able to clean up, no matter what a script or module does. Otherwise, what is the point of being able to host PowerShell in-process in the first place? Then hosting in a external process should be the default and the SDK NuGet package should redistribute the .exe file to allow for this.

[MatejKafka]

I agree it shouldn't matter, but it will likely help pinpoint the issue.

Also, is it possible that some of the cmdlets invoked in the runspace are returning objects, which you hold on to after disposing the runspace, and which internally reference something big from the Exchange module, preventing GC from collecting it? It might help to analyze the heap and see if the memory is really taken up by runspace internals, or by something from the Exchange module.

[arknu]

@MatejKafka Thank you for looking into this.

For some additional details, here is a dump file of the process having run for a couple of days opened in VS:

As you can see, there seems to be some kind of caching of parsed scripts (StatementBlockAst etc.). And this cache seems to stick around after closing the runspace and just keeps growing.

As an experiment, I have added some reflection code to manually clear this cache when disposing:

 Type type = typeof(ScriptBlock);
 FieldInfo info = type.GetField("s_cachedScripts", BindingFlags.NonPublic | BindingFlags.Static);
 var value = info.GetValue(null) as ConcurrentDictionary<Tuple<string, string>, ScriptBlock>;
 if(value != null)
 {
     value.Clear();
 }

I'll leave this running for a few days, but so far the results appear promising.

Maybe the issue is simply that this cache should be cleared when disposing the Runspace?

The issue may be exacerbated by the fact that the Exchange Module creates a temporary module each time it connects, but this would not be an issue if this cache was simply cleared when signaling that I'm done with PowerShell for now by closing the Runspace and disposing it.

[rhubarb-geek-nz]

SDK NuGet package should redistribute the .exe file to allow for this.

I see the PowerShell application (pwsh.exe) as being simply another program that uses the PowerShell SDK rather than being part of the SDK itself.

If it was part of the SDK nuget then any application using the SDK would then also get the exe whether it needed it or not.

If you want to run the pwsh.exe why not simply run pwsh.exe installed on the host system?

[rhubarb-geek-nz]

Garbage Collectors don't clean up statics, they assume they are there for a reason.

Perhaps the script cache should be shared between runtimes by being held as part of InitialSessionState rather than a static.

[arknu]

I found another static variable that should be cleared when closing the runspace:

@rhubarb-geek-nz I found your suggestion about having these caches in IntialSessionState very sensible. That way they could be cleared properly. The current implementation does not really seem geared to hosting PowerShell in a long-process that is sensitive to memory leaks.

I also found (and worked around via reflection) a memory leak in the Exchange Online PowerShell module, but that is for a separate issue.