Loading Modules into Isolated AssemblyLoadContext

[daxian-dbw]

Submit the RFC for the isolated module feature.

[vexx32]

Suggested change

	1. Required modules are loaded into the default ALC as is today.
	The comment to the `RequiredModules` key in module manifest is:
	> Modules that must be imported into the global environment prior to importing this module.
	As indicated by the comment, required moduels are shared by all modules,
	so it makes most sense to always load a required module in the default ALC.
	1. Required modules are loaded into the default ALC as is today.
	The comment to the `RequiredModules` key in module manifest is:
	> Modules that must be imported into the global environment prior to importing this module.
	As indicated by the comment, required modules are shared by all modules,
	so it makes most sense to always load a required module in the default ALC.

Also, part of the problem statement already is:

Basically, when two modules depend on different versions of the same assembly,
if the module that depends on the lower version is loaded first,
the other module cannot be loaded in the same session anymore.

This is a direct contradiction. If required modules are always loaded in the main session, then no two required modules which happen to depend on different versions of the same assembly can possibly be loaded in the same PS session even after this change.

For example:

• Module A
  • Requires Module C
• Module B
  • Requires Module D
• Module C
  • Loads version 1.0 of assembly X
• Module D
  • Loads version 2.0 of assembly X

With this new concept as described (with RequiredModules being always in the default context), you still cannot load module B (depending on D->v2.0 assembly) after loading module A (depending on C->v1.0 assembly), even if you isolate modules A and B themselves.

[daxian-dbw]

This is a good point.

The scenario I thought about was:

• Module A
  • Requires Module C
• Module B
  • Requires Module C

During Import-Module A -Isolated, let's say Module C gets loaded into the same separate ALC.

Later, when Import-Module B, the Module C is already available globally, so Module B will just use the already loaded Module C. However, Module B is loaded in the default ALC while Module C was loaded into a custom ALC, there could be type identity issues when B attempts to call any commands in Module C.

Similarly, Module C was loaded to the global scope, so the user is able to call commands from it directly, then again, type identity issue could happen.

---

I think we may need to change this behavior about the required module:

"required module must be imported to the global environment"

We could import a required module to the local scope of the requesting module (e.g. the context of Module A and Module B) when a user is explicitly loading the requesting module with a separate ALC. Then the required module can be loaded into the same ALC as the requesting module.

However, a required module won't be shared by multiple requesting modules, and the drawbacks of that are:

1. A required module may be imported many times into contexts of different modules (memory load concern)
2. State changes in one required module won't be visible to other requesting modules (breaking change)

This would be a problem to modules like Az modules. Many Az modules share the same required module Az.Account and commands from different Az modules are supposed to work together in a pipeline (meaning that loading different Az modules into different ALC would be problematic). Good news is that the Az PowerShell team is working on addressing the assembly conflicting issue with in the whole set of Az modules, so that Az modules could work properly even without this module isolation feature in PowerShell.

[daxian-dbw]

I put a lot thoughts in how required modules should be handled for isolated modules. There is no perfect solution, but I believe having the required modules to be loaded in the current active ALC is the most suited overall after comparing the pros and cons of 3 candidate solutions.

The details of the analysis are captured in the Handle Required Modules section in the RFC. Please take another look, thanks!

[vexx32]

Are we adding a parameter to Add-Type or updating RequiredAssemblies in a module manifest to select the ALC loading behaviour? Or are we forcing module authors to reach directly into .NET to define the load context behaviour?

I think it's worth updating module manifests to allow module authors to define the default load context behaviour for themselves.

[daxian-dbw]

The list of ways to explicitly load assemblies into a different assembly load context is just to call out that a module author can choose to not respect the module isolation enforced by PowerShell.

But I think in general, a module author doesn't need to care about assembly load context and should just let PowerShell handle that for it. (there will be exceptions for sure, such as the Az modules, they are planning a change to handle assembly conflicts in a finer granularity for all Az modules)

[daxian-dbw]

I think it's worth updating module manifests to allow module authors to define the default load context behavior for themselves.

Any reason why you think a module author should define the assembly loading behavior?
In my opinion, a module author doesn't know the environment where the module will be used in, and thus won't be able to know what would be the right assembly loading behavior in that environment. It should be the user to decide whether the module should be loaded in a separate load context.

[vexx32]

If a module author creates a module that they know will have issues (e.g., it ships a library that's already included in PowerShell, and requires a higher version than the one PowerShell comes with), then placing the burden on end-users of modules to know that they'll always need to explicitly import the module doesn't make much sense to me.

Also, consider a scenario where module A depends on modules B and C, and the author of module A knows there is an incompatibility between B and C; how would they be able to use this? I think supporting it in module manifests makes a lot more sense than just allowing end users to do it.

ALCs are a pretty advanced concept in general. No matter how friendly we make it, I think it's highly likely that module authors will be more familiar with the concept than most end-users, and we should allow for them to make sensible decisions when authoring that can be declared in the module manifest; otherwise, dependencies would end up being 'hidden' in the psm1 since they can only handle it from script, and can't rely on autoloading anymore.

Without some measure of support for customizing how autoloading treats certain modules, I think we're far more likely to cause issues for users.

[daxian-dbw]

If a module author creates a module that they know will have issues (e.g., it ships a library that's already included in PowerShell, and requires a higher version than the one PowerShell comes with)

This is a scenario where allowing the module author to decide the loading behavior makes some sense. However, the module may not have a version conflicting issue with a newer version of PowerShell, but it will still be imported to a separate ALC because it's configured so in the manifest.

This may not seem to be a big deal, but I would argue that many more module authors would choose to always have their module to be loaded into a separate ALC "just to be safe". Is that what we really want? IMHO, we should not encourage loading modules into separate ALC unless it's really needed, so as to avoid type identity issues as much as possible. (type identity issue would cause interoperation issues among commands from different modules)

Also, consider a scenario where module A depends on modules B and C, and the author of module A knows there is an incompatibility between B and C; how would they be able to use this?

Should we really enable user to do this? It seems wrong from the beginning to depend on modules that are not compatible. I think the goal is not to enable arbitrary use of incompatible assemblies/modules in PowerShell. Instead, we should target the common scenarios where a user would likely run into with the "conflicting assembly" issue.

---

But, on the other hand, If we do allow configurations in manifest, what configurations should be provided?

I think we should only allow a module author to decide if the module should be loaded in a isolated ALC and nothing more. I don't think finer granularity controls would worth the complexity.

[vexx32]

Dependency management in PowerShell is already pretty complicated for anything that's dependent on external libraries, and I can see this being useful for module authors if we allow the module author to specify a default behaviour for their module.

I would think that (ideally) we'd add a new key to the module manifest, e.g. LoadIsolated, which specifies whether it simply loads in the global context (defaults to $false) as usual, or whether it loads in a new context (set to $true to enable).

Perhaps a similarly named key could be allowed for each entry of RequiredModules (etc). I think it makes sense for most NestedModules to share the module context, but I'm unsure if it makes sense to create "special" behaviour for some kinds of dependencies that aren't permissible for others; it might be more sensible to just allow it for any referenced modules and/or assemblies.

The issue I have with not providing for it in a module manifest is that a module author may be forced to make a binary module load in a psm1 which it wouldn't otherwise really need, save for requiring the isolation context. It would not be a great UX for users if some modules have to document "auto-loading is broken because of X, Y, Z -- you need to manually use Import-Module with the correct flags to get this module to load" when we could provide the framework for them to give a better UX to their end users.

[iSazonov]

It's a lot of work and a lot of problems. Is it worth it? Are there any real critical scenarios that can to be resolved only by this?

[andyleejordan]

It's a lot of work and a lot of problems. Is it worth it? Are there any real critical scenarios that can to be resolved only by this?

Users of the PowerShell extension would certainly benefit, as there have been many issues due to incompatible dependency versions colliding among modules. It's to the point that the extension follows what's outlined here and attempts to load as many of its dependencies in an isolated Assembly Load Context as possible (specifically our .NET dependencies for the LSP server, like OmniSharp), but we still can't do that for the modules themselves so users are stuck with potential collisions with any of the necessary modules the extension loads.

[rjmholt]

A couple of things that I think are worth elaborating on here:

• How do I use Add-Type in a global way? Will I be able to target an ALC like with Import-Module?
• How does a module author declare that their module should be loaded into its own ALC, or into the global one, or that it's designed to interoperate with another module in the same ALC? Who wins when a module author's configuration conflicts with how the user imports a module?
• What happens when modules are autoloaded?
• Do we need new cmdlets or other mechanisms to handle/manage the new PS ALC concept? We conflate PS ALCs with modules in some places (like with the proposed typename syntax) and I feel like we should design carefully and minimally there if we can.

[rjmholt]

Is there a possibility of us updating the error message here? We could try to catch the conversion error and then look to see if both types exist in known PS ALCs

[daxian-dbw]

I think it's possible. Maybe just need to update the conversion code in LanguagePrimitive.

[iSazonov]

It's very much like self-deception. The only scenario where ALC would work transparently would be if a module uses an assembly exclusively internally and does not expose types from that assembly externally, or repackage them into their own types.
In other cases, conflicts are inevitable and ALC does not guarantee normal operation of the modules. Moreover, this causes unpredictable instability when the script suddenly stops working because someone installed a slightly newer build version for some module. (I.e. now the module just won't load forcing you to look for a workaround, which is more reliable, but afterwards it will load but not guaranteed to work.)

My main objection is that cmdletFrom ModuleA | cmdletFrom ModuleB is not guaranteed to work in general because of type conflicts from different assemblies.
I think this is totally unacceptable.

I would rather invest in what we used to Windows Compatibility feature - PowerShell remoting serialization. This can be seen as a universal repackaging that gets rid of type conflicts.
I suppose we could do it even in-process.

Another point of view. If have two set of modules:

1. A, B, E1
2. C, D, E2
   where modules E1 and E2 have type conflicts. Thus each of the sets could be loaded in separate isolation area. This is very similar to loading modules into different runspaces. This seems much more understandable, predictable (or rather, eliminating unpredictable behavior) and reliable.

[PaulHigin]

Can add a section that summarizes how the ALC works. This would make it easier to understand the in-depth discussions.

[PaulHigin]

Why ReflectionContext and not ModuleLoadContext? The name seems confusing.

[daxian-dbw]

The name is from AssemblyLoadContext.CurrentContextualReflectionContext. It acts in a similar role -- allowing a dev to specify the active assembly load context to use.
But we can always change the name of this property.

[daxian-dbw]

In response to @rjmholt's questions:

How do I use Add-Type in a global way? Will I be able to target an ALC like with Import-Module?

The RFC proposes to add a new parameter -AssemblyLoadContext to Import-Module, so a user can choose to load a module using an existing module's ALC, by Import-Module A -AssemblyLoadContext B.ReflectionContext. (Names of the parameter and property may need to be revisited).

Can you elaborate a bit more on the Add-Type question? The proposal is to make Add-Type aware of the active ALC, so that it loads assembly into the right ALC instead of always loading into the default ALC as of today. I'm not sure what you mean by "in a global way".

How does a module author declare that their module should be loaded into its own ALC, or into the global one, or that it's designed to interoperate with another module in the same ALC? Who wins when a module author's configuration conflicts with how the user imports a module?

The discussion about this is in this thread: #298 (comment).
My personal view is that a module author shouldn't declare where their module should be loaded, and thus there shouldn't be a configuration about this in the manifest. Please share your thoughts in that thread.

What happens when modules are autoloaded?

The behavior proposed in this RFC would be: same as today -- by default module is not loaded in isolated context, so if there is a conflict in assembly loading, it will fail, just like today.

Loading a module in a separate ALC has some implications, including that it may result in type identity issue which would be very confusing to the user, so I don't think we should do this by default, but should always do it when user explicitly asking for it.

We could detect the error when loading a module normally, and then revert back to loading it again in a separate ALC. But again, this magic is not an explicit ask. I guess we could write out a warning message in this case, just like the -WindowsCompat switch does in Import-Module.

Do we need new cmdlets or other mechanisms to handle/manage the new PS ALC concept? We conflate PS ALCs with modules in some places (like with the proposed typename syntax) and I feel like we should design carefully and minimally there if we can.

In this proposal, ALC is supposed to be associated with module. I don't think the goal is to allow arbitrary use of ALC.

[iSazonov]

@daxian-dbw It seems we forget about exception processing coming from different ALC-s.

[PaulHigin]

I feel the potential benefit of this proposal is worth continued investigation. Admittedly it won't solve all version conflict problems, but I think we are at the point where prototyping is needed. I agree with initially keeping the feature as simple as possible. It will be interesting to see how well it solves versioning problems and what, if any, breaking/blocking problems arise.

[JamesWTruher]

great write up, just a few questions

[JamesWTruher]

My assumption is that if a conflicting assembly is attempted to be loaded into a new ALC, we will still see the same error that we see now. Is that right?

[daxian-dbw]

@JamesWTruher The error we see today is that the conflicting assembly cannot be loaded, so we won't see this error when introducing new ALC. But I'm not sure if I understand your question correctly ... can you please elaborate a bit more on the question?

[JamesWTruher]

I am wondering about the debugger experience in this new world. Do you have any thoughts on this?

[daxian-dbw]

I don't think there will be any changes in the debugging experience. Behavior-wise, only the type resolution and possible type identity issues are user visible, but they shouldn't affect the debugging experience.

[JamesWTruher]

Are there potential problems with serialization in this environment? Currently we don't capture version information about the assemblies which are providing the types that are serialized. If a user takes an object out of an isolated ALC and then serializes it via export-CLIXL how will they be able to deserialize it? (especially if the type has changed dramatically)

[daxian-dbw]

I don't think there will be any potential problems with serialization/deserialization. The deserialization will produce a PSObject, not really coupling with a specific version of a type.

For example, say we have a FileInfo object out of a custom ALC, and serialize it with export-clixml. The object produced by import-clixml from the serialization xml file is a PSObject, with the type name Deserialized.System.IO.FileInfo.

[PaulHigin]

We do re-hydrate some core types by constructing that type (Microsoft.PowerShell.DeserializingTypeConverter), and we provide support for custom type re-hydration. This construction may possibly create the 'wrong' type, but may just be something that a custom type converter needs to be aware of. But I don't think this is something to prevent us from pursuing ALCs.

[essentialexch]

Speaking as an admin scripter – which I would guess 90%+ of all PS users are – I have no idea what this is about, how it affects me and my use of PowerShell, and how I would use the new feature. Other than it has something to do with version conflicts. Which I’ve only run into with PS internal modules. Until you can explain it to my level or below, it’ll see miniscule uptake.

[SteveL-MSFT]

I have a couple of concerns:

• Having the user make the choice is never going to work. I think the onus should be on the module author to decide if their module requires isolation
• We also need to think about script modules which can depend on conflicting assemblies
• I agree with @iSazonov that an alternate approach may be investing in improving implicit remoting to have process isolation for modules which has the additional benefit of a better remoting experience (ideally, users should not have to be aware if a cmdlet or object is local or remote)
• For immediate issues with modules using specific versions of assemblies that conflict, it may be good enough to provide a nuget pkg that helps them easily create a ALC for their module (not tied to a specific version of PS as an API)

[daxian-dbw]

I agree with @iSazonov that an alternate approach may be investing in improving implicit remoting to have process isolation for modules which has the additional benefit of a better remoting experience (ideally, users should not have to be aware if a cmdlet or object is local or remote)

IMHO, it's impossible it would be very hard, if possible, to enable method invocation on deserialized objects returned from remoting on top of the PowerShell remoting infrastructure. Here are a few challenges that I can think of:

1. Allowing invoking methods on a deserialized object would require the deserialized object somehow has a reference to the corresponding live objects in the remote end, so that the method invocation could be forward to that live object on the remote end.
2. Method signature marshalling
   • if a parameter type doesn't exists on the client side, how to represent it in the method signature created for the deserialized object on the client side?
   • if a parameter type does exist on the client side, shall it be used as is? If so, when invoking the method on the deserialized object with a live object of such a type, that live object would be deserialized when sending the call request to the remote end, and how would that be rehydrated to a live object? It's possible that objects of some types may not be rehydrated at all.
3. Method resolution -- given the challenge of the signature definition, resolving which method is being called would likely be challenging too.

The implicit remoting approach mostly won't work in module interoperation, unless the object passed between commands from two different modules are values of primitive types. For Az modules, the user context cannot be shared by different Az module, because Az.X and Az.Y would be loaded into two pwsh processes. It also requires spinning up new pwsh process for every module, which would be resource-heavy.

ideally, users should not have to be aware if a cmdlet or object is local or remote

In the ideal case where users don't care the returned object is local or remote, the existing -UseWindowsPowerShell solution already works, and it works sufficiently well for those scenarios.

[essentialexch]

Documentation and discoverability for “-usewindowspowershell” is poor. But this all stands to reinforce my previous comment about this is far too complex.

[daxian-dbw]

Having the user make the choice is never going to work. I think the onus should be on the module author to decide if their module requires isolation

Here are the discussions about updating module manifest to allow module author decide if the module should be loaded into a separate ALC: #298 (comment). @vexx32's argument definitely has a good point. My biggest concern is that the use of isolated module might be abused, which would cause more type identity issues.

[PaulHigin]

@daxian-dbw

I propose that we consider expanding 'implicit remoting execution' to work with local runspaces as well as remote runspaces. In addition, if we allow local runspaces to optionally run in their own ALC then we can leverage 'implicit local execution' to run modules isolated in separate runspaces/ALC, as a way to facilitate running different modules with conflicting binary version dependencies. This way returned objects are 'live' and we don't need to deal with serialization issues. We would still need to consult non-default ALCs when doing type look up (as proposed in this RFC), however.

This would be an inefficient way to load/run modules, but not nearly as inefficient as using implicit remote sessions.

[daxian-dbw]

We also need to think about script modules which can depend on conflicting assemblies

The script module is already taken into consideration. Add-Type will be updated to honor the active ALC. Also, with the updates to the type resolution logic, type reference within the module will resolve the type targeting assemblies loaded in the module's ALC before targeting assemblies loaded in the default ALC.

[daxian-dbw]

For immediate issues with modules using specific versions of assemblies that conflict, it may be good enough to provide a nuget pkg that helps them easily create a ALC for their module (not tied to a specific version of PS as an API)

If we make no change to PowerShell itself, then there is not much that we can help a module to use a separate ALC because it will be very specific to individual modules. This is because assemblies that contain cmdlets needs to be loaded by PowerShell, namely loaded into the default ALC as of today. You will want the potentially conflicting assemblies to be loaded in a separate ALC, and then the problem always boils down to how the conflicting assemblies loaded in a separate ALC can be discoverable/resolvable by the cmdlets assemblies loaded in the default ALC.

There are 2 scenarios that I can see:

1. The conflicting assembly is used purely internally for the module. No types from the conflicting assembly is used for the cmdlet interface (parameter type, returned type, expecting user to create objects of types from it, and etc.).
2. The conflicting assembly is used for the cmdlet interface.

For the scenario (1), it's absolutely fine for a module to use a separate ALC, and there are 2 ways to do that:

1. Use the bridge model described in @rjmholt's blog resolving powershell module assembly dependency conflicts. That's the approach taken by the PSES module. Basically, it requires the cmdlets and all user interface related types to be defined in Cmdet.dll which will be loaded in to the default ALC, and all real business logic to be defined in another assembly, say Engine.dll. Cmdet.dll doesn't have reference to any of the potentially conflicting assembly, but it reference Engine.dll for the real work when cmdlets are running. Cmdlet.dll implements OnImport interface, and will register a handler to AssemblyLoadContext.Default.Resolving to load Engine.dll with a separate ALC. Since Engine.dll is loaded in the separate ALC, its assembly loading requests for those conflicting assemblies would go by the separate ALC first, and thus can be correctly handled.

   pros: it can guarantee the exact required version of a conflicting assembly to be loaded for the module, and it's very scoped -- doesn't affect any assembly loading requests from outside the module.
   cons: obviously, it would involve non-trivial refactoring work for an existing module, and not all modules can be cleanly separated into Cmdlet.dll and Engine.dll.

2. Simply register a handler to AssemblyLoadContext.Default.Resolving event in OnImport, which monitors the loading requests of the potentially conflicting assemblies. If the requested version is what the module depends on, then load the assembly shipped by the module with a separate ALC, and return the loaded assembly.

   pros: it's super simple.
   cons: the added assembly resolution will affect assembly loading requests from outside the module. Also, it could cause type identity issue even though the conflicting assembly is not used for user interface.

For both ways, the event handler and the assembly load context code would be very simple (about 50-ish lines of code), so a NuGet package is not necessary.

For the scenario (2), there will surely be type identity issues when a separate ALC is used by a module for its potentially conflicting assemblies. There is no way for module authors to solve or mitigate the type identity issue in this case, and they have to depend on the isolated module solution provided by PowerShell.

[iSazonov]

The main question for me, to which I have not received an answer, is whether we really have such an important problem for which it is worth so much effort?
I feel that in practice users always find workarounds that are acceptable to them.
So the question is it even worth wasting time on this problem? Can we get an answer to this question in the first place?

In addition to this way (leave it as is), there are three other approaches, each with its own advantages and disadvantages.

1. ALC. The main advantage is the transparency of the modules. The main disadvantage is unpredictable side effects, as well as excessive complexity in implementation, which is more likely to bring more degradation in performance and new bugs than benefits.
2. PowerShell GAC. We could learn PowerShell to register dll-s and load highest (better suited) dll version. It works more transparently and is easier to implement than ALC. But this may make a module inoperable (due to breaking changes in the dll).
3. Implicit remoting. The main drawback is serialization (lost live objects and slow). Nevertheless, we already have WinCompat feature. This approach can also be used to implement sudo. So the development of this area promises many benefits. (Notice, since we are talking about splitting an internal work between child processes this work remains internal and we can use fast binary communication between processes without losing security.)

If we really want to load modules with conflicting assemblies, I would prefer PowerShell GAC approach since the main dev principle is backwards compatibility and it is hoped that this approach will cover most scenarios.
But I would also look closely at implicit remoting which offers even more possibilities not only for this problem but others.

[PaulHigin]

I believe that modules with incompatible binaries is a pretty serious problem for many complex modules. @daxian-dbw can you include some examples in this RFC? I feel implicit remoting is not a good solution, due to serialization and perf issues. And I still feel loading/running modules optionally in isolated runspaces could be a really good compromise.

[daxian-dbw]

I propose that we consider expanding 'implicit remoting execution' to work with local runspaces as well as remote runspaces. In addition, if we allow local runspaces to optionally run in their own ALC then we can leverage 'implicit local execution' to run modules isolated in separate runspaces/ALC, as a way to facilitate running different modules with conflicting binary version dependencies. This way returned objects are 'live' and we don't need to deal with serialization issues. We would still need to consult non-default ALCs when doing type look up (as proposed in this RFC), however.

This would be an inefficient way to load/run modules, but not nearly as inefficient as using implicit remote sessions.

@PaulHigin This is an interesting idea. I did looked into the Rusnpace-level isolation in my previous RFC about this topic. The Runspace-level isolation is doable, and easier than module-level isolation.

Two big challenges that I see are:

1. In order to get live objects back, we need to keep away from the remoting stack. So we cannot use implicit remoting but need something else.
2. An isolated Runspace needs to load all powershell assemblies into a separate ALC (of course, assemblies from modules loaded in that Runspace will be loaded in the same separate ALC). So, any live objects returned from this isolated Runspace will be associating with the assemblies loaded in that specific ALC, and will be incompatible with many types (e.g. all powershell types) in the default Runspace (assuming the default Runspace still uses the default ALC). Due to the type incompatibility, you cannot do anything with it in the default Runspace.

[daxian-dbw]

I believe that modules with incompatible binaries is a pretty serious problem for many complex modules. @daxian-dbw can you include some examples in this RFC?

@PaulHigin @iSazonov The PSES module and Az modules are 2 examples that struggled/is struggling with the conflicting assembly problem. Rob's blog has comments from users about other examples:

You have to implement first class support for assembly management. This issues are being present for years.
Last week we updated the vscode ps extension and suddenly PSScriptAnalyzer stopped working because of that. This is not normal.

If we are running PowerShell 7 (on top of .NET Core 3.1), we can load and run FictionalTools and then load and run FilesystemManager without issue. However, if in a new session we load and run FilesystemManager and then FictionalTools, we will encounter a FileLoadException from the FictionalTools command, because it requires a newer version of Microsoft.Extensions.Logging than the one loaded, but cannot load it because an assembly of the same name has already been loaded.

PowerShell LTS's life cycle is 3 years, it's inevitable for some assemblies shipped in PowerShell to become old versions and potentially conflicting with modules that are updated more frequently. Similarly, modules that are released less frequently could likely cause conflicts with modules that are release more frequently because the former will depend on old version of some libraries.

So, the problem is really not uncommon.

I feel implicit remoting is not a good solution, due to serialization and perf issues. And I still feel loading/running modules optionally in isolated runspaces could be a really good compromise.

I agree with you on the implicit remoting solution.
For the Runspace-level isolation, the returned live object will be incompatible due to type identity issue. I see it more fit in a server-application model, where a single powershell server process can serve multiple client requests with isolation.

PowerShell GAC. We could learn PowerShell to register dll-s and load highest (better suited) dll version. It works more transparently and is easier to implement than ALC. But this may make a module inoperable (due to breaking changes in the dll).

@iSazonov As I already told you in our previous discussion #298 (comment), the PowerShellGet and PowerShell Gallery vNext is going to explore in this space. I'm sure it won't solve all problems, because

1. it cannot guarantee it's always able to load the highest version available of an assembly;
2. major version change could break a module -- users will ask for a way to pin a version of assembly sooner or later.
3. it cannot help when PowerShell itself introduce the assembly dependency of a specific version. (this will be bound to cause conflicts as a version of PowerShell ages)

As for complexity in implementation, you will never know until you really deep dive into the design 😄

[PaulHigin]

@daxian-dbw Implicit remoting is really just command proxies running in a remote runspace, and it would not be too difficult to extend that to include local runspaces.

It seems that only some type instances will be incompatible across different runspace execution, and I am wondering if we can handle those (probably) rare cases. But not sure if it would be worth the effort/complexity.

[daxian-dbw]

Implicit remoting is really just command proxies running in a remote runspace, and it would not be too difficult to extend that to include local runspaces.

@PaulHigin That's good to know. Implicit remoting depends on Invoke-Command to do the actual invocation. I guess Invoke-Command would need to be enhanced to be able to invoke a command targeting a Runspace.

It seems that only some type instances will be incompatible across different runspace execution, and I am wondering if we can handle those (probably) rare cases. But not sure if it would be worth the effort/complexity.

It will be a lot types -- all PowerShell types will be incompatible, because the isolated Runspace needs to load all PowerShell assemblies into a separate ALC. All Runspaces will share the .NET runtime assemblies though.

[PaulHigin]

I guess I don't understand type instance incompatibility. Are type instances incompatible when they are associated with assemblies loaded in different ALCs, even if they are the same assemblies?

[daxian-dbw]

@PaulHigin Yes, even if they are originated from the same assembly file, the types are incompatible because the assemblies loaded into different ALC's are different. Below is an example to demonstrate the problem. Assembly.LoadFile always loads an assembly into a separate ALC, see the source code here.

PS:1> Add-Type -TypeDefinition @'
>> using System;
>> namespace Test
>> {
>>     public class Bar {}
>> }
>> '@ -OutputAssembly C:\arena\tmp\test.dll
PS:2>
PS:2> $asm_test = [System.Runtime.Loader.AssemblyLoadContext]::Default.LoadFromAssemblyPath("C:\arena\tmp\test.dll")
PS:3> $asm_test2 = [System.Reflection.Assembly]::LoadFile("C:\arena\tmp\test.dll")
PS:4> $t1 = $asm_test.GetType("Test.Bar")
PS:5> $t2 = $asm_test2.GetType("Test.Bar")
PS:6>

## Types from those two assemblies are different
PS:6> $t1 -eq $t2
False
PS:7> $t1.FullName
Test.Bar
PS:8> $t2.FullName
Test.Bar
PS:9> $t1.AssemblyQualifiedName
Test.Bar, il4anyda.gjf, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
PS:10> $t2.AssemblyQualifiedName
Test.Bar, il4anyda.gjf, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

## create an object with the second type (loaded with Assembly.LoadFile)
PS:11> $obj2 = $t2::new()

## casting the object to the type [Test.Bar] fails
## [Test.Bar] is resolved using the assembly loaded in default ALC
PS:12> [Test.Bar]$obj2
InvalidArgument: Cannot convert the "Test.Bar" value of type "Test.Bar" to type "Test.Bar".

PS:13>

[isra-fel]

Great work :)

ModuleInfo object will have a new public read-only property named ReflectionContext

I wonder as a module author, how do I access ModuleInfo.ReflectionContext (or equivalent) in the module's code, so that I can build my own assembly loading logic on top of PowerShell, like a second layer of ALC to load assemblies shared by multiple modules to avoid loading them duplicately in each ALC?

[vexx32]

The ModuleInfo for the current module can be accessed from $ExecutionContext.SessionState.Module within the module, so you'd be looking at $ExecutionContext.SessionState.Module.ReflectionContext -- you could do similar with (Get-Module $moduleName).ReflectionContext if you were accessing it from a different module.

[SteveL-MSFT]

Closing this as we've decided on a different direction

[daxian-dbw]

A summary

The type-identity issue and the existing behavior of RequiredModules are the 2 main blockers to the general solution proposed in this RFC. They will make this solution fragile and confusing to end users, and also require significant amount of work in PowerShell engine to just make those 2 issues manageable.

So, we decide to no longer pursue a general solution, but instead, try to make it easy for a binary module to leverage a custom AssemblyLoadContext within itself. The PR PowerShell/PowerShell#16889 is one step toward that. We also updated the section "Assembly resolving handler for side-by-side loading" in this article to point to the repo PowerShell-ALC-Samples that contains the sample code and detailed analysis for the alternative approaches.