fix: Consumption of eager shared modules

[cmath10]

What kind of change does this PR introduce?

Fixes eager consumption of shared dependencies. Full description is in this discussion: #18001 (reply in thread)

Did you add tests for your changes?

No, the reason for no tests attached can be found in the discussion link above. cc @alexander-akait

Does this PR introduce a breaking change?
No, as far as I understand how webpack container works.

What needs to be documented once your changes are merged?
This is yet to be defined, this PR is needed as a starting point.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: cmath10 / name: Kirill Zaytsev (ffaad71)

[webpack-bot]

For maintainers only:

• This needs to be documented (issue in webpack/webpack.js.org will be filed when merged)
• This needs to be backported to webpack 4 (issue will be created when merged)

[alexander-akait]

/cc @ScriptedAlchemy what do you think about these changes

[cmath10]

Any updates?

[alexander-akait]

Don't worry, I will ping you when we need some advices/dicussions, I am fine with your changes

[ScriptedAlchemy]

@2hea1 Do we need to port this back to module-federation/enhanced or did our rewrite already address all these problems?

[2heal1]

@2hea1 Do we need to port this back to module-federation/enhanced or did our rewrite already address all these problems?

@ScriptedAlchemy @cmath10 I use @module-federaion/enhanced , and it has solved the issue .

related pr: cmath10/module-federation-incorrect-version-resolution#1

[cmath10]

@2heal1 Is this part required:
publicPath: 'http://localhost:8080/public/build-legacy/'
buildNext: 'buildNext@http://localhost:8080/build-next/mf-manifest.json',
?

In my app when I build it for production, all files and pages are served by the same NGINX server and I have URLs like https://myapp.domain/mypage and https://myapp.domain/build/runtime.js etc. And URLs with protocol I use only in serve mode in development, and only if my team requires.

[alexander-akait]

@ScriptedAlchemy Are you fine with these changes?

[2heal1]

@2heal1 Is this part required: publicPath: 'http://localhost:8080/public/build-legacy/' buildNext: 'buildNext@http://localhost:8080/build-next/mf-manifest.json', ?

In my app when I build it for production, all files and pages are served by the same NGINX server and I have URLs like https://myapp.domain/mypage and https://myapp.domain/build/runtime.js etc. And URLs with protocol I use only in serve mode in development, and only if my team requires.

@cmath10 ahh i got you , the port can be removed cmath10/module-federation-incorrect-version-resolution@03377bc

[cmath10]

@2heal1 Thanks for the reply! I'll check the solution as soon as possible.

[cmath10]

@2heal1 That works, but now I have a question. As I understand, mf-manifest is loaded via a network to get info about a federated module. I noticed there is no hash in the manifest's name and wonder how that will work if a browser caches it. Or do I understand it incorrectly?

[alexander-akait]

@cmath10 I'm a little out of context, we still need this fix, am I right?

[cmath10]

@alexander-akait yes, if there is no solution for caching.

[2heal1]

@2heal1 That works, but now I have a question. As I understand, mf-manifest is loaded via a network to get info about a federated module. I noticed there is no hash in the manifest's name and wonder how that will work if a browser caches it. Or do I understand it incorrectly?

Yes, this is a question . But since you can control server to dispatch data , so you can use our snapshot to avoid the cache issue.

First you can generate snapshot by invoking generateSnapshotFromManifest api which is provided by @module-federation/sdk , and then you can dispatch the snapshot to html.

Second, set mf runtime plugin to consume the snapshot .

And when the remote need to be loaded , it will just use the remoteEntry , and the real entry has hash ,so it will not have the cache issue .

That's also what we have done in bytedance . I hope it can help you as well.

[2heal1]

@cmath10 I have pushed the commit , you can view this for details .

The start steps:

1. yarn install
2. yarn build
3. yarn serve
4. visit http://localhost:8080/ to see the moduleInfo

[alexander-akait]

@cmath10 Feel free to ping when you try the solutions and decide there is no solution

[cmath10]

I need to port it to PHP first 😅

[cmath10]

@2heal1 this is the only solution?

[2heal1]

@2heal1 this is the only solution?

Yes.. But i feel like this is not very different from the original demo.

[ScriptedAlchemy]

@alexander-akait i think this discussion got a little off topic on the pull request, overall i think this is a problem that should be resolved in webpack as this is a problematic issue. If a module is eager, it should use the best eager version available. typically this would have to be the host as app could already be using it and host may not consume remote before entry is already consuming. This seems like a valid issue and the PR seems to solve some conditions

@cmath10 From what i understand. You have two aspects

1. the original issue which is around multiple versions where some are eager and others are not. And somebody is resolving the promise version rather than the one thats eagerly available? If a remote / host is dependent on an eager module, it should resolve the share that is eagerly available over a promise?

2. the fork we work on, seems to have already fixed this under our sdk/runtime layers, but we use module snapshot and runtime plugins. We are able to vend lock files to the system and can deterministically control module loading, or preload it. This however requires additional infra work vs something webpack runtime can automatically try to solve in a less informed manner.

I believe the main cause of the issue here is related to a synchronous startup in your host, along with a runtime chunk of single set. Without can import(bootstrap), any async setup causes a failure to the sync side of the application.

For this use case, there is one alternative - however if this solution solves eager consumption - Id be interested in seeing if it solves mine.

I chose to integrate an internal async boundary within the runtime code of entrypoints. So that users of single runtime chunk or cannot use import() in entry, do not have to change application architecture to achieve the same effect.
I call the entrypoint manually through the chunk handlers before allowing them to require their entry module.

https://github.com/module-federation/universe/blob/main/packages/enhanced/src/lib/container/AsyncBoundaryPlugin.ts - which makes entire application startup async at the runtime level. Everything starts like an import() would. This means you could depend on async chunks that "eagerly" required, since theres no eager aspect anymore.

Can you ellaborate a little further on you pull request and how the logic here is applied? Is your strategy to search share scope for a already resolved and avaliable version in share scope? or are you able to await a promise and not crash the host with this alteration in consume shared module loading?
Do you notice this specifically when using single runtime chunk or custom chunk split rules?

@alexander-akait i believe that while this solves a specific scope of issue, i do think there is a series of bugs in webpack where single runtime chunk and multiple entry points cause several points of failure.

Single runtime chunk causes container reference modules to only be hoisted to their relative entry points. Routing between multiple entry (like nextjs) will cause container init to fail as webpack runtime is unable to init all the modules for containers referenced in its initExternal() callbacks.

The inability for sync startup is quite painful, eager consumption also will apply to remote module imports too if unwrapped in async boundary somewhere in loading chain, causing call of undefined.
From my implementation, i understand that this is tricky to resolve without returning a promise somewhere - however a way to enable async startup as side effect, where webpack runtime can manage this without a probem, but if end user attempts to access entrypoint manually, it will return promise before export - for several use cases, end users are not accessing entrypoint exports directly. And if its webpack accessing two entrypoints with requrie.e, this is already a async operation and will naturally resolve an inner promise to callbacks, as this is not module.exports of NormalModule(), this is exports of a chunk, and chunk handler seems to accept inner resolve.

Most use cases the user is already loading entrypoint via some promise implementation. if entry startup is async and entrypoiint called its own webpack_require.f() chunk handlers for the entry point, before entry startup runtime module calls its webpack_require(entry.module), this would resolve most eager consumption issues.

[cmath10]

@ScriptedAlchemy,

1. In short, I expect that if my sync build relies on an eager shared module and it is available, it will consume the module immediately, even if there are newer versions of that module that satisfy the boundary specified in my config. I don't see any reason, why this should work any other way.
2. The solution with SDK & runtime might work, but my environment includes PHP (Symfony) & webpack manifest plugin for historical reasons, and for that reason, I can't apply the solution proposed by @2heal1 , without porting runtime at least.

Currently, I use in my project a version of webpack, modified exactly in the same manner I propose here. And it works, despite of tons of legacy I have to deal with. I'll try your alternative to check if it works. Need a few days to finish some tasks before.

[ScriptedAlchemy]

Thats acceptable. This sounds like a legit issue and im pretty use other users have been bitten by this. @alexander-akait we should merge if all requirements are met.

[alexander-akait]

/cc @vankop Can take a look at it too?

[vankop]

I guess we need to add a test case. There is a folder with container test cases test/configCases/container

[alexander-akait]

@cmath10 Yeah, can you add a couple test cases and we can merge

[cmath10]

@alexander-akait

From the original discussion:

I have trouble with the autotest step, not sure where I need to place them and where I can find existing tests for the plugin 😅 (to use them as an example, this is my first time testing things like this 😅).

and from further discussion: #18001 (reply in thread)

Well, I don't refuse, and experience of writing tests for webpack would be a great thing, but I need a little help at least first 😅

[alexander-akait]

@cmath10 Sorry for delay, you can find examples here - https://github.com/webpack/webpack/tree/main/test/configCases/container, as you can see write them is not hard, so just put coumple examples with eager shared modules (which we fixed here), if you need more help feel free to write me

[webpack-bot]

@cmath10 Thanks for your update.

I labeled the Pull Request so reviewers will review it again.

@alexander-akait Please review the new changes.

[cmath10]

@alexander-akait, I just rebased my branch, so I think a new review is not required yet. However, I have some questions:

• Is there any instruction for the tests? How to run them separately, for example.
• My case includes yarn workspaces, how can I organize them inside the test case directory? It is necessary to use two different versions of the same library, and I'm unable to find any suitable example.

I intend to use the following:

1. host's package.json:

{
  "dependencies": {
    "tiny-emitter": "=2.0.0"
  }
}

2. host module:

import TinyEmitter from 'tiny-emitter'

const emitter = new TinyEmitter()

emitter.on('hello', () => console.log('hello[host]'))
emitter.emit('hello')

import('service/emitter').then(({ emitter }) => {
  emitter.emit('hello')
})

3. host configuration:

const { dependencies } = require("./package.json");

// eslint-disable-next-line n/no-unpublished-require
const { ModuleFederationPlugin } = require("../../../../").container;

/** @type {import("../../../../").Configuration} */
const { dependencies } = require("./package.json");

// eslint-disable-next-line n/no-unpublished-require
const { ModuleFederationPlugin } = require("../../../../").container;

/** @type {import("../../../../").Configuration} */
module.exports = {
  plugins: [
    new ModuleFederationPlugin({
      name: "host",
      remoteType: "var",
      remotes: {
        service: "service"
      },
      shared: {
        "tiny-emitter": {
          eager: true,
          singleton: true,
          requiredVersion: dependencies["tiny-emitter"]
        }
      }
    })
  ]
};

4. remote package.json

{
  "dependencies": {
    "tiny-emitter": "^2.1.0"
  }
}

5. remote module

import { TinyEmitter } from 'tiny-emitter'

const emitter = new TinyEmitter()

emitter.on('hello', () => console.log('hello[service]'))

export {
  emitter,
}

6. remote configuration:

const { dependencies } = require("./package.json");

// eslint-disable-next-line n/no-unpublished-require
const { ModuleFederationPlugin } = require("../../../../").container;

/** @type {import("../../../../").Configuration} */
module.exports = {
  plugins: [
    new ModuleFederationPlugin({
      name: "remote",
      filename: "[name].[fullhash].js",
      library: { type: "var", name: "service" },
      exposes: {
        "./emitter": {
          name: "emitter",
          import: "./emitter.js"
        }
      },
      shared: {
        ...dependencies
      }
    })
  ]
};

In my test case, I'm going to use jest.spy to check that the log was invoked two times with strings hello[host] and hello[service]