Use jest worker to parallelize compilation

[janicduplessis]

What did you implement:

Improve build performance by parallelizing webpack compilation using worker threads.

Builds on top of #858, (check only commit 2a776b8). Note there still might be some cleaning up to do and useWorkerThreads option should probably be configurable as it is only usable with node 11+. This also uses async / await which afaik wasn't used in the codebase, but should be fine since it is supported since node 8.

How did you implement it:

Use jest-worker which implements a simple promise based api to execute modules on worker threads.

Most of the changes were needed to avoid passing the webpack config directly, since worker inputs and outputs need to be serializable and webpack configs aren't. What we do to fix this is pass some of the config that we modify (based on the code in validate.js) and merge it with the config that we load again in the worker.

Also uses rechoir and interpret to allow loading webpack configs in other languages like typescript (https://webpack.js.org/configuration/configuration-languages/), this is based on what the webpack cli does. Not sure why it works in the main process without any special config, but not in worker threads so had to add this.

How can we verify it:

Tested in a large project with package: individually on a 2.8 GHz Quad-Core Intel Core i7 Macbook Pro.

Before: builds in ~1000s
After: builds in ~500s

Todos:

• Write tests
• Write documentation
• Fix linting errors
• Make sure code coverage hasn't dropped
• Provide verification config / commands / resources
• Enable "Allow edits from maintainers" for this PR
• Update the messages below

Is this ready for review?: NO
Is it a breaking change?: POSSIBLY

[j0k3r]

@janicduplessis I switched the PR as ready for review by mistake I thought the message said to approve the PR so the test can run. I misread, sorry.

[janicduplessis]

All good, yea I think the best is start with #858 then come back to this one.

[janicduplessis]

Ok I had another look at this and rebased. I think it is good.

Some minor changes that might not be obvious:

• Add log for start / end of compilation (https://github.com/serverless-heaven/serverless-webpack/pull/859/files#diff-f812120496d0391e0cf70e8909b512747733627ad09bf584c4072a2d649d89d3R91)
• Potential breaking change is that lib.serverless (https://github.com/serverless-heaven/serverless-webpack/blob/master/lib/validate.js#L137) won't be set anymore for the webpack config. Not sure if it is documented and used.

Also verified that using enableWorkerThreads: true is fine even for lower node versions as it will fallback to the child_process implementation (https://github.com/facebook/jest/blob/master/packages/jest-worker/src/WorkerPool.ts#L41).

One potential question is do we want to add an option to configure this? Personally I don't think it is needed and concurrency can already be disabled by setting the existing option to 1.

[janicduplessis]

@j0k3r Any chance you could have a look at this?

[j0k3r]

Sorry for being late to reply.

Potential breaking change is that lib.serverlesswon't be set anymore for the webpack config.

It seems that lib.serverless is defined in the README and might be useful if you want to optimize webpack (see https://github.com/serverless-heaven/serverless-webpack#full-customization-for-experts)
Can't you add it when you call worker.compile(? I see you already inject lib.entries & lib.options. Why not lib.serverless?

One potential question is do we want to add an option to configure this? Personally I don't think it is needed and concurrency can already be disabled by setting the existing option to 1.

Do you mean that setting concurrency: 1 will still use jest worker to compile but with only one worker so that the parallelization will be (kind of) disabled?

[janicduplessis]

@j0k3r

It seems that lib.serverless is defined in the README and might be useful if you want to optimize webpack (see https://github.com/serverless-heaven/serverless-webpack#full-customization-for-experts)
Can't you add it when you call worker.compile(? I see you already inject lib.entries & lib.options. Why not lib.serverless?

The problem is everything passed to jest-worker needs to be serializable and lib.serverless isn't. Not sure if possible to recreate this object in the worker.

Do you mean that setting concurrency: 1 will still use jest worker to compile but with only one worker so that the parallelization will be (kind of) disabled?

Yea it will still use jest-worker, but execute only one build at a time. I guess the question was more if we need a way to disable executing in a worker. Maybe if we don't have a way to access lib.serverless in the worker then it could be a use case to execute outside of it.

[j0k3r]

The problem is everything passed to jest-worker needs to be serializable and lib.serverless isn't. Not sure if possible to recreate this object in the worker.

I see.
Maybe we can only inject what user might use, like, in that example, lib.serverless.config: #168 (comment) and inject it as lib.serverlessConfig?

[janicduplessis]

@j0k3r Turns out the config object is also a class instance so what I did is log the serverless object and copy manually the serializable parts that seem useful into a regular object. This covers the use case you listed.

Edit: Copied only props documented in the serverless typescript type definitions and updated our lib typedef to only include props that we copy instead of using the type directly from serverless.

[j0k3r]

Looks like people are using more elements which aren't defined in the serverless typescript definition and/or not documented, like:

• slsw.lib.serverless.configurationInput.provider.stage mode and optimization for webpack configuration is not working #919 (comment)
• slsw.lib.webpack.isLocal
• (even worse) slsw.lib.serverless.configurationInput.custom.MY_CONFIG How to set environment variable during compile/packaging for lambda@edge #918

I don't know how we can properly handle such case and warn people that the element doesn't exist.
We might have a lot of issues about "why XXX is no more part of slsw.lib.serverless..." 😕