Add webpack benchmark #8

jhnns · 2017-10-24T20:26:13Z

This PR adds a simple webpack benchmark based on https://github.com/jhnns/webpack-browser-benchmark

During the benchmark, webpack builds a very small project with an async import() to also cover the code splitting part. The loader pipeline is not executed during this test as it would require a lot of more mocking. Maybe we can add that later.

I don't think that bigger files as fixtures would be reasonable since webpack uses acorn internally which is already benchmarked in another test. However, we could add more files to represent a typical project. Some engine optimization probably only kick in when there are a lot of files.

There is still a problem with this PR: It does not include a polyfill for Error.captureStackTrace() so this won't work in other JS engines than the V8. How should we proceed with it? Why aren't we allowed to polyfill the global object? Maybe it is possible with webpack to inject a special Error object so that we don't have to extend the global object, but I have to check that option...

The Error.captureStackTrace() is not required in the code path that is executed during the benchmark. I tested it in Firefox and Safari and there were no error (and the Error object had no captureStackTrace property)

jhnns · 2017-10-24T20:26:55Z

jest.config.js

+
+module.exports = {
+  watchPathIgnorePatterns: [
+    path.resolve(__dirname, "build")


This is necessary because the webpack test needs to be built with webpack first. This watch ignore pattern fixes an infinite loop issue.

Can you do this as a separate PR? Just adding the jest.config.js file. This seems useful on it's own.

You don't need to ignore that path without the webpack test because other tests are not writing any files. Personally, I wouldn't add that configuration when it's not necessary. What do you think?

Good point.

jhnns · 2017-10-24T20:27:42Z

src/vfs.js

@@ -79,5 +79,18 @@ fs.writeFileSync(
  "third_party/vue.runtime.esm-nobuble-2.4.4.js",
  require("raw-loader!../third_party/vue.runtime.esm-nobuble-2.4.4.js")
 );
+fs.mkdirpSync("/src/fixtures/webpack");


Webpack requires absolute file paths.

How does that work when you run it via node? I.e.

$ node src/cli

bmeurer · 2017-10-25T04:03:52Z

jest.config.js

+
+module.exports = {
+  watchPathIgnorePatterns: [
+    path.resolve(__dirname, "build")


Can you do this as a separate PR? Just adding the jest.config.js file. This seems useful on it's own.

bmeurer · 2017-10-25T04:08:36Z

src/webpack-benchmark.js

+  name: "webpack",
+  defer: true,
+  fn(deferred) {
+    return Promise.all(


This is a problem. I carefully avoided any interaction with the microtask queue, i.e. any use of promises, because that way you no longer measure the core JavaScript bits, but all of a sudden mix in embedder tasks into the measurement (i.e. DOM tasks scheduled on the microtask queue). Also the microtask queue doesn't work the same across different shells (not even across different versions of the same shell).

Please use a synchronous API instead.

Ideally webpack (and rollup FWIW) would provide a simple API entry where you can feed in strings and it would do the tree shaking and bundling on that, and produce a single string output.

Ok, that is going to be interesting 😁 maybe it's possible, but I'm not sure how deep the assumption of an async file system access is buried in the code 😁

We will also leave the regular code path, but that's probably a fair trade-off.

I think that's the only option for this, otherwise we get all kinds of other stuff into the measurement and the results will vary based on whether it's run in Node or the browser.

Ok, I took a look and so far it looks promising.

Basically, we just need to provide synchronous mocks for setImmediate() and process.nextTick(). There are, however, assumptions about the execution order. That's why I had to implement a simple event loop. But with my event loop, it looks like webpack is compiling synchronously. The event loop looks basically like this:

const loop = []; global.process = { nextTick(fn) { const args = Array.prototype.slice.call(arguments, 1); loop.push(() => { fn.apply(null, args); }); } }; global.setImmediate = function (fn) { const args = Array.prototype.slice.call(arguments, 1); loop.push(() => { fn.apply(null, args); }); }; // and then during the benchmark let task; while ((task = loop.shift())) { task(); }

Do you think that would be ok?

Interesting approach. Do you think we could somehow make the setImmediate and process.nextTick mock local to the webpack benchmark? I don't like changing global state. That can easily screw up other benchmarks w/o us noticing it.

Yes, we could inject our process and setImmediate mock to only webpack and virtualfs (virtualfs is using setImmediate to fake async method calls).

However, I would inject these mocks to all modules. Because if any module inside the benchmark is using one of these methods, the benchmark cannot be sync by design so I'd say it's fair to mock these throughout the whole bundle.

That would also make this mocks only available to the local module scope instead of mutating global which ensures that we don't touch the global object.

jhnns · 2017-10-25T22:54:38Z

The benchmark now executes synchronously.

All modules inside the bundle use our process.nextTick, setImmediate and clearImmediate mock now. The global object is not altered.

jhnns · 2017-10-25T22:58:24Z

src/webpack-benchmark.js

+      eventLoop.run();
+
+      if (finished !== true) {
+        throw new Error("Webpack did not finish synchronously");


Just to make sure 😁

bmeurer

Wow, that's a lot of magic, and the magic doesn't work for node unfortunately:

$ node src/cli
Running Web Tooling Benchmark 0.2.0...
--------------------------------------
Encountered error running benchmark webpack, aborting...
Error: Webpack did not finish synchronously
    at payloads.forEach.config (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/src/webpack-benchmark.js:45:15)
    at Array.forEach (<anonymous>)
    at Benchmark.fn (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/src/webpack-benchmark.js:27:14)
    at Benchmark.eval (eval at createCompiled (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/benchmark/benchmark.js:1725:16), <anonymous>:5:124)
    at clock (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/benchmark/benchmark.js:1644:22)
    at clock (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/benchmark/benchmark.js:1818:20)
    at cycle (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/benchmark/benchmark.js:2007:49)
    at Benchmark.run (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/benchmark/benchmark.js:2114:13)
    at execute (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/benchmark/benchmark.js:860:74)
    at invoke (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/benchmark/benchmark.js:970:20)
/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/src/webpack-benchmark.js:34
            throw err;
            ^

EntryModuleNotFoundError: Entry module not found: Error: Can't resolve '/src/fixtures/webpack/a.js' in '/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark'
    at moduleFactory.create (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/webpack/lib/Compilation.js:411:30)
    at factory (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/webpack/lib/NormalModuleFactory.js:235:20)
    at resolver (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/webpack/lib/NormalModuleFactory.js:60:20)
    at asyncLib.parallel (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/webpack/lib/NormalModuleFactory.js:127:20)
    at /usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/async/dist/async.js:3861:9
    at /usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/async/dist/async.js:421:16
    at iteratorCallback (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/async/dist/async.js:996:13)
    at /usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/async/dist/async.js:906:16
    at /usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/async/dist/async.js:3858:13
    at resolvers.normal.resolve (/usr/local/google/home/bmeurer/Projects/web-tooling-benchmark/node_modules/webpack/lib/NormalModuleFactory.js:119:22)

Wouldn't it be easier to just add a synchronous entry point to webpack? That seems to have benefits beyond performance testing, i.e. you could easily unit test it w/o having to deal with promises and stuff.

bmeurer · 2017-10-26T05:11:26Z

src/vfs.js

@@ -79,5 +79,18 @@ fs.writeFileSync(
  "third_party/vue.runtime.esm-nobuble-2.4.4.js",
  require("raw-loader!../third_party/vue.runtime.esm-nobuble-2.4.4.js")
 );
+fs.mkdirpSync("/src/fixtures/webpack");


How does that work when you run it via node? I.e.

$ node src/cli

bmeurer · 2017-10-26T05:12:25Z

webpack.config.js

+        chokidar: require.resolve("./src/mocks/chokidar"),
+        "uglify-js": require.resolve("./src/mocks/dummy"),
+        // These modules are used by virtualfs to fake async fs calls
+        "core-js/library/fn/set-immediate": require.resolve(


How does that work with running the benchmark suite via node?

See also http://facebook.github.io/jest/docs/en/troubleshooting.html#tests-are-extremely-slow-on-docker-and-or-continuous-integration-ci-server

jhnns · 2017-10-26T15:14:01Z

The magic should work 😁

Shouldn't you execute node dist/run.js as CLI? That's important because the CLI needs to be bundled with webpack in order to inject our mocks for process.nextTick and setImmediate:

web-tooling-benchmark add/webpack ✗ f672f33 10h10m ▲ ◒ ➜ node dist/run.js 
Running Web Tooling Benchmark 0.2.0...
--------------------------------------
         acorn:  5.14 runs/sec  ±7.91%
         babel:  6.04 runs/sec  ±7.93%
       babylon:  5.90 runs/sec ±12.13%
         buble:  5.71 runs/sec ±16.31%
          chai:  9.99 runs/sec  ±7.89%
  coffeescript:  5.84 runs/sec ±10.35%
        espree:  1.10 runs/sec  ±8.13%
       esprima:  5.44 runs/sec  ±9.75%
        jshint:  6.84 runs/sec  ±4.97%
         lebab:  6.71 runs/sec ±17.89%
       prepack:  6.63 runs/sec ±12.83%
      prettier:  8.29 runs/sec ±13.73%
    source-map: 14.91 runs/sec ±12.09%
    typescript:  9.07 runs/sec ±14.34%
     uglify-js:  7.02 runs/sec  ±4.99%
     uglify-es: 17.66 runs/sec  ±5.57%
       webpack: 135.50 runs/sec ±15.22%
--------------------------------------
Geometric mean:  7.97 runs/sec

Seems like the beforeAll hook of the webpack-benchmark has a timeout on Travis. I pushed a possible fix using the advice from http://facebook.github.io/jest/docs/en/troubleshooting.html#tests-are-extremely-slow-on-docker-and-or-continuous-integration-ci-server

bmeurer · 2017-10-26T17:03:59Z

That's exactly the thing: The benchmark should work unbundled, just by running node src/cli, since that's the closest configuration to real-world we get, since tools are also run unbundled normally in node. That's why I'm worried about all the magic, which relies on tricks with webpacks bundling.

I really appreciate your contribution and all the time and energy you put into this. But I also need to stress this point a bit, that we need to have a reliable way to just run the webpack core logic in a synchronous way. How hard would it be to modify webpack core to provide such a way? Maybe just doing that on a branch initially, which is then used for benchmarking?

jhnns · 2017-10-26T19:32:34Z

Ok, then it seems like I've misunderstood the goal of the benchmark.

My understanding was that the benchmark is intended to be executable without any dependency on host objects, hence the bundling with webpack because a basic webpack bundle (without code splitting or other fancy stuff) should be executable in any modern JS environment.

I didn't know that it should also be executable in Node.js without touching anything. It would be nice to have a list somewhere about which file should be executable where. Or to have an instruction how to test the benchmark.

that we need to have a reliable way to just run the webpack core logic in a synchronous way. How hard would it be to modify webpack core to provide such a way?

I think I need to talk with @sokra about this. From my POV, it's hard to isolate a core logic because one important part of webpack's core logic is to discover the dependency graph via async file system calls (which also includes async resolving). So we would need to start with a pre-discovered dependency graph, which might be possible, but is a bit far away from the regular execution flow.

There might be a solution in between: Webpack allows to swap out the file system. So we could use a mock file system, but it expects an async API. That's where the sync event loop would come into play. This approach should be executable in Node without bundling it.

There are, however, calls of process.nextTick() in the webpack code where I'm not sure why they are required. To make these synchronous, we would need to patch process.nextTick before the benchmark. Is that an option? If not, could you explain to me why not? I just want to understand the requirements and constraints you have a mind.

bmeurer · 2017-10-27T04:13:49Z

The main problem with all this magic is that the benchmark is now dominated by the fs mock, some use of the vm module and a lot of other unrelated parts. So improvements to this benchmark won't necessarily reflect improvements on webpack in the real world. And this goes against the philosophy of the web tooling benchmark.

mathiasbynens · 2017-11-06T15:56:42Z

It would be nice to have a list somewhere about which file should be executable where. Or to have an instruction how to test the benchmark.

Agreed; sorry we didn’t make this clear before! Now we have: https://github.com/v8/web-tooling-benchmark/blob/master/CONTRIBUTING.md#tests

jhnns · 2017-11-11T12:02:30Z

@mathiasbynens thanks. Is there also a browser list in which browsers it should work?

mathiasbynens · 2017-11-11T12:04:17Z

@jhnns I’d imagine the latest versions of Firefox, Edge, Safari, and Chrome. @bmeurer can confirm. (And yeah, we should document that as well.)

bmeurer · 2017-11-11T17:52:31Z

All browsers, unless there's a good reason to exclude them (i.e. like old internet explorer versions don't need to be supported). But since the tools have to run on Node 6 anyways, that means you automatically support all the relevant browsers.

jhnns commented Oct 24, 2017

View reviewed changes

bmeurer suggested changes Oct 25, 2017

View reviewed changes

jhnns added 2 commits October 25, 2017 23:22

Add webpack benchmark

07c1c4d

Make webpack benchmark synchronous

f672f33

jhnns force-pushed the add/webpack branch from 89596fd to f672f33 Compare October 25, 2017 22:52

jhnns commented Oct 25, 2017

View reviewed changes

bmeurer reviewed Oct 26, 2017

View reviewed changes

Fix slow test execution on Travis

c1160e8

See also http://facebook.github.io/jest/docs/en/troubleshooting.html#tests-are-extremely-slow-on-docker-and-or-continuous-integration-ci-server

mathiasbynens force-pushed the master branch 2 times, most recently from c289d6a to 164db67 Compare May 13, 2018 15:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add webpack benchmark #8

Add webpack benchmark #8

jhnns commented Oct 24, 2017 •

edited

jhnns Oct 24, 2017

bmeurer Oct 25, 2017

jhnns Oct 25, 2017

bmeurer Oct 25, 2017

jhnns Oct 24, 2017

bmeurer Oct 26, 2017

bmeurer Oct 25, 2017

bmeurer Oct 25, 2017

jhnns Oct 25, 2017

bmeurer Oct 25, 2017

jhnns Oct 25, 2017

bmeurer Oct 25, 2017

jhnns Oct 25, 2017 •

edited

jhnns commented Oct 25, 2017

jhnns Oct 25, 2017 •

edited

bmeurer left a comment

bmeurer Oct 26, 2017

bmeurer Oct 26, 2017

jhnns Oct 26, 2017

jhnns commented Oct 26, 2017

bmeurer commented Oct 26, 2017

jhnns commented Oct 26, 2017 •

edited

bmeurer commented Oct 27, 2017

mathiasbynens commented Nov 6, 2017

jhnns commented Nov 11, 2017

mathiasbynens commented Nov 11, 2017

bmeurer commented Nov 11, 2017

Add webpack benchmark #8

Are you sure you want to change the base?

Add webpack benchmark #8

Conversation

jhnns commented Oct 24, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhnns Oct 25, 2017 • edited

Choose a reason for hiding this comment

jhnns commented Oct 25, 2017

jhnns Oct 25, 2017 • edited

Choose a reason for hiding this comment

bmeurer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhnns commented Oct 26, 2017

bmeurer commented Oct 26, 2017

jhnns commented Oct 26, 2017 • edited

bmeurer commented Oct 27, 2017

mathiasbynens commented Nov 6, 2017

jhnns commented Nov 11, 2017

mathiasbynens commented Nov 11, 2017

bmeurer commented Nov 11, 2017

jhnns commented Oct 24, 2017 •

edited

jhnns Oct 25, 2017 •

edited

jhnns Oct 25, 2017 •

edited

jhnns commented Oct 26, 2017 •

edited