Skip to content

Benchmarking your changes

Jump to bottom
Jonas Jenwald edited this page Dec 15, 2023 · 9 revisions

Two options for benchmarking:

  1. Using the Test Runner
  2. Using Firefox Talos

Using the Test Runner

When working on issues related to performance, it is important to provide a performance benchmark for your changes to assess whether or not your change has a performance impact. PDF.js provides tools to do this easily. Normally you would create a simple manifest file has a couple of PDF files you're trying to optimize and run it multiple times, e.g. benchmark.json:

[
  {
    "id": "tracemonkey-eq",
    "file": "pdfs/tracemonkey.pdf",
    "md5": "9a192d8b1a7dc652a19835f6f08098bd",
    "rounds": 50,
    "lastPage": 5,
    "type": "eq"
  }
]

Run the following commands to create a 'baseline' measurement (i.e., before you make your changes):

$ git checkout master
$ gulp generic
$ gulp components
$ cd test
$ node test.mjs --statsFile=stats/results/baseline.json --statsDelay=5000 --manifestFile=benchmark.json

Then apply your changes and create a 'current' measurement (replace <feature-branch> with the name of your branch):

$ cd ..
$ git checkout <feature-branch>
$ gulp generic
$ gulp components
$ cd test
$ node test.mjs --statsFile=stats/results/current.json --statsDelay=5000 --manifestFile=benchmark.json

Finally, you can compare the measurements and see any performance differences:

$ node stats/statcmp.js stats/results/baseline.json stats/results/current.json

As a sanity check, you should do this twice with the same code and compare the results.

For faster benchmarking during development, it could be useful to pass the --noChrome command line parameter to test.js to skip Chrome and only test with Firefox.

Using Firefox Talos

About

Versions of PDF.js that are merged in to Firefox are automatically benchmarked as part of the talos tests to monitor performance regressions/improvements. The talos pdfpaint test benchmarks how long it takes from a PDF being opened to the first page of the PDF being rendered. The advantage of using this system for benchmarking is that it is closer to a real world benchmark where we test the most important thing, which is how long it takes for the first page to render.

Running Locally

Prerequisites

This assumes you've setup a local PDF.js repo and you've setup a build of Firefox locally. We recommend using an non-debug artifact build of Firefox.

Adding a Custom PDF

If you just want to benchmark your changes with the default PDFs you can skip this step.

  1. Copy the PDF file into <firefox_src_dir>/testing/talos/talos/tests/pdfpaint/
  2. Add the file to the manifest in <firefox_src_dir>/testing/talos/talos/tests/pdfpaint/pdfpaint.manifest
  3. (Optional) Remove any other PDF files in the manifest if you only want to benchmark your PDF

Record the Baseline Data

First, we want to run the test PDF without any changes to see how it will compare. In the <firefox_src_dir> run:

./mach talos-test -a pdfpaint

When the test finishes there will be a section in the terminal with ------- Summary: start ------- and -------- Summary: end --------. You'll want to copy that section down somewhere for comparison.

Apply Your Changes & Test Again

  1. In the <pdfjs_src_dir>:
    1. make your changes
    2. build gulp mozcentral
    3. copy the pdf.js build to firefox rsync -a -v <pdfjs_src_dir>/build/mozcentral/browser/extensions/pdfjs/ <firefox_src_dir>toolkit/components/pdfjs/
  2. Back in <firefox_src_dir>:
    1. Build firefox ./mach build faster
    2. Run the tests again ./mach talos-test -a pdfpaint

The talos tests will run and output a test summary. You can now compare these results to see how your changes affected performance.

Running pdfpaint on Mozilla's infrastructure

Similar to above, you'll want to create a baseline result without your changes and another result with your changes. The pdfpaint test is part of the talos "other" group. An example of running the tests 6 times on windows:

./mach try fuzzy -q 'windows10-64-shippable opt-talos-other' --rebuild 6 --artifact

Once the tasks finish, you can then compare the results by using the two commit IDs and replacing <baseline> and <changed> below:

https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=<baseline>&newProject=try&newRevision=<changed>&framework=1&showOnlyImportant=1&page=1
Clone this wiki locally