Mutation testing project

Mutation testing for Servo

Background information: Servo uses the WPT test suite for integration testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT test suite when bugs are deliberately introduced into the code base.

Tracking issue: https://github.com/servo/servo/issues/18529 (please ask questions here)

Initial steps:

• email the mozilla.dev.servo mailing list (be sure to subscribe to it first!) introducing your group and asking any necessary questions
• implement a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
• build the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: hopefully this produces test failures!
• automate this process by writing scripts in a new python/servo/mutation directory, and calling them from scripts in /etc/ci.

Subsequent steps:

• implement other mutations, such as replacing if statements by either if true or if false, deleting or duplicating statements, reordering statements, changing arithmetic operations, and changing atomic string constants.
• improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
• find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
• find search heuristics for identifying mutations that cause no test failures.