EvoMaster: A Tool For Automatically Generating System-Level Test Cases

Summary

EvoMaster (www.evomaster.org) is a tool prototype that automatically generates system-level test cases. Internally, it uses an Evolutionary Algorithm and Dynamic Program Analysis to be able to generate effective test cases. The approach is to evolve test cases from an initial population of random ones, trying to maximize measures like code coverage and fault detection.

At the moment, EvoMaster targets RESTful APIs compiled to JVM 8 and 11 bytecode. The APIs must provide a schema in OpenAPI/Swagger format (either v2 or v3). The tool generates JUnit (version 4 or 5) tests, written in either Java or Kotlin.

A short video (5 minutes) shows the use of EvoMaster on one of the case studies in EMB.

Examples

The following code is an example of one test that was automatically generated by EvoMaster for a REST service called "scout-api" (see EMB repository). The generated test uses the RestAssured library.

@Test
public void test_36_with500() throws Exception {
        
   String location_media_files = "";
        
   String id_0 = given().accept("application/json")
                .header("Authorization", "ApiKey moderator") // moderator
                .contentType("application/json")
                .body(" { " + 
                    " \"uri\": \"hXf3e8B3ikGtuGjT\", " + 
                    " \"name\": \"nkeUfXVTC\", " + 
                    " \"copy_right\": \"\" " + 
                    " } ")
                .post(baseUrlOfSut + "/api/v1/media_files")
                .then()
                .statusCode(200)
                .assertThat()
                .contentType("application/json")
                .body("'uri'", containsString("hXf3e8B3ikGtuGjT"))
                .body("'name'", containsString("nkeUfXVTC"))
                .body("'copy_right'", containsString(""))
                .extract().body().path("id").toString();
                
   location_media_files = "/api/v1/media_files/" + id_0;
        
   given().accept("*/*")
                .header("Authorization", "ApiKey moderator") // moderator
                .get(resolveLocation(location_media_files, baseUrlOfSut + "/api/v1/media_files/1579038228/file"))
                .then()
                .statusCode(500) // se/devscout/scoutapi/resource/MediaFileResource_268_downloadFile
                .assertThat()
                .contentType("application/json")
                .body("'code'", numberMatches(500.0));
}

In this automatically generated test, a new resource is first created with a POST command. The id of this newly generated resource is then extracted from the POST response, and used in the URL of a following GET request on a sub-resource. Such GET request does break the backend, as it returns a 500 HTTP status code. The last line executed in the business logic of the backend is then printed as comment, to help debugging this fault.

The generated tests are self-contained, i.e., they start/stop the REST server by themselves:

    private static SutHandler controller = new em.embedded.se.devscout.scoutapi.EmbeddedEvoMasterController();
    private static String baseUrlOfSut;
    
    
    @BeforeClass
    public static void initClass() {
        baseUrlOfSut = controller.startSut();
        assertNotNull(baseUrlOfSut);
        RestAssured.urlEncodingEnabled = false;
    }
    
    
    @AfterClass
    public static void tearDown() {
        controller.stopSut();
    }
    
    
    @Before
    public void initTest() {
        controller.resetStateOfSUT();
    }

The ability of starting/resetting/stopping the tested application is critical for using the generated tests in Continuous Integration (e.g., Jenkins, Travis and CircleCI). However, it requires to write a driver to tell EvoMaster how to do such start/reset/stop.

A generated test is not only going to be a sequence of HTTP calls toward a running application. EvoMaster can also set up the environment of the application, like automatically adding all the needed data into a SQL database. At the moment, EvoMaster supports H2 and Postgres databases.

Documentation

• Download EvoMaster
• Build EvoMaster from source
• Console options
• OpenApi/Swagger Schema
• Using EvoMaster for Black-Box Testing (easier to setup, but worse results)
• Using EvoMaster for White-Box Testing (harder to setup, but better results)
  • Write an EvoMaster Driver for White-Box Testing
• Academic papers related to EvoMaster
• Slides of presentations/seminars
• Notes for developers contributing to EvoMaster

How to Contribute

There are many ways in which you can contribute. If you found EvoMaster of any use, the easiest way to show appreciation is to star it. Issues and feature requests can be reported on the issues page:

• Bugs: as for any bug report, the more detailed you can be the better. If you are using EvoMaster on an open source project, please provide links to it, as then it is much easier to reproduce the bugs.

• Documentation: if you are trying to use EvoMaster, but the instructions in these notes are not enough to get you started, then it means it is a "bug" in the documentation, which then would need to be clarified.

• Feature Requests: to improve EvoMaster, we are very keen to receive feature requests, although of course we cannot guarantee when they are going to be implemented, if implemented at all. As researchers, we want to know what are the problems that engineers in industry do face, and what could be done to improve EvoMaster to help them.

• Pull Requests: we are keen to receive PRs, as long as you agree with the license of EvoMaster, and as long as you are allowed by your employer to contribute to open-source projects. However, before making a PR, you should read the notes for developers.

• Industry Collaborations: to evaluate the effectiveness of EvoMaster, we need case studies. There are some open-source projects that can be used (e.g., which we selected and aggregated in the EMB repository). But open-source applications are not necessarily representative of software developed in industry. Therefore, we "collaborate" with different companies (e.g., Universitetsforlaget), to apply EvoMaster on their systems.

  • Benefits for us: access to the source code of real, industrial systems (of course, under NDAs). It makes easier to publish academic papers, and to get funding from the research councils to improve EvoMaster even further.

  • Benefits for the industrial collaborators:

    1. getting priority on new features and bug fixing.
    2. "free" human resources (MSc/PhD students and researchers) that try to break your systems and find faults in them.

• Academic Collaborations: we are keen to hear from students and researchers that want to collaborate on topics in which EvoMaster can be used on, or extended for. For example, it is possible to have visiting PhD students in our lab. However, all communications about academic collaborations should not be done here on GitHub, but rather by email, directly to Prof. A. Arcuri.

Funding

EvoMaster has been funded by:

• 2020-2025: a 2 million Euro grant by the European Research Council (ERC), as part of the ERC Consolidator project Using Evolutionary Algorithms to Understand and Secure Web/Enterprise Systems.
• 2018-2021: a 7.8 million Norwegian Kroner grant by the Research Council of Norway (RCN), as part of the Frinatek project Evolutionary Enterprise Testing.

License

EvoMaster's source code is released under the LGPL (v3) license. For a list of the used third-party libraries, you can directly see the root pom.xml file. For a list of code directly imported (and then possibly modified/updated) from other open-source projects, see here.

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.