Run only a test of a class with Parameterized runner

[diogoeag]

When using the Parameterized runner there is no possibility to run only a test method of a test class.

For example when running in maven: mvn test -Dtest=TesClass#testName

No tests are run when using the Parameterized runner.

That would be great to have this feature, has for development is comon to the developers trying to run only the test they are working on.

[migueljbento]

Fixing this would be great, we end up having to comment UTs in large test classes which is undesirable.

[kcooney]

The Maven Surefire plugin allows to to specify globs (see http://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html). Have you tried that?

[diogoeag]

Yes, that's how we would like to use, but it doesn't work. Are you saygin that this is a problem of the surefire plugin and not from junit?

[kcooney]

@diogoeag I believe it's a problem with the Surefire plugin. You can verify that by writing a class with a main method that uses JUnitCore to run the tests of a class that used @Parameterized and have your main method apply a filter to select one of the tests

[diogoeag]

Hi Kevin,

Take a look at these two samples:

First test without parameterized:

package org;

import org.junit.Test;
import org.junit.internal.requests.FilterRequest;
import org.junit.runner.Description;
import org.junit.runner.JUnitCore;
import org.junit.runner.manipulation.Filter;

public class NoParameterTest {

    public static void main(String[] args) {
        new JUnitCore().run(FilterRequest.aClass(NoParameterTest.class).filterWith(new Filter() {
            @Override
            public boolean shouldRun(Description description) {
                System.out.println("Should run test: classname[" + description.getClassName() + "] method name[" + description.getMethodName() + "]");
                return description != null && description.getMethodName() != null && description.getMethodName().equals("test");
            }

            @Override
            public String describe() {
                return null;
            }
        }).getRunner());

    }

    @Test
    public void test() {
        System.out.println("Running Test: test");
    }

    @Test
    public void test2() {
        System.out.println("Running Test: test2");
    }
}

output

Should run test: classname[org.NoParameterTest] method name[test]
Should run test: classname[org.NoParameterTest] method name[test2]
Running Test: test

Second test with parameterized:

package org;

import org.junit.Test;
import org.junit.internal.requests.FilterRequest;
import org.junit.runner.Description;
import org.junit.runner.JUnitCore;
import org.junit.runner.RunWith;
import org.junit.runner.manipulation.Filter;
import org.junit.runners.Parameterized;

import java.util.Arrays;
import java.util.Collection;

@RunWith(Parameterized.class)
public class ParameterTest {

    public static void main(String[] args) {
        new JUnitCore().run(FilterRequest.aClass(ParameterTest.class).filterWith(new Filter() {
            @Override
            public boolean shouldRun(Description description) {
                System.out.println("Should run test: classname[" + description.getClassName() + "] method name[" + description.getMethodName() + "]");
                return description != null && description.getMethodName() != null && description.getMethodName().equals("test");
            }

            @Override
            public String describe() {
                return null;
            }
        }).getRunner());

    }


    @Parameterized.Parameters
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[]{"Paramter 1"}, new Object[]{"Parameter 2"});
    }

    private String localParameter;

    public ParameterTest(String parameter) {

        this.localParameter = parameter;
    }

    @Test
    public void test() {
        System.out.println("Running Test: test");
        System.out.println("Parameter: " + localParameter);
    }

    @Test
    public void test2() {
        System.out.println("Running Test: test2");
        System.out.println("Parameter: " + localParameter);
    }
}

output

Should run test: classname[[0]] method name[null]
Should run test: classname[[1]] method name[null]

The Description passed to the Filter by the Parameterized is not correct right?

[kcooney]

@diogoeag Thanks for the code example. What version of JUnit did you try this on? I think we fixed a related issue in 4.12 but I am not sure.

[diogoeag]

@kcooney, I've tested with the current master 4.12-SNAPSHOT and also with 4.11. Both have the same problem.

[kcooney]

Thanks. Fixing this would like require changes that are not backwards compatible. Setting milestone to 5.0

[diogoeag]

Just to clarify, fixing this would only need to go to BlockJUnit4ClassRunnerWithParameters and remove the override of the #testName method. That is replacing the default

    protected String testName(FrameworkMethod method) {
        return method.getName();
    }

by

    @Override
    protected String testName(FrameworkMethod method) {
        return method.getName() + getName();
    }

And the consequencies would be that the reporting of the tests would not have the parameter name in the testHeader (at least the tests that have failed were only these)?

[kcooney]

@diogoeag but wouldn't we want the parameter name in the description?

I think the fixes would best be made on top of the Description builder we are working on for JUnit 5.0.

[diogoeag]

@kcooney Yes to be correct we should have the parameter name in the description, but the description should not affect the test name. I agree with you that the Description should have a test name that should be the correct name and the display name that should be used in user facing descriptions and include the parameter.

Do you have any estimates on the target date to 5.0 ?

I was looking to the usages of getDisplayName to submit a pull request to 5.0 with the fix, however it is used 23 times in the project (sources and tests) and I don't have the enough knowledge of JUnit source to make such change. Meanwhile I will make a fork for my personal usage with that intermediate fix for me. Having the parameter names is not critical, however allowing us to run single tests saves us many hours in many developers running tests in our source.

Thanks

[diogoeag]

If anyone needs to have access to this temporary fix: feedzai@be41d29

[Tibor17]

@diogoeag
See this. Would it help in surefire 2.18 ?
apache/maven-surefire#46

[diogoeag]

@Tibor17 I'm not sure. Because shoudlRun will use the description also. I don't know enough of the codebase to know if it will work. Only testing it. My sample code can be one of the tests for your fix also. Using parameterized tests has the problem of changing the test name.

[Tibor17]

@diogoeag
In surefire MethodFilter you should be able to specify condition something like this:

String filterBy = JunitVersion < 5.0 & @RunWith = Parameterized.class ? methodName + className : methodName

If you want to tolerate the JUnit issues, you can commit hotfix by yourself in Maven Surefire project.

[kcooney]

@diogoeag we don't yet have a target date for JUnit 5.0, but it would likely be the release after 4.12. We are currently focusing on getting 4.12 out the door

[diogoeag]

@kcooney thanks. We will use our internal release meanwhile.

[Fanch-]

Hello, what about this issue? Is there a workaround to use Parameterized.class and have the possibility to run a single test using maven surefire? Or do we have to wait the 5.0 version for it?

[kcooney]

@Fanch- there is no current fix. We thought would fix this on too of the ImmutableDescription work we were working on, but that effort was abandoned.

[marcphilipp]

I think it would be ok to remove getName() from testName() because it's redundant anyway.

@junit-team/junit-committers What do you think?

[kcooney]

As long as all Description instances are unique for a partameterized test.

[marcphilipp]

We could set the unique ID in the description to make them unique. However, I'm wondering if/why they need to be unique in the whole subtree?

[kcooney]

Yes, Description instances need to be unique. That's all that is passed to the listener to reference a test starting and ending. To build a tree like you showed and update it during the test run you need to know which test started.

[marcphilipp]

Point taken. So, I think we should add a new factory method to Description that creates a test description with an explicit unique ID. We could then pass the current name as unique ID and use just the method's name as its display name. What do you think?

[kcooney]

@marcphilipp there already is a method in Description for creating a test Description with a given unique ID: https://github.com/junit-team/junit4/blob/master/src/main/java/org/junit/runner/Description.java#L109

I don't think that Description should generate unique IDs. Couldn't Parameterized pass a deterministic value to Description.createTestDescription()?

[marcphilipp]

there already is a method in Description for creating a test Description with a given unique ID: https://github.com/junit-team/junit4/blob/master/src/main/java/org/junit/runner/Description.java#L109

Yeah, but that one does not have a class parameter. So we need a different one.

I don't think that Description should generate unique IDs. Couldn't Parameterized pass a deterministic value to Description.createTestDescription()?

I didn't mean that. I meant that Parameterized should pass method.getName() + getName() as unique ID and method.getName() as name.

[kcooney]

SGTM.

Even though we dropped the idea of having immutable Description instances for JUnit 4, should we add a builder to avoid having so many static creational methods?

[marcphilipp]

Sure! 👍

[SqAutoTestTeam]

Are there any estimates when it'll be fixed ?
This bug blocks related bug of JUnit5
junit-team/junit5#549

[kcooney]

@SqAutoTestTeam no one is actively working on this. I did update the Description builder branch recently so an ambitious contributor could start from there. Alternatively, we could add yet another static method to Description.

Fixing this would require new APIs on Description. We wouldn't want JUnit 5 to require those new APIs (currently JUnit 5 would work on just about any version of JUnit 4.x)

[Tibor17]

@diogoeag
Did you read this chapter in Surefire?
http://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html#Multiple_Formats_in_One

Filtering parameterized tests works since of version 2.19. In your case it is -Dtest=TesClass#testName[*] instead of -Dtest=TesClass#testName.

[sbrannen]

@kcooney and @marcphilipp,

What do you guys thinking about "backporting" the fix in https://github.com/junit-team/junit5/pull/1018/files#diff-42ed8b618527eb1521dbe3cd9100e949R48 to JUnit 4?

[kcooney]

@sbrannen that looks like a reasonable change. I would just want to make sure that if we backported those changes we wouldn't make JUnit 4.13 incompatible with JUnit 5, or require projects with JUnit 4-style tests to upgrade to 4.13 before they can run their JUnit 4-style tests using the JUnit 5 runner APIs.

[sbrannen]

@kcooney,

that looks like a reasonable change.

👍

I would just want to make sure that if we backported those changes we wouldn't make JUnit 4.13 incompatible with JUnit 5, or require projects with JUnit 4-style tests to upgrade to 4.13 before they can run their JUnit 4-style tests using the JUnit 5 runner APIs.

In JUnit 4 the change would only be to a single line in Filter#matchMethodDescription(Description), and we can keep the same fix in JUnit Vintage as it is now so that the VintageTestEngine would continue to work with JUnit 4.12.

So, unless I'm overlooking something, that should avoid any compatibility issues.

What do you think?

[kcooney]

@sbrannen SGTM