How to rerun skipped test cases using TestNG?

[Kaveeshhh]

TestNG Version

7.4.0

Expected behavior

I want to re-run testng skipped test cases.

Actual behavior

Getting 'Test ignored.' message.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

   @Test(alwaysRun = true)
    public void skipTest() throws IOException {
        throw new SkipException("Skipping this test case intentionally");
    }

    @AfterSuite(alwaysRun = true)
    public void afterSuite(ITestContext ctx) {
        IResultMap skippedTests = ctx.getSkippedTests();
        int counter = 5;
        while (skippedTests.size() > 0 && (counter--) > 0) {
            List<XmlInclude> includeMethods
                    = skippedTests.getAllMethods()
                    .stream()
                    .map(m -> new XmlInclude(m.getMethodName()))
                    .collect(Collectors.toList());
            XmlClass xmlClass = new XmlClass(this.getClass().getName());
            xmlClass.setIncludedMethods(includeMethods);

            XmlTest test = ctx.getCurrentXmlTest();
            List<XmlClass> classes = test.getXmlClasses();
            classes.remove(0);
            classes.add(xmlClass);

            TestRunner newRunner = new TestRunner(new Configuration(),
                    ctx.getSuite(),
                    test,
                    false,
                    null,
                    new ArrayList<>());
            newRunner.run();

            Set<String> newPassed = newRunner.getPassedTests().getAllResults()
                    .stream()
                    .map(ITestResult::getName)
                    .collect(Collectors.toSet());
            IResultMap passedTests = ctx.getPassedTests();
            Iterator<ITestResult> iter = skippedTests.getAllResults().iterator();
            while (iter.hasNext()) {
                ITestResult result = iter.next();
                if (newPassed.contains(result.getName())) {
                    result.setStatus(ITestResult.SUCCESS);
                    passedTests.addResult(result, result.getMethod());

                    //remove the test from list of skipped tests.
                    iter.remove();
                }
            }
        }
    }

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[krmahadevan]

@Kaveeshhh - Can you please help me understand as to what exactly are you trying to achieve here?

A test method can be skipped due to the following reasons:

1. There was a failure in a configuration method
2. There was a failure in an upstream method, upon which this current test method depends on
3. The test method explicitly throw a SkipException due to whatever reasons
4. The earlier iterations of that method failed and there was a RetryAnalyzeer involved

Please help answer the following questions:

• When you say you would like to retry a skipped test, how do you intend to handle all the above scenarios via the retry mechanism? I ask this because TestNG does not give you insights around what configuration method caused a test to be skipped (It could be a BeforeSuite|BeforeTest|BeforeClass|BeforeMethod failure)
• You are now relying on the internal classes within TestNG to try and re-run a skipped test, which is bound to a lot of failures, because the internals of TestNG will change without any prior notice (Because they are NOT intended to be consumed directly). So today the class TestRunner has a signature in v7.10.2 (latest release as of today) which can change tomorrow and it would be hard for you to keep up. The interface signatures are guaranteed to stay the same, but that's not the case for the implementations.

[Kaveeshhh]

@Kaveeshhh - Can you please help me understand as to what exactly are you trying to achieve here?

A test method can be skipped due to the following reasons:

1. There was a failure in a configuration method
2. There was a failure in an upstream method, upon which this current test method depends on
3. The test method explicitly throw a SkipException due to whatever reasons
4. The earlier iterations of that method failed and there was a RetryAnalyzeer involved

Please help answer the following questions:

• When you say you would like to retry a skipped test, how do you intend to handle all the above scenarios via the retry mechanism? I ask this because TestNG does not give you insights around what configuration method caused a test to be skipped (It could be a BeforeSuite|BeforeTest|BeforeClass|BeforeMethod failure)
• You are now relying on the internal classes within TestNG to try and re-run a skipped test, which is bound to a lot of failures, because the internals of TestNG will change without any prior notice (Because they are NOT intended to be consumed directly). So today the class TestRunner has a signature in v7.10.2 (latest release as of today) which can change tomorrow and it would be hard for you to keep up. The interface signatures are guaranteed to stay the same, but that's not the case for the implementations.

@krmahadevan You're right about the limitations of retrying skipped tests with dependencies. However, for independent test cases that skip due to failures in @BeforeSuite, @BeforeTest, @BeforeClass, or @BeforeMethod annotations, we might not need to pinpoint the exact cause. In this scenario, would it be possible using TestNG to achieve a mechanism that simply retries the entire test class?

[krmahadevan]

@Kaveeshhh - When a test skips due to a configuration failure, TestNG does not tell you as to which config method failure caused the test to be skipped. So we wouldn't be able to run the actual required pre-requisites for a skipped test to be retried.

In this scenario, would it be possible using TestNG to achieve a mechanism that simply retries the entire test class?

No. The current implementation doesnt have any way of doing this from within TestNG.

[krmahadevan]

On the other hand if you would like to retry a failed configuration method (just to rule out flakiness in the setup for e.g.,) you can try doing something like below:

import org.testng.IConfigurable;
import org.testng.IConfigureCallBack;
import org.testng.ITestResult;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class AnotherSample implements IConfigurable {
    private boolean fail = true;

    @Override
    public void run(IConfigureCallBack callBack, ITestResult testResult) {
        callBack.runConfigurationMethod(testResult);
        if (!testResult.isSuccess()) {
            fail = false;
            callBack.runConfigurationMethod(testResult);
        }
    }

    @BeforeClass
    public void beforeClass() {
        if (fail) {
            throw new RuntimeException();
        }
    }

    @Test
    public void test() {
        System.err.println("Hello world");
    }
}

In the sample, we are doing the following:

• Define a class that would provide an implementation of IConfigurable interface. This will ensure that TestNG always calls the run() method and through that the configuration method gets invoked.
• We inspect the status and if it was found to be NOT success, then we retry ( the sample retries just once, but you can do it any number of times)
• Depending on your need, you can either have this logic housed in a common base class from which all test classes extend or you can basically have it reside in just specific classes.

Note: This will handle only configuration retries