Lifecycle hooks with mockk do not work properly on object and java classes with static functions

[testcenter]

Which version of Kotest are you using
5.8.1
Which version of mockk are you using
1.13.10

Steps to reproduce:
Creating an object which will be mocked for some tests:

object TestToMock {
    fun intSum(a: Int, b: Int): Int = a + b
}

or a java file with a static function:

public class TestStaticOnly {
    private TestStaticOnly() {
    }

    public static int intSum(int a, int b) {
        return a + b;
    }
}

Having 2 specs which use this object or java static function:

• mocked with the beforeSpec and unmocked via afterSpec
• without mocking

class MockedTest : DescribeSpec({
    beforeSpec {
        mockkObject(TestToMock)
        mockkStatic(TestStaticOnly::class)
    }
    afterSpec {
        unmockkObject(TestToMock)
        unmockkStatic(TestStaticOnly::class)
    }

    describe("using a mocked object") {
        it("executes the body") {
            assertEquals(4, TestToMock.intSum(3, 1))
        }
        describe("when an answer is provided") {
            it("uses the mock response") {
                every { TestToMock.intSum(any(), any()) } returns 100
                assertEquals(100, TestToMock.intSum(3, 1))
            }
        }
    }

    describe("using a java class with static functions") {
        it("executes the body") {
            assertEquals(4, TestStaticOnly.intSum(3, 1))
        }
        describe("when an answer is provided") {
            it("uses the mock response") {
                every { TestStaticOnly.intSum(any(), any()) } returns 100
                assertEquals(100, TestStaticOnly.intSum(3, 1))
            }
        }
    }
})

class RealTest : DescribeSpec({
    describe("using an already unmocked object") {
        it("executes the body") {
            assertEquals(4, TestToMock.intSum(3, 1))
        }
    }

    describe("using an already unmocked static function") {
        it("executes the body") {
            assertEquals(4, TestStaticOnly.intSum(3, 1))
        }
    }
})

Executing each spec on its own does work as expected.

Having both in the same package and executing the package (or whole test suite) will crash the second test (RealTest) which should use an unmocked state with this assertion:
java.lang.AssertionError: expected:<4> but was:<100>.
Note: It is important, that the RealTest is executed after the mocked + unmocked test for this crash to happen!

It seems there is a problem with mocking/unmocking objects and static functions in combination with lifecycle callbacks.

[testcenter]

FWIW: This seems to be a regression, as with 5.6.2 the tests execute as expected but break with 5.7.0.

[AlexCue987]

this is a well known problem with mocking static functions - it affects the whole JVM, not just one thread. so, whenever tests happen to run in parallel with this mocking and use the function being mocked, they will be broken.

IMO this has nothing to do with kotest. I can reproduce the issue with plain Kotlin. Pls correct me if I am wrong.

[krocard]

Our tests are not configured to run in parallel. Could it be that Kotest runs tests in parallel by default in the new version?

[OliverO2]

No, the default is sequential execution: https://kotest.io/docs/framework/project-config.html#parallelism

[AlexCue987]

we run kotest in gradle builds, and by default gradle runs different specs in parallel. So we can annotate RealTest with @DoNotParallelize and see if that helps. Alternatively, we can design away the need to mock static methods. Instead of the singleton object, can we create a normal class, an instance of which can be mocked via usual means? Then mocking one instance of that thing won't affect mocking anything other tests.

[krocard]

Getting rid of the static mocks would be the best solution, as it would also allow to run the spec in parallel.
Unfortunately it is a significant effort as legacy code makes heavy use of singleton objects and static factories.

by default gradle runs different specs in parallel

@AlexCue987 Do you have any source for this, the Kotest documentation (as linked by @OliverO2) specifies that by default it is not the case. Is gradle overriding kotest somehow?
If true, it would explain the issue.

[AlexCue987]

my understanding was that the following was being set under the hood: system property kotest.framework.parallelism
So can you:

• decorate your broken spec with DoNotParallelize annotation and see if it fixes tests?
• output the value of kotest.framework.parallelism from inside your spec?

[testcenter]

Adding the annotation DoNotParallelize does not change anything.
I am not 100% certain, if I printed the correct value, I used println(System.getProperty("kotest.framework.parallelism")) which prints null.

funny enough: for the example I provided, parallelism would fix the issue, as we would run into a race condition (tested with changing the parallelism to >1 via an AbstractProjectConfig object)

[AlexCue987]

Adding the annotation DoNotParallelize does not change anything.

Then I was wrong, and this issue is not about parallel execution. Let's investigate more.

Maybe:

• afterSpec did not run
• unmockkStatic or unmockkObject did not work

Has the following been invoked:

    afterSpec {
        unmockkObject(TestToMock)
        unmockkStatic(TestStaticOnly::class)
       printlin("I've been invoked")
        // add some more output to make sure it was really unmockked
    }

[testcenter]

Thanks for your input, Alex!

afterSpec is called correctly (got some print output).
Also I added a print statement before the first test of RealTest and it seems the order of execution is correct 👍

I still suspect the problem to be a kotest issue, as when executing as normal junit test, it works. here the snippet I used for testing:
(Note: I added the Order annotation, to create the same order as with the kotest spec)

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
class MockedTestJunit {

    @Test
    @Order(1)
    fun usingAMockedObject() {
        Assertions.assertEquals(4, TestToMock.intSum(3, 1))
    }

    @Test
    @Order(2)
    fun whenAnswerProvidedObject() {
        every { TestToMock.intSum(any(), any()) } returns 100
        Assertions.assertEquals(100, TestToMock.intSum(3, 1))
    }

    @Test
    @Order(3)
    fun usingAMockedStaticFun() {
        Assertions.assertEquals(4, TestStaticOnly.intSum(3, 1))
    }

    @Test
    @Order(4)
    fun whenAnswerProvidedStaticFun() {
        every { TestStaticOnly.intSum(any(), any()) } returns 100
        Assertions.assertEquals(100, TestStaticOnly.intSum(3, 1))
    }

    companion object {
        @JvmStatic
        @BeforeAll
        fun doMock() {
            mockkObject(TestToMock)
            mockkStatic(TestStaticOnly::class)
            println("Mocked")
        }

        @JvmStatic
        @AfterAll
        fun unmock() {
            unmockkObject(TestToMock)
            unmockkStatic(TestStaticOnly::class)
            println("Unmocked")
        }
    }
}

and

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
class RealTestJunit {

    @Test
    @Order(10)
    fun usingAnAlreadyUnmockedObject() {
        Assertions.assertEquals(4, TestToMock.intSum(3, 1))
    }

    @Test
    @Order(11)
    fun usingAnAlreadyUnmockedStaticFun() {
        Assertions.assertEquals(4, TestStaticOnly.intSum(3, 1))
    }
}

[AlexCue987]

returning to "Adding the annotation DoNotParallelize does not change anything" - I would expect that to fix the issue. This is where we should investigate further.

[testcenter]

The default behavior is sequencial, as Oliver mentioned here: #3962 (comment)
So the annotation shouldn't have any effect, or am I missing something? 🤔

[LegoRocks]

I'm having pretty much the same issue. I also don't run tests in parallel (I did not set the parallelism property in the project config and the system property kotest.framework.parallelism is not set as well).
However adding @ DoNotParallelize to the Spec that does the mocking does indeed solve the issue.
Another thing I noticed is, that using afterTest instead of afterSpec also "fixes" this issue though it's not a clean solution.

I'm definitly intereseted in the solution for this problem.