Sporadic results (XCTest) when running simultaneous tests

[bredfield]

Using Xcode 6.1, Swift, XCTest, I've got Nocilla working nicely when tests are run individually. As soon as I try to run multiple tests that use Nocilla stubs, they tests that previously succeeded on their own won't work.

For example, the following test works well on its own:

func testUserLoginFailure404() {
        let expectation = expectationWithDescription("Login Failure")
        let testEmail = "loginfailure@email.com"
        let testPassword = "loginFailure"
        let testToken = "loginFailureToken"
        let loginURLString = BRServer.baseURLString + BRServer.Login(["email":testEmail, "password": testPassword]).fullURL.path! + "/"

        stubRequest("POST", loginURLString)
            .withBody("password=\(testPassword)&grant_type=password&username=\(testEmail)")
            .andReturn(404)
            .withBody("{}")

        BRSession.createSessionFromLogin((email: testEmail, password:testPassword), success: { (token) -> Void in
            XCTFail("Login should not have succeeded")
            expectation.fulfill()
        }) { (statusCode, response) -> Void in
                XCTAssert(statusCode == 404, "should 404")
                expectation.fulfill()
        }

        waitForExpectationsWithTimeout(10.0, handler: nil)
}

However, when run alongside other session tests in the same format, the expectation will sporadically never be met (success/fail blocks aren't executed).

I've made sure that I'm using a separate request URL for each test, so that stubs don't interfere with each other in that way. Any ideas as to why it's not working?

Note: Here's what I'm doing in setUp and tearDown:

    override func setUp() {
        super.setUp()

        LSNocilla.sharedInstance().start()
    }

    override func tearDown() {
        LSNocilla.sharedInstance().clearStubs()
        super.tearDown()
    }

Thanks!

[subbarao]

@bredfield did you find fix for your issues. I am also experiencing similar issues.

[bredfield]

@subbarao I haven't been able to get stable results with Nocilla. I've been using Mockingjay, which has worked well.

[paudav]

Hi @bredfield,

TL;DR: This is probably caused by the exception thrown here. When this gets thrown, subsequent requests in the same test run to correctly stubbed URLs seem to fail.

I'm working in Objective-C, but I've run into a similar issue. Since you've changed test suites, this probably isn't a big deal for you, but I'll leave this post here for anybody else who is seeing inconsistent results during test runs.

In my case, our class under test is using NSURLConnection with delegate callbacks directed to an NSOperationQueue.

As described in issue #103, the exception thrown at https://github.com/luisobo/Nocilla/blob/master/Nocilla/LSNocilla.m#L79 basically wrecks all of the tests that run after the exception is thrown.

In Xcode, the tests seem to run in alphabetical order of the test method names--Depending on the order of tests your XCTest class implementation, this could make things appear to be "randomly" failing. (For fun, reorder the test methods in your file alphabetically, and see if a more clear failure pattern appears).

In our case, the test that caused the exception was intentionally hitting a non-stubbed URL in order to test behavior with non-responsive requests. By design, this test succeeded after checking some queuing behaviors related to a bunch of open requests (to non-stubbed URLs). Inside Nocilla, of course, this throws an exception that we never see, and subsequent tests with correctly stubbed requests fail.

The result, of course, is that if you run each of the tests individually, they pass. If you run them all together, the non-stubbed request in the middle of the test run breaks every subsequent test.

You can detect this condition by setting an exception breakpoint. If you try this and it breaks into LSNocilla.m during the test run, you've sent a request to a non-stubbed URL.

Good Luck,
Paul B. Davis

[TheAdamBorek]

Hi guys,
has anyone resolved this issue ?

[TheAdamBorek]

As @paudav mention above it was connected with exception raising. There is an pull request. Error is returned instead of raising an exception