Test results intermittently disappear in the Xcode 7 UI

[nsforge]

In my project, tests that use XCTestCase directly always show a green or red indicator in the editor gutter UI after running tests, but Specta tests intermittently show an empty grey diamond instead of a green or red diamond. When they disappear in the editor UI, they also disappear from the Issues Navigator.

The Specta test results always show up correctly if I go to the Report Navigator and look at the test results log.

I'm running Xcode 7.0.1 and Specta 1.0.4.

Is this a known issue?

[fatuhoku]

@nsforge I've seen this happen as well. Very very confusing and makes it almost impossible to tell what happened to the tests.

[wsb9]

Adding yet another interesting example: failing test cases get disappeared from Navigator. When I noticed, decided to make small test and inserted XCTFail() call into passing test case. After run, changed case gone away with suite failed and no cue what went wrong.

Xcode 7.2 (7C68), Specta 1.0.5.

[valnoc]

i also get failed tests disappeared from navigator
Any info on "how to fix"?

[kkotar]

Xcode Version 7.3 (7D175) same problems :(

[stephen-francis]

My team has experienced the tests "disappearing" as well, and it makes evaluating results very difficult. Has there been any movement here?

[orta]

The same thing happens in Quick, no-one has figured it out I'm afraid. You're welcome to try dig, but I think you'll need to be looking in Xcode.

[stephen-francis]

Thanks for the quick response @orta, I appreciate it. Has a bug report been filed on this from the Xcode perspective?

[orta]

I haven't raised one, happy to have others do it 👍

[stephen-francis]

@orta: I filed a radar on this: Bug ID 25679988: Test cases disappear from the Test Navigator, on 12 April. Apple responded with: "It seems this issue only manifests when using the Specta framework. Can the developers of Specta resolve this issue?"

In response, on 20 April, I informed them that this issue also manifests itself in Quick, and that we have reason to believe it's an Xcode issue. I haven't gotten another response since.

[modocache]

The issue would manifest with any consumer of +[XCTestCase testInvocations], in which a different set of tests was generated between two test runs.

It would be great to create a sample project that demonstrates this. We could then encourage commenters on this, Quick/Quick#219, and Quick/Quick#427 to "dupe" a master Radar. It's about time we try to effectively communicate what we need to Apple.

No promises, but I'll try to get to this sometime this weekend. If someone beats me to it, I'd be much obliged. ❤️

[stephen-francis]

Thanks for checking in, @modocache, that'd be great.

Do you or @orta happen to know if there's a workaround here? For instance, if we were to run these tests from the command line instead of the Xcode interface, could we avoid this issue?

[modocache]

Oh, hmm, good question... I would assume that running the tests via the command line would not affect the test navigator, so yes, that does seem like a reasonable workaround.

[modocache]

If you use Specta or Quick and are disappointed with tests disappearing in Xcode's Test Navigator, tests lingering in the Test Navigator despite no longer being defined, or even tests not being run, please read and duplicate this radar: https://openradar.appspot.com/26028557.

Xcode does not provide the Specta or Quick maintainers with an API to address these problems. The best way to contribute to fixing these issues is to communicate to the Apple XCTest team that this is a use case we wish they'd support better. The best way I know to do that is to ask you all to re-submit https://openradar.appspot.com/26028557. 🙇

[orta]

Duped 👍

[modocache]

Thanks! Every little bit counts. 💯

[StatusReport]

Great work! Duped as well.

[StatusReport]

I went to the Xcode lab in the WWDC today and found the guy who's responsible for fixing this. I showed him the radar and he was aware of it, but besides raising the awareness for the amount of people that are affected by this bug, I couldn't do much. Let's hope this will get fixed in Xcode 8 GM.

[modocache]

@StatusReport That's awesome, thanks!! 💓 Raising awareness is very valuable -- if more people file radars or mention this to Apple engineers, the problem is more likely to be addressed. Every radar dupe counts.