beforeSuite seems to run always, indecent of focus

[angerman]

Say we have

class ASpec : QuickSpec {
   override func spec {
     beforeSuite { doSomething() }
     describe(...) { ... }
   }
}

and

class BSpec : QuickSpec {
   override func spec {
     describe(...) {
        ...
        fit(...) { ... }
        ...
      }
   }
}

doSomething() is called, even though it's independent of the tests to be executed only.

[modocache]

Ooh, very good point--thanks for bringing this up @angerman! 👍

I'm actually not sure which would be less surprising to users:

1. "My before suites aren't running! Why?" (an hour of debugging later) "What?! Does focusing on an example cause only beforeSuite closures in that example's hierarchy to run?"
2. "I've focused on my example, but beforeSuite closures from other spec classes are still running?!"

I've always thought of beforeSuite and afterSuite to be for setting up and tearing down truly global state. Personally, I'd expect them to run no matter which example was focused.

I'm open to other viewpoints, though! What do you think?

[angerman]

Apart from beforeSuite and afterSuite potentially being expensive and hence might only be wanted on a demand basis, the primary reason this behavior was completely unexpected was due to the perceived scoping.

As I understand it, beforeSuite and afterSuite within a spec, are related to that spec, and that spec only; yet for some reason they are overreaching outside of the spec and class they are defined in.

I never read beforeSuite as beforeTests. Maybe there is some before/after layer missing? I might want to run some setup for my spec, but not for each test, yet that setup might not be required for orthogonal specs (which is why I group my orthogonal testing into different specs in the first place).

As far as I see, I can not not use beforeSuite as beforeEach does not work on a context or description level. At the same time I can not limit beforeSuite into the test direction, unless I start to have multiple TestTargets.

[modocache]

Hmm, you make a good point. As long as beforeSuite and afterSuite appear within the context of a single spec class, it makes sense to associate them with a single spec.

I'd say the solution here would be:

1. Ensure beforeSuite and afterSuite only run if the spec they belong to is not completely filtered out--that is, at least one of the examples in that spec will run.
2. Allow truly global beforeSuite and afterSuite closures to be specified in QuickConfiguration subclasses (this might already be possible). That way, users that want some code to run no matter what's filtered out can place their beforeSuite and afterSuite closures within configuration objects.

@angerman, would you be interested in implementing this behavior? No worries if not, I'm just curious if you'd like to take this opportunity to contribute! I'd be happy to give any pointers you may need. ⛵

[angerman]

That sounds like a great idea!

@modocache I think. I have to decline on implementing it, as I would not be able to commit to it in timely manner :-(

[modocache]

@angerman No worries! If you ever have some spare time and this issue is still open, feel free to comment here. 😄

[karapigeon]

Hey @angerman! Would you be available to implement this behaviour now?

[haitaoli]

I'm trying to create some UI automation with Quick and I found it's necessary to have beforeAll and afterAll blocks because some setup jobs take a long time. I agree beforeSuites and afterSuites can cause confusions because one test spec can affect all others. I looked into the code. It's not an easy task to add this feature. I tried adding two special Examples. But XCTestCase runs in random order even with QuickSpec overriding testInvocations. Another challenge is to make afterAll blocks run even if other tests fail. Another possible solution is to have an extra beforeEach and afterEach that will keep the count and only call beforeAll and afterAll once. It needs to know if all examples have finished in the current group, or if one has failed and continueAfterFailure is false.

Regardless of which solution would work, for UI tests I'd prefer all test cases to run in the order they are defined. So I can divide a workflow into multiple pieces with meaningful descriptions, instead of piling them up in one huge it.

Any other ideas worth trying?

[modocache]

When I need my XCTestCase subclasses to run in a specific order, I name them like so:

Foo_00_TestCase
Foo_01_TestCase
Foo_02_TestCase

This ensures that Foo_02_TestCase runs after Foo_00_TestCase. It's hacky, but it works, because test cases are run in alphabetical order.

[haitaoli]

@modocache does this still work with latest XCode? In my test the order is random, not alphabetically any more.

[jeffh]

IIRC, Xcode randomizes the order unless a failure occurs, then it seems to have some (mostly) constant order. Thanks, Jeff

…

Sent from my iPhone

On May 3, 2017, at 11:41 AM, Haitao Li ***@***.***> wrote: @modocache does this still work with latest XCode? In my test the order is random, not alphabetically any more. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[modocache]

Wow, is that a recent change? I could have sworn this used to work.

[haitaoli]

I believe it's a recent change.