Test filtering does not work with Quick

[modocache]

Someone with Xcode 8 installed should try the new xcodebuild test filters with Quick. Does everything work as expected?

xcodebuild test -only-testing:MyTestBundle/MyTestSuite/MySpec
xcodebuild test -only-testing:MyTestBundle/MyTestSuite
xcodebuild test -only-testing:MyTestBundle

xcodebuild test -skip-testing:MyOtherTestBundle/MyTestSuite/MySpec

[karapigeon]

I can't install betas on my main work partition, but I'm going to setup a partition with macOS and the beta Xcode to test soon.

[modocache]

Awesome, thanks @istx25! Let me know how it goes.

[omnianoureldin]

Hello, I was wondering if there is any progress here. I tried the filtering with xCode 8 and it doesn't seem to work.

[karapigeon]

I still haven't had the chance to get around to this. My Mac was acting up around this and then I forgot. So no progress yet. Sorry!

[MP0w]

I tried it out:

-only-testing:TEST_BUNDLE/MODULE_NAME.SPEC_CLASS_NAME works
-only-testing:TEST_BUNDLE/MODULE_NAME.SPEC_CLASS_NAME/TEST_NAME didn't work
however I was not able to "only test" a single test even for regular (XC)tests. Looks like whatever I add after ...CLASS_NAME/ doesn't matter

[kmahar]

Related to this, is there anyway to use the --filter arg to swift test to only run some tests by name? I have test suites broken up by name (ex. ClientTests), and when ClientTests was an XCTestCase I could pass in --filter ClientTests to just run that group quickly. Now that I've changed ClientTests to be a QuickSpec no tests are run at all when I pass in that option.

[phatblat]

I played around with this using swift 5.2.2 and the full test names can be found using the --list-tests option.

↪ swift test --list-tests
PinKitTests.PostSpec/post__has_a_description
PinKitTests.PostSpec/post__has_an_extended
PinKitTests.PostSpec/post__has_a_hash
PinKitTests.PostSpec/post__has_a_description_2
PinKitTests.PostSpec/post__has_a_meta
PinKitTests.PostSpec/post__is_shared
PinKitTests.PostSpec/post__has_tags
PinKitTests.PostSpec/post__has_a_time
PinKitTests.PostSpec/post__has_a_toread_flag
PinKitTests.PostsAddSpec/posts_add__has_a_result_code
PinKitTests.PostsAddSpec/posts_add__parsing__can_be_parsed_from_json
PinKitTests.PostsDeleteSpec/posts_delete__has_a_result_code
PinKitTests.PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
phatblat at octodec in ~/dev/swift/PinKit on master [✓]

Providing the full test name (or just the ) to the --filter option seems to do less work because there is a lot less output than when running without the filter. However it doesn't appear to actually run any tests due to the "Executed 0 tests" output.

↪ swift test --filter PinKitTests.PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
Test Suite 'Selected tests' started at 2020-06-09 13:28:41.011
Test Suite 'PinKitPackageTests.xctest' started at 2020-06-09 13:28:41.012
Test Suite 'PinKitPackageTests.xctest' passed at 2020-06-09 13:28:41.012.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2020-06-09 13:28:41.012.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds

xcodebuild (v11.5) has the same issue when running a subset of tests. It runs a test suite called 'Selected tests' and the summary shows it "Executed 0 tests".

↪ xcodebuild test -workspace PinKit.xcworkspace -scheme PinKit-Package -only-testing:PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
Command line invocation:
    /Applications/Xcode-11.5.app/Contents/Developer/usr/bin/xcodebuild test -workspace PinKit.xcworkspace -scheme PinKit-Package "-only-testing:PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json"

User defaults from command line:
    IDETestRunOnlyIdentifiers = (
    "PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json"
)

...
Test Suite 'Selected tests' started at 2020-06-09 13:38:47.655
Test Suite 'PinKitTests.xctest' started at 2020-06-09 13:38:47.656
Test Suite 'PinKitTests.xctest' passed at 2020-06-09 13:38:47.656.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2020-06-09 13:38:47.656.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
...

Test session results, code coverage, and logs:
	/Volumes/DerivedData/PinKit-akgpefidgquoryhfyxurznnvcfrr/Logs/Test/Test-PinKit-Package-2020.06.09_13-38-46--0600.xcresult

** TEST SUCCEEDED ** [1.737 sec]

Something in the swift test command is able to see the individual Quick tests. I wonder if that mechanism can be used to help expose the filtered set of "Selected tests" from Quick to the XCTest runtime.

[jessesquires]

Hey there! 👋🏼

I'm a new maintainer for this project and I'm trying to get the next release out ASAP and also clean up old issues and old PRs. I'm closing all issues that no longer seem relevant or are very old / stale.

This does not mean this issue is necessarily being rejected. If you are still interested in contributing the changes specified in this issue, please comment to request it be reopened.

We appreciate you opening this issue and acknowledge the time and effort you put in to contribute! You're awesome. 💯 However, we are all volunteers here with limited capacity working for free. Unfortunately, that means we must close out stale issues and PRs to move the project forward in a reasonable way.

We apologize for the inconvenience and appreciate your understanding! 😎 ✌🏼

[natinusala]

If it adds anything, the issue is not stale or outdated: I confirm that filtering still doesn't work on Swift 5.6 on Linux (I don't know the associated XCTest version) using swift test --filter. It shows the exact same behavior as outlined in the previous comments.

[natinusala]

@jessesquires @phatblat I found out what the issue is, it may only happen on Linux, I don't have access to a Mac computer to test on macOS.

Basically XCTest takes one argument for the list of tests to execute (which is what SwiftPM --filter uses when invoking your test binary).

The thing is, it's not one argument = one test case, it's one argument for all selected test cases, separated by a comma 🙃

So when you have a Quick test that's called Quick.QuickSpec/my thing, when doing this, also does that, XCTest sees "run 3 tests: "Quick.QuickSpec/my thing", "when doing this", "also does that".

The incriminating code is here: https://github.com/apple/swift-corelibs-xctest/blob/swift-5.7-DEVELOPMENT-SNAPSHOT-2022-06-26-a/Sources/XCTest/Private/ArgumentParser.swift#L69

When removing the split call it fixes the issue for me.

I'm not too sure where to go from there, I guess the real fix would be in XCTest to add quotes or whatever, but it's useless if SwiftPM does not also add quotes when using --filter... there is also the macOS version of XCTest to consider that I cannot make a PR to.

What do you recommend? Should I make an XCTest issue? I guess it's normal that commas are not handled properly since test functions are supposed to come from actual function names, but since XCTest allows making them dynamically (through allTestCases) I guess it should be able to handle any string.

In case the answer is "commas are not legal in test cases names", should we consider a workaround in Quick as well?

[younata]

Bug filed with the open source XCTest: apple/swift-corelibs-xctest#458

I also filed FB12489187 for the closed-source version, with contents:

Hello! I maintain the Quick testing framework for Swift and Objective-C. Amongst other things, Quick allows you to use any arbitrary strings as the name of a test. For example, the following code would generate a test named, using xcodebuild test’s filtering syntax, `"SomeSpec"/"can even include /s":

class SomeSpec: QuickSpec {
    override class func spec() {
        it("can even include /s") {
            // the actual test to run
        }
    }
}

In a recent WWDC lab with XCTest engineers, one of them mentioned that allowing slashes in the test name could potentially cause issues with filtering tests. While this isn't an issue when selecting tests to run using test gems, it is an issue if use the -only-testing or -skip-testing options in XCTest. That is, when you try to filter test names that contain slashes, xcodebuild test -only-testing will act as if no test filtering were specified.

This was tested using Xcode 15, beta 2 and Xcode 14.3.1 as in the Mac App Store.

Thanks!

I'll leave this open until it's fixed.

[jasonm23]

Tested with XCode 14.2, -only-testing, -skip-testing work at suite level, not at individual test case level.

Same is true in the XCode UI test navigator.

[bartekpacia]

I understand that -only-testing and skip-testing do not currently work with Quick. But is it a Quick limitation or a more fundamental bug/missing feature in XCTest itself? I asked a question on StackOverflow about this, maybe someone knows the answer 🙏🏻

[younata]

@bartekpacia it's both. As noted above, the swiftpm and xcodebuild tooling assumes that test names are comma-separated, which conflicts with the fact that Quick (as of version 7) uses commas to separate example groups in a single test. You can disable this by setting the QUICK_USE_ENCODED_TEST_SELECTOR_NAMES environment variable, which reverts to the earlier behavior of encoding test selector names (replacing non-alphanumeric characters with _s). I haven't checked if using encoded test selectors fixes the issue, but it's at least a contributing factor.

Additionally, when invoked through the command line, swiftpm and xcodebuild both assume that / characters are used to delimit a test class and a test method. Which is not a valid assumption when used with Quick's unencoded test selectors.

I expect these to get fixed, as the new swift-testing package also allows arbitrary test names. As swift-testing matures, I expect the issues that the swiftpm and xcodebuild CLIs have with Quick's unencoded test selectors to go away.

I also fully expect there to be other issues, not related to unencoded test selectors. The reason I bring that up is because unencoded test selectors are a very new feature of Quick. But, I also haven't done any (recent, at least) research on what exactly the causes of this are.

[bartekpacia]

Thanks for answering @younata!

Actually I was asking becuase I was just wondering if it is possible to make -only-testing work with tests generated dynamically at runtime using XCTestCase.testInvocations – and I learned that Quick is doing that.

I experimented myself with the following test:

import Quick
import XCTest

class QuickExampleUITests: QuickSpec {
  override class func spec() {
    it("testA") {
      NSLog("Executed test A")
      XCTAssertTrue(true)
    }

    it("testB") {
      NSLog("Executed test B")
      XCTAssertTrue(true)
    }
  }
}

That test can be executed with:

xcodebuild test \
  -scheme Example \
  -only-testing 'ExampleUITests/QuickExampleUITests/testA' \
  -configuration Debug \
  -sdk iphoneos -destination 'platform=iOS Simulator,name=iPhone 15'

and it works just fine! So the root source of the problem is indeed Quick using , when concatenating describe() and it().