TDD mode only works on simulator

[oryonatan]

Hi!

I have been trying to get TDD mode working on a physical device (some of our tests require MetalPerformanceShaders and cannot run on simulator).

This didn't work for me ... but the same tests do work on simulator

I added a file Foo.swift

import Foundation

class Foo {
 static func bar() -> Int { 6 }
}

and a test file on another target FooSpec.swift

@testable import SwiftUI_Kit_iOS
import XCTest

final class FooSpec: XCTestCase {
  func testExample() throws {
    XCTAssert(Foo.bar() == 6)
  }
}

and this is the log I am getting when re-saving Foo.swift

💉 Connecting to MBP (169.254.133.190)...
💉 InjectionIII connected /Users/user/SwiftUI-Kit/SwiftUI Kit.xcodeproj
💉 Watching files under the directory /Users/user/SwiftUI-Kit
💉 Compiling /Users/user/SwiftUI-Kit/Shared/Foo.swift
💉 Compiling /Users/user/SwiftUI-Kit/SwiftUI Kit iOSTests/FooSpec.swift
💉 Loading .dylib ...
objc[1705]: Class _TtC15SwiftUI_Kit_iOS3Foo is implemented in both /private/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/SwiftUI Kit iOS (0x104d9aa08) and /private/var/mobile/Containers/Data/Application/1FD7A14A-049A-4760-8DAE-06275590C82B/tmp/eval101.dylib (0x1122243a0). One of the two will be used. Which one is undefined.
💉 Loaded .dylib - Ignore any duplicate class warning ⬆️
💉 Injected type #1 'SwiftUI_Kit_iOS.Foo'
💉 ⚠️ Linking failed (see: /var/folders/8p/wmd1m5b90l1fz661_n52_fgh0000gq/T/command.sh)
clang: error: no such file or directory: '/Users/user/Library/Developer/Xcode/DerivedData/SwiftUI_Kit-bivomvtiddrttdamkikxglswesil/Logs/Build/../../Build/Products/Debug-*/Quick*.o'
clang: error: no such file or directory: '/Users/user/Library/Developer/Xcode/DerivedData/SwiftUI_Kit-bivomvtiddrttdamkikxglswesil/Logs/Build/../../Build/Products/Debug-*/Nimble.o'
clang: error: no such file or directory: '/Users/user/Library/Developer/Xcode/DerivedData/SwiftUI_Kit-bivomvtiddrttdamkikxglswesil/Logs/Build/../../Build/Products/Debug-*/Cwl*.o'

💉 Loading .dylib ...
Error loading /var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/PlugIns/SwiftUI Kit iOSTests.xctest/SwiftUI Kit iOSTests (148):  dlopen(/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/PlugIns/SwiftUI Kit iOSTests.xctest/SwiftUI Kit iOSTests, 0x0109): Library not loaded: @rpath/XCTest.framework/XCTest
  Referenced from: <2143341B-3453-3BF3-AC51-BD9AA2753D76> /private/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/PlugIns/SwiftUI Kit iOSTests.xctest/SwiftUI Kit iOSTests
  Reason: tried: '/usr/lib/swift/XCTest.framework/XCTest' (no such file, not in dyld cache), '/private/preboot/Cryptexes/OS/usr/lib/swift/XCTest.framework/XCTest' (no such file), '/private/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/PlugIns/SwiftUI Kit iOSTests.xctest/Frameworks/XCTest.framework/XCTest' (no such file), '/private/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/Frameworks/XCTest.framework/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/XCTest' (no such file, not in dyld cache), '/private/preboot/Cryptexes/OS/usr/lib/swift/XCTest.framework/XCTest' (no such file), '/private/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/PlugIns/SwiftUI Kit iOSTests.xctest/Frameworks/XCTest.framework/XCTest' (no such file), '/private/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/Frameworks/XCTest.framework/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/XCTest' (no such file, not in dyld cache), '/private/preboot/Cryptexes/OS/usr/lib/swift/XCTest.framework/XCTest' (no such file), '/private/var/containers/Bundle/Application/CFB268B0-4591-43B9-A52A-DEB03F82DF44/SwiftUI Kit iOS.app/Frameworks/XCTest.framework/XCTest' (no such file)
💉 ⚠️ dlopen() error: dlopen(/private/var/mobile/Containers/Data/Application/1FD7A14A-049A-4760-8DAE-06275590C82B/tmp/eval102.dylib, 0x0002): symbol not found in flat namespace '_OBJC_CLASS_$_XCTestCase'
💉 ⚠️ Loading .dylib has failed, This is likely because Swift code being injected references a function using a default argument or a member with access control that is too restrictive or perhaps an XCTest that depends on code not normally linked into your application. Rebuilding and re-running your project (without a build clean) can resolve this.
💉 ⚠️ Injection error: Error Domain=SwiftEval Code=-1 "dlopen() error: dlopen(/private/var/mobile/Containers/Data/Application/1FD7A14A-049A-4760-8DAE-06275590C82B/tmp/eval102.dylib, 0x0002): symbol not found in flat namespace '_OBJC_CLASS_$_XCTestCase'
💉 ⚠️ Loading .dylib has failed, This is likely because Swift code being injected references a function using a default argument or a member with access control that is too restrictive or perhaps an XCTest that depends on code not normally linked into your application. Rebuilding and re-running your project (without a build clean) can resolve this." UserInfo={NSLocalizedDescription=dlopen() error: dlopen(/private/var/mobile/Containers/Data/Application/1FD7A14A-049A-4760-8DAE-06275590C82B/tmp/eval102.dylib, 0x0002): symbol not found in flat namespace '_OBJC_CLASS_$_XCTestCase'
💉 ⚠️ Loading .dylib has failed, This is likely because Swift code being injected references a function using a default argument or a member with access control that is too restrictive or perhaps an XCTest that depends on code not normally linked into your application. Rebuilding and re-running your project (without a build clean) can resolve this.}
💉 Compiling /Users/user/SwiftUI-Kit/SwiftUI Kit iOSTests/FooSpec.swift
💉 ⚠️ Linking failed (see: /var/folders/8p/wmd1m5b90l1fz661_n52_fgh0000gq/T/command.sh)
clang: error: no such file or directory: '/Users/user/Library/Developer/Xcode/DerivedData/SwiftUI_Kit-bivomvtiddrttdamkikxglswesil/Logs/Build/../../Build/Products/Debug-*/Quick*.o'
clang: error: no such file or directory: '/Users/user/Library/Developer/Xcode/DerivedData/SwiftUI_Kit-bivomvtiddrttdamkikxglswesil/Logs/Build/../../Build/Products/Debug-*/Nimble.o'
clang: error: no such file or directory: '/Users/user/Library/Developer/Xcode/DerivedData/SwiftUI_Kit-bivomvtiddrttdamkikxglswesil/Logs/Build/../../Build/Products/Debug-*/Cwl*.o'

💉 Loading .dylib ...
💉 ⚠️ dlopen() error: dlopen(/private/var/mobile/Containers/Data/Application/1FD7A14A-049A-4760-8DAE-06275590C82B/tmp/eval103.dylib, 0x0002): symbol not found in flat namespace '_OBJC_CLASS_$_XCTestCase'
💉 ⚠️ Loading .dylib has failed, This is likely because Swift code being injected references a function using a default argument or a member with access control that is too restrictive or perhaps an XCTest that depends on code not normally linked into your application. Rebuilding and re-running your project (without a build clean) can resolve this.
💉 ⚠️ Injection error: Error Domain=SwiftEval Code=-1 "dlopen() error: dlopen(/private/var/mobile/Containers/Data/Application/1FD7A14A-049A-4760-8DAE-06275590C82B/tmp/eval103.dylib, 0x0002): symbol not found in flat namespace '_OBJC_CLASS_$_XCTestCase'
💉 ⚠️ Loading .dylib has failed, This is likely because Swift code being injected references a function using a default argument or a member with access control that is too restrictive or perhaps an XCTest that depends on code not normally linked into your application. Rebuilding and re-running your project (without a build clean) can resolve this." UserInfo={NSLocalizedDescription=dlopen() error: dlopen(/private/var/mobile/Containers/Data/Application/1FD7A14A-049A-4760-8DAE-06275590C82B/tmp/eval103.dylib, 0x0002): symbol not found in flat namespace '_OBJC_CLASS_$_XCTestCase'
💉 ⚠️ Loading .dylib has failed, This is likely because Swift code being injected references a function using a default argument or a member with access control that is too restrictive or perhaps an XCTest that depends on code not normally linked into your application. Rebuilding and re-running your project (without a build clean) can resolve this.}

trying to do the same on simulator - seems to work fine

[oryonatan]

similar errors are occurring when trying to use it on a My Mac (Designed for iPad) target

[johnno1962]

I'd give up on TDD for Nimble tests. I don't think it will work, certainly not on a device.

[johnno1962]

Have you run the tests previously on the device before you tried to inject it?

[johnno1962]

If you're wondering where the talk about Nimble comes from where a test file contains Spec. it triggers special processing. There is a different problem where XCTestCase needs to come from XCTestCore in the app bundle which would take some time to work out. If you have the time to prepare a small example project with a renamed test file that tests by injection in the simulator but not on a device that would help but I am going to be away from my computer until Jan 6th. In the meantime, see the original nimble issue: #387

[oryonatan]

Yes, I did run the tests on the device before trying to run the app.
Doing the same process on simulator worked well and I got TDD mode running … the only issue is when running on Apple Silicon / iPhone.

I’ll create a small demo project and send it here … also I will try renaming the file to something else the FooSpec, and see if it gives different errors / works.

[johnno1962]

So, basically, injecting tests never worked on a device (I don't think anyone has ever asked for it) but I've pushed a new release candidate where they can when you're using a new version of the copy_bundle.sh script. I appreciate your testing the furtherest reaches of injectionIII and raising issues even if your timing isn't great. Let me know how you get on with the new version; I think even TDD mode works!

[oryonatan]

Hi! sorry for the late response, I've been playing with the RC you've sent and encountered a couple of issues:

1. We are loading Nimble as .framework so I had to change the code in SwiftEval.swift credits goes to Ben

 static let quickFiles = getenv("INJECTION_QUICK_FILES").flatMap {
        String(cString: $0) } ?? "Debug-*/{Quick*,Nimble,Cwl*}.o"

to

    static let quickFiles = getenv("INJECTION_QUICK_FILES").flatMap {
        String(cString: $0) } ?? "Debug-*/Nimble.framework/Nimble"

2. Even after the change, it seems like the code that looks for Nimble.framework always tried finding it in the Debug-iphonesimulator folder instead of Debug-iphoneos.

Fixing this two issues allowed me to run tests in TDD Mode, but it seems like I keep getting NSInvalidArgumentException from pretty much every test ... I am trying to find out why.

[oryonatan]

The issue seems to be coming from the [suite0 performTest:tr] in ClinetBoot.mm
I'm going to check on a smaller project and see if I can reproduce the same issue

[johnno1962]

Probably passing a nil pointer for the test. If you're using a Nimble.framework you'll need to copy it into your app bundle somewhere or in the injection bundle if it's not already there and load it in lazy var loadXCTest...

[johnno1962]

Any progress? Linking with nimble is a bit tricky and you need to link with the -framework Nimble switch and then you need to dynamically load it before you try to load you client library.

[johnno1962]

If you're wondering what's happened to the repo, what started out as "just a couple of changes" for 4.8.1 has mushroomed a bit so I've rebased the recent commits on main onto a new PR branch "codesigning" and deleted them.