AR camera freeze when returning from background on iOS12

[pichirichi]

What's happening?
AR camera will freeze when app is returning from background with an ongoing AR session.

Device tested: iPhone XS Max running iOS12.4

It seems like a deadlock on the “SCNTransaction.commit()” command in LocationNode.swift

How to reproduce:

1. search address in the sample code
2. click on navigate to the address
3. move app to background
4. launch a different app
5. relaunch the sample code
6. app will freeze.

Log messages:

pause
2019-12-01 10:23:38.290421+0200 ARKit+CoreLocation[51187:3346340] [Snapshotting] Snapshotting a view (0x107803a00, UIKeyboardImpl) that is not in a visible window requires afterScreenUpdates:YES.
run
camera did change tracking state: limited, initializing
Camera: <ARCamera: 0x2801c07e0 imageResolution=(1920, 1440) focalLength=(1437.988, 1437.988) principalPoint=(960.241, 715.517) trackingState=Limited-Initializing transform=<translation=(-0.011208 0.063010 -0.039065) rotation=(-5.13° -142.23° -90.69°)>>

Full stack:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00000001984e9018 libsystem_kernel.dylib`__psynch_rw_wrlock + 8
    frame #1: 0x0000000198566784 libsystem_pthread.dylib`_pthread_rwlock_lock_wait + 84
    frame #2: 0x00000001acd89200 SceneKit`C3DTransactionFlush + 328
    frame #3: 0x00000001acd89dac SceneKit`C3DTransactionCommit + 364
    frame #4: 0x000000010176caf8 ARCL`LocationNode.updatePositionAndScale(setup=false, scenePosition=some, nodeLocation=0x0000000283afe4a0, locationManager=0x0000000282d003c0, onCompletion=0x0000000101789240 ARCL`partial apply forwarder for closure #1 () -> () in closure #1 (ARCL.LocationNode) -> () in closure #2 (ARCL.PolylineNode) -> () in ARCL.SceneLocationView.updatePositionAndScaleOfLocationNodes() -> () at <compiler-generated>, self=0x0000000101d94950) at LocationNode.swift:174:24
    frame #5: 0x0000000101786840 ARCL`closure #1 in closure #2 in SceneLocationView.updatePositionAndScaleOfLocationNodes(node=0x0000000101d94950, self=0x000000010802ca00) at SceneLocationView.swift:465:10
    frame #6: 0x000000010177c3d8 ARCL`thunk for @callee_guaranteed (@guaranteed LocationNode) -> (@error @owned Error) at <compiler-generated>:0
    frame #7: 0x0000000101789234 ARCL`thunk for @callee_guaranteed (@guaranteed LocationNode) -> (@error @owned Error)partial apply at <compiler-generated>:0
    frame #8: 0x00000001c7083124 libswiftCore.dylib`(extension in Swift):Swift.Sequence.forEach((A.Element) throws -> ()) throws -> () + 528
    frame #9: 0x0000000101786640 ARCL`closure #2 in SceneLocationView.updatePositionAndScaleOfLocationNodes(node=0x0000000101d913e0, self=0x000000010802ca00) at SceneLocationView.swift:463:23
    frame #10: 0x00000001017847d0 ARCL`thunk for @callee_guaranteed (@guaranteed PolylineNode) -> (@error @owned Error) at <compiler-generated>:0
    frame #11: 0x000000010178919c ARCL`thunk for @callee_guaranteed (@guaranteed PolylineNode) -> (@error @owned Error)partial apply at <compiler-generated>:0
    frame #12: 0x00000001c7083124 libswiftCore.dylib`(extension in Swift):Swift.Sequence.forEach((A.Element) throws -> ()) throws -> () + 528
    frame #13: 0x0000000101786224 ARCL`SceneLocationView.updatePositionAndScaleOfLocationNodes(self=0x000000010802ca00) at SceneLocationView.swift:462:65
    frame #14: 0x00000001017870c4 ARCL`protocol witness for SceneLocationManagerDelegate.updatePositionAndScaleOfLocationNodes() in conformance SceneLocationView at <compiler-generated>:0
    frame #15: 0x0000000101773638 ARCL`SceneLocationManager.updateLocationData(self=0x0000000282d003c0) at SceneLocationManager.swift:83:32
    frame #16: 0x000000010177459c ARCL`closure #1 in SceneLocationManager.run(_0=0x0000000280dda580, self=0x0000000282d003c0) at SceneLocationManager.swift:127:11
    frame #17: 0x0000000101774604 ARCL`thunk for @escaping @callee_guaranteed (@guaranteed NSTimer) -> () at <compiler-generated>:0
    frame #18: 0x000000019934ea78 Foundation`-[_NSTimerBlockTarget fire:] + 40
    frame #19: 0x00000001993f80c8 Foundation`__NSFireTimer + 88
    frame #20: 0x00000001988e71b8 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 32
    frame #21: 0x00000001988e6ee4 CoreFoundation`__CFRunLoopDoTimer + 900
    frame #22: 0x00000001988e66c8 CoreFoundation`__CFRunLoopDoTimers + 252
    frame #23: 0x00000001988e1398 CoreFoundation`__CFRunLoopRun + 1812
    frame #24: 0x00000001988e0964 CoreFoundation`CFRunLoopRunSpecific + 452
    frame #25: 0x000000019ab21d8c GraphicsServices`GSEventRunModal + 108
    frame #26: 0x00000001c5d7d758 UIKitCore`UIApplicationMain + 216
    frame #27: 0x0000000100ee3b1c ARKit+CoreLocation`main at AppDelegate.swift:12:7
    frame #28: 0x000000019839cfd8 libdyld.dylib`start + 4
(lldb) 

[halmueller]

Is this 100% reproducible? Or only occasionally?

What version of ARCL are you building against?

[pichirichi]

It is happening all the time, very easy to reproduce.
I'm running agains the latest develop: SHA 3096c44.

[halmueller]

Ah, so you've tested it within the last couple of hours!

[pichirichi]

I've tested it again now with the latest updates

[halmueller]

I don't have any 12.x hardware left to test on. But maybe someone out there does and can help.

[pichirichi]

I've tried eliminating lines of codes without luck.
Always getting to the deadlock.
I assume that if it is happening on the sample app and in my app all those who are using the library experience the same problem on iOS 12.

[halmueller]

I expect that you’re correct in your assessment. Did it ever work? Can you find an older SHA for ARCL that doesn’t freeze on 12? There aren’t many of us on this project, and we all have other primary jobs. So it will be up to you to track down the commit that caused the failure. Hal Mueller halmueller@gmail.com

…

On Dec 1, 2019, at 11:20 PM, Shai Shamir ***@***.***> wrote:  I've tried eliminating lines of codes without luck. Always getting to the deadlock. I assume that if it is happening on the sample app and in my app all those who are using the library experience the same problem on iOS 12. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[pichirichi]

I don't recall a version it didn't freeze for iOS 12 in the scenario.

[mobile-appz]

Is this 100% reproducible? Or only occasionally?

What version of ARCL are you building against?

Finding that this is 100% reproducible on iPhone SE on iOS 12. The AR camera always freezes on iOS 12 when a screenshot is taken of the screen in a view controller with ARCL implemented. Guess this is the same issue, and can test soon with earlier versions of ARCL to try to find if it ever worked without freezing.

[halmueller]

That would be super helpful! Thank you!

[mobile-appz]

I have just tested this and can confirm that this issue does not occur on the example app in the first release, 1.0, commit 4527709 on an iPhone SE running 12.4.1

However it does occur in an app with the latest ARCL cocoapod with version 1.3.0 of ARCL.

This is reassuring to note that it wasn't always present and has been introduced somewhere along the way. I'm not able determine from that stack trace or log what went wrong sorry. And not sure how to go about locating the commit that broke things. The issue became apparent for me when I was taking screenshots of the app and it was freezing, but if it affects all iOS 12 users - of which there are probably a lot - changing to a background app and then back again this is a bigger issue than I first thought.

[halmueller]

That's good info. Are you familiar with git bisect? Now that you have one reproducible good commit and one reproducible bad commit, you can make git do the heavy lifting of finding the commit that introduced the failure.

[mobile-appz]

That's good info. Are you familiar with git bisect? Now that you have one reproducible good commit and one reproducible bad commit, you can make git do the heavy lifting of finding the commit that introduced the failure.

Wasn't familiar with bisect before but looks like a powerful tool, and a very useful one, I'll try that and report back, thanks.

[mobile-appz]

That's good info. Are you familiar with git bisect? Now that you have one reproducible good commit and one reproducible bad commit, you can make git do the heavy lifting of finding the commit that introduced the failure.

Wasn't familiar with bisect before but looks like a powerful tool, and a very useful one, I'll try that and report back, thanks.

Sorry, unfortunately I haven't been able to find the cause of this issue or the commit that it started occurring on. I very recently had to install iOS 13.3 to check app compatibility with that and thought I would easily be able to go back to iOS 12.4.1, in order to find a fix to this plus another issue on iOS 12.4.3. The separate unrelated issue I was getting was "Error Domain=CMErrorDomain Code=102 "Failed to get true north" on an iPhone SE with iOS 12.4.3. That error was occuring on all ARKit apps, not just this with the ARCL framework, and meant the app froze. I had to upgrade to iOS 13 to see if that upgrade fixed the other error, and it did. However it looks like Apple are now not signing any versions of iOS 12 now so it's impossible to downgrade from iOS 13, once you have installed it.

[brentvvosselen]

I remember having the same issue somewhere during development using ARCL.
The only thing I still remember of what we did to fix this is pausing and restarting the sceneLocationView when the view appears or disappears.
For reference:

    override func viewDidAppear(_ animated: Bool) {
        sceneLocationView.run()
    }

    override func viewWillDisappear(_ animated: Bool) {
        sceneLocationView.pause()
    }