Instrumented test results outputs nothing when test collection failed(process crashed) with AndroidX test orchestrator

[ZSmallX]

Description

My code changes caused exception at Application.onCreate(), and instrumented tests run via AndroidX test orchestrator.

execution command line like:

// prepare AndroidX test orchestrator
....

adb shell 'CLASSPATH=$(pm path androidx.test.services) app_process / androidx.test.services.shellexecutor.ShellMain am instrument -w -r -e coverageFilePath coverage_files/ -e useTestStorage true -e coverage true -e clearPackageData true -e targetInstrumentation com.mypackage.test/androidx.test.runner.AndroidJUnitRunner androidx.test.orchestrator/.AndroidTestOrchestrator'
INSTRUMENTATION_RESULT: stream=

Time: 0

OK (0 tests)


INSTRUMENTATION_CODE: 0

And there is one details file at adb shell run-as androidx.test.orchestrator cat /data/user_de/0/androidx.test.orchestrator/files/testCollection.txt.

INSTRUMENTATION_STATUS: stack=java.io.FileNotFoundException: my_data.json
	at android.content.res.AssetManager.nativeOpenAsset(Native Method)
	at android.content.res.AssetManager.open(AssetManager.java:824)
	at android.content.res.AssetManager.open(AssetManager.java:801)
	at ...
	at com.mypackage.test.ShellApplication.onCreate(ShellApplication.kt:73)
	at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1189)
	at androidx.test.runner.MonitoringInstrumentation.callApplicationOnCreate(MonitoringInstrumentation.java:483)
	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6460)
	at android.app.ActivityThread.access$1300(ActivityThread.java:219)
	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1859)
	at android.os.Handler.dispatchMessage(Handler.java:107)
	at android.os.Looper.loop(Looper.java:214)
	at android.app.ActivityThread.main(ActivityThread.java:7356)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)

INSTRUMENTATION_STATUS: stream=
Process crashed before executing the test(s):
java.io.FileNotFoundException: my_data.json
	at android.content.res.AssetManager.nativeOpenAsset(Native Method)
	at android.content.res.AssetManager.open(AssetManager.java:824)
	at android.content.res.AssetManager.open(AssetManager.java:801)
	at ....
	at com.mypackage.test.ShellApplication.onCreate(ShellApplication.kt:73)
	at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1189)
	at androidx.test.runner.MonitoringInstrumentation.callApplicationOnCreate(MonitoringInstrumentation.java:483)
	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6460)
	at android.app.ActivityThread.access$1300(ActivityThread.java:219)
	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1859)
	at android.os.Handler.dispatchMessage(Handler.java:107)
	at android.os.Looper.loop(Looper.java:214)
	at android.app.ActivityThread.main(ActivityThread.java:7356)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)

INSTRUMENTATION_STATUS_CODE: -2
INSTRUMENTATION_RESULT: shortMsg=Process crashed.
INSTRUMENTATION_CODE: 0

Steps to Reproduce

1. raise exception at Application.onCreate().
2. run tests using AndroidX test orchestrator.

Expected Results

See process crashed details in instrumented tests results outputs to indicate test failure.

Actual Results

No process crashed details in instrumented tests results outputs and indicate test success.

AndroidX Test and Android OS Versions

Android API Level 29
AndroidX Test Orchestrator 1.4.2

Link to a public git repo demonstrating the problem:

None currently.

[ZSmallX]

Looks like that when collectTests was exeuted in a runnable and wait for Android JUnit Runner to addTest to CallbackLogic.

But in process crashed case, the remote Android JUnit Runner was unabled to call addTest, so the test == null and callbackLogic.allTests.isEmpty == true. Then nothing shows up or is presented.

// callback with empty tests.
https://cs.android.com/androidx/android-test/+/master:runner/android_test_orchestrator/java/androidx/test/orchestrator/AndroidTestOrchestrator.java;l=294;bpv=0;bpt=0

// collectTests.
https://cs.android.com/androidx/android-test/+/master:runner/android_test_orchestrator/java/androidx/test/orchestrator/AndroidTestOrchestrator.java;l=265;bpv=0;bpt=0

[ananci]

@ZSmallX - Your explanation looks reasonable but I'm having difficulty reproducing. Would you mind sharing a sample project with a repro?

[ZSmallX]

@ZSmallX - Your explanation looks reasonable but I'm having difficulty reproducing. Would you mind sharing a sample project with a repro?

@ananci Yeah, here is the repro https://github.com/ZSmallX/android_test_issue_1834

And by this repro, I can get the expected test results when running from IDE, and can not when running from command line.