Support AndroidJUnitRunner test filters

[plastiv]

Is your feature request related to a problem? Please describe.
I have mixed onboarding experience as it wasn't directly obvious from docs that Marathon reimplements test builder and doesn't support everything that android test builder supports. My codebase has SdkSuppress annotation which leads Marathon into an infinitive loop.

Describe the solution you'd like
I believe it may improve migration to / experimentation with Marathon especially on existing codebases if Marathon would support "default" test filters https://developer.android.com/training/testing/junit-runner#filtering-tests

Describe alternatives you've considered

• Update android vendor documentation and/or FAQ that AndroidJUnitRunner support is limited / not available.
• Change defaults from infinitive retry to something more sensible

Additional context

• I've found previous discussions here Ignore by default all tests with org.junit.Ignore annotation #186

[Malinskiy]

Hey @plastiv, thanks for submitting this issue

I agree it’s not completely clear what is happening with the tests. However, I do believe this is an issue with the AndroidJUnitRunner and not marathon: the filtering mechanism in JUnit for SdkSuppress should report assumption failure rather than not reporting anything at all. Marathon would correctly identify these annotations out of the box if the behavior would be different.

The docs mention that it’s not possible to support all the platform-specific annotation:

An important thing to mention is that by default platform specific ignore options are not taken into account. This is because a cross-platform test runner cannot account for all the possible test frameworks out there. However, each framework’s ignore option can still be “explained” to marathon, e.g. JUnit’s org.junit.Ignore annotation can be specified in the filtering configuration.

We do however filter the heterogeneous (meaning they don’t depend on device information at all) annotations like Ignore and Suppress.

A significant problem with SdkSuppress Here is if your device pool consists of devices with different SDKs: imagine you have a device with 29 and 30. If your sdksuppress tests with 30 ends up on device 29 - you don’t have a result. If it ends up on 30 - you get the actual test result. A support for this particular type of test is usually achieved in marathon by having a separate pool of devices (e.g. by API level) and then scheduling tests with a particular annotation via test filtering.

In regards to the infinite retry - it’s don’t for your own good because this is an error in the behavior of test environment that you should be able to see. There is a quota for incomplete tests if you want to limit these.

I’m happy to improve the section on the test filtering and be more descriptive of the supported annotations.

@tagantroy @Vacxe thoughs?