Espresso's ViewActions.click() appears to be flaky

[seadowg]

Description

I recently switched a test suite from API 30 to API 34 devices and I'm witnessing occasional fails (roughly 1 in 5 times across ~300 tests) in tests where we click (via onView(withText("blah")).perform(click())) on something and then expect to see the next screen. Inspecting video recordings for these (I'm using Firebase Testlab), I routinely see that the click has never occurred. I've seen this across multiple parts of a large app, so I'm fairly confident this is a problem in the test code rather than some race condition in the application code.

I had wondered if this was click() being executed before setOnClickListener in some circumstances, but as far as I can tell (through some hacky logging and exception throwing) this isn't the case and Espresso is waiting for the Activity/Fragment lifecycle to complete before interacting (as expected).

My suspicion is that there is some flakiness in click() itself and this seems to be more present in API 34 than it was in earlier versions.

I've tried a couple of different solutions to this that all seem to remove the flakes:

1. Using a "try again" mechanism that makes the click, waits for the result it expects, and then tries the click again if it doesn't get it. This is very common in other UI testing frameworks, but not something I've generally had to resort to in Espresso (due to the IdlingResource system).
2. Building a custom OnClickListener that can "signal" the tests when the click has actually occurred. Combining this with a "try again" approach allows your code to be certain that the click event has propagated (rather than needing to wait on some change in the view), but is obviously very invasive (we're changing application code to accommodate tests).
3. Using a custom ViewAction that calls performClick() on the View instead of attempting to inject a tap. This feels like a glaringly simple solution, but it doesn't work with all views. For example, a TextView might be the child of a clickable view rather than clickable itself (like in the case of a view pager title for instance). My guess is that cases like this were the reason (or at least one of the reasons) for building out ViewActions.click() to inject an event in the first place.

Steps to Reproduce

With a view that contains a button that changes screen/alters the view:

onView(withText("button test")).perform(click())
onView(withText("something new after the button click")).matches(isDisplayed())

Expected Results

Espresso will always successfully click on the button.

Actual Results

The click sometimes doesn't happen.

AndroidX Test and Android OS Versions

Espresso 3.5.1 and API 34 running mainly on MediumPhone.arm virtual device on Firebase Test Lab.

[TWiStErRob]

I'm not sure if yours is the same but there was this problem (in the past?) where you get a log:

Overslept and turned a tap into a long press

and that means your onClickListener wouldn't be called, but onLongClickListener does (which is usually not set).

I've had this a lot with option menu ... opening.

Anyway, one of the "solutions" that sometimes made it more stable was .perform(click(click())) which is essentially retrying the click if the outer click turned into a long tap. This and/or increasing the long tap timeout in the emulator you're running in, usually helped.

[seadowg]

I'm not sure if yours is the same but there was this problem (in the past?) where you get a log:

Overslept and turned a tap into a long press

I've never seen that, but I wouldn't be surprised if it's come up and I've just missed it.

and that means your onClickListener wouldn't be called, but onLongClickListener does (which is usually not set).

Ah I've had this problem in the past, but had forgotten about it! I've already increased the long tap timeout for tests, but I'll need to dig in and see if this is still the route of the problem.

Anyway, one of the "solutions" that sometimes made it more stable was .perform(click(click())) which is essentially retrying the click if the outer click turned into a long tap.

I had completely missed that this was an option. Thanks for pointing it out! I guess this ends up being pretty similar to my "Using a try again mechanism that makes the click, waits for the result it expects, and then tries the click again", but it looks like the long tap is actually detected by androidx.test.espresso.action.Tap here (although I could be wrong). That feels a little more definite than "we're not where we expect to be" assuming long clicks are the problem.

[seadowg]

Ah I've had this problem in the past, but had forgotten about it! I've already increased the long tap timeout for tests, but I'll need to dig in and see if this is still the route of the problem.

I've now experimented with this, and it doesn't seem like accidental long presses are the problem sadly. The experiment I performed was to remove all our other workarounds and then add a long click listener to a couple of buttons in the app that get called in many of the tests:

setOnLongClickListener {
    throw IllegalStateException("Long clicked on view!")
}

Running suite several times, I still saw failures due to clicks not firing, but never saw the IllegalStateException so it doesn't look like the "failed" clicks ended up as long clicks. I also tried the click(click) trick, but again still saw some failures.

I'll post the workarounds I'm using once I merge them in (it's all open source stuff thankfully).

[brettchabot]

Hmm I haven't heard any other reports of clicks getting flakier on API 34, and espresso's own tests do not appear to show this symptom. If you find more info please share.

[seadowg]

Hmm I haven't heard any other reports of clicks getting flakier on API 34, and espresso's own tests do not appear to show this symptom. If you find more info please share.

It's hard to know if this was got specifically worse at API 34, but it was in the process of moving from API 30 to 34 that I started keying in on the issue. I should also note that I'm using the ARM emulators as I guess that might have some effect (given we're talking about native event injection). I'll add that to the issue description!

For anyone who want's to investigate further, I'm seeing this in an open source Android app: https://github.com/getodk/collect (at commit 4f54bcc as we're just about to merge work arounds that will hopefully fix this 🤞). I'm confident you'd be able to reproduce the problem running Test Lab (or locally if you're more patient). There's a built in Gradle task (./gradlew testLab), but for the record it's:

gcloud beta firebase test android run \
              --type instrumentation \
              --num-uniform-shards=25 \
              --app collect_app/build/outputs/apk/debug/*.apk \
              --test collect_app/build/outputs/apk/androidTest/debug/*.apk \
              --device model=MediumPhone.arm,version=34,locale=en,orientation=portrait \
              --results-bucket opendatakit-collect-test-results \
              --directories-to-pull /sdcard --timeout 20m \
              --test-targets "notPackage org.odk.collect.android.regression"