Details on DDM Lib Issues?

[cody1024d]

Hey @Malinskiy,

I was wondering if you could expand a bit more on what issues you were facing with the DDM library? We're currently using it in our testing framework (because we're on a pretty low version of Gradle), but I'm seeing some really bad/weird bugs after running ~50 UI Tests.

Specifically, I'm continually seeing Connection Reset By Peer. And once this issue happens, not even an adb kill-server or killing the ADB process fixes it. I've only found restarting my entire machine to do it.

Is this similar to what you were seeing?

[Malinskiy]

Hey @cody1024d

Most of the issues I've seen are related to testing as well. I'll list some from the top of my head:

1. Timeout handling: it's crucial to have the timeouts on specific operations with ddmlib, yet it doesn't set the timeout for the whole operation including device selection, e.g. https://android.googlesource.com/platform/tools/base/+/master/ddmlib/src/main/java/com/android/ddmlib/AdbHelper.java#337
   While the responsibility of handling timeout is on the user of ddmlib, I believe coroutine's timeout handling is much more reliable (setting the timeout for the whole connection rather than just a part) and also allows the client to close the socket gracefully
2. At the time of creation in 2020, there were little to no tests in ddmlib. This improved a bit with time
3. The number of new Thread() calls without saving the reference in ddmlib was astounding. This means that you might have dangling threads without a reference to interrupt them. This also leads to socket timeouts because adb's transport is serial in nature, so if one request is stuck for a specific device - all are stuck (this is true for USB transport AFAIK). In extreme cases the selection thread might be locked completely and adb kill-server will not work since it's a graceful termination. Such a case requires an actual kill of the adb server process.
4. RAM usage was above acceptable when connecting 100 devices to a single test run. With adam I was able to run tests with just under 100Mb heap on a J9 JVM and I don't believe it's possible with ddmlib yet.
5. Documentation for using ddmlib

Some of these are highlighted on the announcement blog post.

[cody1024d]

I guess what I was wondering was if the client itself could cause the device failure, by pushing too much across the socket, etc. Because the device still shows as connected --- I think the underlying socket is just broken, though. My reasoning here, is, in my specific example, the tests run flawlessly when run with the Orchestrator through Android Studio, but has issues when run through Gradle tasks. I have already checked that the actual ADB command that both are sending are identical -- so the only other difference I can think of, is the additional processing that happens through our tasks (pulling code coverage files, screenshots, etc.) that doesn't happen during Android Studio just running through tests.

I'm pulling my hair out hahah

[Malinskiy]

Yes, it's possible to affect the stability of server + adbd via requests, see https://issuetracker.google.com/issues/177868251 for an example of such issue. I've only seen this particular problem before that resulted in a total lock of adb server and nothing could help.

To solve this unfortunately you either need to use safe requests (but there is no definition of such thing unfortunately because the spec is kind of organically evolved) or adb server needs some patches and better handling of TCP connection issues both to adb client and adbd.

I'm sorry I can't be of more help here, debugging such issues depends on having access to your particular setup and codebase. If there is a reproducing code - I'd be happy to look at it.

[cody1024d]

@Malinskiy Thank you for that issuetracker! I just wanted to make sure it wasn't a coding issue on my part that was causing the bug

I ended up resolving my issue, and I do think it was the number of requests I was sending. In my instance, it was because I was trying to pull screenshots from the device after each test, and doing so individually (so if a test had 20 screenshots, it would send 20 requests). This got quite large, quite quickly! Changing this to a bulk import (all of the screenshots, for all tests all at once) resolved the issue.

[Malinskiy]

Glad you've found a solution!
One thing about this trade-off though is you can lose previously finished test artifacts since the device has to survive for the whole duration of test run.

In marathon test runner I chose pulling files after each am instrument call. The files are pulled serially though so as to not have parallel requests.

[cody1024d]

Yeah that is totally fair. If for whatever reason the test process crashes, I won't grab any artifacts --- I'm hoping that just doesn't happen too frequently. But it's better than the socket breaking down :)