[devicelab] catalina bot fails to pick up tasks

[keyonghan]

Pre-health check fails with error

2020-11-10T15:42:51.652195: WARN close-ios-dialog: failed
2020-11-10T15:42:51.652225: WARN ERROR: Command "xcrun xcodebuild -project infra-dialog..." failed with exit code 70.
2020-11-10T15:42:51.652257: WARN #0      fail (package:cocoon_agent/src/utils.dart:139:3)
2020-11-10T15:42:51.652290: WARN #1      closeIosDialog.<anonymous closure> (package:cocoon_agent/src/health.dart:189:9)
2020-11-10T15:42:51.652322: WARN <asynchronous suspension>
2020-11-10T15:42:51.652352: WARN #2      inDirectory (package:cocoon_agent/src/utils.dart:360:24)
2020-11-10T15:42:51.652384: WARN <asynchronous suspension>
2020-11-10T15:42:51.652418: WARN #3      closeIosDialog (package:cocoon_agent/src/health.dart:170:11)
2020-11-10T15:42:51.652451: WARN <asynchronous suspension>
2020-11-10T15:42:51.652486: WARN #4      _captureErrors (package:cocoon_agent/src/health.dart:68:47)
2020-11-10T15:42:51.652518: WARN <asynchronous suspension>
2020-11-10T15:42:51.652551: WARN #5      performHealthChecks.<anonymous closure> (package:cocoon_agent/src/health.dart:49:41)
2020-11-10T15:42:51.652604: WARN <asynchronous suspension>
2020-11-10T15:42:51.652672: WARN #6      _captureErrors (package:cocoon_agent/src/health.dart:68:47)

[keyonghan]

Tried restart device but hit:

flutter@flutter-devicelab-mac-12 agent % idevicediagnostics restart          
ERROR: Could not connect to lockdownd, error code -8

[keyonghan]

Find some similar errors and workaround: libimobiledevice/libimobiledevice#587 (comment), libimobiledevice/libimobiledevice#717 (comment)

But doesn't help here.

@jmagman Do you have any idea how to fix this?

[keyonghan]

Removing close-ios-dialog step in pre-health check for now to by pass the issue, so that the bot continues to pick up tasks.

But task failed with

Task failed with the following reason:
Task failed: DeviceException: No connected iOS devices found.

However

idevice_id -l             
e7184ae5fe05c2ea2ad4654ac70b4ba19c92f2c6

[keyonghan]

From flutter test runner side, it is using /Users/flutter/.cocoon/flutter/bin/flutter devices --machine --suppress-analytics --device-timeout 5. However it returns no ios device:

/Users/flutter/.cocoon/flutter/bin/flutter devices --machine --suppress-analytics --device-timeout 5
[
  {
    "name": "macOS",
    "id": "macos",
    "isSupported": true,
    "targetPlatform": "darwin-x64",
    "emulator": false,
    "sdk": "Mac OS X 10.15 19A582a darwin-x64",
    "capabilities": {
      "hotReload": true,
      "hotRestart": true,
      "screenshot": false,
      "fastStart": false,
      "flutterExit": true,
      "hardwareRendering": false,
      "startPaused": true
    }
  },
  {
    "name": "Web Server",
    "id": "web-server",
    "isSupported": true,
    "targetPlatform": "web-javascript",
    "emulator": false,
    "sdk": "Flutter Tools",
    "capabilities": {
      "hotReload": true,
      "hotRestart": true,
      "screenshot": false,
      "fastStart": false,
      "flutterExit": false,
      "hardwareRendering": false,
      "startPaused": true
    }
  },
  {
    "name": "Chrome",
    "id": "chrome",
    "isSupported": true,
    "targetPlatform": "web-javascript",
    "emulator": false,
    "sdk": "Google Chrome 86.0.4240.193",
    "capabilities": {
      "hotReload": true,
      "hotRestart": true,
      "screenshot": false,
      "fastStart": false,
      "flutterExit": false,
      "hardwareRendering": false,
      "startPaused": true
    }
  }
]

[jonahwilliams]

I think the tool does not use idevice_id , it uses xcdevice

[jmagman]

lockdown error -8 is LOCKDOWN_E_MUX_ERROR, it probably needs a restart. But you can't restart it because you are getting a lockdown error...

xcrun xcdevice list --timeout 10 might give us more info.

[keyonghan]

Here is the info:
xcrun_xcdevice.txt

[jmagman]

e7184ae5fe05c2ea2ad4654ac70b4ba19c92f2c6 isn't even in that list, it seems good and borked, likely needs a restart.

[keyonghan]

filed fxb/64108 to ask for reboot.
In the meantime, marked catalina tasks as flaky to open the tree: #70255

[keyonghan]

Lower the priority to p1 as this is not blocking the tree.

[keyonghan]

Device comes back with fxb/64108 being fixed.
#70328 to remove flakiness.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.