An Analysis of Cloud Mobile Testing Services

[paulsuen]

Mobile App testing is very time-consuming, because there are too many device models both in Android and iOS. Some really good services have popped up that allows testing against multiple devices to be quite simple.Cloud testing seems very attractive, because it can solve 2 problems: time cost and device cost. It can be very good instrument for QA to perform important tests. Then, how to choose a cloud platform?

What testings can cloud test platform do?

Most cloud test platform can do test for native Apps, hybrid Apps and web Apps, include:

1. Compatibility test
   Install/uninstall, launch the App, traversing every page, perform monkey test in real devices, to find errors like ANRs and crashes etc, provide error logs and screenshots to help developers to fix bugs.

2. Functional test
   Write or record testing scripts using script recording tools, and upload scripts to perform functional testing, to cut down manual testing time.

3. Performance Analysis
   Testing report can provide performance data, including battery , CPU, memory consumption etc.

China-based services

Most popular services in China, provided by China leading internet enterprises:

1. MQC(Mobile Quality Center) from Aliyun: www.mqc.yunos.com

2. Testin: www.testin.cn

3. Utest from Tencent: http://utest.qq.com

4. Baidu MTC(Mobile Test Center), from Baidu: http://mtc.baidu.com

5. WeTest(Tencent Quality Open Platform), http://wetest.qq.com

   MQC Testin Utest MTC WeTest
   Android Device Models 1000+ 1000+ 1000+ 1500+ 1000+
   Compatibility Test √ √ √ √ √
   Functional Test(scripts) √ √ × √ ×
   Test Frameworks Robotium, Appium, UIautomator Robotium, Appium,UIautomator Robotium, Robotium, Appium, UIautomator ×
   Scripts Recording Tools MQC-desktop iTestin Xtest recordPlay ×
   Crash report √ √ √ √ √
   Performance Monitoring √ √ √ √ √
   Remote Real Device Debug √ √ √ √ √
   Screen Video Recording × × × √ v
   Individual Pricing Free for 30 random devices everyday Free for 50 random devices everyday
   × Free for Top20 models every day, 1 minute/device Standard Compatibility Test: Free for 50 device/day; Remote Debug: free for 120 minutes
   Enterprise Pricing ￥10 /device for each testing(10 minutes on 1 individual device) Free for 100 random devices everyday;

￥5000 for 1 testing, on 150 selected devices Out of services due to system updating Compatibility test:￥1/device minute, Functional test(scripts) :￥2/device minute Standard Compatibility Test: free for 100 devices/month, ￥3/device for each testing.
Remote Real Device Debug: first 240 minutes for free, and ￥40/hour

I have used free trials of MQC, Testin, , MTC, WeTest. They all provide good service, except WeTest.

Commons:

1. MQC, Testin, MTC all provide Compatibility Test, support Appium/Robotium, provide script recording tools, their testing reports all include screenshot, logcat, performance report. They all provide remote real device debug.

Difference:

1. Only MTC provide screen video recording.
2. MQC and Testin provide better formatted report, give different log types, like error, exception, crash.
   Pricing(for 150 selected devices, 10 minutes on 1 individual device):
3. MQC: ￥1500 for a testing
4. Testin: ￥5000 for a testing
5. MTC: ￥1500 for a standard testing, ￥3000 for a script testing.

Mainstream platform overseas

1. Google Firebase Test Lab(https://firebase.google.com/docs/test-lab/)

2. Amazon Device Farm (https://aws.amazon.com/device-farm/)

3. Testdroid( http://bitbar.com/testing/)

   Google Firebase Amazon Device Farm Testdroid
   Android Device Models 1000+ 1000+ 1000+
   Compatibility Test Support Robo test Support Build-in test: Explorer, Fuzz App Crawler
   Functional Test(scripts) √ √ √
   Test Frameworks Espresso, Robotium, UIAutomator 2.0 Appium Java JUnit, Appium Java TestNG, Appium Pyton, Calabash, Instrumentation, UI Automator Robotium,
   Espresso, Appium, UIAutomator
   Scripts Recording Tools × × Bitbar Testing Recorder
   Crash report √ √ √
   Performance Monitoring √ √ √
   Remote Real Device Debug √ √ √
   Screen Video Recording √ √ √
   Individual Pricing Free for 30 random devices everyday Free for 50 random devices everyday
   ×
   Enterprise Pricing ￥10 /device for each testing(10 minutes on 1 individual device) Free for 100 random devices everyday;

￥5000 for 1 testing, on 150 selected devices Out of services due to system updating

We'll explore your app by analyzing each view with no scripts required.
We'll fuzz test your app by sending random events to it with no scripts

2. Pick test type:
   Please specify test type
   You can upload any test package that is using Android Instrumentation Framework. The best way to create such package is to use Bitbar Testing Recorder which helps you to create very robust test scripts with minimum effort.

In case you don’t have tests, you can use our embedded feature “App Crawler”. App Crawler is an intelligent application crawler that exercises each application systematically by traversing each application through each of the views in the view hierarchy taking screenshots and recording performance data as the crawler progresses through each view.