Shaker

SHAKER is a lightweight approach to detect flakiness in time-constrained tests by adding noise in the execution environment.

How to cite Shaker

@inproceedings{STA:ICSME2020,
	author = {Silva, Denini and Teixeira, Leopoldo and d'Amorim, Marcelo},
	booktitle={2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)},
	title = {{Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker}},
	year = {2020},
	pages={301-311},
	doi={10.1109/ICSME46990.2020.00037}
}

Organization

This repository is organized in the following structure:

• the dataset folder contains the set of flaky tests we used in the paper;
• the raw_results folder contains the raw data produced from the evaluation we performed;
• the evaluation folder contains the toolset developed for running Shaker against the apps listed below and producing the raw data described above.
• the standalone folder contains the instructions for running Shaker in other apps.

Setup Instructions

The minimum requirements for running the tools and scripts we developed for this work are described below. These steps were executed and tested using a fresh install of the Ubuntu 18.04 LTS environment. The approach relies on running an emulator, and some steps might take long to execute, although we paremeterize our scripts so it is not necessary to run the exact number of replications we executed for each experiment in the paper (see the evaluation folder).

The step by step instructions follow below, but these are all automated under the setup.sh script.

Install Linux packages

sudo apt-get update 
sudo apt-get install -y openjdk-8-jdk stress-ng python3-pip unzip r-base

Install required Python libraries:

We assume a working Python3 environment. We require installing the following libraries for executing the Minimal Hitting-Set (MHS) algorithm.

pip3 install python-sat
pip3 install numpy==1.16.1

Install Android Command Line tools

wget https://dl.google.com/android/repository/commandlinetools-linux-6609375_latest.zip
unzip commandlinetools-linux-6609375_latest.zip
rm -f commandlinetools-linux-6609375_latest.zip
mkdir cmdline-tools
mv tools/ cmdline-tools
mkdir Android
mv cmdline-tools/ Android

Configure the Android SDK version and AVD from the command line (Android API version 28)

yes | $HOME/Android/cmdline-tools/tools/bin/sdkmanager --licenses
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "platforms;android-28"
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "system-images;android-28;default;x86"
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "build-tools;28.0.3" 

If you want to use another API version, such as 23, just change the numbers in platforms and system-images:

$HOME/Android/cmdline-tools/tools/bin/sdkmanager "platforms;android-28"
$HOME/Android/cmdline-tools/tools/bin/sdkmanager "system-images;android-28;default;x86"

Create AVD device (Android API version 28):

echo no | $HOME/Android/cmdline-tools/tools/bin/avdmanager create avd --name EmuAPI28 --package "system-images;android-28;default;x86"

If you want to use another API version, such as 23, just change the corresponding numbers:

echo no | $HOME/Android/cmdline-tools/tools/bin/avdmanager create avd --name EmuAPI23 --package "system-images;android-23;default;x86"

Run the emulator

To run the emulator created by the setup.sh script, just execute:

./runemulator.sh

This will run the DEFAULT_EMULATOR defined in the runemulator.sh. If you want to run a particular emulator that you have previously created, with a different name, change the variable inside the script or provide the name as the first argument:

./runemulator.sh AVD_NAME

Finally, if you want to run in headless mode (-no-window), you might execute the following script that runs the default emulator:

./runemulator-nogui.sh

Again, if you want to run a particular emulator that you have previously created, with a different name, change the variable inside the script or provide it as the first argument:

./runemulator-nogui.sh AVD_NAME

After finishing booting the emulator for the first time (wait for the boot complete message in the console), the apps we have used for the evaluation can be installed using the install-apps.sh script.

Executing Shaker

This is all it takes for configuring the environment for running Shaker by using the predefined scripts for each research question from the paper, which are available at the evaluation folder. You can also run the icsme2020-eval.sh or quick-eval.sh to go through all RQs.

Apps used as Objects of Analyses

Name	SHA
AntennaPod	dd5234c
AnyMemo	7e674fb
Espresso	043d028
Flexbox-layout	611c755
FirefoxLite	048d605
Kiss	00011ce
Omni-Notes	b7f9396
Orgzly	d74235e
Paintroid	1f302a2
Susi	17a7031
WiFiAnalyzer	80e0b5d

Issues opened

For each project where we found flaky tests, we opened issues:

• AntennaPod (answered)
• FirefoxLite (answered)
• Espresso
• Kiss
• Omni-Notes
• Orgzly
• Paintroid
• WiFiAnalyzer