Testing is an important part of the application development process. For Android, it is particularly important because the devices are very different from each other in the following ways:
Your Android application should be tested on many devices.
Application testing processes include many types of testing. Let's take a look at manual functional testing. The tester needs to carefully check all the functionality and reset the device to an initial state. The tester repeats these steps for each application and each device. Done manually, this process is very time-consuming.
Automated functional testing can be performed regularly without additional costs. For example, test a build every night on all devices, analyze the results in the morning, and fix the bugs.
In this article I will review several tools for automated functional testing. I'll review only tools included in Android SDK or distributed under Open Source license.
Our goal is to automate actions that are performed manually for maximum precision. Let's review these actions. We'll use several applications and several Android devices.
For each application and each device we need to perform the following steps:
At each step you need to collect and analyze data such as logs and screenshots. Below, we’ll discuss the tools that automate these steps.
First, you need to select the computer that will run the automated test and install the Android SDK on it. I'll use a desktop computer with Linux* as an example. You need to disable lock screen and increase "time before sleep mode" to the maximum level on each device.For some testing methods, you need to disable screen orientation change.
There are two utilities in the Android SDK to control Android devices: ADB and monkeyrunner*. I'll describe in detail how to automate actions that are used in manual testing.
ADB (Android Debug Bridge) is a command line tool to control Android devices. The ADB home page is:Android Debug Bridge
The ADB tool is located in the directory <android_sdk>/platform-tools/.You should put this directory in thePATH environment variable.
Checking the ADB installation
Install and configure the Android SDK, then connect Android devices to your computer and run the command:
adb devices
This command will list all plugged-in Android devices. If the list of devices is not empty, then ADB is working properly.
Working with multiple devices
You should use “-s” parameter to specify which device ADB should use.
adb -s [serial_number] [command]
For example:
adb -s [serial_number] logcat
The serial number of the device can be obtained from the output of «adb devices» command. Parameter -s allows you to work simultaneously with multiple connected devices.
Basic ADB commands
Open a console on the device:
adb shell
Run a command on the device:
adb shell [command]
Many standard Linux utilities are included in Android: ls, cat, dmesg, …
Install the application from the apk file:
adb install example.apk
Uninstall the application:
adb uninstall [package]
Get a package name from the apk file:
aapt dump badging example.apk | grep "Package"
Download a file from the device to the computer:
adb pull [path-on-device] [file]
Upload a file from the computer to the device:
adb push [file] [path-on-device]
Note:
Only read access is allowed to most directories on the Android device. Write access is allowed to /sdcard (but you can’t run programs from this directory) and /data/local/tmp.
Start an application:
adb shell am start -n [package]/[activity]
Run the specified activity.
You can extract the activity name from the apk file:
aapt dump badging example.apk | grep "launchable-activity"
Reading logs
Logcat is a command that reads logs from Android devices.
Logcat homepage:Logcat
Read logs from the device (blocks until you press Ctrl-C):
adb logcat
Clear log buffer on the device:
adb logcat -c
Dump log buffer on the device (shows current buffer contents, non-blocking):
adb logcat -d
Example:
1 | adb logcat -c # clear the buffer log |
2 | # Action |
3 | adb logcat -d > file.log # save the current contents of the log buffer to file.log |
Capture screenshot using screencap
The screencap utility saves the current contents of the screen to a graphic file:
1 | adb shell screencap /sdcard/screen.png |
2 | adb pull /sdcard/screen.png screen.png |
3 | adb shell rm /sdcard /screen.png |
The screencap utility is available on phones with Android 4.x and above. On previous Android versions you can capture screenshots using monkeyrunner.
BASH script to test the application using ADB
Script: app_test.sh
The monkeyrunner tool provides APIs for scripts that control Android devices. You can write a Python* script with monkeyrunner that installs an Android application, launches it, emulates user actions, takes screenshots, and saves them to your computer. Monkeyrunner uses Jython* to run scripts.
The monkeyrunner home page and API reference:monkeyrunner
Reading logs with monkeyrunner
File log.py:
01 | # coding: utf-8 |
02 | from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice |
03 |
04 | def log(fn, device): |
05 | msg = device.shell('logcat -d') |
06 | f_log = open(fn, 'at') |
07 | if msg is None: |
08 | msg = 'None' |
09 | f_log.write(msg.encode('utf-8')) |
10 | f_log.close() |
11 | device.shell('logcat -c') |
12 |
13 | if __name__ == '__main__': |
14 | device = MonkeyRunner.waitForConnection() |
15 | device.shell('logcat -c') # Clear logs buffer |
16 | # ... |
17 | log('example.log', device) # Write logs |
Starting:
monkeyrunner log.py
The script will write logs to a file example.log in the current directory.
Capture screenshot using MonkeyRunner
File screen.py:
1 | # coding: utf-8 |
2 | from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice |
3 |
4 | if __name__ == '__main__': |
5 | device = MonkeyRunner.waitForConnection() |
6 | image = device.takeSnapshot() |
7 | image.writeToFile('screenshot.png','png') |
Starting:
monkeyrunner screen.py
The script takes a screenshot and saves it to a filescreenshot.png in the current directory.
Example of device control using monkeyrunner
Script: monkeyrunner_test.py
Starting:
monkeyrunner monkeyrunner_test.py
Imagine that the device you are testing is given to a very active and creative monkey—the Monkey tool is designed to simulate this situation.The Monkey tool, a part of the Android SDK, sends a stream of random user events.Command line parameters specify the number of user actions, the ratio of each event type, and a name of a package (so Monkey will not go beyond the limits of the tested application and will not start sending SMS to all the contacts in the Address Book).
Examples and a list of parameters are on the Monkey home page:Application Exerciser Monkey
The main advantage of the Monkey tool is zero maintenance costs. Furthermore, stress testing can detect non-trivial bugs.
Disadvantages of testing with the Monkey tool:
Automated testing with Monkey is a good starting point for any application. It is possible that this method will show adequate results for a particular application.If the testing quality is low, you should use other testing methods.
You can not only develop Android device control scripts using MonkeyRunner, you can also write scripts to test the application on a specific device.
Advantages:
Disadvantages:
Developing monkeyrunner scripts takes a lot of time, so this method is usually not justified.However, in special cases, this method might work.
Getevent and sendevent utilities allow the user to record a sequence of events and replay the sequence.Root permissions are not required to run these tools.
Advantages:
Disadvantages:
Record the sequence of events:
01 | # Record event sequence |
02 | # Do actions on the device, press Ctrl-C to finish |
03 | adb shell getevent -t > events.txt |
04 | # Convert event sequence to script |
05 | ./decode_events.py events.txt > events.sh |
06 | # Load the script to the device |
07 | adb push events.sh /data/local/tmp/ |
08 | # Set the permissions |
09 | adb shell chmod 755 /data/local/tmp/events.sh |
10 | # Run script |
11 | adb shell sh /data/local/tmp/events.sh |
Script: decode_events.py
Replay the recorded sequence of events on the device.
Robotium is not a part of Android SDK, and it’s distributed under the Open Source license. The Robotium home page is: User scenario testing for Android.
Robotium scripts define actions on the application UI level rather than the input device level.
For example, a script needs to tap on an «OK» button. The monkeyrunner script will be implemented as “Tap at the screen point (x0, y0)”. A Robotium script will be implemented as “Press the button with the text “OK” .”
When actions are described at the interface level, the testing script can be made independent of interface layout, screen resolution, and orientation.
Additionally, Robotium allows you to check the application response to the action.For example, after clicking the «OK» button, the list with the “Item 1” item should appear.You can check list element names with Robotium. If you check the application state after each step, it is easy to find at which step the error occurred.
Disadvantages:
In general, Robotium allows you to develop the highest quality test cases with adequate cost.
| Testing method | Advantages | Disadvantages |
|---|---|---|
| Monkey – a stream of random user actions | No maintenance costs. | Quality of testing varies from application to application. Monkey doesn’t check the state of the application during testing. |
| monkeyrunner – device control script | Flexibility | The complexity of scripting, even for simple applications. |
| getevent/sendevent – record/replay user actions | Programming skills are not required to record event sequence. | Recorded sequence of actions will only fit one device at a fixed screen orientation. When the application interface changes, the event sequence must be redone. This method doesn’t check state of application during testing. |
| Robotium – test script API to verify the status | Actions are described at the application UI level. Script may be independent of the screen resolution and screen orientation. Script can check application state after an action. | The complexity of writing scripts in Java. If you change the application interface, you will need to modify the script. |
Now we need to analyze the logs and screenshots collected during the automated testing process for errors.
You can do a search for strings:
This list can be extended with error messages that were found during manual testing.
You can prepare a series of screenshots at key moments of testing and compare them to the screen content during automated testing. This will determine if the automated testing process is running correctly.
It's useful to compare the initial screenshot with the screenshot after the application launches. This helps to detect cases when the application silently fails.
Monkeyrunner allows you to compare two screenshots with the specified tolerance in percents:
1 | image1 = device.takeSnapshot() |
2 | # ... |
3 | image2 = device.takeSnapshot() |
4 | if image2.sameAs(image1, 0.1): |
5 | print 'image1 and image2 are the same (10%)' |
Unfortunately, there is no MonkeyImage API to load an image from a file. You can write a custom function to compare images using, for example, the Python* Imaging Library.
Devices should be returned to the initial state after testing.This can be achieved in several ways:
The first option is usually the most appropriate.
Press the Back button multiple times
Press the "Back" button using monkeyrunner:
1 | for i in xrange(0, 10): |
2 | device.press('KEYCODE_BACK', MonkeyDevice.DOWN_AND_UP) |
3 | time.sleep(0.5) |
In practice, this is a good option because it emulates the behavior of a real user.
In this article, we covered some methods of automated testing for Android applications. We reviewed advantages and disadvantages of automated testing methods.
We also discussed the Monkey and monkeyrunner tools included in the Android SDK and the Robotium tool.
Automated testing does not replace other types of testing.A properly constructed testing process that combines different testing methods including automated testing is an essential part of a high quality application development process.
For more such Android resources and tools from Intel, please visit the Intel® Developer Zone