Android CTS (Compatibility Test Suite) introduction--Erin Yueh

Android CTS (Compatibility Test Suite) introduction

What is CTS?
Compatibility Test Suite is a command mode tool to run a series of test cases in Android. They published all CTS source codes from Eclair branch on January 2010.

git repository
http://android.git.kernel.org/?p=platform/cts.git;a=summary

How to run CTS in host machine?
$ cd $MYDROID
$ . build/envsetup.sh
$ make cts

(it would generate all test plans, packages, cases, result report format and then zip to a android-cts.zip file)

• Package CTS: out/host/linux-x86/cts/android-cts.zip
• cts make file: mydroid/build/core/tasks/cts.mk
• run cts program: mydroid/out/host/linux-x86/bin/cts
• test plans: mydroid/out/host/linux-x86/cts/android-cts/repository/plans
• test packages: mydroid/out/host/linux-x86/cts/android-cts/repository/testcases
• test results: mydroid/out/host/linux-x86/cts/android-cts/repository/results
• CTS program settings value: mydroid/cts/tools/utils/host_config.xml

run cts
$ cd $mydroid/out/host/linux-x86/bin/
$ ./cts

Screen would like this:

erin@midnight:~/eclair/mydroid/out/host/linux-x86/bin$ ./cts

Android CTS version 2.1_r1

cts_host > help

Usage: command options

Avaiable commands and options:

Host:

help: show this message

exit: exit cts command line

Plan:

ls --plan: list available plans

ls --plan plan_name: list contents of the plan with specified name

add --plan plan_name: add a new plan with specified name

add --derivedplan plan_name -s/--session session_id -r/--result result_type: derive a plan from the given session

rm --plan plan_name/all: remove a plan or all plans from repository

start --plan test_plan_name: run a test plan

start --plan test_plan_name -d/--device device_ID: run a test plan using the specified device

start --plan test_plan_name -t/--test test_name: run a specific test

start --plan test_plan_name -p/--package java_package_name: run a specific java package

start --plan test_plan_name -t/--test test_name -d/--device device_ID: run a specific test using the specified device

start --plan test_plan_name -p/--package java_package_name -d/--device device_ID: run a specific java package using the specified device

Package:

ls -p/--package: list available packages

ls -p/--package package_name: list contents of the package with specified name

add -p/--package root: add packages from root to repository

rm -p/--package package_name/all: remove a package or all packages from repository

Result:

ls -r/--result: list all result of sessions

ls -r/--result -s/--session session_id: list detail case result of a specified session

ls -r/--result [pass/fail/notExecuted/timeout] -s/--session session_id: list detail cases of a specified session by the specified result.

History:

history/h: list all commands in command history

history/h count: list the latest count records in command history

history/h -e num: run the command designated by 'num' in command history

Device:

ls -d/--device: list available devices



How to add test plan, test packages to CTS?

1. Add test package name to cts.mk, it would generate apk file when we build cts.
2. Add all source folder (includes src java files, Android.mk and its Manifest file) to mydroid/cts/tests/tests
3. If you'd like to create a test plan for this test package. Modify this python script: mydroid/cts/tools/utils/buildCts.py

Adjust CTS program settings?

Modify $mydroid/cts/tools/utils/host_config.xml

• Number of tests executed between reboots. A value <= 0 disables reboots. (maxTestCount)
• Max size [tests] for a package to be run in batch mode. (maxTestInBatchMode)
• Max time [ms] between test status updates. (testStatusTimeoutMs)
• Max time [ms] from start of package in batch mode and the first test status update. (batchStartTimeoutMs)
• Max time [ms] from start of test in individual mode to the first test status update. (individualStartTimeoutMs)
• Timeout [ms] for the signature check. (signatureTestTimeoutMs)
• Timeout [ms] for package installations. (packageInstallTimeoutMs)
• Time to wait [ms] after a package installation or removal. (postInstallWaitMs)

Write your own testing package?

If you would like to write your own testing package, you may reference the Instrumentation Testing document from Android porting guide. Also, you could reference the source code from few Android application, like Browser, Messages, Gallery, Email, Camera, Calculator....etc. You can build its test apk file and upload it to device.

How to write test cases?
Each instrumentation test case is similar to an Android application with the distinction that it starts another application. For example, have a look in the $MYDROID/packages/apps/Music directory.

• There should be a Makefile and an Android Manifest file
• Tests are located in $MYDROID/packages/apps/Music/tests.
• The Instrumentation Test Runner is located at packages/apps/Music/tests/src/com/android/music/MusicPlayerFunctionalTestRunner.java.

Build package apk file

erin@midnight:~/eclair/mydroid/packages/apps/Music/tests$ mm
Install: out/target/product/generic/data/app/MusicTests.apk

Install it to the device
erin@midnight:~/eclair/mydroid/packages/apps/Music/tests$ adb install ../../../../out/target/product/generic/data/app/MusicTests.apk

How to run test cases in device?

Running Tests

erin@midnight:~/$ adb shell pm list instrumentation
instrumentation:com.android.music.tests/.MusicPlayerStressTestRunner (target=com.android.music)
instrumentation:com.android.music.tests/.MusicPlayerFunctionalTestRunner (target=com.android.music)
instrumentation:com.android.music.tests/.MusicPlayerLaunchPerformance (target=com.android.music)

The am command is a command-line interface to the ActivityManager. 'am' is used to start and instrument activities using the adb shell command, as shown in the snippet below:

> adb shell am

usage: am [start|instrument]

      am start [-a <ACTION>] [-d <DATA_URI>] [-t <MIME_TYPE>]

               [-c <CATEGORY> [-c <CATEGORY>] ...]

               [-e <EXTRA_KEY> <EXTRA_VALUE> [-e <EXTRA_KEY> <EXTRA_VALUE> ...]

               [-n <COMPONENT>] [-D] [<URI>]

      am instrument [-e <ARG_NAME> <ARG_VALUE>] [-p <PROF_FILE>]

               [-w] <COMPONENT>



For example, to start the Contacts application you can use

> adb shell am start -n com.google.android.contacts/.ContactsActivity

Eg. verify Music player launcher performance

erin@midnight:~/$ adb shell am instrument -w -r com.android.music.tests/.MusicPlayerLaunchPerformance

Eg. verify Music player stress test

erin@midnight:~/$ adb shell am instrument -w -r com.android.music.tests/.MusicPlayerStressTestRunner

Here is a video about running a cts test case by Android emulator!