End-to-End Test Example

End-to-End Test Example

This tutorial guides you through the construction of a "hello world" Trade Federation test configuration, and gives you a hands-on introduction to the Trade Federation framework. Starting from the TF development environment, it guides you through the process of creating a simple Trade Federation config and gradually adding more features to it.

The tutorial presents the TF test development process as a set of exercises, each consisting of several steps. The exercises demonstrate how to gradually build and refine your configuration, and provide all the sample code you need to complete the test configuration. The title of each exercise is annotated with a letter describing which roles are involved in that step: D for Developer, I for Integrator, and/or R for Test Runner.

When you are finished with the tutorial, you will have created a functioning TF configuration and will have learned many of the most important concepts in the TF framework.

Set up Trade Federation development environment

---

See the Machine Setup page for how to setup the development environment. The rest of this tutorial assumes you have a shell open that has been initialized to the Trade Federation environment.

For simplicity, this tutorial will illustrate adding a configuration and its classes to the Trade Federation framework core library. This can be extended to developing modules outside the source tree by simply compiling the tradefed JAR, and compiling your modules against that JAR.

Creating a test class (D)

---

Lets create a hello world test that just dumps a message to stdout. A tradefed test will generally implement the IRemoteTest interface.

Here's an implementation for the HelloWorldTest:

package com.android.tradefed.example;

import com.android.tradefed.device.DeviceNotAvailableException;
import com.android.tradefed.result.ITestInvocationListener;
import com.android.tradefed.testtype.IRemoteTest;

public class HelloWorldTest implements IRemoteTest {
    @Override
    public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
        System.out.println("Hello, TF World!");
    }
}

Save this sample code to <tree>/tools/tradefederation/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java and rebuild tradefed from your shell:

m -jN

If the build does not succeed, consult the Machine Setup page to ensure that you didn't miss any steps.

Creating a Configuration (I)

---

Trade Federation tests are made executable by creating a Configuration, which is an XML file that instructs tradefed on which test (or tests) to run, as well as which other modules to execute, and in what order.

Lets create a new Configuration for our HelloWorldTest:

<configuration description="Runs the hello world test">
    <test class="com.android.tradefed.example.HelloWorldTest" />
</configuration>

TF will parse the Configuration XML file (aka config), load the specified class using reflection, instantiate it, cast it to a IRemoteTest, and call its run method.

Note that we've specified the full class name of the HelloWorldTest. Save this data to a helloworld.xml file anywhere on your local filesystem (eg /tmp/helloworld.xml).

Running the config (R)

---

From your shell, launch the tradefed console

$ tradefed.sh

Ensure that a device is connected to the host machine and is visible to tradefed

tf >list devices
Serial            State      Product   Variant   Build   Battery  
004ad9880810a548  Available  mako      mako      JDQ39   100

Configurations can be executed using the run <config> console command. Try this:

tf> run /tmp/helloworld.xml
05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!

You should see "Hello, TF World!" outputted on the terminal.

Adding the config to the Classpath (D, I, R)

---

For convenience of deployment, you can also bundle configs into the tradefed jars themselves. Tradefed will automatically recognize all configurations placed in 'config' folders on the classpath.

Lets illustrate this now by moving the helloworld.xml into the tradefed core library.

Move the helloworld.xml file into <tree>/tools/tradefederation/prod-tests/res/config/example/helloworld.xml.

Rebuild tradefed, and restart the tradefed console.

Ask tradefed to display the list of configurations from the classpath:

tf> list configs
[…]
example/helloworld: Runs the hello world test

You can now run the helloworld config via the following command

tf >run example/helloworld
05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!

Interacting with a Device (D, R)

---

So far our hello world test isn't doing anything interesting. Tradefed's specialty is running tests using Android devices, so lets add an Android device to the test.

Tests can get a reference to an Android device by implementing the IDeviceTest interface.

Here's a sample implementation of what this looks like:

public class HelloWorldTest implements IRemoteTest, IDeviceTest {
    private ITestDevice mDevice;
    @Override
    public void setDevice(ITestDevice device) {
        mDevice = device;
    }

    @Override
    public ITestDevice getDevice() {
        return mDevice;
    }
…
}

The Trade Federation framework will inject the ITestDevice reference into your test via the IDeviceTest#setDevice method, before the IRemoteTest#run method is called.

Let's modify the HelloWorldTest print message to display the serial number of the device.

@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
    System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());
}

Now rebuild tradefed, and check the list of devices:

$ tradefed.sh
tf >list devices
Serial            State      Product   Variant   Build   Battery  
004ad9880810a548  Available  mako      mako      JDQ39   100

Take note of the serial number listed as Available above. That is the device that should be allocated to HelloWorld.

tf >run example/helloworld
05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548

You should see the new print message displaying the serial number of the device.

Sending Test Results (D)

---

IRemoteTests report results by calling methods on the ITestInvocationListener instance provided to their #run method. Note that the TF framework itself is responsible for reporting the start and end of each Invocation, (via the ITestInvocationListener#invocationStarted and ITestInvocationListener#invocationEnded methods, respectively).

A test run is a logical collection of tests. To report test results, IRemoteTests are responsible for reporting the start of a test run, the start and end of each test, and the end of the test run.

Here's what the HelloWorldTest implementation might look like with a single failed test result.

@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
    System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());

    TestIdentifier testId = new TestIdentifier("com.example.TestClassName", "sampleTest");
    listener.testRunStarted("helloworldrun", 1);
    listener.testStarted(testId);
    listener.testFailed(TestFailure.FAILURE, testId, "oh noes, test failed");
    listener.testEnded(testId, Collections.emptyMap());
    listener.testRunEnded(0, Collections.emptyMap());
}

Note that Trade Federation also includes several IRemoteTest implementations that you can reuse instead of writing your own from scratch. These include, for instance, InstrumentationTest, which can run an Android application's tests remotely on an Android device, parse the results, and forward them to the ITestInvocationListener). See the Test Types documentation for more details.

Storing Test Results (I)

---

By default, a TF config will use the TextResultReporter as the test listener implementation. TextResultReporter will dump the results of an invocation to stdout. To illustrate, try running the hello-world config from the previous section:

$ ./tradefed.sh
tf >run example/helloworld
05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests
Test FAILURE: com.example.TestClassName#sampleTest 
 stack: oh noes, test failed 
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms

If you want to store the results of an invocation elsewhere, such as in a file, you need to specify a custom ITestInvocationListener implementation by using the result_reporter tag in your configuration.

Trade Federation includes the XmlResultReporter listener, which will write test results to an XML file, in a format similar to that used by the ant JUnit XML writer.

Let's specify the result_reporter in the configuration now. Edit the …/res/config/example/helloworld.xml config like this:

<configuration description="Runs the hello world test">
    <test class="com.android.tradefed.example.HelloWorldTest" />
    <result_reporter class="com.android.tradefed.result.XmlResultReporter" />
</configuration>

Now rebuild tradefed and re-run the hello world sample:

tf >run example/helloworld
05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt
05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt
05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0

Notice the log message stating that an XML file has been generated. The generated file should look like this:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost">
  <properties />
  <testcase name="sampleTest" classname="com.example.TestClassName" time="0">
    <failure>oh noes, test failed
    </failure>
  </testcase>
</testsuite>

You can also write your own custom invocation listeners. It just needs to implement the ITestInvocationListener interface.

Also note that tradefed supports multiple invocation listeners, meaning that you can send test results to multiple independent destinations. Just specify multiple <result_reporter> tags in your config to do this.

Logging (D, I, R)

---

TradeFederation includes two logging facilities:

1. ability to capture logs from the device (aka device logcat)
2. ability to record logs from the TradeFederation framework running on the host machine (aka the host log)

Lets focus on #2 for now. Trade Federation's host logs are reported using the CLog wrapper for the ddmlib Log class.

Let's convert the previous System.out.println call in HelloWorldTest to a CLog call:

@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
    CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber());

Note that CLog handles string interpolation directly, akin to String.format. At this point, when you rebuild and rerun TF, you should see the log message on stdout.

tf> run example/helloworld
…
05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
…

By default, tradefed will output host log messages to stdout. TF also includes a log implementation that will write messages to a file: FileLogger. To add file logging, add a logger tag to the config, specifying the full class name of FileLogger.

<configuration description="Runs the hello world test">
    <test class="com.android.tradefed.example.HelloWorldTest" />
    <result_reporter class="com.android.tradefed.result.XmlResultReporter" />
    <logger class="com.android.tradefed.log.FileLogger" />
</configuration>

Now rebuild and run the helloworld example again.

tf >run example/helloworld 
…
05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt
05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…

Note the log message indicating the path of the host log. View the contents of that file, and you should see your HelloWorldTest log message

$ more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…
05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548

The TradeFederation framework will also automatically capture the logcat from the allocated device, and send it the invocation listener for processing. XmlResultReporter will save the captured device logcat as a file.

Option Handling (D, I, R)

---

Objects loaded from a Trade Federation Configuration (aka Configuration objects) also have the ability to receive data from command line arguments.

This is accomplished via the @Option annotation. To participate, a Configuration object class would apply the @Option annotation to a member field, and provide it a unique name. This would allow that member field's value to be populated via a command line option, and would also automatically add that option to the configuration help system.

Note: Not all field types are supported. See the OptionSetter javadoc for a description of supported types.

Let's add an @Option to the HelloWorldTest:

@Option(name="my_option",
        shortName='m',
        description="this is the option's help text",
        // always display this option in the default help text
        importance=Importance.ALWAYS)
private String mMyOption = "thisisthedefault";

And let's add a log message to display the value of the option in HelloWorldTest, so we can demonstrate that it was received correctly.

@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
    …
    CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption);

Rebuild TF and run helloworld; you should see a log message with my_option's default value.

tf> run example/helloworld
…
05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault'

Passing Values from the Command Line

Now pass in a value for my_option: you should see my_option getting populated with that value

tf> run example/helloworld --my_option foo
…
05-24 18:33:44 I/HelloWorldTest: I received option 'foo'

TF configurations also include a help system, which automatically displays help text for @Option fields. Try it now, and you should see the help text for my_option:

tf> run --help example/helloworld
Printing help for only the important options. To see help for all options, use the --help-all flag

  cmd_options options:
    --[no-]help          display the help text for the most important/critical options. Default: false.
    --[no-]help-all      display the full help text for all options. Default: false.
    --[no-]loop          keep running continuously. Default: false.

  test options:
    -m, --my_option      this is the option's help text Default: thisisthedefault.

  'file' logger options:
    --log-level-display  the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error.

Note the message at the top about "printing only the important options." To reduce option help clutter, TF uses the Option#importance attribute to determine whether to show a particular @Option field's help text when --help is specified. --help-all will always show help for all @Option fields, regardless of importance. See the Option.Importance javadoc for details.

Passing Values from a Configuration

You can also specify an Option's value within the config by adding a <option name="" value=""> element. Let's see how this looks in helloworld.xml:

<test class="com.android.tradefed.example.HelloWorldTest" >
    <option name="my_option" value="fromxml" />
</test>

Re-building and running helloworld should now produce this output:

05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml'

The configuration help should also be updated to indicate my_option's new default value:

tf> run --help example/helloworld
  test options:
    -m, --my_option      this is the option's help text Default: fromxml.

Also note that other configuration objects included in the helloworld config, such as FileLogger, also accept options. The option --log-level-display is interesting because it filters the logs that show up on stdout. You may have noticed from earlier in the tutorial that the "Hello, TF World! I have device …' log message stopped being displayed on stdout once we switched to using FileLogger. You can increase the verbosity of logging to stdout by passing in the --log-level-display arg.

Try this now, and you should see the 'I have device' log message reappear on stdout, in addition to being logged to a file.

tf >run --log-level-display info example/helloworld
…
05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548

That's All, Folks!

---

As a reminder, if you're stuck on something, the Trade Federation source code has a lot of useful information that isn't exposed in the documentation. And if all else fails, try asking on the android-platform Google Group, with "Trade Federation" in the message subject.