testNG 不错的单元测框架--我喜欢

最近使用了一些testNG单元测试框架，感觉很是灵活方便，可以定义测试方法之间顺序和分组，用起来也很简单直观，利用JDK5的标注来进行测试，不错，我喜欢

Table of Contents

1 - Introduction
2 - Annotations
3 - testng.xml
4 - Running TestNG
5 - User's manual
5.1 - Test groups
5.2 - Groups of groups
5.3 - Exclusion groups
5.4 - Partial groups
5.5 - Parameters
5.5.1 - From testng.xml
5.5.2 - From DataProviders
5.5.3 - Parameters in the reports
5.6 - Dependent methods
5.7 - Factories
5.8 - Parallel running and time-outs
5.9 - Rerunning failed tests
5.10 - JUnit tests
5.11 - JDK 1.4
5.12 - Running TestNG programmatically
5.13 - BeanShell and advanced group selection
5.14 - Annotation transformers
5.15 - Migrating to JDK annotations from javadoc annotations
6 - Test results
6.1 - Success, failure and assert
6.2 - Logging and results
6.2.1 - Listeners
6.2.2 - Reporters
6.2.3 - JUnitReports
6.2.4 - Reporter API
6.2.5 - XML Reports

1 - Introduction

TestNG is a testing framework designed to simply a broad range of testing needs, from unit testing (testing a class in isolation of the others) to integration testing (testing entire systems made of several classes, several packages and even several external frameworks, such as application servers).

Writing a test is typically a three-step process:

• Write the business logic of your test and insert TestNG annotations in your code.
• Add the information about your test (e.g. the class name, the groups you wish to run, etc...) in a testng.xml file or in build.xml.
• Run TestNG.

You can find a quick example on the Welcome page.

The concepts used in this documentation are as follows:

• A suite is represented by one XML file. It can contain one or more tests and is defined by the <suite> tag.
• A test is represented by <test> and can contain one or more TestNG classes.
• A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the <class> tag and can contain one or more test methods.
• A test method is a Java method annotated by @Test in your source.

A TestNG test can be configured by @BeforeXXX and @AfterXXX annotations which allows to perform some Java logic before and after a certain point, these points being either of the items listed above.

The rest of this manual will explain the following:

• A list of all the annotations with a brief explanation. This will give you an idea of the various functionalities offered by TestNG but you will probably want to consult the section dedicated to each of these annotations to learn the details.
• A description of the testng.xml file, its syntax and what you can specify in it.
• A detailed list of the various features and how to use them with a combination of annotations and testng.xml.

2 - Annotations

Here is a quick overview of the annotations available in TestNG along with their attributes.

@BeforeSuite @AfterSuite @BeforeTest @AfterTest @BeforeGroups @AfterGroups @BeforeClass @AfterClass @BeforeMethod @AfterMethod		Configuration information for a TestNG class: @BeforeSuite: The annotated method will be run before all tests in this suite have run. @AfterSuite: The annotated method will be run after all tests in this suite have run. @BeforeTest: The annotated method will be run before the test. @AfterTest: The annotated method will be run after the test. @BeforeGroups: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked. @AfterGroups: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked. @BeforeClass: The annotated method will be run before all the tests in the test class have been run. @AfterClass: The annotated method will be run after all the tests in the test class have been run. @BeforeMethod: The annotated method will be run before any test method is invoked. @AfterMethod: The annotated method will be run after any test method is invoked.
	alwaysRun	For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to. For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.
	dependsOnGroups	The list of groups this method depends on.
	dependsOnMethods	The list of methods this method depends on.
	enabled	Whether methods on this class/method are enabled.
	groups	The list of groups this class/method belongs to.
	inheritGroups	If true, this method will belong to groups specified in the @Test annotation at the class level.
@DataProvider		Marks a method as supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.
	name	The name of this DataProvider.
@Factory		Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[].
@Parameters		Describes how to pass parameters to a @Test method.
	value	The list of variables used to fill the parameters of this method.
@Test		Marks a class or a method as part of the test.
	alwaysRun	If set to true, this test method will always be run even if it depends on a method that failed.
	dataProvider	The name of the data provider for this test method.
	dataProviderClass	The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class.
	dependsOnGroups	The list of groups this method depends on.
	dependsOnMethods	The list of methods this method depends on.
	description	The description for this method.
	enabled	Whether methods on this class/method are enabled.
	expectedExceptions	The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
	groups	The list of groups this class/method belongs to.
	invocationCount	The number of times this method should be invoked.
	successPercentage	The percentage of success expected from this method
	sequential	If set to true, all the methods on this test class are guaranteed to run sequentially, even if the tests are currently being run with parallel="true". This attribute can only be used at the class level and it will be ignored if used at the method level.
	timeOut	The maximum number of milliseconds this test should take.
	threadPoolSize	The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount. Note: this attribute is ignored if invocationCount is not specified

3 - testng.xml

You can invoke TestNG in several different ways:

• With a testng.xml file
• With ant
• From the command line

This section describes the format of testng.xml (you will find documentation on ant and the command line below).

The current DTD for testng.xml can be found on the main Web site: http://testng.org/testng-1.0.dtd (for your convenience, you might prefer to browse the HTML version).

Here is an example testng.xml file:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Suite1"  verbose="1" >
<test name="Nopackage" >
<classes>
<class name="NoPackageTest" />
</classes>
</test>

<test name="Regression1"  >
<classes>
<class name="test.sample.ParameterSample" />
<class name="test.sample.ParameterTest" />
</classes>
</test>
</suite>

You can specify package names instead of class names:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Suite1" verbose="1" >
<test name="Regression1"  >
<packages>
<package name="test.sample" />
</packages>
</test>
</suite>

In this example, TestNG will look at all the classes in the package test.sample and will retain only classes that have TestNG annotations.

You can also specify groups and methods to be included and excluded:

<test name="Regression1">
<groups>
<run>
<exclude name="brokenTests" />
<include name="checkinTests" />
</run>
</groups>

<classes>
<class name="test.IndividualMethodsTest">
<methods>
<include name="testMethod" />
</methods>
</class>
</classes>
</test>

You can also define new groups inside testng.xml and specify additional details in attributes, such as whether to run the tests in parallel, how many threads to use, whether you are running JUnit tests, etc... Please see the DTD for a complete list of the features, or read on.

4 - Running TestNG

TestNG can be invoked in different ways:

• Command line
• ant
• Eclipse
• IntelliJ's IDEA

This section only explains how to invoke TestNG from the command line. Please click on one of the links above if you are interested in one of the other ways.

Assuming that you have TestNG in your class path, the simplest way to invoke TestNG is as follows:

java org.testng.TestNG testng1.xml [testng2.xml testng3.xml ...]

You need to specify at least one XML file describing the TestNG suite you are trying to run. Additionally, the following command-line switches are available:

Command Line Parameters

Option	Argument	Documentation
-d	A directory	The directory where the reports will be generated (defaults to test-output).
-sourcedir	A semi-colon separated list of directories.	The directories where your javadoc annotated test sources are. This option is only necessary if you are using javadoc type annotations. (e.g. "src/test" or "src/test/org/testng/eclipse-plugin;src/test/org/testng/testng").
-testclass	A comma-separated list of classes that can be found in your classpath.	A list of class files separated by commas (e.g. "org.foo.Test1,org.foo.test2").
-groups	A comma-separated list of groups.	The list of groups you want to run (e.g. "windows,linux,regression").
-excludegroups	A comma-separated list of groups.	The list of groups you want to be excluded from this run.
-testrunfactory	A Java classes that can be found on your classpath.	Lets you specify your own test runners. The class needs to implement org.testng.ITestRunnerFactory .
-listener	A comma-separated list of Java classes that can be found on your classpath.	Lets you specify your own test listeners. The classes need to implement org.testng.ITestListener
-parallel	methods|tests	If specified, sets the default mechanism used to determine how to use parallel threads when running tests. If not set, default mechanism is not to use parallel threads at all. This can be overridden in the suite definition.
-threadcount	The default number of threads to use when running tests in parallel.	This sets the default maximum number of threads to use for running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.
-suitename	The default name to use for a test suite.	This specifies the suite name for a test suite defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different suite name. It is possible to create a suite name with spaces in it if you surround it with double-quotes "like this".
-testname	The default name to use for a test.	This specifies the name for a test defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different test name. It is possible to create a test name with spaces in it if you surround it with double-quotes "like this".
-reporter	The extended configuration for a custom report listener.	Similar to the -listener option, except that it allows the configuration of JavaBeans-style properties on the reporter instance. Example: -reporter com.test.MyReporter:methodFilter=*insert*,enableFiltering=true You can have as many occurences of this option, one for each reporter that needs to be added.

This documentation can be obtained by invoking TestNG without any arguments.

You can also put the command line switches in a text file, say c:\command.txt, and tell TestNG to use that file to retrieve its parameters:

  C:> more c:\command.txt
  -d test-output testng.xml
  C:> java org.testng.TestNG @c:\command.txt

Additionally, TestNG can be passed properties on the command line of the Java Virtual Machine, for example

java -Dtestng.test.classpath="c:/build;c:/java/classes;" org.testng.TestNG testng.xml

Here are the properties that TestNG understands:

System properties

Property	Type	Documentation
testng.test.classpath	A semi-colon separated series of directories that contain your test classes.	If this property is set, TestNG will use it to look for your test classes instead of the class path. This is convenient if you are using the package tag in your XML file and you have a lot of classes in your classpath, most of them not being test classes.

Example:

java org.testng.TestNG -groups windows,linux -testclass org.test.MyTest

Note that the ant task and testng.xml allow you to launch TestNG with more parameters (methods to include, specifying parameters, etc...), so you should consider using the command line only when you are trying to learn about TestNG and you want to get up and running quickly.

5 - Test methods, Test classes and Test groups

5.1 - Test groups

TestNG allows you to perform sophisticated groupings of test methods. Not only can you declare that methods belong to groups, but you can also specify groups that contain other groups. Then TestNG can be invoked and asked to include a certain set of groups (or regular expressions) while excluding another set. This gives you maximum flexibility in how you partition your tests and doesn't require you to recompile anything if you want to run two different sets of tests back to back.

For example, it is quite common to have at least two categories of tests

• Check-in tests. These tests should be run before you submit new code. They should typically be fast and just make sure no basic functionality was broken.
  
• Functional tests. These tests should cover all the functionalities of your software and be run at least once a day, although ideally you would want to run them continuously.

Typically, check-in tests are a subset of functional tests. TestNG allows you to specify this in a very intuitive way with test groups. For example, you could structure your test by saying that your entire test class belongs to the "functest" group, and additionally that a couple of methods belong to the group "checkintest":

public class Test1 {
@Test(groups = { "functest", "checkintest" })
public void testMethod1() {
}

@Test(groups = {"functest", "checkintest"} )
public void testMethod2() {
}

@Test(groups = { "functest" })
public void testMethod3() {
}

}

Invoking TestNG with

<test name="Test1">
<groups>
<run>
<include name="functest"/>
</run>
</groups>
<classes>
<class name="example1.Test1"/>
</classes>
</test>

will run all the test methods in that classes, while invoking it with checkintest will only run testMethod1() and testMethod2().

Here is another example, using regular expressions this time. Assume that some of your test methods should not be run on Linux, your test would look like:

@Test
public class Test1 {
@Test(groups = { "windows.checkintest" }) 
public void testWindowsOnly() {
}

@Test(groups = {"linux.checkintest"} )
public void testLinuxOnly() {
}

@Test(groups = { "windows.functest" )
public void testWindowsToo() {
}
}

You could use the following testng.xml to launch only the Windows methods:

<test name="Test1">
<groups>
<run>
<include name="windows.*"/>
</run>
</groups>

<classes>
<class name="example1.Test1"/>
</classes>
</test>

Note: TestNG uses regular expressions, and not wildmats. Be aware of the difference (for example, "anything" is matched by ".*" -- dot star -- and not "*").

Method groups

You can also exclude or include individual methods:

<test name="Test1">
<classes>
<class name="example1.Test1">
<methods>
<include name=".*enabledTestMethod.*"/>
<exclude name=".*brokenTestMethod.*"/>
</methods>
</class>
</classes>
</test>

This can come in handy to deactivate a single method without having to recompile anything, but I don't recommend using this technique too much since it makes your testing framework likely to break if you start refactoring your Java code (the regular expressions used in the tags might not match your methods any more).

5.2 - Groups of groups

Groups can also include other groups.These groups are called "MetaGroups". For example, you might want to define a group "all" that includes "checkintest" and "functest". "functest" itself will contain the groups "windows" and "linux" while "checkintest will only contain "windows". Here is how you would define this in your property file:

<test name="Regression1">
<groups>
<define name="functest">
<include name="windows"/>
<include name="linux"/>
</define>

<define name="all">
<include name="functest"/>
<include name="checkintest"/>
</define>

<run>
<include name="all"/>
</run>
</groups>

<classes>
<class name="test.sample.Test1"/>
</classes>
</test>

5.3 - Exclusion groups

TestNG allows you to include groups as well as exclude them.

For example, it is quite usual to have tests that temporarily break because of a recent change, and you don't have time to fix the breakage yet. 4 However, you do want to have clean runs of your functional tests, so you need to deactivate these tests but keep in mind they will need to be reactivated.

A simple way to solve this problem is to create a group called "broken" and make these test methods belong to it. For example, in the above example, I know that testMethod2() is now broken so I want to disable it:

@Test(groups = {"checkintest", "broken"} )
public void testMethod2() {
}

All I need to do now is to exclude this group from the run:

<test name="Simple example">
<groups>
<run>
<include name="checkintest"/>
<exclude name="broken"/>
</run>
</groups>

<classes>
<class name="example1.Test1"/>
</classes>
</test>

This way, I will get a clean test run while keeping track of what tests are broken and need to be fixed later.

Note: you can also disable tests on an individual basis by using the "enabled" property available on both @Test and @Before/After annotations.

5.4 - Partial groups

You can define groups at the class level and then add groups at the method level:

@Test(groups = { "checkin-test" })
public class All {

@Test(groups = { "func-test" )
public void method1() { ... }

public void method2() { ... }
}

In this class, method2() is part of the group "checkin-test", which is defined at the class level, while method1() belongs to both "checkin-test" and "func-test".

5.5 - Parameters

Test methods don't have to be parameterless. You can use an arbitrary number of parameters on each of your test method, and you instruct TestNG to pass you the correct parameters with the @Parameters annotation.

There are two ways to set these parameters: with testng.xml or programmatically.

5.5.1 - Parameters from testng.xml

If you are using simple values for your parameters, you can specify them in your testng.xml:

@Parameters({ "first-name" })
@Test
public void testSingleString(String firstName) { 
System.out.println("Invoked testString " + firstName);
assert "Cedric".equals(firstName);
}

In this code, we specify that the parameter firstName of your Java method should receive the value of the XML parameter called first-name. This XML parameter is defined in testng.xml:

<suite name="My suite">
<parameter name="first-name" value="Cedric"/>
<test name="Simple example">
 <-- ... -->

The same technique can be used for @Before/After and @Factory annotations:

@Parameters({ "datasource", "jdbcDriver" })
@BeforeMethod
public void beforeTest(String ds, String driver) {
 m_dataSource = ...;               // look up the value of datasource
 m_jdbcDriver = driver;
}

This time, the two Java parameter ds and driver will receive the value given to the properties datasource and jdbc-driver respectively.

The @Parameters annotation can be placed at the following locations:

• On any method that already has a @Test, @Before/After or @Factory annotation.
• On at most one constructor of your test class. In this case, TestNG will invoke this particular constructor with the parameters initialized to the values specified in testng.xml whenever it needs to instantiate your test class. This feature can be used to initialize fields inside your classes to values that will then be used by your test methods.

Notes:

• The XML parameters are mapped to the Java parameters in the same order as they are found in the annotation, and TestNG will issue an error if the numbers don't match.
• Parameters are scoped. In testng.xml, you can declare them either under a <suite> tag or under <test>. If two parameters have the same name, it's the one defined in <test> that has precedence. This is convenient if you need to specify a parameter applicable to all your tests and override its value only for certain tests.

5.5.2 - Parameters with DataProviders

Specifying parameters in testng.xml might not be sufficient in the following cases:

• You are not using a testng.xml.
• You need to pass complex parameters, or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc...).

In this case, you can use a Data Provider to supply the values you need to test. A Data Provider is a method on your class that returns an array of array of objects. This method is annotated with @DataProvider:

//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
return new Object[][] {
{ "Cedric", new Integer(36) },
{ "Anne", new Integer(37)}, 
};
}

//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
System.out.println(n1 + " " + n2);
} 

will print

Cedric 36
Anne 37

A @Test method specifies its Data Provider with the dataProvider attribute. This name must correspond to a method on the same class annotated with @DataProvider(name="...") with a matching name.

By default, the data provider will be looked for in the current test class or one of its base classes. If you want to put your data provider in a different class, it needs to be a static method and you specify the class where it can be found in the dataProviderClass attribute:

public static class StaticProvider {
@DataProvider(name = "create")
public static Object[][] createData() {
return new Object[][] {
new Object[] { new Integer(42) }
}
}
}

public class MyTest {
@Test(dataProvider = "create", dataProviderClass = StaticProvider.class)
public void test(Integer n) {
// ...
}
}

The Data Provider method can return one of the following two types:

• An array of array of objects (Object[][]) where the first dimension's size is the number of times the test method will be invoked and the second dimension size contains an array of objects that must be compatible with the parameter types of the test method. This is the cast illustrated by the example above.
• An Iterator<Object[]>. The only difference with Object[][] is that an Iterator lets you create your test data lazily. TestNG will invoke the iterator and then the test method with the parameters returned by this iterator one by one. This is particularly useful if you have a lot of parameter sets to pass to the method and you don't want to create all of them upfront.

Here is an example of this feature for both JDK 1.4 and JDK5 (note that the JDK 1.4 example does not use Generics):

/**
* @testng.data-provider name="test1"
*/
public Iterator createData() {
return new MyIterator(DATA);
)

@DataProvider(name = "test1")
public Iterator createData() {
return new MyIterator(DATA);
} 

If you declare your @DataProvider as taking a java.lang.reflect.Method as first parameter, TestNG will pass the current test method for this first parameter. This is particularly useful when several test methods use the same @DataProvider and you want it to return different values depending on which test method it is supplying data for.

For example, the following code prints the name of the test method inside its @DataProvider:

@DataProvider(name = "dp")
public Object[][] createData(Method m) {
System.out.println(m.getName()); // print test method name
return new Object[][] { new Object[] { "Cedric" }};
}

@Test(dataProvider = "dp")
public void test1(String s) {
}

@Test(dataProvider = "dp")
public void test2(String s) {
}

and will therefore display:

test1
test2

5.5.3 - Parameters in reports

Parameters used to invoke your test methods are shown in the HTML reports generated by TestNG. Here is an example:

5.6 - Dependent methods

Sometimes, you need your test methods to be invoked in a certain order. This is useful for example

• To make sure a certain number of test methods have completed and succeeded before running more test methods.
• To initialize your tests while wanting this initialization methods to be test methods as well (methods tagged with @Before/After will not be part of the final report).

In order to do this, you can use the attributes dependsOnMethods or dependsOnGroups, found on the @Test annotation.

There are two kinds of dependencies:

• Hard dependencies. All the methods you depend on must have run and succeeded for you to run. If at least one failure occurred in your dependencies, you will not be invoked and marked as a SKIP in the report.
• Soft dependencies. You will always be run after the methods you depend on, even if some of them have failed. This is useful when you just want to make sure that your test methods are run in a certain order but their success doesn't really depend on the success of others. A soft dependency is obtained by adding "alwaysRun=true" in your @Test annotation.

Here is an example of a hard dependency:

@Test