一.背景:
承接上一篇自动化二期CQC(TAOBAO TOAST框架二次开发)---支持自定义测试环境;
maven工程使用surefire插件,执行"mvn -Dtest=测试类 test"命令,stdout并不支持输出成功用例和skipped用例(JUnit中被注解为@Ignore的类或方法)的信息,只输出Failed和error的用例信息,如下:
,故前端无法根据stdout解析出成功的和skipped的用例详细。
尝试1:修改maven surefire插件源码让它支持输出success and fail的用例信息,查看源码,surefire(https://github.com/apache/maven-surefire/tree/master/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire)工程,见另一篇文章
修改maven surefire源码,支持stdout出成功和skipped用例,并自定义stdout用例格式
尝试2:用surefire生成的xml文件(在项目target/surefire-reports)进行解析,surefire生成的xml中包含了成功的和skipped用例,但是surefire是根据单独一个测试类生成一个xml结果,而我们的用例,使用了@ClassnameFilters聚合了同一功能的测试类(正则匹配),比如评论功能
@ClassnameFilters({"com.weibo.cases.xiaoyu.*CommentTest", "com.weibo.cases.wanglei16.*CommentTest","com.weibo.cases.xuelian.*CommentTest","com.weibo.cases.hugang.*CommentTest","com.weibo.cases.maincase.*CommentTest"})
这样会生成很多不同被匹配类的xml,不好统计某一功能的结果。
当然可以尝试surefire-report功能,但是这就脱离了TOAST自己解析。
二.stdout支持输出成功用例和skipped用例详情
第二种尝试失败后,转向底层用例实现上。
1.stdout出执行成功用例
通过查看JUnit apidoc(http://junit.org/apidocs/index.html),JUnit提供了类
org.junit.rules.TestWatcher, 记录了测试用例执行过程中的行为,包括用例开始,成功,失败,skipped(a test is skipped due to a failed assumption,而不是@Ignore注释的), 结束等行为。
@Rule
public TestWatcher testWatcher = new TestWatcher() {
@Override
protected void succeeded(Description description) {
System.out.println("Success: " + description.getDisplayName());
}
};
输出格式如下:
Success: testDestroyLikes(com.weibo.cases.wanglei16.LikesGroupTest)
2.stdout出skipped的用例(被注释为@Ignore的测试类或方法)
runnerForClass方法,使用工厂类,创建Runner, 自定义一个新的Runner,通过反射找出带有注解@Ignore的类或方法并stdout。如果类有@Ignore,则只输出类名,因为Junit最后统计结果时,会把@Ignore类记为1个用例,不是计算该类下面的测试方法数(实验验证过);否则,遍历方法,如果方法有@Ignore,则输出该方法。
主要代码如下,完整版请查看多线程执行用例这篇文章:
public ConcurrentSuite(final Class<?> klass) throws InitializationError {
// 调用父类ClasspathSuite构造函数
// AllDefaultPossibilitiesBuilder根据不同的测试类定义(@RunWith的信息)返回Runner,使用职责链模式
super(klass, new AllDefaultPossibilitiesBuilder(true) {
@Override
public Runner runnerForClass(Class<?> testClass) throws Throwable {
List<RunnerBuilder> builders = Arrays
.asList(new RunnerBuilder[] {
// 创建Runner, 工厂类,
// 自定义自己的Runner,找出注解为@Ignore,并输出@Ignore的类和方法名
new RunnerBuilder() {
@Override
public Runner runnerForClass(
Class<?> testClass)throws Throwable {
// 获取类的所有方法
Method[] methods = testClass.getMethods();
// 如果类有@Ignore,则只输出类名,因为Junit最后计算结果时,会把@Ignore类记为1个用例,
// 不是计算类下面的测试方法数(实验验证过)
// 否则,遍历方法,如果方法有@Ignore,则输出该方法
if (testClass.isAnnotationPresent(Ignore.class)) {
System.out.println("Ignore: "
+ testClass.getName());
} else {
for (Method method : methods) {
if (method.isAnnotationPresent(Ignore.class)) {
System.out.println("Ignore: " + testClass.getName() + "." + method.getName());
}
}
}
return null;
}
}, ignoredBuilder(), annotatedBuilder(),
suiteMethodBuilder(), junit3Builder(),
junit4Builder() });
for (RunnerBuilder each : builders) {
// 根据不同的测试类定义(@RunWith的信息)返回Runner
Runner runner = each.safeRunnerForClass(testClass);
if (runner != null)
// 方法级别,多线程执行
// return MulThread(runner);
return runner;
}
return null;
}
});
Ignore输出格式:
Ignore: com.weibo.cases.wanglei16.LikeObjectRpcTest
Ignore: com.weibo.cases.wanglei16.LikesGroupTest.test4
结合mvn test stdout出的fail,error用例格式,完整的用例详情stdout格式如下:
skipped用例
Ignore: com.weibo.cases.wanglei16.LikeObjectRpcTest
Ignore: com.weibo.cases.wanglei16.LikesGroupTest.test4
pass用例
Success: testLikeStatus(com.weibo.cases.wanglei16.LikeObjectRpcTest)
Success: testLikesUpdate(com.weibo.cases.wanglei16.LikeObjectRpcTest)
Failed tests:
LikeObjectRpcTest.testLikesUpdate:153
Expected: is "12312"
but: was "1042018:10012099744"
LikesByMeBatchTest.testSingleType:183
Expected: is an empty collection
but: <[com.weibo.model.Objects@42a6f5df]>
Tests in error:
LikesByMeBatchTest.testMultiType:190 » ArrayIndexOutOfBounds -1
对应的正则表达式:
fail
$failPattern = "#\s{2}(\w+\.\w+:\d+)\s[^\S]#";
error
$errorPattern = "#\s{2}(\w+\.\w+:\d{1,}\s[^\s].*)#";
Ignore
$skipPattern = "#Ignore:\s(.*)#";
success
$passPattern = "#Success:\s(.*)#";
<?php
/*
*@author hugang
*parse case info
*/
include_once('BaseParser.php');
class JUnitMvnParser extends BaseParser
{
protected function parseCaseAmount()
{
$amountPattern = '#Tests run: (\d+), Failures: (\d+), Errors: (\d+), Skipped: (\d+)[\r\n]+#';
preg_match_all($amountPattern, $this->output, $amountMatches);
$this->parserInfo->case_total_amount = array_sum($amountMatches[1]);
$this->parserInfo->case_failed_amount = array_sum($amountMatches[2]) + array_sum($amountMatches[3]);
$this->parserInfo->case_skipped_amount = array_sum($amountMatches[4]);
$this->parserInfo->case_passed_amount = $this->parserInfo->case_total_amount - $this->parserInfo->case_failed_amount - $this->parserInfo->case_skipped_amount;
}
protected function parseCases()
{
// 下行没有空格,自行清掉;[ INFO ]中了博客关键字
$idPattern = "#\ [ INFO \ ][^\n]*\nCASE\sID:\s(\d*)#s";
preg_match_all($idPattern, $this->output, $idMatches);
// fail case
$failPattern = "#\s{2}(\w+\.\w+:\d+)\s[^\S]#";
preg_match_all($failPattern, $this->output, $fmatches);
for($i = 0; $i < count($fmatches[1]); $i++) {
$caseInfo = new CaseInfo();
if (isset($idMatches[1][$i])) {
$caseInfo->id = trim($idMatches[1][$i]);
} else if (isset($idMatches[1])) {
$idx = count($idMatches[1]);
if (isset($idMatches[1][$idx - 1])) {
$caseInfo->id = trim($idMatches[1][$idx - 1]);
}
}
$caseInfo->name = trim($fmatches[1][$i]);
$caseInfo->info = trim($fmatches[1][$i]);
$caseInfo->result = CaseInfo::RESULT_FAILED;
if (!empty($caseInfo->id)) {
$testcase = TestCase::model()->findByPk($caseInfo->id);
if ($testcase !== null) {
$caseInfo->name = $testcase->name;
}
}
$this->parserInfo->cases[] = $caseInfo;
}
// fail Expected info
$failExpectPattern = "#(Expected:\sis.*)#";
preg_match_all($failExpectPattern, $this->output, $fematches);
for($j = 0; $j < count($fematches[1]); $j++){
if(isset($this->parserInfo->cases[$j])){
$caseInfo = $this->parserInfo->cases[$j];
$caseInfo->info .= "\n<b>" .$fematches[1][$j] . "</b>";
$this->parseInfo->cases[$j] = $caseInfo;
}
}
// fail But info
$failButPattern = "#\s{6}(but:\s.*)#";
preg_match_all($failButPattern, $this->output, $fbmatches);
for($m = 0; $m < count($fbmatches[1]); $m++){
if(isset($this->parserInfo->cases[$m])){
$caseInfo = $this->parserInfo->cases[$m];
$caseInfo->info .= "<b> " .$fbmatches[1][$m] . "</b>";
$this->parseInfo->cases[$m] = $caseInfo;
}
}
// error cases info
$errorPattern = "#\s{2}(\w+\.\w+:\d{1,}\s[^\s].*)#";
preg_match_all($errorPattern, $this->output, $fematches);
for($i = 0; $i < count($fematches[1]); $i++) {
$caseInfo = new CaseInfo();
if (isset($idMatches[1][$i])) {
$caseInfo->id = trim($idMatches[1][$i]);
} else if (isset($idMatches[1])) {
$idx = count($idMatches[1]);
if (isset($idMatches[1][$idx - 1])) {
$caseInfo->id = trim($idMatches[1][$idx - 1]);
}
}
$caseInfo->name = trim($fematches[1][$i]);
$caseInfo->info = trim($fematches[1][$i]);
$caseInfo->result = CaseInfo::RESULT_FAILED;
if (!empty($caseInfo->id)) {
$testcase = TestCase::model()->findByPk($caseInfo->id);
if ($testcase !== null) {
$caseInfo->name = $testcase->name;
}
}
$this->parserInfo->cases[] = $caseInfo;
}
// ignore cases info
$skipPattern = "#Ignore:\s(.*)#";
preg_match_all($skipPattern, $this->output, $smatches);
foreach ($smatches[1] as $key => $smatch) {
$caseInfo = new CaseInfo();
if (isset($idMatches[1][$key])) {
$caseInfo->id = trim($idMatches[1][$key]);
} else if (isset($idMatches[1])) {
$idx = count($idMatches[1]);
if (isset($idMatches[1][$idx - 1])) {
$caseInfo->id = trim($idMatches[1][$idx - 1]);
}
}
$caseInfo->name = trim($smatches[1][$key]);
$caseInfo->info = trim($smatches[1][$key]);
$caseInfo->result = CaseInfo::RESULT_SKIPPED;
if (!empty($caseInfo->id)) {
$testcase = TestCase::model()->findByPk($caseInfo->id);
if ($testcase !== null) {
$caseInfo->name = $testcase->name;
}
}
$this->parserInfo->cases[] = $caseInfo;
}
// success cases info
$passPattern = "#Success:\s(.*)#";
preg_match_all($passPattern, $this->output, $pmatches);
foreach ($pmatches[1] as $key => $pmatch)
{
$caseInfo = new CaseInfo();
if(isset($idMatches[1][$key]))
{
$caseInfo->id = trim($idMatches[1][$key]);
}
else if(isset($idMatches[1]))
{
$idx = count($idMatches[1]);
if(isset($idMatches[1][$idx-1]))
{
$caseInfo->id = trim($idMatches[1][$idx-1]);
}
}
$caseInfo->name = trim($pmatches[1][$key]);
$caseInfo->info = trim($pmatch);
$caseInfo->result = CaseInfo::RESULT_PASSED;
if(!empty($caseInfo->id))
{
$testcase = TestCase::model()->findByPk($caseInfo->id);
if($testcase !== null)
{
$caseInfo->name = $testcase->name;
}
}
$this->parserInfo->cases[] = $caseInfo;
}
}
}
?>
结果展示如下,展示顺序(1.failed cases->2.error cases->3.skipped cases->4.success cases):