自动化二期CQC（TAOBAO TOAST框架二次开发）---支持自定义测试环境

 一.背景

小组内决定自动化二期，添加前端调用，展示运行结果。

一期是由Ant构建，过程：1.Compile Java Junit Cases->2.JUnit target->3.JUnit Report target->4.Report Copy to Tomcat target->5.Email notify Result Link target；

通过不同的build.xml指定不同功能的class， 使用shell编写脚本，选择测试环境，再选择相应的模块；shell会先Svn拉下最新的用例工程代码，重置用户，根据用户选择的环境修改用例工程中配置文件内容，再根据选择的模块去运行不同的build.xml。

在github寻找自动化测试开源框架，只有TOAST和我们现有的工程能很好的整合。

比如：

百度的ITEST（https://github.com/BaiduQA/itest）： 就跟我们的自动化一期的类似，也是执行ant，得到结果，也没用到前端调用；

网易的Dagger（https://github.com/NetEase/Dagger）： a light, robust Web UI auto test framework ， 使用了Selenium（浏览器兼容性测试，ThoughWorks）和TestNg，也并不适合我们。

TOAST（https://github.com/alibaba/toast ) ：是由taobao etao测试团队开发的一套自动化测试框架，支持单元测试，功能测试，集成测试；由web端，Controller端，Agent端三部分组成，web端由PHP YII框架（MVC）编写的前端，Controller端和Agent端都是C++编写的，通过gcc编译生成二进制执行脚本，用户通过web端自定义测试内容，当用户点击运行某一测试任务，web端将需要执行的内容写入协议文件，/tmp/toast/Run-Created_***_*_*******.ini；controller端实时扫描目录/tmp/toast， 如果有新文件（Run-Created_***_*_*******.ini）将文件内容读取，解析出执行命令，传递给命令中对应机器（TestBox）的agent，对应的agent去执行相应的脚本，以功能测试为例：前端设置相应的SVN 地址（测试工程地址），对应的测试类，点击运行；controller端监控到新的协议文件：

2015-04-03 16:26:01 [INFO]: (./toastcontroller,,7770,0)There is command file: /tmp/toast/Run_Create_141_0_1428049561.ini
2015-04-03 16:26:01 [DEBUG]: (./toastcontroller,,7770,0)requestStr:
{"RunID":"141","Commands":[{"CommandID":"141","TestBox":"10.210.230.26","TestCommand":"python \/data1\/toast\/plugin\/case_run\/run_case -t mvn -u https:\/\/svn1.intra.sina.com.cn\/weibo_QA\/TEST\/work\/AutoTestContent -c 6 -f LikesGroupTest;","Timeout":"127","Sudoer":"root"}],"TestType":"Regress"}

第一行为controller监控到的文件（web端写入），第二行到末尾为请求的内容，RunID为这次运行的id:141，TestBox为指定的测试机器，TestCommand在测试机上执行的命令：python \/data1\/toast\/plugin\/case_run\/run_case -t mvn -u https:\/\/svn1.intra.sina.com.cn\/weibo_QA\/TEST\/work\/AutoTestContent -c 6 -f LikesGroupTest;","Timeout":"127","Sudoer":"root"， 执行python脚本run_case，参数-t：运行测试用例的方式，-u:  测试代码的svn地址，-c: 用例ID, -f: 测试类名。

如下，controller端将命令发送给agent端(controller和agent建立了一个TCP链接，通过该链接controller发送命令，接收命令执行结果，agent每3分钟向controller端发送heart beat信息，实现上heart beat与机器信息结合，如cpu, 内存，网络利用率等发送给controller端)。

2015-04-03 16:26:01 [DEBUG]: (./toastcontroller,,7770,0)Send command to agent id 141, type 1, timeout 127
2015-04-03 16:26:01 [DEBUG]: (./toastcontroller,,7770,0)command: python /data1/toast/plugin/case_run/run_case -t mvn -u https://svn1.intra.sina.com.cn/weibo_QA/TEST/work/AutoTestContent -c 6 -f LikesGroupTest;

agent端接收到controller端命令：

2015-04-03 16:26:01 [DEBUG]: (,,4585,0)Receive command id 141, type 1, timeout 127
2015-04-03 16:26:01 [DEBUG]: (,,4585,0)account length 4, command length 144
2015-04-03 16:26:01 [DEBUG]: (,,4585,0)account: root command: python /data1/toast/plugin/case_run/run_case -t mvn -u https://svn1.intra.sina.com.cn/weibo_QA/TEST/work/AutoTestContent -c 6 -f LikesGroupTest;
2015-04-03 16:26:01 [INFO]: (,,4585,0)Send command 141 starting message

2015-04-03 16:26:01 [DEBUG]: (,,4585,0)Command 141, processing processid: 28292, ttyfd: 6
2015-04-03 16:26:01 [DEBUG]: (,,4585,0)GetCommandOutput fd 6
2015-04-03 16:26:28 [INFO]: (,,4585,0)Send command 141 result: 2, return code: 0 result string:
2015-04-03 16:26:28 [DEBUG]: (,,4585,0)Command 141 , result status is: 0

agent端执行用例结束后，向controller端发送结束信息，controller端接收到运行结束信息，并通过API：/toast/run/updaterun通知web端：状态为status为200，RunId:141运行结束。

2015-04-03 16:26:28 [INFO]: (./toastcontroller,,7770,0)Task 141 is run completed
2015-04-03 16:26:28 [DEBUG]: (./toastcontroller,,7770,0)URL: http://10.13.1.139/toast/run/updaterun
2015-04-03 16:26:28 [DEBUG]: (./toastcontroller,,7770,0)Content: id=141&status=200&return_value=0&desc_info=
2015-04-03 16:26:28 [INFO]: (./toastcontroller,,7770,0)Curl result: Receive update command run info with id#141 info: {"id":"141","status":"200","return_value":"0","desc_info":""} source is (10.13.1.139)

controller端将agent执行用例时的 stdout 写入/tmp/toast_output文件，以这次RunId为名的log文件：141.log；前端php根据141.log，通过正则表达式（输出结果中信息较固定）进行解析运行结果。

以上就是一个完成的执行流程。

TOAST能很好的结合我们现有的测试工程，进行调度和展示结果，而测试的运行速度完全依赖于我们自身测试工程的JUnit优化（多线程执行，同时支持类和方法级别的并发），但是TOAST已有2年未更新，现有的部分功能代码并没有实现，如解析stdout结果，前端解析文件（php）并没有正常解析，源码中把所有case都赋予失败，这个是未完成的功能； 还有我们需要指定测试的环境，这个是新的需求，原有TOAST不支持。

二. 功能测试支持自定义测试环境

agent端执行用例，实际上是执行python脚本run_case，新增-e(--env) 参数，指定测试环境。

#/usr/bin/python2.6
# Filename: run_case
# -*- coding: utf-8 -*-

#
#   Copyright (C) 2007-2013 Alibaba Group Holding Limited
#
#   This program is free software;you can redistribute it and/or modify
#   it under the terms of the GUN General Public License version 2 as
#   published by the Free Software Foundation.
#

import sys
import getopt
import os
TOOL = ["mmt", "mvn"]


def usage():
    '''
    @summary: run_case manual.
    '''
    print "Usage: run_case -t tool -u url -c case_id [-f] ...\n"
    print "Misc:\n",
    print "	-h, --help	Print this for help, then exit.\n"
    print "Operation:\n",
    print "	-t, --tool	The test tool to run the test case.\n",
    print "	-u, --url	The URL for checking out the test case, support SVN only now.\n \
                                For maven project, this is maven project base svn url",
    print "	-c, --caseid	The caseid which comes from TOAST, and it will be printed.\n \
                                For maven project, this is the test calss will be run.",
    print "	-f, --function	The function which will be run, some test tools no need it(such as MMT).\n"
    print " -e, --env  test environment.\n"
    print "Example:\n",
    print "	run_case -t mmt -u http://xxx.xxx.xxx/svn/case1.php -c 1\n"


def get_options(argv):
    '''
    @summary: get options and check options.
    '''
    message = "Use run_case --help to get usage information.\n"
    options = {
        "tool": "",
        "url": "",
        "function": "",
        "env":"",
        "caseid": ""
    }
    try:
        opts, args = getopt.getopt(argv[1:], "ht:u:c:e:f:", ["help", "tool=", "url=", "caseid=", "env=","function="])
        for o, a in opts:
            if o in ("-h", "--help"):
                usage()
                exit(0)
            if o in ("-t", "--tool"):
                if a in TOOL:
                    options["tool"] = a
                else:
                    print "The test tool is required, and support " + ", ".join(TOOL) + " only. " + message
                    exit(1)
            if o in ("-u", "--url"):
                if "" != a:
                    options["url"] = a
                else:
                    print "The URL is required. " + message
                    exit(1)
            if o in ("-c", "--caseid"):
                if "" != a:
                    options["caseid"] = a
                else:
                    print "The caseid is required. " + message
                    exit(1)
            if o in("-e", "--env"):
                if "" != a:
                    options["env"] = a
                else:
                    print "The env is required. " + message
                    exit(1)
            if o in ("-f", "--function"):
                if "" != a:
                    options["function"] = a
                else:
                    print "The function is required. " + message
                    exit(1)
        return options
    except getopt.GetoptError:
        print message
        exit(1)

if __name__ == "__main__":
    if 1 == len(sys.argv):
        usage()
        exit(0)
    else:
        options = get_options(sys.argv)
        try:
            if options["tool"] == 'mvn':
               #from './tool/runmvn' import run_mvn_case
               sys.path.append(os.path.join(os.path.dirname(__file__),"./tool"))
               from runmvn import run_mvn_case
               cfg_file = os.path.splitext(os.path.abspath(__file__))[0] + ".conf"
               print cfg_file + options["caseid"] + options["url"]
               runner = run_mvn_case(cfg_file, options["function"], options["url"], options["caseid"], options["env"])
               try:
                   runner.get_code()
                   print 'code has checked out'
                   if "" != options["env"]:
                        runner.set_env(options["env"])
                   runner.run_a_case(options["function"])
               except Exception, ex:
                   print Exception,":",ex
                   print traceback.format_exc()
               finally:
                   runner.cleanup()
                   sys.exit(0)
            else: 
                toollib = __import__("tool." + options["tool"], globals(), locals(), ['Mmt'])
                Tool = getattr(toollib, options["tool"].capitalize())
                tool = Tool(options["url"], options["caseid"])
                tool.execute()
                sys.exit(0)
        except ImportError, e:
            print "Import tool module error."
            print e
            exit(1)

新增set_env(env)函数，修改配置文件内容，替换成自定义环境：

#!/usr/bin/python
# call mvn command run mvn case
# 1. checkout code in the svn
# 2. mvn test to run the specify case
# Infact it's just a mvn wapper

#
#   Copyright (C) 2007-2013 Alibaba Group Holding Limited
#
#   This program is free software;you can redistribute it and/or modify
#   it under the terms of the GUN General Public License version 2 as
#   published by the Free Software Foundation.
#

import ConfigParser
import string, os, sys
import subprocess
import uuid
import shutil
class run_mvn_case:
    def __init__(self):
        self.options       = {}
        self.CONFILE       = ""
        self.casetorun     = ""
        self.mvnprojectsvn = ""
        self.configer      =NULL
        self.local_path = ""
        self.env = ""
    def __init__(self, cfg_file, casetorun, mvnprojectsvn, caseid, env):
        self.CONFILE = cfg_file
        self.casetorun = casetorun
        self.mvnprojectsvn=mvnprojectsvn
        self.configer = ConfigParser.ConfigParser()
        self.configer.read(self.CONFILE)
        self.local_path = "/tmp/" + str(uuid.uuid4())
        self.id = caseid
        self.env = env
        self.cleanup()
    def get_code(self):
        svn_account = self.configer.get('svn', 'account')
        svn_password = self.configer.get('svn', 'password')
        svn_command = "svn co " + "--username " + svn_account + " --password " + svn_password + " --no-auth-cache " + " --non-interactive " + self.mvnprojectsvn + " " + self.local_path
        print svn_command
        pipe = subprocess.Popen(svn_command, bufsize=4096, shell=True, stderr=subprocess.STDOUT, stdout = subprocess.PIPE, close_fds=True)
        while True:
            line = pipe.stdout.readline(4096)
            if not line:
                break
            sys.stdout.write(line)
        return pipe.wait()
    #reset environment
    def set_env(self, env):
        print 'change environment'
        filepath = self.local_path + "/src/test/java/global.properties"
        host, port = env.split(':', 1)
        with open(filepath, 'w') as f:
            f.write("host=" + host + "\nport=" + port + "\nsource=2975945008\n")
        with open(filepath, 'r') as fr:
            print fr.read()
    def run_a_case(self, case):
        print 'start to run case'
        command = "cd " + self.local_path + "; mvn -Dtest=" + case + " test"
        print command
        pipe = subprocess.Popen(command, bufsize=4096, shell=True, stderr=subprocess.STDOUT, stdout = subprocess.PIPE, close_fds=True)
        while True:
            line = pipe.stdout.readline()
            if not line:
                break
            sys.stdout.write(line)
        print "CASE ID: " + self.id + "\n"
        return pipe.wait()

    def cleanup(self):
        if os.path.exists(self.local_path):
            shutil.rmtree(self.local_path)


def usage():
    print "run mvn\n" \
        "-h --help print this help message\n" \
        "-c --class test calass want to run\n" \
        "-u --svnurl the maven project base svn url\n"

if __name__ == '__main__':
    import getopt
    
    if len(sys.argv) < 2:
        usage()
        sys.exit(1)
    try:
        opts,args = getopt.getopt(sys.argv[1:], "hc:u:", ["help", "class=", "svnurl="])
    except getopt.GetoptError as err:
        print str(err)
        usage()
        system.exit(2)    
    runclass = ""     
    svnurl = ""                 
    for o, a in opts:
        if o in("-h", "--help"):
            usage()
            sys.exit()
        elif o in ("-c", "--class"):
            runclass = a
        elif o in("-u", "--svnurl"):
            svnurl = a
        else:
            assert False, "unhandled option"
    cfg_file = os.path.splitext(os.path.abspath(__file__))[0] + ".conf"
    print cfg_file
    print runclass    
    print svnurl
    runner = run_mvn_case(cfg_file, runclass, svnurl)
    try:
        runner.get_code()
        print 'code has checked out'
        runner.run_a_case(runclass)
    except Exception, ex:
        print Exception,":",ex
        print traceback.format_exc()
    finally:
        runner.cleanup()
        sys.exit(0)

该脚本流程如下：1.在/tmp目录下生成一个临时文件（由uuid.uuid4()生成，32字节字符串）->2.svn co拉测试用例工程放在step1生成的临时文件中->3.判断是否有参数-e，如果有修改配置文件中内容为相应环境，如果没有直接默认跑线上->4.进入step1创建的目录，执行controller发来的命令， mvn -Dtest=LikeObjectRpcTest test，执行相应的测试用例（要求测试工程为maven并安装surefire插件）。

该脚本执行的同时，agent端会将该stdout通过socket传给controller端，controller端写入stdout文件夹中，以RunId命名，eg:141.log；web端读取stdout文件中的141.log进行结果分析，php使用全局正则表达式进行匹配，用例总数，失败数，skipped数。

解析用例执行结果（原有TOAST，功能并未正常实现，将用例全赋为 $caseInfo->result = CaseInfo::RESULT_FAILED(web端JUnitMvnParser.php)；并根据该文件中正则表达式，反推taobao的JUnit case设计时，会在每个case中，System.out.println()出每个用例的信息，步骤和用例图示（按照统一的规则输出），类似于：

System.out.println("Results : casename(caseinfo)Tests run:");
System.out.println("[step case=0 number=\"10000001\"]步骤1[/step]");
System.out.println("[img case=0]*.img[/img]");

这个值得学习，但是如果JUnit支持用例重试机制，stdout就会有重复，解析就会出现重复的现象，当然可以包一层，滤重的过程，暂不输出用例信息 ）。

在实际执行用例时，标准的stdout并没有打印出成功和skipped用例的内容，支持自定义stdout全部用例信息：自动化二期（TAOBAO TOAST框架二次开发）---支持结果展示。

InfoQ上有一篇文章值得一读，支付宝AQC：支付宝分层与端到端回归平台建设实践，可惜没开源，文章中的理念还是值得学习。