念念不忘,必有回响
引言
- 我们要做java代码覆盖率检查
- 我们要对增量代码的覆盖率检查
- 我们要merge代码的时候自动检查覆盖率
出于以上目的,我们开始讲如何做,很多代码是站在前人的肩上进行的
思路参考:https://www.cnblogs.com/cocc/p/12365950.html
大致思路
- 本地查看代码覆盖率
- 提交gitlab上
- merge_requests时触发pipeline进行检查,不通过禁止merge(gitlb版本要求14.9以上,低于这个版本的也有解决方案,也会提到,gitlab域名/help可以查看版本)
环境准备
pom文件
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.7</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.2.0</version>
<scope>test</scope>
</dependency>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${project.parent.version}</version>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.7</version>
<executions>
<execution>
<id>default-prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>default-report</id>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
执行mvn install,会在target/site/下看到jacoco的目标,用网页打开就就可以看到单元测试的代码覆盖率
图一
图二
增量代码检查
上边得到的全量,现在如何将全量代码的变成增量的?
python代码参考的https://www.cnblogs.com/cocc/p/12365950.html,可以去拷贝
代码不全,缺少两个文件
我自己编了两份:
复制图一jacoco-resource中的report.css添加一下内容
/*覆盖的 绿色*/
pre.source span.fc-diff {
background-color:#34ce00;
}
pre.source span.bfc-diff {
background-image: url(diff.gif);
background-repeat: no-repeat;
background-position: 2px center;
}
pre.source span.bfc-diff:hover {
background-color:#4dff4d;
}
/*未覆盖的 红色*/
pre.source span.nc-diff {
background-color:#ff0000;
}
pre.source span.bnc-diff {
background-image: url(diff.gif);
background-repeat: no-repeat;
background-position: 2px center;
}
pre.source span.bnc-diff:hover {
background-color:#ff1c55;
}
/*覆盖的if 黄色*/
pre.source span.pc-diff {
background-color:#3d7a04;
}
/*覆盖的if 变换的颜色 深黄色*/
pre.source span.bpc-diff {
background-image: url(diff.gif);
background-repeat: no-repeat;
background-position: 2px center;
}
pre.source span.bpc-diff:hover {
background-color:#ffff34;
}
giff.gif是随便图的
这个时候,代码就可以跑起来了,此时只能出现以下效果
图三
而我们需要的是返回结果
修改两处代码
1、 diff_processor.py
class DiffProcessor():
def __init__(self, diffpath,code_path,coco_path):
'''
:param diffpath:gitdiff代码文件路径
:param code_path: 源码路径
:param coco_path: jacoco全量覆盖率报告路径
'''
self.diffpath = diffpath
self.code_path = code_path
self.coco_path = coco_path
self.res = 0 """======>这个是新增的"""
def Del_Dr(self, htmlpath, dirlist, ret, filetype, *kwargs):
'''
该函数的目的是为了修改html页面的内容
:param htmlpath:要修改的html文件的路径
:param dirlist:html页面要保留的类名列表或文件
:param ret:ret
:param diff_results:diff文件过滤的字典
:param filetype:需要修改的类型(root指文件根目录,package指包目录下,file指文件类型)
'''
with open(htmlpath, 'r') as e:
html_doc = "".join(e.readlines())
soup = BeautifulSoup(html_doc, 'lxml')
a_list = soup.select("a") # 获取html页面所有的a标签
for a_s in a_list:
a_s_text = a_s.text.strip("\n").strip(" ").strip("\n") # 循环获取a标签的text属性并过滤掉\n和空格
if filetype == "file":
a_s_text = a_s_text.split("(")[0]
if str(a_s_text) not in dirlist and a_s.parent.parent.name == "tr": # 如果text不等于要保留的类名,则直接删除该节点所属的tr标签
a_s.parent.parent.extract()
del_td = soup.find_all("tr")[0].find_all("td")[1:]
for td in del_td:
td.extract()
# 新增td行Add lines
new_tr = soup.new_tag("td")
new_tr.string = "Add lines"
soup.thead.tr.append(new_tr)
new_tr.attrs = {'class': 'sortable'}
# 新增td行Overlay lines
overlay_tr = soup.new_tag("td")
overlay_tr.string = "Overlay lines"
soup.thead.tr.append(overlay_tr)
overlay_tr.attrs = {'class': 'sortable'}
# 新增td行Coverage
coverage_tr = soup.new_tag("td")
coverage_tr.string = "Coverage"
soup.thead.tr.append(coverage_tr)
coverage_tr.attrs = {'class': 'sortable'}
pack_tr_list = soup.find_all("tbody")[0].find_all("tr") # 获取tbody中tr组成的列表
for tpack in pack_tr_list: # 删除tbody中tr中除类名或文件名的其他列
for pa_td in tpack.find_all("td")[1:]:
pa_td.extract()
tfoot_list = soup.find_all("tfoot")[0].find_all("td")[1:] # 删除tfoot中除Total外的其他列
for tfoot in tfoot_list:
tfoot.extract()
packageAddlines = 0 """======>这个是新增的"""
packageCovliness = 0 """======>这个是新增的"""
for npack in pack_tr_list:
pack_name = npack.find_all("a")[0].string.strip("\n").strip(" ").strip("\n")
addlines = 0 """======>这个是新增的"""
covlines = 0 """======>这个是新增的"""
if filetype == "package": # 如果是包名下的index.html文件做如下处理
addlines = ret[pack_name]['new']
covlines = ret[pack_name]['cover']
elif filetype == "root":
for k, v in enumerate(ret[pack_name]):
addlines += ret[pack_name][v]['new']
covlines += ret[pack_name][v]['cover']
elif filetype == "file":
pack_void_name = pack_name.split("(")[0]
filename, diff_dict = kwargs
filename_new_list = filename.split("src/main/java/")[-1].split("/")
filename_new = ".".join(filename_new_list[:-1])
class_name = filename_new_list[-1].split(".")[0]
if filename in diff_dict.keys() and class_name in ret[filename_new].keys():
void_lines_list = diff_dict[filename]['diff_voids'][pack_void_name]
new_line_list = list(
set(ret[filename_new][class_name]['new_lines']).intersection(set(void_lines_list)))
cover_line_list = list(
set(ret[filename_new][class_name]['cover_lines']).intersection(set(void_lines_list)))
addlines = len(new_line_list)
covlines = len(cover_line_list)
if addlines == 0:
coverage = '{:.2%}'.format(0)
else:
coverage = '{:.2%}'.format(covlines / addlines) # 覆盖率
addlines_tr = soup.new_tag("td")
if addlines:
addlines_tr.string = "%s" % addlines
npack.append(addlines_tr)
covlines_tr = soup.new_tag("td")
covlines_tr.string = "%s" % covlines
npack.append(covlines_tr)
coverage_tr = soup.new_tag("td")
coverage_tr.string = "%s" % coverage
npack.append(coverage_tr)
else:
npack.extract()
if str(npack).find("el_package") != -1: """======>这个是新增的"""
packageAddlines += addlines """======>这个是新增的"""
packageCovliness += covlines """======>这个是新增的"""
if packageAddlines != 0: """======>这个是新增的"""
self.res = packageCovliness/packageAddlines """======>这个是新增的"""
else: """======>这个是新增的"""
self.res = 1 """======>这个是新增的"""
# 重新生成index.html页面
html_path_new = htmlpath + "_bat"
with open(html_path_new, 'w+') as f:
f.write(html_parser.unescape(soup.prettify()))
os.remove(htmlpath)
os.rename(html_path_new, htmlpath)
main.py
# 生成增量报告
"""======>以下是修改的jacoco"""
processor = DiffProcessor(opts.giffdir, opts.dir, os.path.join(opts.jareport, 'jacoco'))
processor.diff_file()
# 拷贝css和图片资源
"""======>以下是修改的"""
shutil.copy('report.css', os.path.join(opts.jareport, "jacoco/jacoco-resources/"))
shutil.copy('diff.gif', os.path.join(opts.jareport, "jacoco/jacoco-resources/"))
"""======>以下是新增的"""
print('代码覆盖率为{:.2%}'.format(processor.res))
if processor.res < 0.5 :
print("检查不通过")
os._exit()
else:
print("检查通过")
return 0
有了上边的代码就可以执行了
通过就接受,不通过就报错。
CI执行
现在我们可以在本地做增量代码的检查了,开始做CI,
从本地dev分支合并到test分支。
参考:https://docs.gitlab.cn/jh/ci/pipelines/merge_request_pipelines.html
创建runner就不说了,runner的tag为bigdata02,只有在test的merge_requests时才进行检查
stages:
- deploy
job:
stage: deploy
tags:
- bigdata02
only:
- merge_requests
script: # 执行的shell命令
- bash bin/checkCode.sh
checkCode.sh
#!/bin/bash
set -e
code_path=`pwd`
echo $code_path
echo $CI_COMMIT_REF_NAME # 打印当前分支
echo $GITLAB_USER_EMAIL # 打操作用户的邮箱
git branch -av # 查看所有分支
git checkout remotes/origin/test # 切到test分支
git checkout remotes/origin/$CI_COMMIT_REF_NAME # 切到当前分支
mvn install # 当前生成代码覆盖率的报告
mkdir -p $CI_COMMIT_REF_NAME/gitdiff # 创建临时文件,存放新增代码
git diff remotes/origin/test remotes/origin/$CI_COMMIT_REF_NAME > $CI_COMMIT_REF_NAME/gitdiff/gitdiff.txt # 比较当前分支比test多了哪些代码
cat $CI_COMMIT_REF_NAME/gitdiff/gitdiff.txt # 打印看一下
echo "git diff已完成,生成增量代码文件,请查看"
start.sh $code_path $code_path/$CI_COMMIT_REF_NAME/gitdiff/gitdiff.txt $code_path/target/site # 执行脚本,start.sh就是 python3 main.py -d -g -j ....
rm -rf $CI_COMMIT_REF_NAME/ # 删除临时文件
echo "完成"
此时就可以在执行merge_requests的时候进行增量代码覆盖率的检查了
other
低于14.9的看这里,就是在每次提交代码的时候进行检查,push时执行pipeline,设置pipeline不通过就不让合并。
stages:
- deploy
job:
stage: deploy
tags:
- bigdata02
script: # 执行的shell命令
- bash bin/checkCode.sh
checkCode.sh
#!/bin/bash
set -e
code_path=`pwd`
echo $code_path
echo $CI_COMMIT_REF_NAME
echo $GITLAB_USER_EMAIL
git branch -av
merger=`git branch -av | head -n 1 | grep "Merge branch" | wc -l`
echo $merger
if [ $CI_COMMIT_REF_NAME = "test" ]; then
echo "test分支不需要检查"
elif [ $CI_COMMIT_REF_NAME = "master" ]; then
echo "master分支不需要检查"
elif [ $merger -eq 1 ]; then
echo "Merge不需要检查"
else
git checkout remotes/origin/test
git checkout remotes/origin/$CI_COMMIT_REF_NAME
mvn install
mkdir -p $CI_COMMIT_REF_NAME/gitdiff
git diff remotes/origin/test remotes/origin/$CI_COMMIT_REF_NAME > $CI_COMMIT_REF_NAME/gitdiff/gitdiff.txt
cat $CI_COMMIT_REF_NAME/gitdiff/gitdiff.txt
echo "git diff已完成,生成增量代码文件,请查看"
start.sh $code_path $code_path/$CI_COMMIT_REF_NAME/gitdiff/gitdiff.txt $code_path/target/site
rm -rf $CI_COMMIT_REF_NAME/
echo "完成"
fi