jacoco增量覆盖率实践
https://segmentfault.com/a/1190000039027049
ray_duan发布于 1 月 19 日
能找到这里,说明对jacoco的原理和使用有了一定的了解,而我写这边文章主要是网络上基本没有完整文档加代码的jaocco增量覆盖说明,所以我想分享些东西让需要这方面的人快速去实现自己想要的功能,那么如果想实现增量代码覆盖率需要做到哪些工作呢?
大家在网络上找到的实现方式无外乎三种
- 获取到增量代码,在jacoco进行插桩时判断是否是增量代码后再进行插桩,这样需要两个步骤,一是获取增量代码,二是找到jacoco的插桩逻辑进行修改
- 获取增量代码,在report阶段去判断方法是否是增量,再去生成报告
- 获取差异代码,解析生成的report报告,再过滤出差异代码的报告
首先第一种需要对java字节码操作比较熟悉,难度较高,我们不谈,第三种去解析生成的报告,可能存在误差
所以我们一般选择第二种,而网络上所有的增量实现基本是基于第二种,我们先看看下面的
上图说明了jacoco测试覆盖率的生成流程,而我们要做的是在report的时候加入我们的逻辑
根据我们的方案,我们需要三个动作
- 计算出两个版本的差异代码(基于git)
- 将差异代码在jacoco的report阶段传给jacoco
- 修改jacoco源码,生成报告时判断代码是否是增量代码,只有增量代码才去生成报告
下面我们逐步讲解上述步骤
计算差异代码
计算差异代码我实现了一个简单的工程:差异代码获取
主要用到了两个工具类
1. <dependency> 2. <groupId>org.eclipse.jgit</groupId> 3. <artifactId>org.eclipse.jgit</artifactId> 4. </dependency> 6. <!-- https://mvnrepository.com/artifact/com.github.javaparser/javaparser-core --> 7. <dependency> 8. <groupId>com.github.javaparser</groupId> 9. <artifactId>javaparser-core</artifactId> 10. </dependency>
org.eclipse.jgit主要用于从git获取代码,并获取到存在变更的文件
javaparser-core是一个java解析类,能将class类文件解析成树状,方便我们去获取差异类
1. /** 2. * 获取差异类 3. * 4. * @param diffMethodParams 5. * @return 6. */ 7. public List<ClassInfoResult> diffMethods(DiffMethodParams diffMethodParams) { 8. try { 9. //原有代码git对象 10. Git baseGit = cloneRepository(diffMethodParams.getGitUrl(), localBaseRepoDir + diffMethodParams.getBaseVersion(), diffMethodParams.getBaseVersion()); 11. //现有代码git对象 12. Git nowGit = cloneRepository(diffMethodParams.getGitUrl(), localBaseRepoDir + diffMethodParams.getNowVersion(), diffMethodParams.getNowVersion()); 13. AbstractTreeIterator baseTree = prepareTreeParser(baseGit.getRepository(), diffMethodParams.getBaseVersion()); 14. AbstractTreeIterator nowTree = prepareTreeParser(nowGit.getRepository(), diffMethodParams.getNowVersion()); 15. //获取两个版本之间的差异代码 16. List<DiffEntry> diff = nowGit.diff().setOldTree(baseTree).setNewTree(nowTree).setShowNameAndStatusOnly(true).call(); 17. //过滤出有效的差异代码 18. Collection<DiffEntry> validDiffList = diff.stream() 19. //只计算java文件 20. .filter(e -> e.getNewPath().endsWith(".java")) 21. //排除测试文件 22. .filter(e -> e.getNewPath().contains("src/main/java")) 23. //只计算新增和变更文件 24. .filter(e -> DiffEntry.ChangeType.ADD.equals(e.getChangeType()) || DiffEntry.ChangeType.MODIFY.equals(e.getChangeType())) 25. .collect(Collectors.toList()); 26. if (CollectionUtils.isEmpty(validDiffList)) { 27. return null; 28. } 29. /** 30. * 多线程获取旧代码和新代码的差异类及差异方法 31. */ 32. List<CompletableFuture<ClassInfoResult>> priceFuture = validDiffList.stream().map(item -> getClassMethods(getClassFile(baseGit, item.getNewPath()), getClassFile(nowGit, item.getNewPath()), item)).collect(Collectors.toList()); 33. return priceFuture.stream().map(CompletableFuture::join).filter(Objects::nonNull).collect(Collectors.toList()); 34. } catch (GitAPIException e) { 35. e.printStackTrace(); 36. } 37. return null; 38. }
以上代码为获取差异类的核心代码
2. /** 3. * 获取类的增量方法 4. * 5. * @param oldClassFile 旧类的本地地址 6. * @param mewClassFile 新类的本地地址 7. * @param diffEntry 差异类 8. * @return 9. */ 10. private CompletableFuture<ClassInfoResult> getClassMethods(String oldClassFile, String mewClassFile, DiffEntry diffEntry) { 11. //多线程获取差异方法,此处只要考虑增量代码太多的情况下,每个类都需要遍历所有方法,采用多线程方式加快速度 12. return CompletableFuture.supplyAsync(() -> { 13. String className = diffEntry.getNewPath().split(".")[0].split("src/main/java/")[1]; 14. //新增类直接标记,不用计算方法 15. if (DiffEntry.ChangeType.ADD.equals(diffEntry.getChangeType())) { 16. return ClassInfoResult.builder() 17. .classFile(className) 18. .type(DiffEntry.ChangeType.ADD.name()) 19. .build(); 20. } 21. List<MethodInfoResult> diffMethods; 22. //获取新类的所有方法 23. List<MethodInfoResult> newMethodInfoResults = MethodParserUtils.parseMethods(mewClassFile); 24. //如果新类为空,没必要比较 25. if (CollectionUtils.isEmpty(newMethodInfoResults)) { 26. return null; 27. } 28. //获取旧类的所有方法 29. List<MethodInfoResult> oldMethodInfoResults = MethodParserUtils.parseMethods(oldClassFile); 30. //如果旧类为空,新类的方法所有为增量 31. if (CollectionUtils.isEmpty(oldMethodInfoResults)) { 32. diffMethods = newMethodInfoResults; 33. } else { //否则,计算增量方法 34. List<String> md5s = oldMethodInfoResults.stream().map(MethodInfoResult::getMd5).collect(Collectors.toList()); 35. diffMethods = newMethodInfoResults.stream().filter(m -> !md5s.contains(m.getMd5())).collect(Collectors.toList()); 36. } 37. //没有增量方法,过滤掉 38. if (CollectionUtils.isEmpty(diffMethods)) { 39. return null; 40. } 41. ClassInfoResult result = ClassInfoResult.builder() 42. .classFile(className) 43. .methodInfos(diffMethods) 44. .type(DiffEntry.ChangeType.MODIFY.name()) 45. .build(); 46. return result; 47. }, executor); 48. }
以上代码为获取差异方法的核心代码
大家可以下载代码后运行,下面我们展示下,运行代码后获取到的差异代码内容(参数可以是两次commitId,也可以是两个分支,按自己的业务场景来)
1. { 2. "code": 10000, 3. "msg": "业务处理成功", 4. "data": [ 5. { 6. "classFile": "com/dr/application/InstallCert", 7. "methodInfos": null, 8. "type": "ADD" 9. }, 10. { 11. "classFile": "com/dr/application/app/controller/Calculable", 12. "methodInfos": null, 13. "type": "ADD" 14. }, 15. { 16. "classFile": "com/dr/application/app/controller/JenkinsPluginController", 17. "methodInfos": null, 18. "type": "ADD" 19. }, 20. { 21. "classFile": "com/dr/application/app/controller/LoginController", 22. "methodInfos": [ 23. { 24. "md5": "2C9D2AE2B1864A2FCDDC6D47CEBEBD4C", 25. "methodName": "captcha", 26. "parameters": "HttpServletRequest request,HttpServletResponse response" 27. }, 28. { 29. "md5": "3D6DFADD2171E893D99D3D6B335B22EA", 30. "methodName": "login", 31. "parameters": "@RequestBody LoginUserParam loginUserParam,HttpServletRequest request" 32. }, 33. { 34. "md5": "90842DFA5372DCB74335F22098B36A53", 35. "methodName": "logout", 36. "parameters": "" 37. }, 38. { 39. "md5": "D0B2397D04624D2D60E96AB97F679779", 40. "methodName": "testInt", 41. "parameters": "int a,char b" 42. }, 43. { 44. "md5": "34219E0141BAB497DCB5FB71BAE1BDAE", 45. "methodName": "testInt", 46. "parameters": "String a,int b" 47. }, 48. { 49. "md5": "F9BF585A4F6E158CD4475700847336A6", 50. "methodName": "testInt", 51. "parameters": "short a,int b" 52. }, 53. { 54. "md5": "0F2508A33F719493FFA66C5118B41D77", 55. "methodName": "testInt", 56. "parameters": "int[] a" 57. }, 58. { 59. "md5": "381C8CBF1F381A58E1E93774AE1AF4EC", 60. "methodName": "testInt", 61. "parameters": "AddUserParam param" 62. }, 63. { 64. "md5": "64BF62C11839F45030198A8D8D7821C5", 65. "methodName": "testInt", 66. "parameters": "T[] a" 67. }, 68. { 69. "md5": "D091AB0AD9160407AED4182259200B9B", 70. "methodName": "testInt", 71. "parameters": "Calculable calc,int n1,int n2" 72. }, 73. { 74. "md5": "693BBA0A8A57F2FD19F61BA06F23365C", 75. "methodName": "display", 76. "parameters": "" 77. }, 78. { 79. "md5": "F9DFE0E75C78A31AFB6A8FD46BDA2B81", 80. "methodName": "a", 81. "parameters": "InnerClass a" 82. } 83. ], 84. "type": "MODIFY" 85. }, 86. { 87. "classFile": "com/dr/application/app/controller/RoleController", 88. "methodInfos": null, 89. "type": "ADD" 90. }, 91. { 92. "classFile": "com/dr/application/app/controller/TestController", 93. "methodInfos": [ 94. { 95. "md5": "B1840C873BF0BA74CB6749E1CEE93ED7", 96. "methodName": "getPom", 97. "parameters": "HttpServletResponse response" 98. }, 99. { 100. "md5": "9CEE68771972EAD613AF237099CD2349", 101. "methodName": "getDeList", 102. "parameters": "" 103. } 104. ], 105. "type": "MODIFY" 106. }, 107. { 108. "classFile": "com/dr/application/app/controller/UserController", 109. "methodInfos": [ 110. { 111. "md5": "7F2AD08CE732ADDFC902C46D238A9EB3", 112. "methodName": "add", 113. "parameters": "@RequestBody AddUserParam addUserParam" 114. }, 115. { 116. "md5": "D41D8CD98F00B204E9800998ECF8427E", 117. "methodName": "get", 118. "parameters": "" 119. }, 120. { 121. "md5": "2B35EA4FB5054C6EF13D557C2ACBB581", 122. "methodName": "list", 123. "parameters": "@ApiParam(required = true, name = "page", defaultValue = "1", value = "当前页码") @RequestParam(name = "page") Integer page,@ApiParam(required = true, name = "pageSize", defaultValue = "10", value = "每页数量") @RequestParam(name = "pageSize") Integer pageSize,@ApiParam(name = "userId", value = "用户id") @RequestParam(name = "userId", required = false) Long userId,@ApiParam(name = "username", value = "用户名") @RequestParam(name = "username", required = false) String username,@ApiParam(name = "userSex", value = "性别") @RequestParam(name = "userSex", required = false) Integer userSex,@ApiParam(name = "mobile", value = "手机号") @RequestParam(name = "mobile", required = false) String mobile" 124. } 125. ], 126. "type": "MODIFY" 127. }, 128. { 129. "classFile": "com/dr/application/app/controller/view/RoleViewController", 130. "methodInfos": null, 131. "type": "ADD" 132. }, 133. { 134. "classFile": "com/dr/application/app/controller/view/UserViewController", 135. "methodInfos": [ 136. { 137. "md5": "9A1DDA3F41B36026FC2F3ACDAE85C1DB", 138. "methodName": "user", 139. "parameters": "" 140. } 141. ], 142. "type": "MODIFY" 143. }, 144. { 145. "classFile": "com/dr/application/app/param/AddRoleParam", 146. "methodInfos": null, 147. "type": "ADD" 148. }, 149. { 150. "classFile": "com/dr/application/app/vo/DependencyVO", 151. "methodInfos": null, 152. "type": "ADD" 153. }, 154. { 155. "classFile": "com/dr/application/app/vo/JenkinsPluginsVO", 156. "methodInfos": null, 157. "type": "ADD" 158. }, 159. { 160. "classFile": "com/dr/jenkins/vo/DeviceVo", 161. "methodInfos": null, 162. "type": "ADD" 163. }, 164. { 165. "classFile": "com/dr/jenkins/vo/GoodsVO", 166. "methodInfos": null, 167. "type": "ADD" 168. }, 169. { 170. "classFile": "com/dr/jenkins/vo/JobAddVo", 171. "methodInfos": null, 172. "type": "ADD" 173. }, 174. { 175. "classFile": "com/dr/repository/user/dto/query/RoleQueryDto", 176. "methodInfos": null, 177. "type": "ADD" 178. }, 179. { 180. "classFile": "com/dr/repository/user/dto/query/UserQueryDto", 181. "methodInfos": null, 182. "type": "ADD" 183. }, 184. { 185. "classFile": "com/dr/repository/user/dto/result/MenuDTO", 186. "methodInfos": null, 187. "type": "ADD" 188. }, 189. { 190. "classFile": "com/dr/repository/user/dto/result/RoleResultDto", 191. "methodInfos": null, 192. "type": "ADD" 193. }, 194. { 195. "classFile": "com/dr/repository/user/dto/result/UserResultDto", 196. "methodInfos": null, 197. "type": "ADD" 198. }, 199. { 200. "classFile": "com/dr/user/service/impl/RoleServiceImpl", 201. "methodInfos": [ 202. { 203. "md5": "D2AAADF53B501AE6D2206B2951256329", 204. "methodName": "getRoleCodeByUserId", 205. "parameters": "Long id" 206. }, 207. { 208. "md5": "47405162B3397D02156DE636059049F2", 209. "methodName": "getListByPage", 210. "parameters": "RoleQueryDto roleQueryDto" 211. } 212. ], 213. "type": "MODIFY" 214. }, 215. { 216. "classFile": "com/dr/user/service/impl/UserServiceImpl", 217. "methodInfos": [ 218. { 219. "md5": "D41D8CD989ABCDEFFEDCBA98ECF8427E", 220. "methodName": "selectListByPage", 221. "parameters": "UserQueryDto userQueryDto" 222. } 223. ], 224. "type": "MODIFY" 225. } 226. ] 227. }
data部分为差异代码的具体内容
将差异代码传递到jaocco
大家可以参考:jacoco增量代码改造
我们只需要找到Report类,加入可选参数
@Option(name = "--diffCode", usage = "input file for diff", metaVar = "<file>") String diffCode;
这样,我们就可以在jacoco内部接受到传递的参数了,如果report命令加上--diffCode就计算增量,不加则计算全量,不影响正常功能,灵活性高
我们这里改造了analyze方法,将增量代码塞给CoverageBuilder对象,我们需要用时直接去获取
1. private IBundleCoverage analyze(final ExecutionDataStore data, 2. final PrintWriter out) throws IOException { 3. CoverageBuilder builder; 4. // 如果有增量参数将其设置进去 5. if (null != this.diffCode) { 6. builder = new CoverageBuilder(this.diffCode); 7. } else { 8. builder = new CoverageBuilder(); 9. } 10. final Analyzer analyzer = new Analyzer(data, builder); 11. for (final File f : classfiles) { 12. analyzer.analyzeAll(f); 13. } 14. printNoMatchWarning(builder.getNoMatchClasses(), out); 15. return builder.getBundle(name); 16. }
差异代码匹配
jacoco采用AMS类去解析class类,我们需要去修改org.jacoco.core包下面的Analyzer类
1. private void analyzeClass(final byte[] source) { 2. final long classId = CRC64.classId(source); 3. final ClassReader reader = InstrSupport.classReaderFor(source); 4. if ((reader.getAccess() & Opcodes.ACC_MODULE) != 0) { 5. return; 6. } 7. if ((reader.getAccess() & Opcodes.ACC_SYNTHETIC) != 0) { 8. return; 9. } 10. // 字段不为空说明是增量覆盖 11. if (null != CoverageBuilder.classInfos 12. && !CoverageBuilder.classInfos.isEmpty()) { 13. // 如果没有匹配到增量代码就无需解析类 14. if (!CodeDiffUtil.checkClassIn(reader.getClassName())) { 15. return; 16. } 17. } 18. final ClassVisitor visitor = createAnalyzingVisitor(classId, 19. reader.getClassName()); 20. reader.accept(visitor, 0); 22. }
主要是判断如果需要的是增量代码覆盖率,则匹配类是否是增量的(这里是jacoco遍历解析每个类的地方)
然后修改ClassProbesAdapter类的visitMethod方法(这个是遍历类里面每个方法的地方)
整个比较的代码逻辑在这里,注释写的你叫详细了
修改完成后,大家只要构建出org.jacoco.cli-0.8.7-SNAPSHOT-nodeps.jar包,然后report时传入增量代码即可
全量报告
增量报告
所遇到问题
- 差异方法的参数匹配
由于我们使用javaparser解析出的参数格式为String a,int b
而ASM解析出的 为Ljava/lang/String,I;在匹配参数的时候遇到了问题,最终我找到了Type类的方法
Type.getArgumentTypes(desc)
然后
argumentTypes[i].getClassName()
将AmS的参数解析成String,int(做了截取),然后再去匹配,就能正确匹配到参数的格式了
- 为什么不将整个生成报告做成一个平台
jacoco生成报告的时候,需要传入源码,编译后的class文件,而编译这些东西我们一般都有自己的ci平台去做,我们可以将我们的覆盖率功能集成到我们的devops平台,从那边去获取源码或编译出的class文件,而且可以做业务上的整合,所以没有像supper-jacoco那样做成一个平台
考资料: super-jacoco 里面有些bug,使用的时候请注意