以前在工作中,生产开发中出现了一些问题,比较严重,构建的时候居然要1个小时以上,没法子只能排查,尝试改maven源码的方式进行优化,和大佬研究一番后验证没问题发出来作为分享!!
1、问题重现
Jenkins使用的maven版本3.8.5(我没想明白为啥要用这么新的,给我idea都换最新的了)
版本:apache-maven3.8.5
现象重现下:
当我把项目拉到本地,用maven去reimport的时候,发现内存非常高,并且cpu也被吃满了,于是我用jvm的调试工具看了下,发现内存不断涨,开发经常说本地配置了20G都不够,真头疼!
我因为本地配的是8G,这种就直接往上飙升,不带释放的,打了线程快照发现:
"main" #1 prio=5 os_prio=0 tid=0x000001ea3dcba000 nid=0x46c4 runnable [0x00000044ccef9000]
java.lang.Thread.State: RUNNABLE
at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
at java.util.AbstractMap.equals(AbstractMap.java:495)
at java.util.Collections$UnmodifiableMap.equals(Collections.java:1493)
at java.util.Objects.equals(Objects.java:59)
at org.eclipse.aether.artifact.AbstractArtifact.equals(AbstractArtifact.java:203)
at org.eclipse.aether.internal.impl.collect.DataPool$ConstraintKey.equals(DataPool.java:314)
at java.util.HashMap.getNode(HashMap.java:573)
at java.util.HashMap.get(HashMap.java:558)
at org.eclipse.aether.internal.impl.collect.DataPool.getConstraint(DataPool.java:155)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.cachedResolveRangeResult(DefaultDependencyCollector.java:616)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:392)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:506)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:362)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.process(DefaultDependencyCollector.java:349)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:254)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:284)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:170)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:339)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:293)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:211)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:165)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:157)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:121)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:127)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
截图就很容易发现一个问题,这四个方法一直在循环;应该是这出现问题,但是是什么问题,暂时不清楚。
现象有了,我们一探究竟?
2、探路开始
先下载maven的源码:
下载地址:Index of /dist/maven/maven-3/3.8.5/source
下载下来后用idea打开,这里注意下idea的版本,要2022版本及以上可以
为了保证版本一致,也要下载下maven3.8.5的发行版(配置在idea和本地电脑的环境变量里)
下载release的地址:Maven – Download Apache Maven
下载下来后配置好环境变量(这里就不演示怎么配置了),先用idea打开maven源码,等待自己编译好后,可以斗把地主。
根据上面堆栈的截图,发现有个类DefaultDependencyCollector的4个方法一直在循环,doRecurse->process->processDependency->processDependency(重载)
接下来我们要用idea新建一个远程debug的application,记得端口改成监听8000,因为要用到mvnDebug工具,端口是8000。
然后这个时候突然发现DefaultDependencyCollector是在 maven-resolver-impl-1.6.3.jar这个依赖里的,所以我顺便把这个jar也下载下来了。
下载地址:Index of /dist/maven/resolver
同样,idea打开这个项目:
那么既然都到这里了,其实上面的maven源码就暂时可以不需要了,就先用maven-resolver-impl 这个源码进行调试,同样这里设定个remote debug:
调试开始,我首先打几个debug在那四个卡的方法上,于是进到我的项目里,执行下命令:
mvnDebug dependency:tree -Dverbose 其中-Dverbose 将当前所有的依赖关系都展示出来,包括来自不同处的依赖项。
回车后,此时进到maven-resolver-impl项目里,debug开启走起:
断点进来了,跟着走看看到底在干啥?显示几个参数
args:暂时说不好,但是知道意思
results:解析的结果
dependencies:项目解析的pom依赖
repositories:远程仓库的地址
depSelector:看样子是个选择器,来遍历每个dependency的坐标依赖深度的
depManager:依赖管理器,负责迭代深度用的
depTraverser:依赖的遍历器
verFilter:更不知道什么玩意
于是,把上面所有的参数带进入process后,发现要遍历所有的pom里的dependency坐标
@SuppressWarnings("checkstyle:parameternumber")
private void process(final Args args, Results results, List<Dependency> dependencies,
List<RemoteRepository> repositories, DependencySelector depSelector,
DependencyManager depManager, DependencyTraverser depTraverser, VersionFilter verFilter) {
for (Dependency dependency : dependencies) {
processDependency(args, results, repositories, depSelector, depManager, depTraverser, verFilter,
dependency);
}
}
具体的debug过程就不写了,蛮复杂的,有兴趣找我要源码,我注释写在里面了...
3、原因剖析(重点)
debug后我打了个日志,发现解析slf4j依赖的次数,高达9w次,这么恐怖的吗?
说下maven主要解析的思路:
debug后发现maven在解析依赖的时候,会解析到exclude这个结点,如果有1个项目A,引用了B和C的业务jar包,而业务jar包中出现exclude坐标,那么maven会把这个业务jar包中所有的内部依赖,全部都去排除这个jar,而不断的循环中出现了Spring的依赖在不断的解析,大概率是出现业务包里一旦排除了类似某些包,是在spring全家桶里的(springboot、springcloud等),譬如slf4j,logback等,那么依赖的B和C的业务jar中依赖全部都要排除,那么此时需要的spring依赖,maven会认为这些都不一样,会不断的放到迭代器里去递归解析,进而不断的解析里面的依赖关系,这就非常耗时,也就说明上面四个方法根本出不来的原因了,但是最后生成的spring相关的jar是一样的。
举个简单例子:B业务的pom依赖了spring,springcloud等,mybatis,其他第三方的,,包括自己业务代码....
而A项目中出现B的坐标,并且exclude slf4j的依赖,当去解析B这个依赖的时候,会去把B的pom全部解析一遍,并且里面的每个坐标,都剔除掉slf4j这个依赖,假设某个依赖spring-boot-starter里有这个jar(只是举例子),那么在去build里面的所有spring依赖,都会被认为是不一样的,正常来说只要发现了一个spring依赖(比如spring-core),就应该放在缓存里作为key,后面从缓存取,如果取到了就不去遍历,跳出循环,而真实情况是,B业务排除了slf4j,C业务排除了log4j,就会导致两个去遍历的时候,根据artifactId和exclude解析,认为的spring不会认为是你排除后,要得到的是同一个spring(但是实际上是一个spring),这个机制就是这样,自然每个都会去解析,从而递归去得到包,而最后的包,都是spring的包,没有不一样。
那么优化的思路就是针对自己的业务进行优化:遍历坐标,如果遇到我们的artifactId的坐标,就会认为不进行排除exclude,不走了exclude逻辑,就不会去每个业务包里都去执行exclude逻辑,最终的效果是会多出一些包,这些包原本应该是被exclude掉的,通过maven手动处理就行。
Maven本身机制没有问题,只是场景需要吧。
4、发包
接下来改好了源码后,我要打包,这里又是坑涉及到apache的rat证书,因为apache有一定的规范不让打包,不多解释了(mvn clean install -DskipTests -Drat.skip=true -Dcheckstyle.skip=true)
这个时候我就发布了一个包,验证后,本地内存4g不到就可以解析了,环境上从1h起步,到几分钟的优化,如果war中多出了一些包,用这个方法排除下就行:
<!-- 剔除无用的依赖 -->
<plugin>
<artifactId>maven-war-plugin</artifactId>
<version>3.2.3</version>
<configuration>
<packagingExcludes>
WEB-INF/lib/slf4j-simple-1.7.28.jar
</packagingExcludes>
</configuration>
</plugin>
很久之前搞了一晚上的调试,自己研究研究后写出来大家分享下调试过程,记得感兴趣可以了解下!!