maven 解决依赖冲突

依赖冲突的表现

ClassNotFound

// class 名字会随着冲突的不同而不同
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tracing.SpanReceiverHost
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 21 more

NoSuchMethod

java.lang.NoSuchMethodError: org.elasticsearch.client.RestHighLevelClient.bulkAsync(Lorg/elasticsearch/action/bulk/BulkRequest;Lorg/elasticsearch/client/RequestOptions;Lorg/elasticsearch/action/ActionListener;)Lorg/elasticsearch/client/Cancellable;
	at org.apache.flink.streaming.connectors.elasticsearch7.Elasticsearch7ApiCallBridge.lambda$createBulkProcessorBuilder$0(Elasticsearch7ApiCallBridge.java:82)
	at org.elasticsearch.action.bulk.Retry$RetryHandler.execute(Retry.java:202)
	at org.elasticsearch.action.bulk.Retry.withBackoff(Retry.java:62)

如何查找冲突

查找依赖冲突的来源

在解决冲突之前，我们需要知道冲突的 Jar 是从哪里引入的

如果在提交作业时，提示「可能有冲突」的 jar，则需要排查项目中哪里依赖了对应的 jar 包
如果是 NoSuchMethod 或者 ClassNotFound 等异常，则根据报错的 class 来查找对应的 jar 包（比如上文中 ClassNotFound 示例中需要查找 SpanReceiverHost 从哪个 jar 中引入，NoSuchMethod 异常示例中，则需要查找 org.elasticsearch.client.RestHighLevelClient 从哪里引入-- 从依赖栈我们知道是 elasticserach7 引入而来）

首先到项目的目录执行如下命令

mvn dependency:tree -Dverbose

会得到类似如下的结果

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Flink-Demo 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ Flink-Demo ---
[INFO] org.example:Flink-Demo:jar:1.0-SNAPSHOT
[INFO] +- org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile
[INFO] |  +- org.apache.flink:flink-annotations:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-metrics-core:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[INFO] |  |  +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] |  |  \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.15:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  \- org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile
[INFO] +- org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.commons:commons-math3:jar:3.5:compile - omitted for duplicate)
[INFO] |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-5.0:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] |  |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-hadoop-fs:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-5.0:compile
[INFO] |  +- org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile
[INFO] |  +- (org.apache.flink:flink-shaded-asm:jar:5.0.4-5.0:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-shaded-jackson:jar:2.7.9-5.0:compile
[INFO] |  +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] |  +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |  +- org.javassist:javassist:jar:3.19.0-GA:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] |  +- com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile
[INFO] |  |  +- com.typesafe:config:jar:1.3.0:compile
[INFO] |  |  \- org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0:compile
[INFO] |  +- com.typesafe.akka:akka-stream_2.11:jar:2.4.20:compile
[INFO] |  |  +- (com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile - omitted for duplicate)
[INFO] |  |  +- org.reactivestreams:reactive-streams:jar:1.0.0:compile
[INFO] |  |  \- com.typesafe:ssl-config-core_2.11:jar:0.2.1:compile
[INFO] |  |     \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] |  +- com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20:compile
[INFO] |  +- com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20:compile
[INFO] |  |  +- (com.typesafe.akka:akka-actor_2.11:jar:2.4.20:compile - omitted for duplicate)
[INFO] |  |  \- (org.slf4j:slf4j-api:jar:1.7.16:compile - omitted for conflict with 1.7.15)
[INFO] |  +- org.clapper:grizzled-slf4j_2.11:jar:1.3.2:compile
[INFO] |  |  \- (org.slf4j:slf4j-api:jar:1.7.9:compile - omitted for conflict with 1.7.15)
[INFO] |  +- com.github.scopt:scopt_2.11:jar:3.5.0:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.7.6:compile
[INFO] |  |  \- com.twitter:chill-java:jar:0.7.6:compile
[INFO] |  +- com.github.oshi:oshi-core:jar:3.4.0:compile
[INFO] |  |  +- net.java.dev.jna:jna-platform:jar:4.2.2:compile
[INFO] |  |  |  \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] |  |  \- (org.slf4j:slf4j-api:jar:1.7.22:compile - omitted for conflict with 1.7.15)
[INFO] |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] +- org.apache.flink:flink-streaming-java_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- org.apache.flink:flink-clients_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- org.apache.flink:flink-optimizer_2.11:jar:1.7-tdflink-0.2.40:compile
[INFO] |  |  |  +- (org.apache.flink:flink-core:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-runtime_2.11:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.apache.flink:flink-shaded-jackson:jar:2.7.9-5.0:compile - omitted for duplicate)
[INFO] |  |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (org.apache.flink:flink-java:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  |  +- (commons-cli:commons-cli:jar:1.3.1:compile - omitted for duplicate)
[INFO] |  |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] |  +- (org.apache.flink:flink-shaded-guava:jar:18.0-5.0:compile - omitted for duplicate)
[INFO] |  +- org.apache.commons:commons-math3:jar:3.5:compile
[INFO] |  +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] |  \- (org.apache.flink:force-shading:jar:1.7-tdflink-0.2.40:compile - omitted for duplicate)
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.14:compile
[INFO] |  +- (org.slf4j:slf4j-api:jar:1.7.14:compile - omitted for conflict with 1.7.15)
[INFO] |  \- (log4j:log4j:jar:1.2.17:compile - omitted for duplicate)
[INFO] \- log4j:log4j:jar:1.2.17:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.437 s
[INFO] Finished at: 2021-11-25T11:22:25+08:00
[INFO] Final Memory: 13M/309M
[INFO] ------------------------------------------------------------------------

在上述输出的日志中，打印出项目的依赖树以及版本冲突情况，其中

version managed from xxx, omitted for duplicated：这个依赖和另一个依赖版本冲突了，使用xxx为最终的版本，并忽略这个依赖
omitted for duplicated：这个依赖已经存在了，忽略这个依赖
omitted for conflict with xxx：这个依赖和另一个依赖版本冲突了，使用xxx为最终的版本，并忽略这个依赖

如何解决依赖冲突

将依赖的 scope 改成 provided

<dependency>
   <groupId>org.apache.flink</groupId>
   <artifactId>flink-core</artifactId>
   <version>${flink.version}</version>
   <scope>provided</scope>           # 将 org.apache.flink:flink-core 对应的 jar 改成 provided 的
</dependency>

显示的 exclusive 掉相关依赖

 <dependency>
    <groupId>com.alibaba.occea</groupId>
    <artifactId>TDB</artifactId>
    <version>1.2.14</version>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
    </exclusions>
</dependency>

对 class 进行 shade 操作

<build>
 <plugins>
   ...
   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-shade-plugin</artifactId>
       <version>3.1.1</version>
       <executions>
          <execution>
             <id>shade-plugin</id>
             <phase>package</phase>
             <goals>
                <goal>shade</goal>
             </goals>
             <configuration>
               <shadedArtifactAttached>false</shadedArtifactAttached>
               <filters>
                   <filter>
                       <artifact>*:*</artifact>
                       <excludes>
                           <exclude>*.jar</exclude>
                           <exclude>META-INF/maven/**</exclude>
                       </excludes>
                   </filter>
               </filters>
               <artifactSet>
                   <includes>
                       <include>org.json4s:json4s-ast_${scala.binary.version}</include>
                       <include>org.json4s:json4s-core_${scala.binary.version}</include>
                       <include>org.json4s:json4s-jackson_${scala.binary.version}</include>
                       <include>org.json4s:json4s-scalap_${scala.binary.version}</include>
                   </includes>
               </artifactSet>
               <relocations>
                   <relocation>
                       <pattern>org.json4s</pattern>
                       <shadedPattern>org.apache.livy.shaded.json4s</shadedPattern>
                   </relocation>
               </relocations>
           </configuration>
          </execution>
       </executions>
       <configuration>
 	     <relocations>
 	        <relocation>
 		       <pattern>org.yaml</pattern>
 		       <shadedPattern>test</shadedPattern>
 	        </relocation>
         </relocations>
       </configuration>
     </plugin>
 </plugins>
</build>