flink编译过程中下载的依赖包很多,在 apache-maven-3.5.4/conf/settings.xml 配置ali和huawei两个mirror,下载会加速不少。
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>*,!hadoop</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
<mirror>
<id>huaweimaven</id>
<name>huawei maven</name>
<url>https://mirrors.huaweicloud.com/repository/maven/</url>
<mirrorOf>central</mirrorOf>
</mirror>
编译问题汇总:
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :flink-mapr-fs
报错缺失flink-mapr-fs,需要手动下载安装。
通过手动下载maprfs-5.2.1-mapr.jar包,下载地址地址:https://repository.mapr.com/nexus/content/groups/mapr-public/com/mapr/hadoop/maprfs/5.2.1-mapr/
.上传至主节点
将下载的maprfs-5.2.1-mapr.jar包上传至主节点的/home/hadoop/downloads目录下。
3.手动安装
手动安装缺少的包到本地仓库
mvn install:install-file -DgroupId=com.mapr.hadoop -DartifactId=maprfs -Dversion=5.2.1-mapr -Dpackaging=jar -Dfile=/home/hadoop/downloads/maprfs-5.2.1-mapr.jar
4.继续编译
使用maven继续编译flink(可以排除刚刚已经安装的包)
mvn clean install -Dmaven.test.skip=true -Dhadoop.version=2.7.3 -rf :flink-mapr-fs
[ERROR] /home/hadoop/opensource/flink/flink-filesystems/flink-mapr-fs/src/main/java/org/apache/flink/runtime/fs/maprfs/MapRFileSystem.java:[73,93] cannot find symbol
[ERROR] symbol: class Configuration
缺失org.apache.hadoop.fs包,报错找不到。
flink-mapr-fs模块的pom文件中添加如下依赖:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
又报错:
[ERROR] Failed to execute goal on project flink-avro-confluent-registry: Could not resolve dependencies for project org.apache.flink:flink-avro-confluent-registry:jar:1.6.2: Could not find artifact io.confluent:kafka-schema-registry-client:jar:3.3.1 in nexus-osc (http://maven.aliyun.com/nexus/content/repositories/central) -> [Help 1]
同样的方法结果:
mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=3.3.1 -Dpackaging=jar -Dfile=/opt/tools/installed/kafka-schema-registry-client-3.3.1.jar
继续编译
mvn clean install -DskipTests -Dhadoop.version=3.1.1 -Dscala-2.11 -Dpython2.7 -rf :flink-avro-confluent-registry
[INFO] flink-yarn-tests ................................... SUCCESS [ 42.978 s]
[INFO] flink-fs-tests ..................................... SUCCESS [ 0.595 s]
[INFO] flink-docs 1.8.1 ................................... SUCCESS [ 2.857 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:38 min
[INFO] Finished at: 2020-04-19T14:10:44+08:00
[INFO] ------------------------------------------------------------------------
编译完成:
flink-release-1.8.1/flink-dist/target/flink-1.8.1-bin