spark
spark下载地址:
https://www.apache.org/dyn/closer.lua/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
spark源码编译
mvn clean install -Dmaven.test.skip=true -Dhadoop.version=2.7.2 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true
编译打包
./dev/make-distribution.sh --name 2.7.2 --tgz -Pyarn -Phive -Phive-thriftserver -Pscala-2.11 -Phadoop-2.7 -Dhadoop.version=2.7.2
-Dmaven.clean.failOnError=false
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<failOnError>false</failOnError>
</configuration>
</plugin>
远程调试spark
#调试Master,在master节点的spark-env.sh中添加```code
#SPARK_MASTER_OPTS变量
export SPARK_MASTER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10000"
#调试Worker,在worker节点的spark-env.sh中添加```code
#SPARK_WORKER_OPTS变量
export SPARK_WORKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10001"
flink
编译
mvn checkstyle:checkstyle
mvn clean package -DskipTests
flink 编译报错
compiler-plugin:3.8.0:compile (default-compile) on project flink-avro-confluent-registry: Compilation failure: Compilation failure:
解决
mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=3.3.1 -Dpackaging=jar -Dfile=/Users/AllenBai/Downloads/kafka-schema-registry-client-3.3.1.jar
2、编译flink-shaded
指定hadoop版本2.7.2, 把编译好的包打到本地maven仓库,后面编译flink
mvn clean install -DskipTests -Dhadoop.version=2.7.2
3、编译 flink
指定hadoop 版本 2.7.2
mvn clean package -DskipTests -Dhadoop.version=2.7.2