spark和flink源码编译

spark

spark下载地址：

https://www.apache.org/dyn/closer.lua/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz

spark源码编译

mvn clean install -Dmaven.test.skip=true -Dhadoop.version=2.7.2 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true

编译打包

./dev/make-distribution.sh --name 2.7.2 --tgz -Pyarn -Phive -Phive-thriftserver -Pscala-2.11 -Phadoop-2.7 -Dhadoop.version=2.7.2  
 -Dmaven.clean.failOnError=false

 <plugin>
      <artifactId>maven-clean-plugin</artifactId>
      <version>3.0.0</version>
      <configuration>
        <failOnError>false</failOnError>
      </configuration>
    </plugin>

远程调试spark

#调试Master，在master节点的spark-env.sh中添加```code

#SPARK_MASTER_OPTS变量
export SPARK_MASTER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10000"

#调试Worker，在worker节点的spark-env.sh中添加```code

#SPARK_WORKER_OPTS变量
export SPARK_WORKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10001"

flink

编译

mvn checkstyle:checkstyle
mvn clean package -DskipTests

flink 编译报错

 compiler-plugin:3.8.0:compile (default-compile) on project flink-avro-confluent-registry: Compilation failure: Compilation failure:

解决

 mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-schema-registry-client -Dversion=3.3.1 -Dpackaging=jar  -Dfile=/Users/AllenBai/Downloads/kafka-schema-registry-client-3.3.1.jar

2、编译flink-shaded

指定hadoop版本2.7.2， 把编译好的包打到本地maven仓库，后面编译flink

mvn clean install -DskipTests -Dhadoop.version=2.7.2

3、编译 flink

指定hadoop 版本 2.7.2

mvn clean package -DskipTests -Dhadoop.version=2.7.2