spark2.3.1+jdk1.8向下兼容1.7（生产机器为jdk1.7）

最新推荐文章于 2022-04-06 16:43:49 发布

cjlion

最新推荐文章于 2022-04-06 16:43:49 发布

阅读量1.5k

点赞数

分类专栏： spark

本文链接：https://blog.csdn.net/cjlion/article/details/83308810

版权

spark 专栏收录该内容

5 篇文章 1 订阅

订阅专栏

1、打包

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>${spark.version}</version>
    <!-- 编译环境能用，运行环境不可用(打包时不打进来) -->
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>${spark.version}</version>
    <!-- 编译环境能用，运行环境不可用 -->
    <scope>provided</scope>
</dependency>

2、上传到生产集群/opt/core/spark-2.3.1-bin-hadoop2.6/bin/spark-submit --class com.jiupaipay.wordcount.WordCount --master local[*] ./sparkwordcount-1.0-SNAPSHOT-jar-with-dependencies.jar

/opt/core/spark-2.3.1-bin-hadoop2.6/bin/spark-submit --queue root.line.pay --class com.jiupaipay.wordcount.WordCount --master yarn-client ./sparkwordcount-1.0-SNAPSHOT-jar-with-dependencies.jar