上传本地jar包到spark集群的jars目录下:
cd /software/spark-3.1.2-bin-hadoop2.7/jars
idea打包项目生成jar包的方式:
添加maven的pom依赖,根据中文注释更改值
<build>
<sourceDirectory>src/main</sourceDirectory>
<testSourceDirectory>src/main</testSourceDirectory>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<!--jar包的文件名 -->
<finalName>${project.groupId}</finalName>
<archive>
<manifest>
<!--这里要替换成jar包main方法所在类 -->
<mainClass>src.spark.demo02</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase> <!-- 指定在打包节点执行jar包合并操作 -->
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<!-- see http://davidb.github.com/scala-maven-plugin -->
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<useFile>false</useFile>
<disableXmlReport>true</disableXmlReport>
<!-- If you have classpath issue like NoDefClassError,... -->
<!-- useManifestOnlyJar>false</useManifestOnlyJar -->
<includes>
<include>**/*Test.*</include>
<include>**/*Suite.*</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
maven-package生成jar包
master是Spark,Mesos或YARN集群的URL,或者一个特殊的“local [*]”字符串来让程序以本地模式运行
当在集群上运行程序时,不需要在程序中硬编码master参数,而是使用spark-submit提交应用程序并将master的URL以脚本参数的形式传入。但是,对于本地测试和单元测试,您可以通过“local[*]”来运行Spark程序(请确保本地系统中的cpu核心数够用)
/software/spark-3.1.2-bin-hadoop2.7/bin/spark-submit
--class application.Application
--master spark://hadoop01:7077
/software/spark-3.1.2-bin-hadoop2.7/jars/EcommerceAnalysis-1.0-SNAPSHOT.jar
10