一、在idea中 写一个 wordcount 案例:
这里一定不能设置 .setMaster(“local[*]”)
这里一定不能设置 .setMaster(“local[*]”)
这里一定不能设置 .setMaster(“local[*]”)
重要的事情要说三遍!!!!否则 在启动yarn 模式的时候会报错!!!
val conf = new SparkConf().setMaster(“local[*]”).setAppName(“WordCount”)
package submit_test
import org.apache.spark.{SparkConf, SparkContext, SparkFiles}
object WordCount {
def main(args: Array[String]): Unit = {
//scala代码实现spark入口 (这里没有指明是什么模式,在提交时指明)
//yarn这里一定不能设置.setMaster("local[*]")
//val conf = new SparkConf().setMaster("local[*]").setAppName("WordCount")
val conf = new SparkConf().setAppName("WordCount")
val sc = new SparkContext(conf)
//加载hdfs文件到SparkRDD
val lines = sc.textFile(args(0))
//WordCount实现
val result = lines.flatMap(x=>x.split(" ")).map((_,1)).reduceByKey(_+_)
//结果保存在hdfs
result.saveAsTextFile(args(1))
sc.stop()
}
}
二、使用idea打包:
2.1 默认都点ok (有的依赖不需要 ,可以自选)
2.2build 和 rebuild 都可以:
2.3 将spark-submit.jar 上传到 虚拟机上:
三、虚拟机上的操作
[root@cdh01 sparktest]# pwd
/opt/sparktest
[root@cdh01 sparktest]# ll
总用量 178308
-rw-r--r-- 1 root root 182583469 3月 23 20:51 spark-submit.jar
[root@cdh01 sparktest]# zip -d spark-submit.jar META-INF/*.RSA META-INF/*.DSA META-INF/*.SF
zip warning: name not matched: META-INF/*.RSA
deleting: META-INF/DUMMY.SF
deleting: META-INF/DUMMY.DSA
[root@cdh01 sparktest]#
四、在hdfs 上创建一个 文件夹 sparktest
在sparktest文件夹下创建一个 word.txt文件
文件内容如下:
hello world
hello print
五、提交任务 :
[root@cdh01 sparktest]# spark-submit --master yarn --deploy-mode cluster --driver-memory 500m --executor-memory 500m --executor-cores 2 --class submit_test.WordCount /opt/sparktest/spark-submit.jar hdfs://192.168.2.112:8020/sparktest/word.txt hdfs://192.168.2.112:8020/sparktest/wcOut
其中 这个输入、输出路径 的ip 是高可用活跃的那天机器的ip
hdfs://192.168.2.112:8020/sparktest/word.txt
hdfs://192.168.2.112:8020/sparktest/wcOut
输出的文件夹如下:
wordcount 的结果:
备注:
我用这种打包方式 spark-submit 也可以:
[root@cdh01 sparktest]# ll
总用量 366372
-rw-r--r-- 1 root root 194196468 3月 23 21:13 spark-submit-1.0-SNAPSHOT.jar
-rw-r--r-- 1 root root 180963015 3月 23 20:53 spark-submit.jar
[root@cdh01 sparktest]# spark-submit --master yarn --deploy-mode cluster --driver-memory 500m --executor-memory 500m --executor-cores 2 --class submit_test.WordCount /opt/sparktest/spark-submit-1.0-SNAPSHOT.jar hdfs://192.168.2.112:8020/sparktest/word.txt hdfs://192.168.2.112:8020/sparktest/wcOut333
pom.xml如下:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.atguidian.sparksubmit</groupId>
<artifactId>spark-submit</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<spark.version>2.4.0</spark.version>
<scala.version>2.11</scala.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<!-- scala-maven-plugin:编译scala程序的Maven插件 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
</plugin>
<!-- maven-compiler-plugin:编译java程序的Maven插件 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<!-- 编译scala程序的Maven插件的一些配置参数 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- 编译java程序的Maven插件的一些配置参数 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- maven-shade-plugin:打jar包用的Mavne插件 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>