Spark程序运行方式

Spark程序运行方式

本文主要介绍Spark上传集群运行的过程及shell脚本的编写

脚本文件编写参数介绍

在linux环境下 spark-submit指令打印如下

[hadoop@hadoop01 MyShell]$ spark-submit
Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]

Options:
  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                              on one of the worker machines inside the cluster ("cluster")
                              (Default: client).
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --name NAME                 A name of your application.
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --packages                  Comma-separated list of maven coordinates of jars to include
                              on the driver and executor classpaths. Will search the local
                              maven repo, then maven central and any additional remote
                              repositories given by --repositories. The format for the
                              coordinates should be groupId:artifactId:version.
  --exclude-packages          Comma-separated list of groupId:artifactId, to exclude while
                              resolving the dependencies provided in --packages to avoid
                              dependency conflicts.
  --repositories              Comma-separated list of additional remote repositories to
                              search for the maven coordinates given with --packages.
  --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
                              on the PYTHONPATH for Python apps.
  --files FILES               Comma-separated list of files to be placed in the working
                              directory of each executor. File paths of these files
                              in executors can be accessed via SparkFiles.get(fileName).

  --conf PROP=VALUE           Arbitrary Spark configuration property.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.

  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).

  --proxy-user NAME           User to impersonate when submitting the application.
                              This argument does not work with --principal / --keytab.

  --help, -h                  Show this help message and exit.
  --verbose, -v               Print additional debug output.
  --version,                  Print the version of current Spark.
*************************************这条线上边的是local和其他模式公用的参数下边是standalone模式独有的***********************
 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: 1).

 Spark standalone or Mesos with cluster deploy mode only:
  --supervise                 If given, restarts the driver on failure.
  --kill SUBMISSION_ID        If given, kills the driver specified.
  --status SUBMISSION_ID      If given, requests the status of the driver specified.

 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.

 Spark standalone and YARN only:
  --executor-cores NUM        Number of cores per executor. (Default: 1 in YARN mode,
                              or all available cores on the worker in standalone mode)
****************************************下边是yarn独有的参数**************************************************************
 YARN-only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode
                              (Default: 1).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: 2).
                              If dynamic allocation is enabled, the initial number of
                              executors will be at least NUM.
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.
  --principal PRINCIPAL       Principal to be used to login to KDC, while running on
                              secure HDFS.
  --keytab KEYTAB             The full path to the file that contains the keytab for the
                              principal specified above. This keytab will be copied to
                              the node running the Application Master via the Secure
                              Distributed Cache, for renewing the login tickets and the
                              delegation tokens periodically.

1.local本地运行

代码如下

package com.wordcount

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}


/**
  * 使用scala实现wordCount,读取HDFS的数据
  * 注意这里在运行的时候用到了高可用的额组名,这样的话就需要将core-site.xml和hdfs-site.xml放在classPath的目录下
  * 放在resources的目录下运行就可以了
  *
  */
object _03wordCountHDFS {
  def main(args: Array[String]): Unit = {
    //创建配置对象
    val conf: SparkConf = new SparkConf()
      .setAppName(s"${_03wordCountHDFS.getClass.getSimpleName}")
      .setMaster("local")
    val sc = new SparkContext(conf)

    //生成RDD
    val textRDD: RDD[String] = sc.textFile("hdfs://bd1901/wordcount/in/word.txt")
    //进行transformation操作和action操作
    val retRDD: RDD[(String, Int)] = textRDD.flatMap(_.split("\\.")).map((_,1)).reduceByKey(_+_)
    //将最终的数据进行输出
   // retRDD.foreach(x=>println(x._1+"------->"+x._2))
    retRDD.saveAsTextFile("hdfs://bd1901/sparkwordout")
    //关闭资源
    sc.stop()
  }
}

将项目打成jar包上传到linux服务器本地目录
编写shell脚本文件

#!/bin/sh


SPARK_BIN=/home/hadoop/apps/spark/bin

${SPARK_BIN}/spark-submit \
--class com.wordcount._03wordCountHDFS \  //运行的类的全限定名
--master local \     //指定本地运行
--deploy-mode client \  //client模式
--executor-memory 600m \   //内存600m
/home/hadoop/jars/spark-core-1.0-SNAPSHOT.jar  //jar包的地址

2.standalone运行

这是采用Spark运行的方式
master的写法如下:
master: spark://bigdata01:7077或者 spark://bigdata01:7077,bigdata02:7077

• 单机模式:spark://bigdata01:7077

• 如果是HA的集群，spark://bigdata01:7077,bigdata02:7077

deploy-mode: 为cluster方式和client方式
两种方式的区别:

• client :driver在本地启动，提交spark作业的那台机器就是driver，sparkcontext就在这台机器创建
• cluster :driver不在本地启动，driver会在spark集群中被启动，driver也在worker节点上面

代码如下:

package com.wordcount

import org.apache.hadoop.fs.FileSystem
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}


/**
  * 使用scala实现wordCount,读取HDFS的数据
  * 注意这里在运行的时候用到了高可用的额组名,这样的话就需要将core-site.xml和hdfs-site.xml放在classPath的目录下
  * 放在resources的目录下运行就可以了
  *
  */
object _04wordCountHDFSjar{
  def main(args: Array[String]): Unit = {
    //创建配置对象
    val conf: SparkConf = new SparkConf()
      .setAppName(s"${_04wordCountHDFSjar.getClass.getSimpleName}")
  //.set("HADOOP_USER_NAME","hadoop").set("fs.defaultFS", "hdfs://bd1901:8020")
    val sc = new SparkContext(conf)

    //生成RDD
    val textRDD: RDD[String] = sc.textFile("hdfs://bd1901/wordcount/in/word.txt")
    //进行transformation操作和action操作
    val retRDD: RDD[(String, Int)] = textRDD.flatMap(_.split("\\.")).map((_,1)).reduceByKey(_+_)
    //将最终的数据进行输出
    retRDD.saveAsTextFile("hdfs://bd1901/sparkwordout")
    //关闭资源
    sc.stop()
  }
}

将代码打成jar包上传集群运行既可,这种方式应将jar包放在hdfs上,这样不同节点均能访问
编写脚本如下
client的脚本

#!/bin/sh


SPARK_BIN=/home/hadoop/apps/spark/bin

${SPARK_BIN}/spark-submit \
--class com.wordcount._04wordCountHDFSjar \
--master spark://hadoop01:7077,hadoop02:7077 \
--deploy-mode client \
--executor-memory 1000m \
--total-executor-cores 1 \
--executor-cores 1 \
hdfs://bd1901/jars/spark-core-1.0-SNAPSHOT.jar

cluster方式的脚本的编写

#!/bin/sh


SPARK_BIN=/home/hadoop/apps/spark/bin

${SPARK_BIN}/spark-submit \
--class com.wordcount._04wordCountHDFSjar \
--master spark://hadoop01:7077,hadoop02:7077 \
--deploy-mode cluster \
--executor-memory 1000M \
--total-executor-cores 1 \
--executor-cores 1 \
hdfs://bd1901/jars/spark-core-1.0-SNAPSHOT.jar

运行相应对的脚本即可

3.yarn的方式运行

master的写法 -------->master:yarn
deploy-mode:

client :driver在本地启动，提交spark作业的那台机器就是driver，sparkcontext就在这台机器创建

cluster :driver不在本地启动，driver会在yarn集群中被启动，driver也在nodemanager节点上面
代码如下

package com.wordcount

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}


/**
  * 使用scala实现wordCount,读取HDFS的数据
  * 注意这里在运行的时候用到了高可用的额组名,这样的话就需要将core-site.xml和hdfs-site.xml放在classPath的目录下
  * 放在resources的目录下运行就可以了
  *这种使用参数的形式进行
  */
object _05wordCountHDFSjarargs{
  def main(args: Array[String]): Unit = {
    if(args==null||args.length<3){
      println(
            """
              |Parameter Errors! Usage:<inputpath> <sleep> <output>
              |inputpath:   程序输入参数
              |sleep:   休眠时间
              |output:  程序输出参数
            """.stripMargin
      )
    }
    val Array(inputpath,sleep,outputpath)=args//模式匹配
    //创建配置对象
    val conf: SparkConf = new SparkConf()
      .setAppName(s"${_04wordCountHDFSjar.getClass.getSimpleName}")
    val sc = new SparkContext(conf)

    //生成RDD
    val textRDD: RDD[String] = sc.textFile(inputpath)
    //进行transformation操作和action操作
    val retRDD: RDD[(String, Int)] = textRDD.flatMap(_.split("\\.")).map((_,1)).reduceByKey(_+_)
    //将最终的数据进行输出
    retRDD.saveAsTextFile(outputpath)
    Thread.sleep(sleep.toLong)
    //关闭资源
    sc.stop()
  }
}

将代码打成jar包上传到hdfs上,这样就可以不同节点共享
编写shell脚本

client方式的脚本的编写

#!/bin/sh


SPARK_BIN=/home/hadoop/apps/spark/bin
OUT_PATH=hdfs://bd1901/out/spark/wc
`hdfs dfs -test -z ${OUT_PATH}`

if [ $? -eq 0 ]
then
  echo 1111
  `hdfs dfs -rm -R ${OUT_PATH}`
fi

${SPARK_BIN}/spark-submit \
--class com.wordcount._05wordCountHDFSjarargs \
--master yarn \
--deploy-mode client \
--executor-memory 1000m \
--num-executors 1 \ 
--executor-cores 1 \
hdfs://bd1901/jars/spark-core-1.0-SNAPSHOT.jar \
hdfs://bd1901/wordcount/in/word.txt 1000 ${OUT_PATH}

cluster的方式的shell编写

#!/bin/sh


SPARK_BIN=/home/hadoop/apps/spark/bin
OUT_PATH=hdfs://bd1901/out/spark/wc     //指定输出的路径
`hdfs dfs -test -z ${OUT_PATH}`          //判断输出路径是否存再

if [ $? -eq 0 ]                   //如果存在的话将此路径的文件进行删除,避免报错
then
  echo 1111
  `hdfs dfs -rm -R ${OUT_PATH}`
fi

${SPARK_BIN}/spark-submit \
--class com.wordcount._05wordCountHDFSjarargs \     //运行的类的完全限定名
--master yarn \                        //yarn模式
--deploy-mode cluster \               //cluster模式
--executor-memory 1000m \          //内存1000m
--num-executors 1 \ 
--executor-cores 1 \
hdfs://bd1901/jars/spark-core-1.0-SNAPSHOT.jar \      //jar包路径
hdfs://bd1901/wordcount/in/word.txt 1000 ${OUT_PATH}      //手动输入程序的参数

运行相应的脚本,注意这里需要在脚本中传入相应的参数给运行的程序,这样程序更加健壮

注意:
在打jar包的过程中需要注意,打成jar包之后需要检查一下是否包含所要运行的类,如果不包含的话就不要上传,否则也要报错,如果不包含所有的类的话有可能是maven的打包的插件不正常
可以引入如下的插件

<build>
	<pluginManagement>
		<plugins>
			<plugin>
				<groupId>net.alchim31.maven</groupId>
				<artifactId>scala-maven-plugin</artifactId>
				<version>3.2.2</version>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.5.1</version>
			</plugin>
		</plugins>
	</pluginManagement>
	<plugins>
		<plugin>
			<groupId>net.alchim31.maven</groupId>
			<artifactId>scala-maven-plugin</artifactId>
			<executions>
				<execution>
					<id>scala-compile-first</id>
					<phase>process-resources</phase>
					<goals>
						<goal>add-source</goal>
						<goal>compile</goal>
					</goals>
				</execution>
				<execution>
					<id>scala-test-compile</id>
					<phase>process-test-resources</phase>
					<goals>
						<goal>testCompile</goal>
					</goals>
				</execution>
			</executions>
		</plugin>

		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<executions>
				<execution>
					<phase>compile</phase>
					<goals>
						<goal>compile</goal>
					</goals>
				</execution>
			</executions>
		</plugin>

		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-shade-plugin</artifactId>
			<version>2.4.3</version>
			<executions>
				<execution>
					<phase>package</phase>
					<goals>
						<goal>shade</goal>
					</goals>
					<configuration>
						<filters>
							<filter>
								<artifact>*:*</artifact>
								<excludes>
									<exclude>META-INF/*.SF</exclude>
									<exclude>META-INF/*.DSA</exclude>
									<exclude>META-INF/*.RSA</exclude>
								</excludes>
							</filter>
						</filters>
					</configuration>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>