Spark1.5.2 on yarn fair scheduler 配置

原创 2015年11月19日 15:20:26

1.fair scheduler 配置

  
<?xml version="1.0"?>
<allocations>
  <pool name="production">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>2</minShare>
  </pool>
  <pool name="test">
    <schedulingMode>FIFO</schedulingMode>
    <weight>2</weight>
    <minShare>3</minShare>
  </pool>
  <pool name="spark">
    <schedulingMode>FAIR</schedulingMode>
    <weight>2</weight>
    <minShare>3</minShare>
  </pool>
  <pool name="ztai">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>3</minShare>
  </pool>
</allocations>

2.代码提交

mport org.apache.spark.SparkConf
import org.apache.spark.SparkContext

/**
 * Created by root on 11/18/15.
 */

object HiveUtility {
  def createTable(sc: SparkContext, db: String, table: String, system_user_name: String) = {
    val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
    println("user.name " + System.getProperty("user.name"))
    println("HADOOP_USER_NAME " + System.getProperty("HADOOP_USER_NAME"))
    sqlContext.sql("use " + db)
    sqlContext.sql("CREATE TABLE IF NOT EXISTS " + db + "." + table + "(key INT, value STRING)")
    sqlContext.sql("LOAD DATA  INPATH 'hdfs:///user/spark/examples/src/main/resources/kv1.txt' INTO TABLE " + db + "." + table)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").count()
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 20").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 5").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 15").collect().foreach(println)
  }

  def main(args: Array[String]) {
    if (args.length < 3) {
      print("useage:HiveUtility.jar db table system_user_name")
    }
    System.setProperty("user.name", args(2))
    System.setProperty("HADOOP_USER_NAME", args(2))
    val conf = new SparkConf()
   // conf.set("spark.scheduler.allocation.file", "hdfs:///user/spark/fairscheduler.xml")
    conf.set("spark.scheduler.mode", "FAIR") // set scheduler mode
    val sc = new SparkContext(conf)
    sc.setLocalProperty("spark.scheduler.pool", "spark") // set fair scheduler
    createTable(sc, args(0), args(1), args(2))
  }
}

提交命令:
mainclass=com.suning.spark.hive.HiveUtility
mainjar=HiveUtility-v1.jar
hadoop fs -rm  /home/spark/software/spark/examples/src/main/resources/kv1.txt
hadoop fs -put /home/spark/software/spark/examples/src/main/resources/kv1.txt examples/src/main/resources
export HADOOP_USER_NAME=bi
djars=/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar,/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar,/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar,/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar
spark-submit --master yarn-cluster \
             --queue spark       \
             --files /home/spark/software/spark/conf/hive-site.xml,/home/spark/software/spark/conf/fairscheduler.xml \
             --jars $djars            \
             --class $mainclass $mainjar  bi_dm  hive3   bi

3.结果验证





日志:
5/11/19 15:16:51 INFO Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
15/11/19 15:16:51 INFO Client: Setting up container launch context for our AM
15/11/19 15:16:51 INFO Client: Setting up the launch environment for our AM container
15/11/19 15:16:51 INFO Client: Preparing resources for our AM container
15/11/19 15:16:51 INFO Client: Uploading resource file:/home/spark/software/spark-1.5.2-bin-2.4.0/lib/spark-assembly-1.5.2-hadoop2.4.0.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/spark-assembly-1.5.2-hadoop2.4.0.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/workspace/tersort/HiveUtility-v1.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/HiveUtility-v1.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-api-jdo-3.2.6.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-core-3.2.10.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-rdbms-3.2.9.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/mysql-connector-java-5.1.27-bin.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/hive-site.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/hive-site.xml
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/fairscheduler.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/fairscheduler.xml
15/11/19 15:16:54 INFO Client: Uploading resource file:/data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0/__spark_conf__6425041649655761245.zip -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/__spark_conf__6425041649655761245.zip
15/11/19 15:16:54 INFO SecurityManager: Changing view acls to: spark,bi
15/11/19 15:16:54 INFO SecurityManager: Changing modify acls to: spark,bi
15/11/19 15:16:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark, bi); users with modify permissions: Set(spark, bi)
15/11/19 15:16:54 INFO Client: Submitting application 6897 to ResourceManager
15/11/19 15:16:54 INFO YarnClientImpl: Submitted application application_1447783148941_6897
15/11/19 15:16:55 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:55 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.spark
	 start time: 1447917414919
	 final status: UNDEFINED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:16:56 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:57 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:58 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:59 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:17:00 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:00 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 10.27.1.143
	 ApplicationMaster RPC port: 0
	 queue: root.spark
	 start time: 1447917414919
	 final status: UNDEFINED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:17:01 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:02 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:03 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:04 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:05 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:06 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:07 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:09 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:10 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:11 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:12 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:13 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:14 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:15 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:16 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:17 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:18 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:19 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:20 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:21 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:22 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:23 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:24 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:25 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:26 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:27 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:28 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:29 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:30 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:31 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:32 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:33 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:34 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:35 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:36 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:37 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:38 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:39 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:40 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:41 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:42 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:43 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:44 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:45 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:46 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:47 INFO Client: Application report for application_1447783148941_6897 (state: FINISHED)
15/11/19 15:17:47 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 10.27.1.143
	 ApplicationMaster RPC port: 0
	 queue: root.spark
	 start time: 1447917414919
	 final status: SUCCEEDED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:17:47 INFO ShutdownHookManager: Shutdown hook called
15/11/19 15:17:47 INFO ShutdownHookManager: Deleting directory /data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0




参考:http://spark.apache.org/docs/latest/job-scheduling.html


版权声明:本文为博主原创文章,未经博主允许不得转载。

yarn中资源调度fair schedule详解

Fair Scheduler将整个Yarn的可用资源划分成多个资源池,每个资源池中可以配置最小和最大的可用资源(内存和CPU)、最大可同时运行Application数量、权重、以及可以提交和管理Ap...
  • sinat_29581293
  • sinat_29581293
  • 2017年02月27日 17:26
  • 1607

yarn Fairscheduler与Capacityscheduler

Capacityscheduler允许多个组织共享整个集群,每个组织可以获得集群的一部分计算能力。通过为每个组织分配专门的队列,然后再为每个队列分配一定的集群资源,这样整个集群就可以通过设置多个队列的...
  • Nougats
  • Nougats
  • 2017年05月01日 20:29
  • 991

yarn fair scheduler 之概述

所有的调度器都是基于事件模型,需要处理的一个重要事件是nodeUpdate,处理各个节点的心跳。 对于公平调度器,在处理某一Node心跳事件时,先处理刚刚启动的Container和已完成任务的Con...
  • wujun8
  • wujun8
  • 2014年06月19日 16:52
  • 1838

Yarn参数优化(Fair Scheduler版本)

YARN自从hadoop2.0之后, 我们可以使用apache yarn 来对集群资源进行管理。yarn把可以把资源(内存,CPU)以Container的方式进行划分隔离。YARN会管理集群中所有机器...
  • leone911
  • leone911
  • 2016年06月07日 17:08
  • 754

Hadoop权威指南读书笔记(2) — Yarn简介及Capacity & Fair Scheduler

Yarn简介Yarn(Yet Another Resource Negotiator)是hadoop2中引入的一个集群资源管理系统。...
  • labud
  • labud
  • 2016年07月27日 19:18
  • 1204

Fair Scheduler

与Capacity Scheduler类似,Fair Scheduler以队列为单位划分资源,每个队列可设定一定比例的资源最低保证和使用上限,同时,每个用户也可设定一定的资源使用上限以防止资源滥用;当...
  • a118170653
  • a118170653
  • 2015年02月10日 11:00
  • 904

基于yarn的公平调度实例

场景描述       假设遇到这样的客户,需要在公司内部的集群上进行任务提交运行,客户的任务是每天跑取一些比较简单的mr程序(凌晨提交上来,需要在当天的6点之前运行结束),而公司内部自己需要用集群做相...
  • JThink_
  • JThink_
  • 2015年12月01日 17:19
  • 2829

CDH5.4.4基于yarn的公平调度器配置

公平调度器介绍 简介      公平调度器是一种赋予作业(app)资源的方法,目的是让所有的作业随着时间的推移,都能获得平均的资源。hadoop NG有能力调度多种不同类型的资源(CPU co...
  • JThink_
  • JThink_
  • 2015年11月30日 14:59
  • 3175

hadoop2.0 公平调度器(fair-scheduler)配置

如果是第一次接触hadoop2.0版本的 Fairshare scheduler , 最好先看一下他的官方文档: http://hadoop.apache.org/docs/r2.2.0/hadoop...
  • baiyangfu
  • baiyangfu
  • 2013年11月01日 17:21
  • 6013

Fair Scheduler 队列设置

先附一个官网链接 由于yarn只能使用一种调度器,所以要想使用Fair Scheduler首先要在yarn-site.xml中将配置参数 yarn.resourcemanager.scheduler....
  • lumingkui1990
  • lumingkui1990
  • 2016年08月10日 17:15
  • 814
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:Spark1.5.2 on yarn fair scheduler 配置
举报原因:
原因补充:

(最多只允许输入30个字)