Spark1.5.2 on yarn fair scheduler 配置

原创 2015年11月19日 15:20:26

1.fair scheduler 配置

  
<?xml version="1.0"?>
<allocations>
  <pool name="production">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>2</minShare>
  </pool>
  <pool name="test">
    <schedulingMode>FIFO</schedulingMode>
    <weight>2</weight>
    <minShare>3</minShare>
  </pool>
  <pool name="spark">
    <schedulingMode>FAIR</schedulingMode>
    <weight>2</weight>
    <minShare>3</minShare>
  </pool>
  <pool name="ztai">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>3</minShare>
  </pool>
</allocations>

2.代码提交

mport org.apache.spark.SparkConf
import org.apache.spark.SparkContext

/**
 * Created by root on 11/18/15.
 */

object HiveUtility {
  def createTable(sc: SparkContext, db: String, table: String, system_user_name: String) = {
    val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
    println("user.name " + System.getProperty("user.name"))
    println("HADOOP_USER_NAME " + System.getProperty("HADOOP_USER_NAME"))
    sqlContext.sql("use " + db)
    sqlContext.sql("CREATE TABLE IF NOT EXISTS " + db + "." + table + "(key INT, value STRING)")
    sqlContext.sql("LOAD DATA  INPATH 'hdfs:///user/spark/examples/src/main/resources/kv1.txt' INTO TABLE " + db + "." + table)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").count()
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 20").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 5").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 15").collect().foreach(println)
  }

  def main(args: Array[String]) {
    if (args.length < 3) {
      print("useage:HiveUtility.jar db table system_user_name")
    }
    System.setProperty("user.name", args(2))
    System.setProperty("HADOOP_USER_NAME", args(2))
    val conf = new SparkConf()
   // conf.set("spark.scheduler.allocation.file", "hdfs:///user/spark/fairscheduler.xml")
    conf.set("spark.scheduler.mode", "FAIR") // set scheduler mode
    val sc = new SparkContext(conf)
    sc.setLocalProperty("spark.scheduler.pool", "spark") // set fair scheduler
    createTable(sc, args(0), args(1), args(2))
  }
}

提交命令:
mainclass=com.suning.spark.hive.HiveUtility
mainjar=HiveUtility-v1.jar
hadoop fs -rm  /home/spark/software/spark/examples/src/main/resources/kv1.txt
hadoop fs -put /home/spark/software/spark/examples/src/main/resources/kv1.txt examples/src/main/resources
export HADOOP_USER_NAME=bi
djars=/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar,/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar,/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar,/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar
spark-submit --master yarn-cluster \
             --queue spark       \
             --files /home/spark/software/spark/conf/hive-site.xml,/home/spark/software/spark/conf/fairscheduler.xml \
             --jars $djars            \
             --class $mainclass $mainjar  bi_dm  hive3   bi

3.结果验证





日志:
5/11/19 15:16:51 INFO Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
15/11/19 15:16:51 INFO Client: Setting up container launch context for our AM
15/11/19 15:16:51 INFO Client: Setting up the launch environment for our AM container
15/11/19 15:16:51 INFO Client: Preparing resources for our AM container
15/11/19 15:16:51 INFO Client: Uploading resource file:/home/spark/software/spark-1.5.2-bin-2.4.0/lib/spark-assembly-1.5.2-hadoop2.4.0.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/spark-assembly-1.5.2-hadoop2.4.0.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/workspace/tersort/HiveUtility-v1.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/HiveUtility-v1.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-api-jdo-3.2.6.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-core-3.2.10.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-rdbms-3.2.9.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/mysql-connector-java-5.1.27-bin.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/hive-site.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/hive-site.xml
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/fairscheduler.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/fairscheduler.xml
15/11/19 15:16:54 INFO Client: Uploading resource file:/data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0/__spark_conf__6425041649655761245.zip -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/__spark_conf__6425041649655761245.zip
15/11/19 15:16:54 INFO SecurityManager: Changing view acls to: spark,bi
15/11/19 15:16:54 INFO SecurityManager: Changing modify acls to: spark,bi
15/11/19 15:16:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark, bi); users with modify permissions: Set(spark, bi)
15/11/19 15:16:54 INFO Client: Submitting application 6897 to ResourceManager
15/11/19 15:16:54 INFO YarnClientImpl: Submitted application application_1447783148941_6897
15/11/19 15:16:55 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:55 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.spark
	 start time: 1447917414919
	 final status: UNDEFINED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:16:56 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:57 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:58 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:59 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:17:00 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:00 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 10.27.1.143
	 ApplicationMaster RPC port: 0
	 queue: root.spark
	 start time: 1447917414919
	 final status: UNDEFINED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:17:01 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:02 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:03 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:04 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:05 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:06 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:07 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:09 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:10 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:11 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:12 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:13 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:14 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:15 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:16 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:17 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:18 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:19 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:20 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:21 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:22 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:23 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:24 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:25 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:26 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:27 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:28 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:29 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:30 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:31 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:32 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:33 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:34 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:35 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:36 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:37 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:38 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:39 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:40 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:41 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:42 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:43 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:44 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:45 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:46 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:47 INFO Client: Application report for application_1447783148941_6897 (state: FINISHED)
15/11/19 15:17:47 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 10.27.1.143
	 ApplicationMaster RPC port: 0
	 queue: root.spark
	 start time: 1447917414919
	 final status: SUCCEEDED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:17:47 INFO ShutdownHookManager: Shutdown hook called
15/11/19 15:17:47 INFO ShutdownHookManager: Deleting directory /data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0




参考:http://spark.apache.org/docs/latest/job-scheduling.html


版权声明:本文为博主原创文章,未经博主允许不得转载。

相关文章推荐

Spark调度模式-FIFO和FAIR

Spark中的调度模式主要有两种:FIFO和FAIR。使用哪种调度器由参数spark.scheduler.mode来设置,默认为FIFO...

spark的FairSchedule

spark 内置TaskScheduler级别的调度算法,分别是队列与公平,默认是队列方式。具体源码可参考SchedulableBuilder 准确来讲spark有两层公平,一层是公平调度,一层是公平...

Hadoop Yarn多用户资源管理–Fair Scheduler介绍与配置

Hadoop Yarn多用户资源管理–Fair Scheduler介绍与配置 时间 2015-10-26 17:33:54  极客头条 原文  http://lxw1234.com/archi...

Hadoop YARN配置参数—Fair Scheduler相关参数

首先在yarn-site.xml中,将配置参数yarn.resourcemanager.scheduler.class设置为org.apache.hadoop.yarn.server.resource...
  • LW_GHY
  • LW_GHY
  • 2016年06月22日 23:34
  • 545

Hadoop Yarn多用户资源管理–Fair Scheduler介绍与配置

Hadoop Yarn多用户资源管理–Fair Scheduler介绍与配置 时间 2015-10-26 17:33:54  极客头条 原文  http://lxw1234.com/archives...

Spark1.5.2 on Hadoop2.4.0 安装配置

一、           前言  本文简单介绍yarn安装,主要介绍spark1.5.2on yarn模式安装,仅供参考。   二、           yarn配置 1.     yarne.xml...

Clouder Manager: Yarn Fair scheduler config

直接上配置: drf drf 0 mb,0 vcores 308020 mb,124 vcores 100 300 1...
  • mtj66
  • mtj66
  • 2017年07月05日 17:45
  • 110

yarn fair scheduler 之公平排序算法

这篇文章主要分析公平调度器的公平排序算法,基于hadoop-2.3.0-cdh5.0.0 首先,了解这几个概念: 资源需求量:当前队列或者应用希望获得的资源...
  • wujun8
  • wujun8
  • 2014年06月19日 10:50
  • 2959

Hadoop权威指南读书笔记(2) — Yarn简介及Capacity & Fair Scheduler

Yarn简介Yarn(Yet Another Resource Negotiator)是hadoop2中引入的一个集群资源管理系统。...
  • labud
  • labud
  • 2016年07月27日 19:18
  • 927

Yarn参数优化(Fair Scheduler版本)

YARN自从hadoop2.0之后, 我们可以使用apache yarn 来对集群资源进行管理。yarn把可以把资源(内存,CPU)以Container的方式进行划分隔离。YARN会管理集群中所有机器...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:Spark1.5.2 on yarn fair scheduler 配置
举报原因:
原因补充:

(最多只允许输入30个字)