关闭

Spark1.5.2 on yarn fair scheduler 配置

709人阅读 评论(0) 收藏 举报
分类:

1.fair scheduler 配置

  
<?xml version="1.0"?>
<allocations>
  <pool name="production">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>2</minShare>
  </pool>
  <pool name="test">
    <schedulingMode>FIFO</schedulingMode>
    <weight>2</weight>
    <minShare>3</minShare>
  </pool>
  <pool name="spark">
    <schedulingMode>FAIR</schedulingMode>
    <weight>2</weight>
    <minShare>3</minShare>
  </pool>
  <pool name="ztai">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>3</minShare>
  </pool>
</allocations>

2.代码提交

mport org.apache.spark.SparkConf
import org.apache.spark.SparkContext

/**
 * Created by root on 11/18/15.
 */

object HiveUtility {
  def createTable(sc: SparkContext, db: String, table: String, system_user_name: String) = {
    val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
    println("user.name " + System.getProperty("user.name"))
    println("HADOOP_USER_NAME " + System.getProperty("HADOOP_USER_NAME"))
    sqlContext.sql("use " + db)
    sqlContext.sql("CREATE TABLE IF NOT EXISTS " + db + "." + table + "(key INT, value STRING)")
    sqlContext.sql("LOAD DATA  INPATH 'hdfs:///user/spark/examples/src/main/resources/kv1.txt' INTO TABLE " + db + "." + table)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").count()
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 20").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 5").collect().foreach(println)
    sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 15").collect().foreach(println)
  }

  def main(args: Array[String]) {
    if (args.length < 3) {
      print("useage:HiveUtility.jar db table system_user_name")
    }
    System.setProperty("user.name", args(2))
    System.setProperty("HADOOP_USER_NAME", args(2))
    val conf = new SparkConf()
   // conf.set("spark.scheduler.allocation.file", "hdfs:///user/spark/fairscheduler.xml")
    conf.set("spark.scheduler.mode", "FAIR") // set scheduler mode
    val sc = new SparkContext(conf)
    sc.setLocalProperty("spark.scheduler.pool", "spark") // set fair scheduler
    createTable(sc, args(0), args(1), args(2))
  }
}

提交命令:
mainclass=com.suning.spark.hive.HiveUtility
mainjar=HiveUtility-v1.jar
hadoop fs -rm  /home/spark/software/spark/examples/src/main/resources/kv1.txt
hadoop fs -put /home/spark/software/spark/examples/src/main/resources/kv1.txt examples/src/main/resources
export HADOOP_USER_NAME=bi
djars=/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar,/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar,/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar,/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar
spark-submit --master yarn-cluster \
             --queue spark       \
             --files /home/spark/software/spark/conf/hive-site.xml,/home/spark/software/spark/conf/fairscheduler.xml \
             --jars $djars            \
             --class $mainclass $mainjar  bi_dm  hive3   bi

3.结果验证





日志:
5/11/19 15:16:51 INFO Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
15/11/19 15:16:51 INFO Client: Setting up container launch context for our AM
15/11/19 15:16:51 INFO Client: Setting up the launch environment for our AM container
15/11/19 15:16:51 INFO Client: Preparing resources for our AM container
15/11/19 15:16:51 INFO Client: Uploading resource file:/home/spark/software/spark-1.5.2-bin-2.4.0/lib/spark-assembly-1.5.2-hadoop2.4.0.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/spark-assembly-1.5.2-hadoop2.4.0.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/workspace/tersort/HiveUtility-v1.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/HiveUtility-v1.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-api-jdo-3.2.6.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-core-3.2.10.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-rdbms-3.2.9.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/mysql-connector-java-5.1.27-bin.jar
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/hive-site.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/hive-site.xml
15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/fairscheduler.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/fairscheduler.xml
15/11/19 15:16:54 INFO Client: Uploading resource file:/data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0/__spark_conf__6425041649655761245.zip -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/__spark_conf__6425041649655761245.zip
15/11/19 15:16:54 INFO SecurityManager: Changing view acls to: spark,bi
15/11/19 15:16:54 INFO SecurityManager: Changing modify acls to: spark,bi
15/11/19 15:16:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark, bi); users with modify permissions: Set(spark, bi)
15/11/19 15:16:54 INFO Client: Submitting application 6897 to ResourceManager
15/11/19 15:16:54 INFO YarnClientImpl: Submitted application application_1447783148941_6897
15/11/19 15:16:55 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:55 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.spark
	 start time: 1447917414919
	 final status: UNDEFINED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:16:56 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:57 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:58 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:16:59 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED)
15/11/19 15:17:00 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:00 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 10.27.1.143
	 ApplicationMaster RPC port: 0
	 queue: root.spark
	 start time: 1447917414919
	 final status: UNDEFINED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:17:01 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:02 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:03 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:04 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:05 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:06 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:07 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:09 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:10 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:11 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:12 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:13 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:14 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:15 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:16 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:17 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:18 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:19 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:20 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:21 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:22 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:23 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:24 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:25 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:26 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:27 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:28 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:29 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:30 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:31 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:32 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:33 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:34 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:35 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:36 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:37 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:38 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:39 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:40 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:41 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:42 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:43 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:44 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:45 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:46 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING)
15/11/19 15:17:47 INFO Client: Application report for application_1447783148941_6897 (state: FINISHED)
15/11/19 15:17:47 INFO Client: 
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: 10.27.1.143
	 ApplicationMaster RPC port: 0
	 queue: root.spark
	 start time: 1447917414919
	 final status: SUCCEEDED
	 tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/
	 user: bi
15/11/19 15:17:47 INFO ShutdownHookManager: Shutdown hook called
15/11/19 15:17:47 INFO ShutdownHookManager: Deleting directory /data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0




参考:http://spark.apache.org/docs/latest/job-scheduling.html


1
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:58793次
    • 积分:1116
    • 等级:
    • 排名:千里之外
    • 原创:49篇
    • 转载:1篇
    • 译文:6篇
    • 评论:7条
    最新评论