spark 相关

最新推荐文章于 2023-03-29 11:29:15 发布

飞鱼丶灬

最新推荐文章于 2023-03-29 11:29:15 发布

阅读量238

点赞数

分类专栏：大数据文章标签： spark

本文链接：https://blog.csdn.net/m0_37859032/article/details/125102991

版权

大数据专栏收录该内容

20 篇文章 0 订阅

订阅专栏

spark-submit  --master yarn --deploy-mode  cluster  --name  bbbbbbbbb  --queue  root.xy_yarn_pool.development   --verbose   --class  com.xinyan.bigdata.buddha.TestContext  /home/xy_hpt/onlytest0731.jar

Adding default property: spark.lineage.log.dir=/var/log/spark/lineage
Adding default property: spark.serializer=org.apache.spark.serializer.KryoSerializer
Adding default property: spark.hadoop.mapred.output.compress=true
Adding default property: spark.yarn.jars=local:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/jars/*,local:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/hive/*
Adding default property: spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8
Adding default property: spark.eventLog.enabled=true
Adding default property: spark.shuffle.service.enabled=true
Adding default property: spark.extraListeners=com.cloudera.spark.lineage.NavigatorAppListener
Adding default property: spark.yarn.appMasterEnv.MKL_NUM_THREADS=1
Adding default property: spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native
Adding default property: spark.lineage.enabled=true
Adding default property: spark.yarn.historyServer.address=http://bigdata-3.baofoo.cn:18088
Adding default property: spark.yarn.historyServer.allowTracking=true
Adding default property: spark.ui.enabled=true
Adding default property: spark.ui.killEnabled=true
Adding default property: spark.dynamicAllocation.schedulerBacklogTimeout=1
Adding default property: spark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native
Adding default property: spark.yarn.config.gatewayPath=/opt/cloudera/parcels
Adding default property: spark.sql.queryExecutionListeners=com.cloudera.spark.lineage.NavigatorQueryListener
Adding default property: spark.dynamicAllocation.maxExecutors=6
Adding default property: spark.yarn.config.replacementPath={{HADOOP_COMMON_HOME}}/../../..
Adding default property: spark.executorEnv.MKL_NUM_THREADS=1
Adding default property: spark.driver.extraJavaOptions=-Dfile.encoding=UTF-8
Adding default property: spark.executorEnv.OPENBLAS_NUM_THREADS=1
Adding default property: spark.submit.deployMode=client
Adding default property: spark.shuffle.service.port=7337
Adding default property: spark.master=yarn
Adding default property: spark.authenticate=false
Adding default property: spark.network.crypto.enabled=false
Adding default property: spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native
Adding default property: spark.io.encryption.enabled=false
Adding default property: spark.eventLog.dir=*********(redacted)
Adding default property: spark.dynamicAllocation.enabled=true
Adding default property: spark.dynamicAllocation.minExecutors=1
Adding default property: spark.dynamicAllocation.initialExecutors=1
Adding default property: spark.yarn.appMasterEnv.OPENBLAS_NUM_THREADS=1
Adding default property: spark.dynamicAllocation.executorIdleTimeout=60
Adding default property: spark.hadoop.mapred.output.compression.codec=snappy
Parsed arguments:
  master                  yarn
  deployMode              cluster
  executorMemory          null
  executorCores           null
  totalExecutorCores      null
  propertiesFile          /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/conf/spark-defaults.conf
  driverMemory            null
  driverCores             null
  driverExtraClassPath    null
  driverExtraLibraryPath  /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native
  driverExtraJavaOptions  -Dfile.encoding=UTF-8
  supervise               false
  queue                   root.xy_yarn_pool.development
  numExecutors            null
  files                   null
  pyFiles                 null
  archives                null
  mainClass               com.xinyan.bigdata.buddha.TestContext
  primaryResource         file:/home/xy_hpt/onlytest0731.jar
  name                    bbbbbbbbb
  childArgs               []
  jars                    null
  packages                null
  packagesExclusions      null
  repositories            null
  verbose                 true

Spark properties used, including those specified through
 --conf and those from the properties file /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/conf/spark-defaults.conf:
  (spark.sql.queryExecutionListeners,com.cloudera.spark.lineage.NavigatorQueryListener)
  (spark.executor.extraLibraryPath,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native)
  (spark.hadoop.mapred.output.compression.codec,snappy)
  (spark.executorEnv.MKL_NUM_THREADS,1)
  (spark.lineage.enabled,true)
  (spark.authenticate,false)
  (spark.yarn.jars,local:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/jars/*,local:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/hive/*)
  (spark.driver.extraLibraryPath,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native)
  (spark.yarn.am.extraLibraryPath,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native)
  (spark.yarn.historyServer.address,http://bigdata-3.baofoo.cn:18088)
  (spark.eventLog.enabled,true)
  (spark.executorEnv.OPENBLAS_NUM_THREADS,1)
  (spark.yarn.historyServer.allowTracking,true)
  (spark.dynamicAllocation.schedulerBacklogTimeout,1)
  (spark.yarn.appMasterEnv.OPENBLAS_NUM_THREADS,1)
  (spark.yarn.config.gatewayPath,/opt/cloudera/parcels)
  (spark.ui.killEnabled,true)
  (spark.serializer,org.apache.spark.serializer.KryoSerializer)
  (spark.dynamicAllocation.maxExecutors,6)
  (spark.executor.extraJavaOptions,-Dfile.encoding=UTF-8)
  (spark.dynamicAllocation.executorIdleTimeout,60)
  (spark.dynamicAllocation.initialExecutors,1)
  (spark.dynamicAllocation.minExecutors,1)
  (spark.shuffle.service.enabled,true)
  (spark.yarn.config.replacementPath,{{HADOOP_COMMON_HOME}}/../../..)
  (spark.ui.enabled,true)
  (spark.io.encryption.enabled,false)
  (spark.driver.extraJavaOptions,-Dfile.encoding=UTF-8)
  (spark.submit.deployMode,client)
  (spark.shuffle.service.port,7337)
  (spark.network.crypto.enabled,false)
  (spark.eventLog.dir,*********(redacted))
  (spark.hadoop.mapred.output.compress,true)
  (spark.extraListeners,com.cloudera.spark.lineage.NavigatorAppListener)
  (spark.master,yarn)
  (spark.lineage.log.dir,/var/log/spark/lineage)
  (spark.yarn.appMasterEnv.MKL_NUM_THREADS,1)
  (spark.dynamicAllocation.enabled,true)

    
Main class:
org.apache.spark.deploy.yarn.Client
Arguments:
--jar
file:/home/xy_hpt/onlytest0731.jar
--class
com.xinyan.bigdata.buddha.TestContext
System properties:
(spark.yarn.queue,root.xy_yarn_pool.development)
(spark.sql.queryExecutionListeners,com.cloudera.spark.lineage.NavigatorQueryListener)
(spark.hadoop.mapred.output.compression.codec,snappy)
(spark.executor.extraLibraryPath,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native)
(spark.lineage.enabled,true)
(spark.executorEnv.MKL_NUM_THREADS,1)
(spark.authenticate,false)
(spark.driver.extraLibraryPath,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native)
(spark.yarn.jars,local:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/jars/*,local:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/spark/hive/*)
(spark.yarn.am.extraLibraryPath,/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native)
(spark.yarn.historyServer.address,http://bigdata-3.baofoo.cn:18088)
(spark.eventLog.enabled,true)
(spark.executorEnv.OPENBLAS_NUM_THREADS,1)
(spark.yarn.historyServer.allowTracking,true)
(spark.dynamicAllocation.schedulerBacklogTimeout,1)
(SPARK_SUBMIT,true)
(spark.yarn.appMasterEnv.OPENBLAS_NUM_THREADS,1)
(spark.yarn.config.gatewayPath,/opt/cloudera/parcels)
(spark.ui.killEnabled,true)
(spark.serializer,org.apache.spark.serializer.KryoSerializer)
(spark.dynamicAllocation.maxExecutors,6)
(spark.executor.extraJavaOptions,-Dfile.encoding=UTF-8)
(spark.app.name,bbbbbbbbb)
(spark.shuffle.service.enabled,true)
(spark.dynamicAllocation.minExecutors,1)
(spark.dynamicAllocation.initialExecutors,1)
(spark.dynamicAllocation.executorIdleTimeout,60)
(spark.yarn.config.replacementPath,{{HADOOP_COMMON_HOME}}/../../..)
(spark.ui.enabled,true)
(spark.io.encryption.enabled,false)
(spark.driver.extraJavaOptions,-Dfile.encoding=UTF-8)
(spark.submit.deployMode,cluster)
(spark.shuffle.service.port,7337)
(spark.network.crypto.enabled,false)
(spark.eventLog.dir,*********(redacted))
(spark.hadoop.mapred.output.compress,true)
(spark.extraListeners,com.cloudera.spark.lineage.NavigatorAppListener)
(spark.master,yarn)
(spark.dynamicAllocation.enabled,true)
(spark.yarn.appMasterEnv.MKL_NUM_THREADS,1)
(spark.lineage.log.dir,/var/log/spark/lineage)
Classpath elements:
file:/home/xy_hpt/onlytest0731.jar