spark on yarn---{{PWD}}<CPS>{{PWD}}

采用yarn-client提交job的时候,发现在application的日志中出现以下内容,一直以为  {{PWD}}<CPS>{{PWD}}出现错误,但是后来通过google发现这是正常的,在派发container的时候,
会调用hadoop-yarn-nodemanager-jar包将{{,CPS和}}进行更改。
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/318/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.4.0-91/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/05/14 01:09:37 INFO util.SignalUtils: Registered signal handler for TERM
18/05/14 01:09:37 INFO util.SignalUtils: Registered signal handler for HUP
18/05/14 01:09:37 INFO util.SignalUtils: Registered signal handler for INT
18/05/14 01:09:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/05/14 01:09:38 INFO yarn.ApplicationMaster: Preparing Local resources
18/05/14 01:09:40 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
18/05/14 01:09:40 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1526284770995_0001_000001
18/05/14 01:09:40 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
18/05/14 01:09:40 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
18/05/14 01:09:40 INFO spark.SecurityManager: Changing view acls groups to:
18/05/14 01:09:40 INFO spark.SecurityManager: Changing modify acls groups to:
18/05/14 01:09:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hdfs); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hdfs); groups with modify permissions: Set()
18/05/14 01:09:41 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
18/05/14 01:09:41 INFO yarn.ApplicationMaster: Driver now available: 192.168.9.18:61776
18/05/14 01:09:41 INFO client.TransportClientFactory: Successfully created connection to /192.168.9.18:61776 after 99 ms (0 ms spent in bootstraps)
18/05/14 01:09:41 INFO yarn.ApplicationMaster:
===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>/usr/hdp/2.6.4.0-91/hadoop/conf<CPS>/usr/hdp/2.6.4.0-91/hadoop/*<CPS>/usr/hdp/2.6.4.0-91/hadoop/lib/*<CPS>/usr/hdp/current/hadoop-hdfs-client/*<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/*<CPS>/usr/hdp/current/hadoop-yarn-client/*<CPS>/usr/hdp/current/hadoop-yarn-client/lib/*<CPS>/usr/hdp/current/ext/hadoop/*<CPS>/usr/hdp/2.6.4.0-91/hbase/lib/*<CPS>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.4.0-91/hadoop/lib/hadoop-lzo-0.6.0.2.6.4.0-91.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*
    SPARK_YARN_STAGING_DIR -> *********(redacted)
    SPARK_USER -> *********(redacted)
    SPARK_YARN_MODE -> true

  command:
    {{JAVA_HOME}}/bin/java \
      -server \
      -Xmx1024m \
      '-Dhdp.version=2.6.4.0-91' \
      -Djava.io.tmpdir={{PWD}}/tmp \
      '-Dspark.driver.port=61776' \
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
      -XX:OnOutOfMemoryError='kill %p' \
      org.apache.spark.executor.CoarseGrainedExecutorBackend \
      --driver-url \
      spark://CoarseGrainedScheduler@192.168.9.18:61776 \
      --executor-id \
      <executorId> \
      --hostname \
      <hostname> \
      --cores \
      1 \
      --app-id \
      application_1526284770995_0001 \
      --user-class-path \
      file:$PWD/__app__.jar \
      1><LOG_DIR>/stdout \
      2><LOG_DIR>/stderr
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值