spark on yarn cluster查看日志

spark on yarn cluster查看日志

[hadoop@hadoop001 shell]$ yarn logs -applicationId application_1420997455428_0005
15/01/12 04:34:51 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
/tmp/logs/hadoop/logs/application_1420997455428_0005does not exist.
Log aggregation has not completed or is not enabled.
原因是聚合日志没有开启

解决如下：

[hadoop@hadoop001 hadoop]$ vi yarn-site.xml
  <!--开启聚合日志--> 
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
      <!--设置聚合日志放置的位置--> ///tmp/logs是hdfs分布式系统上面的，自己创建自定义的文件夹
    <property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/tmp/logs</value>
    </property>
    <!--设置聚合日志存放的时间，单位为秒，会自动移除-->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
  <value>3600</value>
    </property>

配置完yarn-site.xml之后重启节点，再次查看日志

[hadoop@hadoop001 shell]$ yarn logs -applicationId application_1420997455428_0005
15/01/12 09:53:43 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Container: container_1421027534751_0001_01_000003 on hadoop001_40205
======================================================================
LogType:stderr
Log Upload Time:Mon Jan 12 09:53:04 +0800 2015
LogLength:7912
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/filecache/10/__spark_libs__7968601071120350016.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/01/12 09:52:58 INFO executor.CoarseGrainedExecutorBackend: Started daemon with process name: 31008@hadoop001
15/01/12 09:52:58 INFO util.SignalUtils: Registered signal handler for TERM
15/01/12 09:52:58 INFO util.SignalUtils: Registered signal handler for HUP
15/01/12 09:52:58 INFO util.SignalUtils: Registered signal handler for INT
15/01/12 09:52:59 INFO spark.SecurityManager: Changing view acls to: hadoop
15/01/12 09:52:59 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/01/12 09:52:59 INFO spark.SecurityManager: Changing view acls groups to: 
15/01/12 09:52:59 INFO spark.SecurityManager: Changing modify acls groups to: 

注：1.查看某个应用的日志

yarn logs -applicationId application_1420997455428_0005

2.查看某个应用的状态

yarn application -status application_1420997455428_0005

3.杀掉某个应用
（直接在UI界面或者是终端kill掉任务都是不对的，该任务可能还会继续执行下去，所以要用如下命令才算完全停止该应用的执行）

yarn application -kill application_1420997455428_0005