hadoop运行随记4

今天生产环境中，有一个节点的tasktracker进程退出了，从日志发现，退出前有以下频繁报错信息：

2012-12-25 05:36:51,279 WARN org.apache.hadoop.mapred.TaskTracker: Exception while localization ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
        at org.apache.hadoop.mapred.JobLocalizer.createJobDirs(JobLocalizer.java:222)
        at org.apache.hadoop .mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:184)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
        at java.lang.Thread.run(Thread.java:662)

2012-12-25 05:36:51,279 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ha doop cause:ENOENT: No such file or directory

 

导致系统退出的日志：

 2012-12-25 05:37:48,215 FATAL org.apache.hadoop.mapred.JvmManager: Inconsistent state!!! JVM Manager reached an unstable state while reaping a JVM for task: attempt_201212241613_0203_m_000019_0 Number of active JVMs:5
  JVMId jvm_201212241613_0207_m_1186750853 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0207_m_000002_0
  JVMId jvm_201212241613_0202_m_854191839 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0202_m_000010_1
  JVMId jvm_201212241613_0203_m_-1853247367 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0203_m_000018_0
  JVMId jvm_201212241613_0210_m_804157000 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0210_m_000004_0
  JVMId jvm_201212241613_0202_m_1004523329 #Tasks ran: 0 Currently busy? true Currently running: attempt_2012122416 13_0202_m_000009_0. Aborting.
2012-12-25 05:37:48,219 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at slave3/192.168.2.23
************************************************************/

 

目前采用方式是调整child的内存使用大小，已运行两天没出问题了，将继续关注……