今天生产环境中,有一个节点的tasktracker进程退出了,从日志发现,退出前有以下频繁报错信息:
2012-12-25 05:36:51,279 WARN org.apache.hadoop.mapred.TaskTracker: Exception while localization ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at org.apache.hadoop.mapred.JobLocalizer.createJobDirs(JobLocalizer.java:222)
at org.apache.hadoop .mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:184)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
at java.lang.Thread.run(Thread.java:662)
2012-12-25 05:36:51,279 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ha doop cause:ENOENT: No such file or directory
导致系统退出的日志:
2012-12-25 05:37:48,215 FATAL org.apache.hadoop.mapred.JvmManager: Inconsistent state!!! JVM Manager reached an unstable state while reaping a JVM for task: attempt_201212241613_0203_m_000019_0 Number of active JVMs:5
JVMId jvm_201212241613_0207_m_1186750853 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0207_m_000002_0
JVMId jvm_201212241613_0202_m_854191839 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0202_m_000010_1
JVMId jvm_201212241613_0203_m_-1853247367 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0203_m_000018_0
JVMId jvm_201212241613_0210_m_804157000 #Tasks ran: 0 Currently busy? true Currently running: attempt_201212241613_0210_m_000004_0
JVMId jvm_201212241613_0202_m_1004523329 #Tasks ran: 0 Currently busy? true Currently running: attempt_2012122416 13_0202_m_000009_0. Aborting.
2012-12-25 05:37:48,219 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at slave3/192.168.2.23
************************************************************/
目前采用方式是调整child的内存使用大小,已运行两天没出问题了,将继续关注……