从RM上看到有unhealy的NM,查了下NM的日志发现:
2018-04-12 20:05:56,277 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /webser/hadoop-hadoop/tmp/nm-local-dir error, used space above threshold of 90.0%, removing from list of valid directories
2018-04-12 20:05:56,277 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /home/hadoop/hadoop2/logs/userlogs error, used space above threshold of 90.0%, removing from list of valid directories
2018-04-12 20:05:56,277 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed: 1/1 local-dirs are bad: /webser/hadoop-hadoop/tmp/nm-local-dir; 1/1 log-dirs are bad: /home/hadoop/hadoop2/logs/userlogs
2018-04-12 20:05:56,277 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs are bad: /webser/hadoop-hadoop/tmp/nm-local-dir; 1/1 log-dirs are bad: /home/hadoop/hadoop2/logs/userlogs
说明yarn的本地目录已经到90%了,不再向里写数据了
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 99G 84G 9.5G 90% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
清理服务器上的垃圾即可。