Hadoop YARN: 1/1 local-dirs are bad: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir; 1/1 log-dirs are

Hadoop YARN: 1/1 local-dirs are bad: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers hdfs硬盘90% yarn unhealthy

1 /1  local - dirs  are bad: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir ; 1 /1  log- dirs  are bad: /var/log/hadoop-yarn/containers

Node Manager logs

1
2
3
yarn.server.nodemanager.DirectoryCollection: Directory /var/lib/hadoop-yarn/cache/yarn/nm-local-dir  error, used space above threshold of 90.0%, removing from list of valid directories
2014-11-17 17:45:00,713 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /var/log/hadoop-yarn/containers  error, used space above threshold of 90.0%, removing from list of valid directories
2014-11-17 17:45:00,713 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed: 1 /1  local - dirs  are bad: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir ; 1 /1  log- dirs  are bad: /var/log/hadoop

 

Resource Manager logs

1
2014-11-17 16:57:07,301 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Node localhost:34650 reported UNHEALTHY with details: 1 /1  local - dirs  are bad: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir ; 1 /1  log- dirs are bad: /var/log/hadoop-yarn/containers
 

遇到上面这样的问题,我的解决办法是:

 

在yarn-site.xml 里面配置

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage
95.0

 

重启yarn即可解决!

 

yarn-daemon.sh stop nodemanager

yarn-daemon.sh start nodemanager


Hadoop YARN最近版本中增加的几个非常有用的特性,包括:

(1)ResourceManager HA

在apache hadoop 2.4或者CDH5.0.0版本之后,增加了ResourceManger HA特性,支持基于Zookeeper的热主备切换,具体配置参数可以参考Cloudera的文档:ResourceManager HA配置

需要注意的是,ResourceManager HA只完成了第一个阶段的设计,即备ResourceManager启动后,会杀死之前正在运行的Application,然后从共享存储系统中读取这些Application的元数据信息,并重新提交这些Application。启动ApplicationMaster后,剩下的容错功能就交给ApplicationMaster实现了,比如MapReduce的ApplicationMaster会不断地将完成的任务信息写到HDFS上,这样,当它重启时,可以重新读取这些日志,进而只需重新运行那些未完成的任务。ResourceManager HA第二个阶段的任务是,备ResourceManager接管主ResourceManager后,无需杀死那些正在运行的Application,让他们像任何事情没有发生一样运行下去。

(2) 磁盘容错

在apache hadoop 2.4或者CDH5.0.0版本之后,增加了几个对多磁盘非常友好地参数,这些参数允许YARN更好地使用NodeManager上的多块磁盘,相关jira为:YARN-1781,主要新增了三个参数:

yarn.nodemanager.disk-health-checker.min-healthy-disks:NodeManager上最少保证健康磁盘比例,当健康磁盘比例低于该值时,NodeManager不会再接收和启动新的Container,默认值是0.25,表示25%;

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage:一块磁盘的最高使用率,当一块磁盘的使用率超过该值时,则认为该盘为坏盘,不再使用该盘,默认是100,表示100%,可以适当调低;

yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb:一块磁盘最少保证剩余空间大小,当某块磁盘剩余空间低于该值时,将不再使用该盘,默认是0,表示0MB。

(3)资源调度器

Fair Scheduler:Fair Scheduler增加了一个非常有用的新特性,允许用户在线将一个应用程序从一个队列转移到另外一个队列,比如将一个重要作业从一个低优先级队列转移到高优先级队列,操作命令是:bin/yarn application -movetoqueue appID -queue targetQueueName,相关jira为:YARN-1721

Capacity Scheduler:Capacity Scheduler中资源抢占功能经过了充分的测试,可以使用了。


  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException java.lang.NullPointerException at org.json.JSONObject.<init>(JSONObject.java:144) at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43) at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:867) at org.apache.sqoop.mapreduce.JobBase.putSqoopOptionsToConfiguration(JobBase.java:393) at org.apache.sqoop.mapreduce.JobBase.createJob(JobBase.java:379) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:255) at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:747) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:536) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:633) at org.apache.sqoop.Sqoop.run(Sqoop.java:146) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:182) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:233) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:242) at org.apache.sqoop.Sqoop.main(Sqoop.java:251) Log Type: stdout Log Upload Time: Mon Jul 24 10:47:38 +0800 2023 Log Length: 74530 Showing 4096 bytes of 74530 total. Click here for the full log. 35517561_3806_01_000001: PRELAUNCH_OUT=/yarn/container-logs/application_1683335517561_3806/container_1683335517561_3806_01_000001/prelaunch.out: NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=: NM_PORT=8041: HADOOP_YARN_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn: USER=admin: CLASSPATH=/yarn/nm/usercache/admin/appcache/application_1683335517561_3806/container_1683335517561_3806_01_000001:/yarn/nm/usercache/admin/appcache/application_1683335517561_3806/container_1683335517561_3806_01_000001/*:/etc/hadoop/conf.cloudera.yarn:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn/lib/*:: PRELAUNCH_ERR=/yarn/container-logs/application_1683335517561_3806/container_1683335517561_3806_01_000001/prelaunch.err: HADOOP_TOKEN_FILE_LOCATION=/yarn/nm/usercache/admin/appcache/application_1683335517561_3806/container_1683335517561_3806_01_000001/container_tokens: LOCAL_USER_DIRS=/yarn/nm/usercache/admin/: OOZIE_ACTION_CONF_XML=/yarn/nm/usercache/admin/appcache/application_1683335517561_3806/container_1683335517561_3806_01_000001/action.xml: SHLVL=2: HOME=/home/: CONTAINER_ID=container_1683335517561_3806_01_000001: MALLOC_ARENA_MAX=4:怎么回事
07-25
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值