yarn LocalResourcesTrackerImpl remove resource Error

1. NM异常日志 (/data0/hadoop/log/yarn/yarn-mapred-nodemanager-yz5202.hadoop.data.sina.com.cn.log)

 

ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/tez-mapreduce-0.5.0-incubating-SNAPSHOT.jar, 1470452069832, FILE, null },pending,[],1163127461340,DOWNLOADING} with non-zero refcount 2016 - 10 - 25  00 : 07 : 59 , 843  ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/snappy-java-1.0.4.1.jar, 1470452070141, FILE, null },pending,[],1163127475524,DOWNLOADING} with non-zero refcount 2016 - 10 - 25  00 : 07 : 59 , 843  ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/hadoop-mapreduce-client-core-2.4.0.jar, 1470452070471, FILE, null },pending,[],1163127488563,DOWNLOADING} with non-zero refcount 2016 - 10 - 25  00 : 07 : 59 , 843  ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/hadoop-mapreduce-client-common-2.4.0.jar, 1470452071821, FILE, null },pending,[],1163127500891,DOWNLOADING} with non-zero refcount 2016 - 10 - 25  00 : 07 : 59 , 843  ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/guice-3.0.jar, 1470452070637, FILE, null },pending,[],1163127516204,DOWNLOADING} with non-zero refcount 2016 - 10 - 25  00 : 07 : 59 , 843  ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/tez-dag-0.5.0-incubating-SNAPSHOT.jar, 1470452072079, FILE, null },pending,[],1163127534824,DOWNLOADING} with non-zero refcount 2016 - 10 - 25  00 : 07 : 59 , 843  ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/tez-runtime-internals-0.5.0-incubating-SNAPSHOT.jar, 1470452069673, FILE, null },pending,[],1163127555160,DOWNLOADING} with non-zero refcount 2016 - 10 - 25  00 : 07 : 59 , 843  ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs: //ns1/user/huali1/.staging/job_1470311300058_64103/libjars/jsr305-2.0.3.jar, 1470452070933, FILE, null },pending,[],1163127573133,DOWNLOADING} with non-zero refcount

2.异常问题分析

If a container is in the process of localizing when it is stopped/killed then resources are left in the DOWNLOADING state. If no other container comes along and requests these resources they linger around with no reference counts but aren't cleaned up during normal cache cleanup scans since it will never delete resources in the DOWNLOADING state even if their reference count is zero

container 在localizing 过程中如果被kill或者stopped，使得资源处于DOWNLOADING 状态，并且没有其他的container使用这些资源即rsrc.getRefCount() <= 0；

在进行资源清理的时候，处于DOWNLOADING 状态的资源不会被清理，即使RefCount=0；

在上述状态下会出现异常：ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource

异常代码位置

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

public  boolean  remove(LocalizedResource rem, DeletionService delService) { // current synchronization guaranteed by crude RLS event for cleanup     LocalizedResource rsrc = localrsrc.get(rem.getRequest());     if  ( null  == rsrc) {       LOG.error( "Attempt to remove absent resource: "  + rem.getRequest()           +  " from "  + getUser());       return  true ;     }     if  (rsrc.getRefCount() >  0         || ResourceState.DOWNLOADING.equals(rsrc.getState()) || rsrc != rem) {       // internal error       LOG.error( "Attempt to remove resource: "  + rsrc           +  " with non-zero refcount" );       return  false ;     }  else  {  // ResourceState is LOCALIZED or INIT       localrsrc.remove(rem.getRequest());       if  (ResourceState.LOCALIZED.equals(rsrc.getState())) {         delService.delete(getUser(), getPathToDelete(rsrc.getLocalPath()));       }       decrementFileCountForLocalCacheDirectory(rem.getRequest(), rsrc);       LOG.info( "Removed "  + rsrc.getLocalPath() +  " from localized cache" );       return  true ;     }   }

3.解决办法

过滤条件中把ResourceState.DOWNLOADING.equals(rsrc.getState()) 条件去掉，patch如下，patch 内容见Reference；

Index: LocalResourcesTrackerImpl.java =================================================================== --- LocalResourcesTrackerImpl.java  (revision  270924 ) +++ LocalResourcesTrackerImpl.java  (working copy) @@ - 230 , 8  + 230 , 7  @@             +  " from "  + getUser());         return  true ;       } -     if  (rsrc.getRefCount() >  0 -        || ResourceState.DOWNLOADING.equals(rsrc.getState()) || rsrc != rem) { +     if  (rsrc.getRefCount() >  0  || rsrc != rem) {         // internal error         LOG.error( "Attempt to remove resource: "  + rsrc             +  " with non-zero refcount" );

4.Reference 

 JIRA 地址 ：https://issues.apache.org/jira/browse/YARN-2902

 patch 地址：https://issues.apache.org/jira/secure/attachment/12685562/YARN-2902.patch

YARN-2902_LocalResourcesTrackerImpl_Error.patch