Hadoop0.21内存泄漏问题：数据块映射管理的一个bug

最新推荐文章于 2023-04-22 15:50:10 发布

mango_song

最新推荐文章于 2023-04-22 15:50:10 发布

阅读量1.3k

点赞数

分类专栏： hadoop 文章标签： NameNode Memory leak 内存泄漏 HashMap

本文链接：https://blog.csdn.net/mango_song/article/details/42551907

版权

hadoop 专栏收录该内容

46 篇文章 1 订阅

订阅专栏

我们的HDFS生产环境是Hadoop-0.21,机器规模200台，block在7KW左右. 集群每运行几个月，NameNode就会频繁FGC,最后不得不restart NameNode. 因此怀疑NameNode存在内存泄漏问题，我们dump出了NameNode进程在重启前后的对象统计信息。

07-10重启前:

num     #instances         #bytes class name
----------------------------------------------
   1:      59262275     3613989480 [Ljava.lang.Object;
...
10:       8549361      615553992 org.apache.hadoop.hdfs.server.namenode.BlockInfoUnderConstruction
11:       5941511      427788792 org.apache.hadoop.hdfs.server.namenode.INodeFileUnderConstruction
...

07-10重启后:

num     #instances         #bytes class name
----------------------------------------------
   1:      44188391     2934099616 [Ljava.lang.Object;
...
23:        721763       51966936 org.apache.hadoop.hdfs.server.namenode.BlockInfoUnderConstruction
24:        620028       44642016 org.apache.hadoop.hdfs.server.namenode.INodeFileUnderConstruction

...

从上面的信息可以看出，NameNode节点重启前最占内存的对象是[Ljava.lang.Object、[C、org.apache.hadoop.hdfs.server.namenode.INodeFile、org.apache.hadoop.hdfs.server.namenode.BlockInfo、[B、org.apache.hadoop.hdfs.server.namenode.BlockInfoUnderConstruction$ReplicaUnderConstruction等，它们的引用关系如下:

其中，根据NameNode节点的内部处理逻辑，INodeFileUnderConstruction和BlockInfoUnderConstruction都属于中间状态，当文件的写关闭之后，INodeFileUnderConstruction会变成INodeFile，BlockInfoUnderConstruction会变成BlockInfo，而集群的文件写压力不可能在100W/s级别，因此，NameNode节点可能存在内存泄漏。

文件在close时会调用NameNode的complete方法来关闭，此时BlocksMap的映射就会从BlockInfoUnderConstruction-->BlockInfoUnderConstruction 变成BlockInfo-~~>BlockInfo. (我们暂且描述为oldBlock>oldBlock 替换为newBlock~~->newBlock) BlocksMap对这一状态转变的处理逻辑是:

 BlockInfo replaceBlock(BlockInfo newBlock) {
    BlockInfo currentBlock = map.get(newBlock);
 
    assert currentBlock != null : "the block if not in blocksMap";
    // replace block in data-node lists
    for(int idx = currentBlock.numNodes()-1; idx >= 0; idx--) {
      DatanodeDescriptor dn = currentBlock.getDatanode(idx);
      Log.info("Replace Block[" + newBlock + "] to Block[" + currentBlock + "] in DataNode[" + dn + "]");
      dn.replaceBlock(currentBlock, newBlock);
    }
    
    // replace block in the map itself
    map.put(newBlock, newBlock);
    return newBlock;
  }

Block重写了hashCode和equals方法，使得newBlock和oldBlock有相同的hashCode,而且newBlock.equals(oldBlock)=true.

上述代码的原意是将map中的Entry(oldBlock,oldBlock)替换成(newBlock,newBlock). 而HashMap在处理put的时候，如果key相同(注意：这里的相同是指(newKey.hashCode==oldKey.hashCode && (oldKey==newKey || oldkey.equals(newKey))). 只会将对应的value替换，导致oldBlock->oldBlock被替换成oldBlock->newBlock，也就是oldBlock依然没有被释放，也就是所谓的内存泄漏。

请参考HashMap的代码:

    /**
     * Associates the specified value with the specified key in this map.
     * If the map previously contained a mapping for the key, the old
     * value is replaced.
     *
     * @param key key with which the specified value is to be associated
     * @param value value to be associated with the specified key
     * @return the previous value associated with <tt>key</tt>, or
     *         <tt>null</tt> if there was no mapping for <tt>key</tt>.
     *         (A <tt>null</tt> return can also indicate that the map
     *         previously associated <tt>null</tt> with <tt>key</tt>.)
     */
    public V put(K key, V value) {
        if (key == null)
            return putForNullKey(value);
        int hash = hash(key.hashCode());
        int i = indexFor(hash, table.length);
        for (Entry<K,V> e = table[i]; e != null; e = e.next) {
            Object k;
            if (e.hash == hash && ((k = e.key) == key || key.equals(k))) {
                V oldValue = e.value;
                e.value = value;
                e.recordAccess(this);
                return oldValue;
            }
        }

        modCount++;
        addEntry(hash, key, value, i);
        return null;
    }

建议将BlocksMap修复如下，已经提交patch，请见:https://issues.apache.org/jira/browse/HDFS-7592

 BlockInfo replaceBlock(BlockInfo newBlock) {

+   /**
+    * change to fix bug about memory leak of NameNode by huahua.xu  
+    * 2013-08-17 15:20
+    */
 
    BlockInfo currentBlock = map.get(newBlock);
 
    assert currentBlock != null : "the block if not in blocksMap";
    // replace block in data-node lists
    for(int idx = currentBlock.numNodes()-1; idx >= 0; idx--) {
      DatanodeDescriptor dn = currentBlock.getDatanode(idx);
      dn.replaceBlock(currentBlock, newBlock);
    }
    
    // replace block in the map itself


+   BlockInfo currentBlock = map.remove(newBlock);     
    map.put(newBlock, newBlock);
    return newBlock;
  }

截止到目前为止，该patch已经正式更新到线上集群，解决了内存泄漏问题。

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

这是公司生产环境中遇到的较为严重的bug,目前已经提交社区：https://issues.apache.org/jira/browse/HDFS-7592 ，在此和大家share一下.