今天收到报警Datanode is dead，登录上去看下发现datanode进程还“活着”，没有高负载，内存也正常，datanode日志发现只有几处block传输异常，之后就是在接收block，但是心跳超时导致被NN认为死亡：


WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException inBlockReceiver.run():
java.net.SocketException:Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1003)
at java.lang.Thread.run(Thread.java:662)

查看了ThreadDump（见附1）后终于发现原因：

在pipeline失败后，需要创建一个新的DataXceiver进行pipeline recovery,1.调用recoverRbw方法会锁定FsDatasetImp；2.然后调用stopWriter:即在进行错误恢复时，如果该数据节点的写线程DataReceiver如果还没有结束，要首先中断并等待该线程退出，保证后续恢复：

public void stopWriter() throws IOException {
  if (writer != null && writer != Thread.currentThread() && writer.isAlive()) {
    writer.interrupt();
    try {
      writer.join();
    } catch (InterruptedException e) {
      throw new IOException("Waiting for writer thread is interrupted.");
    }
  }
}

3.同时老的写入线程可能正在提交block，因此需要对FsDatasetImpl进行锁定，这时死锁发生了！！！

4.而Datanode发送Heartbeat前也需要获取FsDatasetImpl的锁定获取节点容量，因此发生超时！


其实这是一个Bug，在CDH4.4.0已经修复了，引入隐藏参数dfs.datanode.xceiver.stop.timeout.millis默认1min，对stopWriter引入超时join.

附1：

"DataXceiverfor client xxx at xxx [Receiving block xxx]" daemon prio=10tid=0x00000000406bc000 nid=0x612d in Object.wait() [0x00007ffdcb314000]
   java.lang.Thread.State: WAITING (on objectmonitor)
        at java.lang.Object.wait(Native Method)
        atjava.lang.Thread.join(Thread.java:1186)
        - locked <0x00000007ca882a30> (aorg.apache.hadoop.util.Daemon)
        atjava.lang.Thread.join(Thread.java:1239)
        atorg.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:157)
        atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverRbw(FsDatasetImpl.java:707)
        - locked <0x0000000784acf478> (aorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
        atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverRbw(FsDatasetImpl.java:91)
        atorg.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:163)
        atorg.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:457)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:103)
        atorg.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:67)
        atorg.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
        at java.lang.Thread.run(Thread.java:662)


"PacketResponder:xxx, type=HAS_DOWNSTREAM_IN_PIPELINE" daemon prio=10tid=0x00007ffde0248000 nid=0x5ed7 waiting for monitor entry [0x00007ffdccbf9000]
   java.lang.Thread.State: BLOCKED (on objectmonitor)
        atorg.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:846)
        - waiting to lock<0x0000000784acf478> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
        atorg.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:964)
        atjava.lang.Thread.run(Thread.java:662)