java.nio.channels.SocketChannel[connected local=/192.168.10.51:45306 remote=slave1/192.168.10.51:500

经过日志的初步分析，发现3600s这个线索，从job的configuration中，初步查找出参数dfs.client.socket-timeout，单位毫秒。

-Ddfs.client.socket-timeout=3600000

试验性地将这个参数修改为60ms，可以看出出现超时的概率非常大，但会不断重试以继续:

2014-09-26 12:53:03,184 WARN [main] org.apache.hadoop.hdfs.DFSClient: Failed to connect to /192.168.7.22:50010 for block, add to deadNodes and continue.
 java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : 
 java.nio.channels.SocketChannel[connected local=/192.168.7.17:22051 remote=/192.168.7.22:50010] 
 java.net.SocketTimeoutException: 60 millis timeout while waiting for channel to be ready for read. ch : 
 java.nio.channels.SocketChannel[connected local=/192.168.7.17:22051 remote=/192.168.7.22:50010]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1490)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:392)
at org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:131)
at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1108)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:533)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749) 
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:601)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at com.xxx.app.MzSequenceFile$Reader.init(MzSequenceFile.java:521)
at com.xxx.app.MzSequenceFile$Reader.<init>(MzSequenceFile.java:515)
at com.xxx.app.MzSequenceFile$Reader.<init>(MzSequenceFile.java:505)
at com.xxx.yo.io.CombineFileRawLogReader.<init>(CombineFileRawLogReader.java:146)
at com.xxx.yo.io.CampaignRawLogReader.next(CampaignRawLogReader.java:64)
at com.xxx.yo.io.CampaignRawLogReader.next(CampaignRawLogReader.java:22)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:197)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:183)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

于是，最终将这个参数设置为60s，这其实也是集群最终默认的超时时间，由于之前一次不明就里的优化，导致了后续这些问题的发生，因此在调整参数时，一定要注意了解清楚该参数造成的影响。

简要分析的结论

在Mapper端读取HDFS上的文件时，可能由于网络原因（由于我们的Split切分地比较大，因此不可能做到完全数据本地化）导致读取数据超时，原来居然设置成1个小时，而任务的超时时间仅设置为20分钟，因此只要发生读取数据超时，就必然会引起任务超时。

通过这次分析过程，学到了很多查找问题的方式，包括通过现象分析规律，得到线索，最终查找问题的原因。快速测试，不能忽略哪怕一个小的Exception，不行就是分析hadoop的源码，掌握其运行时行为。

如果task tracker在一段时间（默认是10min，可以通过mapred.task.timeout属性值来设置，单位是毫秒）内一直没有收到它的进度报告，则把它标记为失效。

2. java.net.SocketTimeoutException: Call to datanode1/172.16.0.122:60020 failed because java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.0.126:55277 remote=datanode1/172.16.0.122:60020]

问题: 主服务器和从服务器上hbase-site.xml中，zookeeper.session.timeout配置的时间不一致

解决方案：把主服务器和从服务器上hbase-site.xml中的zookeeper.session.timeout配置一致，统一成120000