记一次集群job运行缓慢问题

业务人员反馈部分job运行缓慢，于是进行了统计观察，resource manager页面观察到如下现象

1.作业比以前运行的时间变长

具体大概是个别job平时1小时搞定，偶尔会出现2小时左右，集群计算资源也没打满状态。

 

2.Job的日志中有以下WARN的信息

2018-04-18 00:16:11,632 WARN [ResponseProcessor for block BP-<pool_id>:blk_<block_id>] org.apache.hadoop.hdfs.DFSClient: Slow ReadProcessor read fields took 57485ms (threshold=30000ms);

ack: seqno: 4 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 3284342, targets:

[DatanodeInfoWithStorage[x.x.x.x:50010,DS-26391dd6-c34d-4f7a-a6ff-6b9d264a6edd,DISK],

DatanodeInfoWithStorage[x.x.x.x:50010,DS-1840e064-e616-49d5-8ead-91f65bb3af93,DISK],

DatanodeInfoWithStorage[x.x.x.x:50010,DS-e884e0d2-b1a1-414d-925c-5d6efd1258e4,DISK]]

 

3.Datanode的日志中有以下WARN信息

2018-04-17 06:23:48,796 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 341ms (threshold=300ms)<