大数据集群yarn任务延迟问题调查

最新推荐文章于 2024-05-14 00:25:41 发布
最新推荐文章于 2024-05-14 00:25:41 发布
阅读量1.6k 收藏 2
点赞数
分类专栏： 分布式数据处理 大数据 文章标签： hadoop java 大数据
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qingchen191/article/details/128308905
版权

一、背景

        BI集群，有60多个节点，2P+数据，机器都已经运行了3年以上。

二、现象

        1、yarn任务严重延迟，有时候甚至超时失败

        2、yarn任务手动kill后重跑大多数时候会成功

三、调查思路

        一开始怀疑是跑任务的当时资源不足导致，一直当作资源不足处理。

        1、查看延迟任务日志

        2、查看节点日志

        3、分析任务执行的task以及container情况，发现有一个节点执行任务时耗时特别长

        4、定位到可能是这台机器有问题，重点调查这台机器的问题：

        通过跟踪节点的日志，yarn日志基本正常，hadoop datanode日志有异常，根据异常日志搜索出来的问题，没有结论。

2021-09-27 16:03:12,467 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475378156_3517112822, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.InterruptedIOException: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[connected 
local=/10.xx.xx.xx:50010 remote=/10.204.114.146:55280]. 447424 millis timeout left.
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:352)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1389)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1328)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1249)
	at java.lang.Thread.run(Thread.java:745)





2021-09-27 16:26:06,623 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1382344001-10.204.25.17-1458873906864:blk_4475387788_3517121113 src: /10.ee.ee.ee:19545 dest: /10.xx.xx.xx:50010
2021-09-27 16:26:07,228 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.ee.ee.ee:19545, dest: /10.xx.xx.xx:50010, bytes: 5730, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-857996562_138, offset: 0, srvID: ff8d66b8-7176-4c2e-a530-6b5038d64e52, blockid: BP-1382344001-10.204.25.17-1458873906864:blk_4475387788_3517121113, duration: 604633141
2021-09-27 16:26:07,228 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475387788_3517121113, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2021-09-27 16:26:07,399 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475385852_3517120714, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2207)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1165)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:26:07,406 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-1382344001-10.204.25.17-1458873906864:blk_4475385852_3517120714
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:467)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:781)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:761)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:237)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:26:07,406 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.run(): 
java.nio.channels.ClosedByInterruptException
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:478)
	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1389)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1328)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1249)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:26:07,406 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475385852_3517120714, type=HAS_DOWNSTREAM_IN_PIPELINE
java.nio.channels.ClosedByInterruptException
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:478)
	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1389)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1328)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1249)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:26:07,406 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475385852_3517120714, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-09-27 16:26:07,406 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-1382344001-10.204.25.17-1458873906864:blk_4475385852_3517120714 received exception java.io.IOException: Connection reset by peer
2021-09-27 16:26:07,406 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: datanode88.bi:50010:DataXceiver error processing WRITE_BLOCK operation  src: /10.ee.ee.ee:31015 dst: /10.xx.xx.xx:50010
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:467)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:781)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:761)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:237)
	at java.lang.Thread.run(Thread.java:745)


2021-09-27 16:30:39,560 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-1382344001-10.204.25.17-1458873906864:blk_4475378492_3517117022
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:467)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:781)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:761)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:237)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:30:39,561 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.run(): 
java.nio.channels.ClosedByInterruptException
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:478)
	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1389)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1328)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1249)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:30:39,561 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475378492_3517117022, type=HAS_DOWNSTREAM_IN_PIPELINE
java.nio.channels.ClosedByInterruptException
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:478)
	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1389)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1328)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1249)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:30:39,561 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475378492_3517117022, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-09-27 16:30:39,561 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-1382344001-10.204.25.17-1458873906864:blk_4475378492_3517117022 received exception java.io.IOException: Connection reset by peer
2021-09-27 16:30:39,561 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: datanode88.bi:50010:DataXceiver error processing WRITE_BLOCK operation  src: /10.ee.ee.ee:2043 dst: /10.xx.xx.xx:50010
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:467)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:781)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:761)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:237)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 16:30:39,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1382344001-10.204.25.17-1458873906864:blk_4475389081_3517122514 src: /10.ee.ee.ee:34679 dest: /10.xx.xx.xx:50010
2021-09-27 16:30:40,252 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1382344001-10.204.25.17-1458873906864:blk_4475378492_3517117022 src: /10.ee.ee.ee:6123 dest: /10.xx.xx.xx:50010
2021-09-27 16:30:40,252 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover RBW replica BP-1382344001-10.204.25.17-1458873906864:blk_4475378492_3517117022
2021-09-27 16:30:40,252 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_4475378492_3517117022, RBW
  getNumBytes()     = 44179902
  getBytesOnDisk()  = 44179902
  getVisibleLength()= 44179902
  getVolume()       = /data/dfs/data/current
  getBlockFile()    = /data/dfs/data/current/BP-1382344001-10.204.25.17-1458873906864/current/rbw/blk_4475378492
  bytesAcked=44179902
  bytesOnDisk=44179902




2021-09-27 18:27:02,533 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2207)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1165)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 18:27:02,535 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755
java.io.IOException: Premature EOF from inputStream
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:467)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:781)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:761)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:237)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 18:27:02,535 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.run(): 
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1389)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1328)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1249)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 18:27:02,535 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1389)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1328)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1249)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 18:27:02,535 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-09-27 18:27:02,535 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755 received exception java.io.IOException: Premature EOF from inputStream
2021-09-27 18:27:02,536 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: datanode88.bi:50010:DataXceiver error processing WRITE_BLOCK operation  src: /10.ee.ee.ee:3268 dst: /10.xx.xx.xx:50010
java.io.IOException: Premature EOF from inputStream
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:467)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:781)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:761)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:237)
	at java.lang.Thread.run(Thread.java:745)
2021-09-27 18:27:02,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer: Transmitted BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755 (numBytes=6635939) to /10.216.5.16:50010
2021-09-27 18:27:02,759 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755 src: /10.ee.ee.ee:3340 dest: /10.xx.xx.xx:50010
2021-09-27 18:27:02,759 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover RBW replica BP-1382344001-10.204.25.17-1458873906864:blk_4475546259_3517286755
2021-09-27 18:27:02,759 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_4475546259_3517286755, RBW
  getNumBytes()     = 6635939
  getBytesOnDisk()  = 6635939
  getVisibleLength()= 6635939
  getVolume()       = /data/dfs/data/current
  getBlockFile()    = /data/dfs/data/current/BP-1382344001-10.204.25.17-1458873906864/current/rbw/blk_4475546259
  bytesAcked=6635939
  bytesOnDisk=6635939
2021-09-27 18:27:03,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1382344001-10.204.25.17-1458873906864:blk_4475550789_3517286896 src: /10.ee.ee.ee:25889 dest: /10.xx.xx.xx:50010
2021-09-27 18:27:03,670 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1382344001-10.204.25.17-1458873906864:blk_4475550790_3517286897 src: /10.ee.ee.ee:48873 dest: /10.xx.xx.xx:50010
2021-09-27 18:27:03,673 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.ee.ee.ee:48873, dest: /10.xx.xx.xx:50010, bytes: 19994, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1365176392_177806, offset: 0, srvID: ff8d66b8-7176-4c2e-a530-6b5038d64e52, blockid: BP-1382344001-10.204.25.17-1458873906864:blk_4475550790_3517286897, duration: 2383047

        1）、怀疑有数据倾斜，某个任务用到的数据分布不均匀，都在这个节点上，导致处理量大，耗时特别长。后来排除这种情况。

        2）、怀疑这台机器的配置有问题，检查各种配置，发现jdk版本不一致，这台机器上用的是openjdk1.8，其他机器用的oraclejdk1.8，也有的机器上用openjdk1.8，为了排除jdk的问题，把这台机器的jdk改为oraclejdk1.8，重启服务后问题依然存在。（过程中发现整个集群的jdk有好几个小版本，感慨一下，维护的同学在集群扩容的时候就不考虑跟之前保持一致吗？）

        中间也怀疑过centos系统的linux内核版本不一致导致的，查看了一下没有问题，据说有个小版本是有问题的。

        3）、因为机器很老了，怀疑这台机器硬盘有坏掉的情况，进行磁盘检测，硬件故障检测，各项硬件都正常，排除硬盘问题。

        4）、重启这台机器，问题依然存在，但是有一个发现，这台机器重启的时候，集群任务跑的很快，基本可以断定是这台机器导致整个集群的问题。

        5）、怀疑网络问题，一开始检查都很正常，后来一直ping，发现有1%的丢包。

        6）、可能是网线或者网口的问题，最后换了一个光口（从光口A换到光口B）后问题解决。

四、结论

        1、可以通过一直ping来判断网络是否正常。

        2、网络问题可能会导致很多意想不到的异常。

        yarn任务异常，异常日志在datanode日志里，最后发现是光口的问题。调查问题的结局总是让人意想不到。

轻尘
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Hadoop Yarn延迟调度分析（Delay Schedulering）
weixin_35792948的博客
08-09 1083
大规模集群中，对数据本地化（data loality）的优化，可以减少很多网络IO，变成本地的磁盘IO，而磁盘的IO比网络的IO快很多，这对降低集群的IO负载以及增加集群的吞吐量是很有益处的，所以我对Yarn中的延迟调度进行分析，尝试提高作业中任务所在容器的调度本地化命中率。 我们从自带的延迟调度的单元测试入手进行分析： @Test public void testDelaySchedul...
yarn任务执行慢
zhijun666
12-18 1197
之前执行一个spark作业，设置的的executor-memory为4G，num-executors为5 ，executor-cores为6之后，查看saprkui界面中的executors只有两个executor加一个driver， 之后考虑可能是内存不够，设置的的executor-memory为2G，num-executors为6 ，executor-cores为4，查看saprkui界面中的executors有三个executor加一个driver， 快了十分钟 看看yarn中的资源情况 看看
正确解决java.io.InterruptedIOException异常的有效解决方法
最新发布
wbajsjhhhhh的博客
05-14 1809
正确解决java.io.InterruptedIOException异常的有效解决方法
换npm yarn的源让install超时去死吧
djph26741的博客
05-12 1591
安装npm install时，长时间停留在fetchMetadata: sill mapToRegistry uri http://registry.npmjs.org/whatwg-fetch处， 此处需要在对应的URL处下载一些文件，只要网络稳定，一般花不了多长时间 但是，最近两次操作都是长时间停留，在网上找了一些原因，才发现后面的URL地址可以由淘宝的源代替，原来是资源问题导致...
解决Yarn慢的问题
weixin_44772835的博客
10-15 2457
如果觉得安装速度慢，安装源和原来 npm 是一样的，可以通用，修改方法如下： 在国内不管使用npm还是yarn，安装速度都很慢。因为镜像源地址在国外，有qiang的存在，导致速度很慢，这里换上淘宝的镜像源，即可加速安装速度这里示例使用的是yarn，使用npm的用户将yarn改为npm即可。查看当前设置的镜像源地址yarn ...
yarn安装依赖超时
lzj2504476514的博客
08-01 2536
报错原因：由于yarn默认的配置源是https://registry.yarnpkg.com，下载依赖包容易超时， 使用yarn config list查看yarn的配置源 解决方案：将配置源设置为国内镜像源 执行yarn config registry https://registry.npm.taobao.org --global ...
yarn的原生web页面加载数据显示历史job列表缓慢
jingyi1219的博客
03-31 697
问题描述： yarn的原生web页面加载数据显示历史job列表缓慢 问题原因： 这是由于yarn提交的历史任务数量太大（接近1w）导致 解决方法 方法一： 在网上看到有人说有相关的配置，去官网上找了一下，应该是这个yarn的timeline时序数据库有关的配置，截图如下 这两个参数的翻译一下分别是： 时间轴存储区中最近读取的实体开始时间的缓存大小（以实体数为单位）默认为10000 时间线存储区中最近写入的实体开始时间的缓存大小（以实体数为单位）默认为10000 这里的实体entity应该就是job 官网配
flink集群搭建包.zip
06-14
Hadoop 2.x版本提供了YARN资源管理系统，Flink可以运行在YARN之上，实现集群资源的有效管理。"shaded"意味着这个库已经包含了所有依赖的jar包，避免了类冲突的问题，"uber"则表示这是一个聚合的jar包，方便直接使用...
java 大数据大数据笔记.doc
01-13
3. YARN（Yet Another Resource Negotiator）是Hadoop的资源管理系统，负责集群内资源（如内存、CPU核数）的管理和调度。它为其他计算框架如Spark、Tez等提供了运行环境，保证了多任务的公平性和资源利用率。 总的...
第13章 大数据平台监控命令.pptx
11-28
本章聚焦于大数据平台的监控命令，主要针对Hadoop及其相关组件，包括YARN、HDFS、HBase、Hive、ZooKeeper、Sqoop和Flume等。监控的目标不仅在于掌握告警和日志信息，还要了解平台运行、资源和服务的状态，以及能够...
大数据性能调优
08-16
8. **网络通信优化**：优化网络I/O，例如使用Netty框架，减少网络延迟，提升大数据集群间通信效率。 9. **分布式存储优化**：对于HDFS这样的分布式文件系统，优化块大小、副本数量和数据分布策略，能有效提高读写...
yarn集群NodeManager日志聚合慢问题解决方案
阿华田的博客
10-28 1067
正常情况作业提交到 Yarn 集群时，作业完成或者失败后，每个 NM 节点都会对每个 app 作业进行日志聚合操作，存储到hdfs指定的目录下，但是最近发现越来越多的任务通过yarn logs命令无法查询，经过排查发现很多任务的日志聚合变慢了，需要半小时甚至更多时间才能聚合完成。
Spark以yarn-client提交任务时报错超时，Connection to 192.168.. /has been quiet forms while there are outstanding...
weixin_30566111的博客
07-13 498
报错信息如上，具体是运行FusionInsight给的样例SparkPi，在local环境下是可以的，但是如果以yarn-client模式就会卡住，然后120s以后超时，其实以yarn-cluster模式也是会报错的，开始在spark-default-conf 中加上了driver的spark.driver.host = $客户端IP，没用，将服务器各个主机免密登陆，没用，再将客户端的ip添...
yarn install或者 npm install 下载安装超时的解决办法
Yoga99的博客
12-14 2511
2.如果按照上述情况还没有解决超时问题，那可能是网络问题，你很有可能使用了默认超时。那这种情况下我们可以在相关的配置文件。
hadoop任务优化-调整Yarn队列资源
wisgood的专栏
09-24 8997
hadoop任务优化1-调整Yarn队列资源
解决用海豚调度时,yarn处理大量并发任务很慢的问题
qq_43382750的博客
04-22 2847
公司用的海豚调度,工作流里同时运行了二十多个插入sql,yarn只有两个任务同时运行,并且一直在卡这,到获取tez session这不动,检查了下tez的超时时间,600s改成了10s, 同时在ambari上给yarn增加了一个队列hive,各分50 添加了并刷新保存 ...
Hive任务被阻塞，配置yarn多队列
qq_42672132的博客
08-25 929
一、出现的问题 hive客户端开着，shell脚本中的hive任务启动不起来。 二、解决方法 在Yarn中额外配置一条队列，hive客户端中的任务提交到这个新的队列中 vim $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml <!--修改：添加hive队列，然后把容量调整一下，看你个人--> <property> <name>yarn.scheduler.capacity.root.queues</name&g
Yarn容量调度器并发度问题，解决一直卡在Running job
For_dongyang的专栏
03-04 3275
一、问题描述 Yarn默认调度器为CapacityScheduler（容量调度器），且默认只有一个队列——default。如果队列中执行第一个任务资源不够，就不会再执行第二个任务，一直等到第一个任务执行完毕。 （1）创建一张测试表 create table student(id int, name string); （2）启动1个hive客户端，执行以下插入数据的sql语句 insert into table student values(1,'abc'); 执行该语句，hive会初始化..
Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+ES+Redash等详细安装部署
06-22
1、内容概要：Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+Elasticsearch+Redash等大数据集群及组件搭建指南（详细搭建步骤+实践过程问题总结）。 2、适合人群：大数据运维、大数据相关技术及组件初学者。 3、能学到啥：大数据集群及相关组件搭建的详细步骤，了解大数据各组件的用途，深入认识各大数据组件工作原理及优化方案。 4、阅读建议：使用过以上大数据组件进行简单开发、了解以上大数据组件的相关功能。 5、资源价值：大数据组件搭建的详细实践步骤、一次性提供较全面的常用大数据集群及组件安装部署内容、资源是博主结合官网文档+网上各类搭建文档+本人亲自实践后总结整合的文档（包括过程踩坑记录+对应解决方案）。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值