hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073741891_1096java.io.IOException: Got

已于 2023-03-15 11:11:55 修改
阅读量1.7k 收藏 4
点赞数 2
分类专栏： 大数据 文章标签： hdfs hadoop
于 2023-03-14 19:58:54 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43652205/article/details/129537430
版权

目录

一、异常产生过程

二、解决方案

三、思考原因

今天在配置Hadoop3.x纠删码（Erasure Coding）时，发生了以下报错，

[mingyu@hadoop102 hadoop-3.1.3]$ hadoop fs -put 1.pptx /input
2023-03-14 19:40:43,384 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2023-03-14 19:40:43,567 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,584 INFO hdfs.DataStreamer: Exception in createBlockOutputStream #1: blk_-9223372036854775423_1166
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
	at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:106)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,588 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.13.105:9866,DS-461042db-82e2-48a4-94dd-5d6e3d783035,DISK]
2023-03-14 19:40:43,592 WARN hdfs.DataStreamer: DataStreamer Exception
java.io.IOException: Unable to create new block.#1: failed, block==null
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:114)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,593 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,755 WARN hdfs.DataStreamer: Could not get block locations. Source file "/input/1.pptx._COPYING_" - Aborting...#1: failed, block==null
2023-03-14 19:40:43,853 INFO hdfs.DataStreamer: Exception in createBlockOutputStream #3: blk_-9223372036854775421_1166
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
	at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:106)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,853 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.13.106:9866,DS-91b65fc7-a547-41ad-b8dc-b9729969c950,DISK]
2023-03-14 19:40:43,854 WARN hdfs.DataStreamer: DataStreamer Exception
java.io.IOException: Unable to create new block.#3: failed, block==null
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:114)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,854 WARN hdfs.DataStreamer: Could not get block locations. Source file "/input/1.pptx._COPYING_" - Aborting...#3: failed, block==null
2023-03-14 19:40:43,854 WARN hdfs.DFSOutputStream: Failed: oldBytes=0, len=1048576, DFSStripedOutputStream:#3: failed, block==null
java.io.IOException: Could not get block locations. Source file "/input/1.pptx._COPYING_" - Aborting...#3: failed, block==null
	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477)
	at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,858 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,971 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,974 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,974 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:44,058 WARN hdfs.DFSOutputStream: Block group <1> failed to write 2 blocks. It's at high risk of losing data.

一、异常产生过程

当时的操作如下

1、启用RS-3-2-1024k纠删码

hdfs ec -enablePolicy -policy RS-3-2-1024k

  

2、配置纠删码生效路径为/input

hdfs ec -setPolicy -path /input -policy RS-3-2-1024k

 3、通过hadoop102上传文件到/input

hadoop fs -put 1.pptx /input

然后就遇到了异常

[mingyu@hadoop102 hadoop-3.1.3]$ hadoop fs -put 1.pptx /input
2023-03-14 19:40:43,384 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2023-03-14 19:40:43,567 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,584 INFO hdfs.DataStreamer: Exception in createBlockOutputStream #1: blk_-9223372036854775423_1166
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
	at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:106)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,588 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.13.105:9866,DS-461042db-82e2-48a4-94dd-5d6e3d783035,DISK]
2023-03-14 19:40:43,592 WARN hdfs.DataStreamer: DataStreamer Exception
java.io.IOException: Unable to create new block.#1: failed, block==null
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:114)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,593 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,755 WARN hdfs.DataStreamer: Could not get block locations. Source file "/input/1.pptx._COPYING_" - Aborting...#1: failed, block==null
2023-03-14 19:40:43,853 INFO hdfs.DataStreamer: Exception in createBlockOutputStream #3: blk_-9223372036854775421_1166
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
	at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
	at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:106)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,853 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.13.106:9866,DS-91b65fc7-a547-41ad-b8dc-b9729969c950,DISK]
2023-03-14 19:40:43,854 WARN hdfs.DataStreamer: DataStreamer Exception
java.io.IOException: Unable to create new block.#3: failed, block==null
	at org.apache.hadoop.hdfs.StripedDataStreamer.nextBlockOutputStream(StripedDataStreamer.java:114)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,854 WARN hdfs.DataStreamer: Could not get block locations. Source file "/input/1.pptx._COPYING_" - Aborting...#3: failed, block==null
2023-03-14 19:40:43,854 WARN hdfs.DFSOutputStream: Failed: oldBytes=0, len=1048576, DFSStripedOutputStream:#3: failed, block==null
java.io.IOException: Could not get block locations. Source file "/input/1.pptx._COPYING_" - Aborting...#3: failed, block==null
	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477)
	at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
2023-03-14 19:40:43,858 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,971 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,974 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:43,974 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-14 19:40:44,058 WARN hdfs.DFSOutputStream: Block group <1> failed to write 2 blocks. It's at high risk of losing data.

然后在HDFS网页端查看上传的文件

 在hadoop105、hadoop106上面没有

经过几个小时的排查，终于找到了解决方案😁😁

二、解决方案

将hadoop105、hadoop106的防火墙关掉就好了

systemctl status firewalld


systemctl stop firewalld

systemctl disable firewalld

 

 再次进行上传

hadoop fs -put 1.pptx /input/2.pptx

 

 

 全都有了

三、思考原因

在向服务器集群添加新服务器的时候没有仔细地将防火墙关掉

在新增服务器的这篇文章已经新增这一步骤

Hadoop服役新服务器_一抹鱼肚白的博客-CSDN博客

一抹鱼肚白
关注
  • 2
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Hadoop上传文件异常：Exception in createBlockOutputStreamjava.net.NoRouteToHostException: No route to host
qq_64066344的博客
03-14 1044
Hadoop上传文件异常：Exception in createBlockOutputStreamjava.net.NoRouteToHostException: No route to host
Exception in createBlockOutputStream
qq_30657195的博客
03-07 2240
过程描述： 杀死 namenode ，使用namesecondary数据恢复后（方法：手动恢复，无误），数据丢失，且第一次上传数据出现：Exception in createBlockOutputStream 日志： 19/03/06 02:31:58 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFExce...
Spring Cloud中怎么使用Resilience4j Retry对OpenFeign进行重试
最新发布
冥胖胖9的博客
08-01 610
通过上述配置和代码示例，我们在Spring Cloud项目中成功集成了Resilience4j的Retry机制，为OpenFeign的服务调用提供了自动重试功能。Resilience4j提供的灵活配置使得我们可以根据实际需求，精细控制重试策略，从而提高系统的可靠性和稳定性。希望这篇文章能帮助你更好地理解和使用Resilience4j Retry对OpenFeign进行重试。如果你有任何问题或建议，欢迎在评论区留言交流。
org.apache.hadoop.hdfs.DataStreamer: Exception in createBlockOutputStream问题
jshazhang的专栏
10-15 1778
2020-08-24 19:12:15,596 INFO [Thread-925] org.apache.hadoop.hdfs.DataStreamer: Exception in createBlockOutputStream blk_1090328244_16602678 java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatch
报错：INFO hdfs.DataStreamer: Exception in createBlockOutputStream解决办法
hsg77的专栏
07-12 9560
报错：INFO hdfs.DataStreamer: Exception in createBlockOutputStream解决办法 错误信息如下： [hadoop3@master ~]$ ls app bin cnWin10.iso data part-r-00000 test_fenfu2000_1w.json tools zookeeper.out [hadoop3@...
hdfs.DataStreamer: Exception in createBlockOutputStream XXXXX
weixin_46039745的博客
12-23 526
该报错比较简单，直观的看就是防火墙没关。禁止firewall开机启动。停止firewall。
hdfs.DataStreamer: Exception in createBlockOutputStream
weixin_30779691的博客
04-11 1724
在上传文件至 HDFS 提示如下信息 [root@h136 jdk1.8.0_202]# hadoop fs -put javafx-src.zip / 19/04/11 23:19:36 INFO hdfs.DataStreamer: Exception in createBlockOutputStream java.io.IOException: Got error, status=...
23/05/31 22:28:48 WARN BlockReaderFactory: I/O error constructing remote block reader. java.io.IOException: Got error, status message opReadBlock BP-1569217759-192.168.100.101-1685011952795:blk_1073741834_1010 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-1569217759-192.168.100.101-1685011952795:blk_1073741834_1010, for OP_READ_BLOCK, self=/192.168.100.1:64373, remote=/192.168.100.101:9866, for file /hive/warehouse/user_project.db/mediamatch_userevent/mediamatch_userevent.csv, for pool BP-1569217759-192.168.100.101-1685011952795 block 1073741834_1010
06-01
这是一个 Java IO 异常，可能是由于读取远程块数据时发生了错误。错误信息显示 ReplicaNotFoundException，可能是因为找不到所需的块副本。这个错误是在尝试读取文件 /hive/warehouse/user_project.db/mediamatch_...
hdfs相关问题及解决方案归档
weixin_44976835的博客
08-26 2691
目录问题1：reduce预处理阶段shuffle时获取已完成的map的输出失败次数超过上限问题2：Too many fetch-failures问题3：处理MR速度特别的慢问题4：能够启动datanode，但无法访问，也无法结束问题5：节点断联问题6：JVM内存不够问题7：Hadoop添加节点的方法问题8：给单个node新加硬盘问题9：IO写操作出现问题问题10：HDFS退服节点的方法问题11：IO异常问题12：status of 255 error问题13：sort by value问题14：restar
Hadoop_三台服务器搭建集群- 启动HDFS在fs appendToFile 报错（从失败稀里糊涂Ok了）
yidianyidei的博客
07-15 1158
无法使用appenToFile一 初始终止集群 stop-dfs.sh错误2另起炉灶-不明所以就OK 了 一 初始 群起hdfs 集群-start-hdfs.sh hadoop fs -appendToFile 3 /2 20/07/15 09:29:18 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Got error, status message , ack with firstBadL
优化Hadoop Balancer运行速度
数据之路
05-17 6837
  1. 修改dfs.datanode.max.transfer.threads = 4096 (如果运行hbase的话建议为16384)，指定用于在DataNode间传输block数据的最大线程数，老版本的对应参数为dfs.datanode.max.xcievers 2. 修改dfs.datanode.balance.bandwidthPerSec  = 52428800，指定DataNod...
hadoop那些事儿_1
tawcl的专栏
03-31 1978
一、背景 1、上周生产集群加入几台节点，执行start-balancer后进度十分地缓慢，连续几天未完成。 2、屋漏偏逢连夜雨，周六供电线路被施工挖断，机房UPS在坚持几个小时后，集群整体宕掉。 3、周一供电正常后，集群再次启动。 二、问题 1、症状 （1）hadoop，hdfs启动后上传文件正常，日志中未发现异常。 （2）hbase，可以启动，但是启动后很多表的regions无法正
HDFS中javaAPI创建文件写入内容报错：INFO hdfs.DFSClient: Exception in createBlockOutputStream
登峰造极胡子球手
11-11 3607
19/11/11 20:33:26 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.net.ConnectException: Connection refused: no further information at sun.nio.ch.SocketChannelImpl.checkConnect(Na...
INFO hdfs.DFSClient: Exception in createBlockOutputStream错误
fengyiyangziyou_的博客
11-13 5429
从本地往hdfs文件系统上传文件，出现如下错误：  17/11/12 22:44:10 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink as 192.168.0.101:50010 at org.apache.had
hdfs中Exception in createBlockOutputStream以及MapReduce中I/O error constructing remote block reader
学习是一种信仰的博客
11-13 1347
问题描述 windows电脑向aliyun进行hdfs api文件追加操作时出现下方错误： DEBUG [Thread-3] (DFSOutputStream.java:1503) - Connecting to datanode *.*.*.*:9866 DEBUG [IPC Client (927327686) connection to centos8/*.*.*.*:9820 from root] (Client.java:1192) - IPC Client (927327686) conne..
Hadoop Exception in createBlockOutputStream
九师兄
11-08 1735
简介： Hadoop Exception in createBlockOutputStream 报错： 获取FileSystem的时候 # dfs.data.transfer.protection这个配置添加上面就可以了 conf.set("dfs.data.transfer.protection","integrity"); FileSystem fs = FileSystem.g...
hdfs.DFSClient: Exception in createBlockOutputStream
Faith_xzc
07-07 662
问题描述： 向hadoop集群上传文件报错 [xzc@hadoop105 hadoop-2.7.2]$ hadoop fs -put wcinput/wc.input /user/atguigu/input/ 原因： 防火墙没有关闭 解决办法： 关闭防火墙（注意需要切换root用户） 重新上传： [xzc@hadoop105 hadoop-2.7.2]$ hadoop fs -put wcinput/wc.input /user/atguigu/input/ ...
INFO hdfs.DFSClient: Exception in createBlockOutputStream
mo
09-30 2115
今天在用shell命令进行put时出现了如下的报错： 这两个ip地址都是我的datanode，后来联想到是我没有禁止这两个datanode的防火墙所以会出现上述的错误，我当时只是进行了关闭而已，过了一段时间再次开启hadoop时肯定这两个子节点的防火墙又变成了默认的开启状态，一查状态还真的是这样。 于是果断的用systemctl disable firewalld.service把这两个子节点的防火墙的状态禁止了，在去put时就可以看到不报错了，网上还有说如果不是防火墙的原因的话可能是selinux的
hdfs.datastreamer: exception
06-28
这个问题可能与 Hadoop 的 HDFS（分布式文件系统）有关，具体错误原因需要查看更详细的日志信息才能确定。一般来说，这个异常可能涉及到 HDFS 数据传输时出现了问题，可能是由于网络连接不稳定、存储设备故障等原因...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值