Hadoop异常--There are 3 datanode(s) running and 3 node(s) are excluded in this operation.

最新推荐文章于 2024-07-24 08:30:24 发布
最新推荐文章于 2024-07-24 08:30:24 发布
阅读量5.9k 收藏 5
点赞数 2
分类专栏： # Hadoop 文章标签： hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_44672474/article/details/104631281
版权

一、问题描述

在向Hadoop集群上传文件的时候，出现如下错误：
错误的提示：put: File /install.log._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation.

[root@node01 ~]# hdfs dfs -put install.log /
20/03/03 13:22:29 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
	at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1606)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1404)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587)
20/03/03 13:22:29 INFO hdfs.DFSClient: Abandoning BP-643568444-192.168.21.101-1583206346968:blk_1073741843_1019
20/03/03 13:22:29 INFO hdfs.DFSClient: Excluding datanode 192.168.21.103:50010
20/03/03 13:22:29 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
	at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1606)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1404)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587)
20/03/03 13:22:29 INFO hdfs.DFSClient: Abandoning BP-643568444-192.168.21.101-1583206346968:blk_1073741844_1020
20/03/03 13:22:29 INFO hdfs.DFSClient: Excluding datanode 192.168.21.104:50010
20/03/03 13:22:29 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
	at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1606)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1404)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587)
20/03/03 13:22:29 INFO hdfs.DFSClient: Abandoning BP-643568444-192.168.21.101-1583206346968:blk_1073741845_1021
20/03/03 13:22:29 INFO hdfs.DFSClient: Excluding datanode 192.168.21.102:50010
20/03/03 13:22:29 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /install.log._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1559)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)

	at org.apache.hadoop.ipc.Client.call(Client.java:1470)
	at org.apache.hadoop.ipc.Client.call(Client.java:1401)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1528)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1345)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587)
put: File /install.log._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.

二、问题原因

我的slave机器的防火墙没有关闭，否则主机无法跟其它机器进行通信，由此导致错误的发生。

三、解决办法

关闭其它机器的防火墙，这里以node02为例；

[root@node02 etc]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@node02 etc]# chkconfig iptables off

关闭selinux；

[root@node02 etc]# vi /etc/selinux/config

将SELINUX设置为disabled
在这里插入图片描述

四、再次上传文件

[root@node01 ~]# ls
anaconda-ks.cfg  hadoop-2.6.5.tar.gz  install.log  install.log.syslog  jdk-8u181-linux-x64.rpm
[root@node01 ~]# hdfs dfs -put install.log /
20/03/03 13:49:44 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 192.168.21.103:50010
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1456)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587)
20/03/03 13:49:44 INFO hdfs.DFSClient: Abandoning BP-643568444-192.168.21.101-1583206346968:blk_1073741846_1022
20/03/03 13:49:44 INFO hdfs.DFSClient: Excluding datanode 192.168.21.103:50010
20/03/03 13:49:44 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
	at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1606)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1404)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587)
20/03/03 13:49:44 INFO hdfs.DFSClient: Abandoning BP-643568444-192.168.21.101-1583206346968:blk_1073741847_1023
20/03/03 13:49:44 INFO hdfs.DFSClient: Excluding datanode 192.168.21.104:50010
[root@node01 ~]#

打开浏览器，出现如下界面，表示上传成功！
在这里插入图片描述

aldalee
关注
  • 2
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 7
    评论
专栏目录
跨集群拷贝Distcp性能测试
m0_46168848的博客
07-13 3172
distcp+shell脚本实现跨集群数据迁移
【Docker x Hadoop x Java API】xxx could only be written to 0 of the 1 minReplication nodes,There are 3
Ice__Clean的博客
10-07 1607
脑阔疼 本人使用 docker 搭建的 hadoop 集群：【Docker x Hadoop】使用 Docker 搭建 Hadoop 集群（从零开始保姆级） 今天在使用 Java API 对 Docker 搭建的 Hadoop 集群进行操作时 创建和删除文件夹可以，但一到了上传和下载文件，就疯狂报错 上传的文件在集群中可以看到，但大小均为 0 ，主要的异常报错主要如下： File /idea/warn.log could only be written to 0 of the 1 minReplica.
could only be replicated to 1 nodes instead of minReplication (=2). There are 3 datanode(s) running
chaoge_to_change的博客
10-26 625
在一个bash中写了很多连续的hive脚本，总是跑一段时间后出现 org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hadoop-yarn/staging/mqq/.staging/job_1540025341471_0619/libjars/mail-1.4.1.jar could only be repl...
HDFS could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3
AdamShyly的博客
04-14 1824
我在测试HDFS的写数据压测的时候，在测试程序还没运行完我就按了Ctrl + C强制退出，导致HDFS上的/tmp目录下存放的临时数据文件过大，直接将HDFS文件空间占满，什么都传不上去。所以直接将HDFS的/tmp目录下之前运行程序的临时文件删掉就好了（可以根据当时运行程序的时机来判断该不该删）。 之前有看到很多人都说是由于ClusterID不一致，所以需要格式化集群，但我发现我集群上的NameNode与DataNode的ClusterID都是一样的。所以有时候别人的意见就只能是参考参考，别轻易地就往自
HIVE_DATAX取数供数问题记录
悢七的CSDN博客
05-11 656
Hadoop本身的用户和组的关系，都是同步Linux系统中的，但是HDFS和Linux的超级用户组又有一点差别，HDFS中的超级用户组是supergroup，但是Linux中默认是没有supergoup这个组，这个时候只需要在Linux中增加supergroup这个组，然后将要在HDFS中加入到supergroup中的用户加到这个组中，再同步HDFS用户和组即可。解决办法：在hive的配置文件hive-site.xml中开启hiveserver2服务的高可用，即掉线了会马上再开启一个，配置如下。
Hadoop 本地上传文件报错： could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s)
huting1990的博客
12-02 5144
背景：Hadoop集群 是部署在腾讯云，需要把云服务器的 防火墙关闭，且外网端口都放开 使用Hadoop中 fileSystem.moveFromLocalFile(new Path(“file:///Users/Documents/b.txt”),new Path("/")); 讲本地文件上传到集群，报错：org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /kkb/a.txt.zip could only be written
hadoop环境调错
cuichunchi的博客
11-19 263
File /user/cuichunchi/hadoop/NOTICE.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation. 上...
hadoop-3.1.1.3.1.4.0-315.tar.gz
10-11
ambari-2.7.5 编译过程中四个大包下载很慢，所以需要提前下载，包含：hbase-2.0.2.3.1.4.0-315-bin.tar.gz ，hadoop-3.1.1.3.1.4.0-315.tar.gz ， grafana-6.4.2.linux-amd64.tar.gz ，phoenix-5.0.0.3.1.4.0-315....
flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar.tar.gz
11-14
在这个特定的兼容包中，我们可以看到两个文件：flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar（实际的兼容库）和._flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar（可能是Mac OS的元数据文件，通常...
Hadoop下载 hadoop-3.3.3.tar.gz
07-06
Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进 Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不...
flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar
11-16
Flink-1.11.2与Hadoop3集成JAR包，放到flink安装包的lib目录下，可以避免Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.这个报错，实现...
hadoop.dll-and-winutils.exe-for-hadoop2.7.3-on-windows_X64
12-16
Hadoop 2.7.3 Windows64位 编译bin（包含winutils.exe, hadoop.dll），自己用的，把压缩包里的winutils.exe, hadoop.dll 放在你的bin 目录 在重启eclipse 就好了
There are 0 datanode(s) running and no node(s) are excluded in this operation.
大黄的博客
11-19 2402
今天特码的用hadoop提交job时候，发现莫名其妙出现这个错误。 最后发现其实就是datanode没有成功启动啊，玛德，jps虽然显示slave节点datanode进程存在，其实是处于假死状态。意思就是所有slave节点的datanode都启动失败。 原因如下：datanode VERSION文件中的clusterID与namenode VERSION文件中的clusterID不一样！！！ datanode VERSION文件的详细位置为/home/lemaker/data/hdfstmp/dfs/
【Flink】FLink 提交报错 instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are
九师兄
12-02 609
1.概述 flink任务提交报错，错误如下，看到红色标签的时候，第一反应是权限问题 2. 随便put一个文件 su - hdfs vi aa.txt hdfs dfs -put aa.txt /user/hdfs/.flink/ 报错一样的。猜测和项目无关 2. 尝试权限 修改权限为所有人可操作，但是运行仍然报错这个 hdfs dfs -chmod -R 777 /user/hdfs/.flink 因此此方案行不通 3. safemode 难道是安全模式，根据命令 【hadoop】A.
Spark SQL 报错：File xxx could only be written to 0 of the 1 minReplication nodes.
weixin_46684938的博客
06-19 971
文章目录1.访问不到节点 File xxx could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 node(s) are excluded in this operation.问题原因：解决方案： 1.访问不到节点 File xxx could only be written to 0 of the 1 minReplication nodes. There are
云服务搭建hadoop集群 ，往hdfs文件系统中put文件的时候报错0 nodes instead of minReplication (=1).  There are 3 datanode(s)
zhangshk_的博客
09-13 1011
报错信息： put: File /test/README.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.   原因： ...
dfsput操作报“There are 0 datanode(s) running and no node(s) are excluded in this operation”
u010719917的专栏
06-27 6277
.  There are 0 datanode(s) running and no node(s) are excluded in this operation.   $ bin/hdfs dfs -mkdir /user   $ bin/hdfs dfs -mkdir /user/   $bin/hdfs dfs -put etc/hadoop input  报如下错：   17/06/
There are 0 datanode(s) running，解决Hadoop启动时，没有启动datanode
weixin_44724856的博客
06-12 1704
向Hadoop上传文件时出现错误：There are 0 datanode(s) running and no node(s) are excluded in this operation.，用jps命令查看发现datanode没有启动。 解决：
大数据处理：大数据处理框架Hadoop、Spark
最新发布
Chujun123528的博客
07-24 925
大数据处理是当代信息技术领域的一个重要分支，它涉及到海量数据的存储、管理和分析。为了高效地应对大数据处理的挑战，多种框架被开发出来，其中Hadoop和Spark是最为知名和广泛应用的两种。以下将详细介绍这两种框架以及它们在大数据处理中的应用和优势。
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /salary1.csv could only be written to 0 of the 1 minReplication nodes. There are 0 datanode(s) running and 0 node(s) are excluded in this operation.
07-09
这个错误是由于在写入文件"/salary1.csv"时，只有0个副本节点可用，没有满足最小副本数（minReplication）的要求。此外，当前没有正在运行的数据节点（datanode）以及被排除的节点（excluded nodes）。这可能是由于Hadoop集群的配置问题引起的。请确保所有的数据节点都处于运行状态，并且没有被排除在写入操作之外。你可以检查Hadoop配置文件以及集群状态来解决这个问题。
评论 7
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值