hadoop put异常

最新推荐文章于 2024-04-18 19:30:43 发布
最新推荐文章于 2024-04-18 19:30:43 发布
阅读量550 收藏
点赞数
分类专栏： hadoop 文章标签： Hadoop Java Apache SUN Access

09/01/19 17:32:41 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:42 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 4
09/01/19 17:32:43 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:43 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 3
09/01/19 17:32:44 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:44 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 2
09/01/19 17:32:45 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:45 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 1
09/01/19 17:32:48 WARN dfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:48 WARN dfs.DFSClient: Error Recovery for block null bad datanode[0]
put: Could not get block locations. Aborting...
Exception closing file /user/hadoop/input/1.test
java.io.IOException: Could not get block locations. Aborting...
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2151)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1897)

 

Hadoop DFSClient警告NotReplicatedYetException信息

有时，当你申请到一个HOD集群后马上尝试上传文件到HDFS时，DFSClient会警告NotReplicatedYetException。通常会有一个这样的信息 -

WARN hdfs.DFSClient: NotReplicatedYetException sleeping <filename> retries left 3
08/01/25 16:31:40 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException:java.io.IOException: File <filename> could only be replicated to 0 nodes, instead of 1

当你向一个DataNodes正在和NameNode联络的集群上传文件的时候，这种现象就会发生。在上传新文件到HDFS之前多等待一段时间就可以解决这个问题，因为这使得足够多的DataNode启动并且联络上了NameNode。

iteye_17729
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
HDFS NotReplicatedYetException
渴死的火烈鸟
07-21 2768
I encounter exception below when I use copyFromLocal to copy several big files (10G) to HDFS. Hadoop guy’s word to explain this:  “I noticed the same recently. For me it happened since the datanodes ...
Hadoop常见异常
02-03
Hadoop常见异常，以及hadoop配置，等资料
hadoop在用命令将文件放入文件夹中的时报错,请教
Rio520的博客
08-21 514
[hadoop@NameNode196 ~]$ hadoop fs -ls Found 1 items drwxr-xr-x - hadoop supergroup 0 2020-08-21 21:24 input [hadoop@NameNode196 ~]$ ll 总用量 330164 drwxr-xr-x. 4 hadoop hadoop 30 8月 21 18:15 dfs -rw-rw-r--. 1 hadoop hadoop 84 8月
put: File /user/hadoop/test/.bashrc._COPYING_ could only be written to 0 of the ...
最新发布
m0_64473560的博客
04-18 360
启动完成后，可以通过命令 jps 来判断是否成功启动，若成功启动则会列出如下进程: "NameNode"、"DataNode" 和 "SecondaryNameNode"（如果 SecondaryNameNode 没有启动，运行 sbin/stop-dfs.sh 关闭进程，然后再次尝试启动尝试）。如果没有 NameNode 或 DataNode ，那就是配置不成功，检查之前步骤，或通过查看启动日志排查原因。hadoop找不到DataNode，DataNode 没有启动。结果：运行命令之后成功将文件上传。
hive报错Could not get block locations. Source file “/user/hive/warehouse/xxx
博客
12-24 3222
hive报错 Could not get block locations. Source file "/user/hive/warehouse/xxx... 以及 Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error while processing row (tag=0)... 是因为链接超时了设置set mapred.task.timeout=600000;为10分钟或更长即可 ...
hadoop无法上传hdfs方法解决
qq_56572867的博客
04-18 6224
首先，我遇到的是两种情况第一种是namenode没有开启，秒挂的情况，这个会出现这种提示 put: File /hdfsdata/NOTICE.txt._COPYING_ could only be written to 0 of the 1 minReplication nodes. There are 0 datanode(s) running and 0 node(s) are excluded in this operation. 注意就是下面提醒了datanode是0，通过jps查看，
hadoop复制文件出现There are 0 datanode(s) running and 0 node(s) are excluded in this operation.解决方案
seamoon的博客
05-21 583
hadoop复制文件出现There are 0 datanode(s) running and 0 node(s) are excluded in this operation.解决方案
hadoop 3.1.4
10-03
Hadoop是Apache旗下的一个用java语言实现开源软件框架，是一个开发和运行处理大规模数据的软件平台。允许使用简单的编程模型在大量计算机集群上对大型数据集进行分布式处理。
hadoop3.3.4 winutils
12-18
在windows11平台，基于hadoop3.3.4源码包自编译，按需自取，内含hadoop.dll等文件以及全部源码等
hadoop-3.1.3安装包
07-28
Hadoop实现了一个分布式文件系统(Hadoop Distributed File System)，简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的(low-cost)硬件上;而且它提供高吞吐量(high throughput)来访问应用程序的数据，适合...
hadoop2.7.7
07-20
hadoop2.7.7
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9000/user/
AStar的博客
02-22 1643
根源是：   FileSystem fs = FileSystem.get(conf);   fs.delete(new Path(args[2]),true);                             //删除输出路径继续循环                             FileInputFormat.addInputPath(job, new Path(
[初学hadoop]错误信息：hdfs://192.168.0.16:9000/report.bak, expected: file:///
menghuannvxia的专栏
03-24 2820
在本地运行hadoop Map/Reduce程序时，在涉及到hdfs文件操作的时候，往往会出现上面提到的错误，例如，下面这段代码： 1 Path clear_path = new Path(args[1]); 2 FileSystem fs = FileSystem.get(conf); 3 fs.delete(clear_path);   通过试验发现，在分布式ha
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException
itm的专栏
09-02 4161
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/nnThroughputBenchmark/addblock/AddblockBenchDir0/Addbl
hadoop常用命令
qq_32812063的博客
01-21 204
hadoop fs,hadoop dfs以及hdfs dfs区别： The FileSystem (FS) shell is invoked by bin/hadoop fs. All the FS shell commands take path URIs as arguments. The URI format is scheme://autority/path. For HDFS the s...
hadoop 初识
bingdongguke的专栏
08-05 757
hadoop无法正常启动(1) 执行  $ bin/hadoop start-all.sh之后,无法启动. 异常一  Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for
HDFS 客户端常见报错整理
eli的博客
11-16 3825
向 NameNode 进行 RPC 请求向 DataNode 进行 IO 读写。无论哪个过程，如果出现异常，一般都不会导致业务失败，也都有重试机制，实际上，业务想要失败是很难的。在实际使用过程中，客户端和 NN 之间的 RPC 交互一般不会有什么报错，大部分报错都出现在和 DN 的 IO 交互过程中，这篇文章主要总结一下常见的 DN IO 报错。
(转)记hadoop故障一例：BlockAlreadyExistsException
weixin_33963594的博客
08-31 428
Posted on 2011-08-15 by 54chen hive版本：0.7.0 hadoop版本：0.20.2在线上跑了一个季度了，基本上没什么问题，今天突然出了问题。 在hive执行时留下的蛛丝马迹： Failed with exception org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetExceptio...
hadoop fs -put 用法
07-08
hadoop fs -put命令用于将本地文件或目录复制到Hadoop分布式文件系统（HDFS）中。 语法如下： ``` hadoop fs -put <localsrc> ``` 其中，`<localsrc>` 是本地文件或目录的路径，`<dst>` 是HDFS目标路径。 例如...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值