1.文件append的问题
hadoop的版本1.0.4以后,API中已经有了追加写入的功能,但不建议在生产环境中使用,原因如下:Does HDFS allow appends to files? This is currently set to false because there are bugs in the "append code" and is not supported in any prodction cluster. 如果要测试的话,需要把dfs.support.appen的参数设置为true,不然客户端写入的时候会报错:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: Append to hdfs not supported. Please refer to dfs.support.append configuration parameter.
解决:修改namenode节点上的hdfs-site.xml。
<span style="font-size:18px;"><property>
<name>dfs.support.append</name>
<value>true</value>
</property></span>
2.在Hadoop-1.0.4和Hadoop-2.2的使用append时,需求:追加写入文件,如果文件不存在,需求先创建。
异常:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /huangq/dailyRolling/mommy-dailyRolling for DFSClient_-1456545217 on client 10.1.85.243 because current leaseholder is trying to recreate file.at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1374)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1246)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1426)
at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:643)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
出现上述报错的代码:
<span style="font-size:18px;">FileSystem fs = FileSystem.get(conf);
Path dstPath = new Path(dst);
if (!fs.exists(dstPath)) {
fs.create(dstPath);
}
FSDataOutputStream fsout = fs.append(dstPath);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fsout));</span>
异常的原因:fs句柄的原因,改成下面就ok了。
解决:创建完文件之后,关闭流fs,再次获得一次新的fs。
<span style="font-size:18px;">FileSystem fs = FileSystem.get(conf);
Path dstPath = new Path(dst);
if (!fs.exists(dstPath)) {
fs.create(dstPath);
fs.close();
fs = FileSystem.get(conf);
}
FSDataOutputStream fsout = fs.append(dstPath);</span>