hadoop复制文件java_hadoop – 将大文件复制到HDFS中

我正在尝试将大文件(32 GB)复制到HDFS中.我从未在HDFS中复制文件时遇到任何麻烦,但这些都比较小.我正在使用hadoop fs -put< myfile> < myhdfsfile>并且高达13,7 GB一切顺利,但后来我得到了这个例外：

hadoop fs -put * /data/unprocessed/

Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: Input/output error

at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:150)

at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)

at java.io.BufferedInputStream.read(BufferedInputStream.java:334)

at java.io.DataInputStream.read(DataInputStream.java:149)

at org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:384)

at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:217)

at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)

at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)

at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)

at java.io.DataInputStream.read(DataInputStream.java:100)

at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)

at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)

at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)

at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)

at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:191)

at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1183)

at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:130)

at org.apache.hadoop.fs.FsShell.run(FsShell.java:1762)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)

Caused by: java.io.IOException: Input/output error

at java.io.FileInputStream.readBytes(Native Method)

at java.io.FileInputStream.read(FileInputStream.java:242)

at org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.read(RawLocalFileSystem.java:91)

at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:144)

... 20 more

当我检查日志文件(在我的NameNode和DataNodes上)时,我看到文件上的租约已被删除,但没有指定原因.根据日志文件一切顺利.以下是我的NameNode日志的最后几行：

2013-01-28 09:43:34,176 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /data/unprocessed/AMR_EXPORT.csv. blk_-4784588526865920213_1001

2013-01-28 09:44:16,459 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.1.6.114:50010 is added to blk_-4784588526865920213_1001 size 30466048

2013-01-28 09:44:16,466 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on file /data/unprocessed/AMR_EXPORT.csv from client DFSClient_1738322483

2013-01-28 09:44:16,472 INFO org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile: file /data/unprocessed/AMR_EXPORT.csv is closed by DFSClient_1738322483

2013-01-28 09:44:16,517 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 168 Total time for transactions(ms): 26Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0

有没有人对此有所了解？我已经检查了core-default.xml和hdfs-default.xml以查找我可以覆盖的属性,这些属性会扩展租约,但是找不到.