访问Hadoop的HDFS文件系统的Java实现

Hadoop版本

 Hadoop-0.19.1-core.jar

1、创建HDFS对象

Configuration config = new Configuration();
config.set("fs.default.name","hdfs://127.0.0.1:9000/");
FileSystem dfs = FileSystem.get(config);

2、新建目录

String dirName = "TestDirectory";
Path src = new Path(dfs.getWorkingDirectory()+"/"+dirName);
dfs.mkdirs(src);

String subDirName = "subDirectory";
Path src = new Path(dfs.getWorkingDirectory()+"/TestDirectory/"+ subDirName);
dfs.mkdirs(src);

3、删除目录

String dirName = "TestDirectory";
Path src = new Path(dfs.getWorkingDirectory()+"/"+dirName);
Dfs.delete(src);

4、复制文件到本地或者复制文件到HDFS文件系统指定目录中

Path src = new Path("E://HDFS/file1.txt");

Path dst = new Path(dfs.getWorkingDirectory()+"/TestDirectory/subDirectory/");

dfs.copyFromLocalFile(src, dst);

Path src = new Path(dfs.getWorkingDirectory()+"/TestDirectory/subDirectory/file1.txt");

Path dst = new Path("E://HDFS/");

dfs.copyToLocalFile(src, dst);

5、创建并读写文件

Path src = new Path(dfs.getWorkingDirectory()+"/TestDirectory/subDirectory/file2.txt");

dfs.createNewFile(src);

写文件

Path src = new Path(dfs.getWorkingDirectory()+"/TestDirectory/subDirectory/file2.txt");

FileInputStream fis = new FileInputStream("E://HDFS/file1.txt");

int len = fis.available();

byte[] btr = new byte[len];

fis.read(btr);

FSDataOutputStream fs = dfs.create(src);

fs.write(btr);

fs.close();

读取文件

Path src = new Path(dfs.getWorkingDirectory()+"/TestDirectory/subDirectory/file1.txt");

FSDataInputStream fs = dfs.open(src);

String str = null;

while ((str = fs.readline())!= null)
{
System.out.println(str);
}

6、其他操作（和Java文件操作基本一致）

Path src = new Path(dfs.getWorkingDirectory()+"/TestDirectory/HDFS/file1.txt");

System.out.println(dfs.exists(src));

System.out.println(dfs.getDefaultBlockSize());

System.out.println(dfs.getDefaultReplication());

Path src = new Path(dfs.getWorkingDirectory()+"/TestDirectory/subDirectory/file1.txt");
System.out.println(dfs.isDirectory(src));
System.out.println(dfs.isFile(src));