通过Java读取HDFS数据

 

Accessing the HDFS from Java

When we are working with the Hadoop Distributed File System,  sometimes we need access to the files content within file system to be manipulated by any application. Due to the HDFS is written in Java we can interact with it through the Java API. Thus, we can use the Java FileSystem class to perform file system operations. Hadoop also provides a C library called libhdfs that mirrors the JavaFileSystem interface and can be used to access the HDFS and although I haven’t tried it I hope do it soon.

Suppose we need to access to the content of a file with a cat-type application, so the code that we should write could be like next:

import java.io.InputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class HadoopFScat {
	public static void main(String[] args) throws Exception {
		String uri = args[0];
		Configuration conf = new Configuration();
		FileSystem fileSystem = FileSystem.get(URI.create(uri), conf);
		InputStream inputStream = null;
		try{
			inputStream = fileSystem.open(new Path(uri));
			IOUtils.copyBytes(inputStream, System.out, 4096, false);
		}finally{
			IOUtils.closeStream(in);
		}
	}
}

The above code takes the URL file as argument to read data from the HDFS. One of the simplest ways to read data from the Hadoop file system is by using a java.net.URL object which opens a stream, but also we can use the FileSystem API to do it. If we use the last, we need a Configurationobject to encapsulate server’s configuration which is taken from conf/core-site.xml file. We also need use the IOUtil class to print on screen the content file and finally close it. The sample to run it should be:

% hadoop HadoopFScat hdfs://127.0.0.1/user/luis/sample.txt
This is the content of the sample.txt file that
is stored in the Hadoop Distributed File System

We can use another forms to access a file in a specific position according to a offset, so I’ll try write about it in the next post and run an example.

Greetings

原文 http://www.undercloud.org/?p=82 

这是一个简单的通过Java读取HDFS中数据的例子 运行过程

1.javac -cp hadoop-core-1.2.1.jar HadoopFScat.java

2.hadoop HadoopFScat sample.txt