hdfs 常用java api,hdfs 常用java API---代码篇（一）

一、hadoop环境准备(我安装的hadoop版本是hadoop-1.2.1)

首先要配置hadoop环境变量HADOOP_CLASSPATH(我的hadoop安装在了/home/grid/hadoop-1.2.1)：

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are

# optional.  When running a distributed configuration it is best to

# set JAVA_HOME in this file, so that it is correctly defined on

# remote nodes.

# The java implementation to use.  Required.

export JAVA_HOME=/usr/jdk1.6.0_45

# Extra Java CLASSPATH elements.  Optional.

export HADOOP_CLASSPATH=/home/grid/hadoop-1.2.1/myclass

我在hadoop安装目录下创建了myclass目录

确认hadoop_classpath环境变量生效

[grid@h1 conf]$ source hadoop-env.sh

[grid@h1 conf]$ echo $HADOOP_CLASSPATH

/home/grid/hadoop-1.2.1/myclass

二、编写并编译URLCat源代码

1、代码功能

调用hadoop的java API接口，读取hdfs中的文件

2、代码内容

[grid@h1 myclass]$ cat  URLCat.java

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;

import org.apache.hadoop.io.IOUtils;

import java.io.InputStream;

import java.net.URL;

public class URLCat {

static

{

URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());

}

public static void main(String[] args) throws Exception

{

InputStream in = null;  //定义一个输入stream in

try {

in = new URL(args[0]).openStream(); // 传递参数给 in

IOUtils.copyBytes(in, System.out, 4096, false);  //读取IO in的内容并打印到屏幕(System.out)

}

finally {

IOUtils.closeStream(in);

}

}

}

3、代码编译及运行

[grid@h1 myclass]$ javac URLCat.java

URLCat.java:1: 软件包 org.apache.hadoop.fs 不存在

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;

^

URLCat.java:2: 软件包 org.apache.hadoop.io 不存在

import org.apache.hadoop.io.IOUtils;

^

URLCat.java:9: 找不到符号

符号： 类 FsUrlStreamHandlerFactory

位置： 类 URLCat

URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());

^

URLCat.java:16: 找不到符号

符号： 变量 IOUtils

位置： 类 URLCat

IOUtils.copyBytes(in, System.out, 4096, false);

^

URLCat.java:19: 找不到符号

符号： 变量 IOUtils

位置： 类 URLCat

IOUtils.closeStream(in);

^

5 错误

需要指定classpath进行编译

[grid@h1 myclass]$ javac -cp  /home/grid/hadoop-1.2.1/hadoop-core-1.2.1.jar  URLCat.java

[grid@h1 myclass]$ ls

URLCat.class  URLCat.java

编译成功在$HADOOP_CLASSPATH目录下产生了.class的编译文件

运行代码：

[grid@h1 myclass]$ hadoop fs -ls in

Found 4 items

-rw-r--r--   2 grid supergroup        101 2013-08-24 12:50 /user/grid/in/VERSION

-rw-r--r--   2 grid supergroup          7 2013-08-23 21:30 /user/grid/in/test3.txt

-rw-r--r--   2 grid supergroup         12 2013-08-23 21:30 /user/grid/in/text1.txt

-rw-r--r--   2 grid supergroup         13 2013-08-23 21:30 /user/grid/in/text2.txt

[grid@h1 myclass]$ hadoop fs -cat in/test3.txt

hadoop

[grid@h1 myclass]$ hadoop URLCat hdfs://h1:9000/user/grid/in/test3.txt

hadoop

4、代码用到的API解读

1、hadoop基础类

org.apache.hadoop.fs Class FsUrlStreamHandlerFactory

org.apache.hadoop.fs.FsUrlStreamHandlerFactory

All Implemented Interfaces:

public class FsUrlStreamHandlerFactory

extends Object

Factory for URL stream handlers. There is only one handler whose job is to create UrlConnections. A FsUrlConnection relies on FileSystem to choose the appropriate FS implementation. Before returning our handler, we make sure that FileSystem knows an implementation for the requested scheme/protocol.

FsUrlStreamHandlerFactory：URL流处理工具，具体的作用要使用java.net.URL的 setURLStreamHandlerFactory()方法设置URLStreamHandlerFactory，这时需要传递一个 FsUrlStreamHandlerFactory。这个操作对一个jvm只能使用一次。

org.apache.hadoop.io.IOUtils

public class IOUtils

extends Object

An utility class for I/O related functionality.

一个I/O相关功能的实用类