Hadoop CDH4.5 HTTPfs部署

为什么80%的码农都做不了架构师？>>>   

Apache Hadoop HttpFS服务提供了通过HTTP访问HDFS，HttpFS自带REST HTTP API， 支持所有HDFS文件系统的操作。

HttpFS一般用于以下情况：

*) Read and write data in HDFS using HTTP utilities (such as curl or wget) and HTTP libraries from languages other than Java (such as Perl).

*) Transfer data between HDFS clusters running different versions of Hadoop (overcoming RPC versioning issues), for example using Hadoop DistCp.

*) Read and write data in HDFS in a cluster behind a firewall. (The HttpFS server acts as a gateway and is the only system that is allowed to send and receive data through the firewall).

        The webhdfs client file system implementation can access HttpFS via the Hadoop filesystem command

1    安装HttpFS

apt-get install hadoop-httpfs

2    配置HttpFS，HttpFS从core-site.xml和hdfs-site.xml读取HDFS配置

        HttpFS安装完后，目录结构如下

Binaries           /usr/lib/hadoop-httpfs/
Configuration      /etc/hadoop-httpfs/conf/
Documentation      /usr/share/doc/hadoop-httpfs/
Data               /var/lib/hadoop-httpfs/
Logs               /var/log/hadoop-httpfs/
temp               /var/tmp/hadoop-httpfs/
PID file           /var/run/hadoop-httpfs/

3    配置HttpFS代理用户

        配置core-site.xml文件，定义一个linux用户来运行HttpFS

<property>
 <name>hadoop.proxyuser.httpfs.hosts</name>
 <value>*</value>
</property>

<property>
 <name>hadoop.proxyuser.httpfs.groups</name>
 <value>*</value>
</property>

4    启动HttpFS服务

service hadoop-httpfs start
http://<HTTPFS_HOSTNAME>:14000/webhdfs/v1
curl "http://localhost:14000/webhdfs/v1?op=gethomedirectory&user.name=babu"

挂载HDFS，你想挂载分区那样挂载HDFS文件系统吗？操作过程如下：

apt-get install hadoop-hdfs-fuse
mkdir -p <mount_point>
hadoop-fuse-dfs dfs://<name_node_hostname>:<namenode_port> <mount_point>
umount <mount_point>

为了重启后也生效，把挂载命令加入/etc/fstab中

dfs://<name_node_hostname>:<namenode_port> <mount_point> fuse allow_other,usetrash,rw 2 0

实战

1   U-5安装hadoop-hdfs-fuse

apt-get install hadoop-hdfs-fuse

2    dfs挂在到/mnt目录下

hadoop-fuse-dfs dfs://mycluster:8020 /mnt

转载于:https://my.oschina.net/guol/blog/265804