How to mount HDFS using FUSE

How to mount HDFS using FUSE

Posted on June 3, 2012 by Dan Nanni 6 Comments

Hadoop Distributed File System (HDFS) is a distributed, scalable filesystem developed as the back-end storage for data-intensive Hadoop applications. As such, HDFS is designed to handle very large files with "write-once-read-many" access model. As HDFS is not a full-fledged POSIX compliant filesystem, it cannot be directly mounted by the operating system, and file access with HDFS is done via HDFS shell commands.

However, one can leverage FUSE to write a userland application that exposes HDFS via a traditional filesystem interface. fuse-dfs is one such FUSE-based application which allows you to mount HDFS as if it were a traditional Linux filesystem. If you would like to mount HDFS on Linux, you can install fuse-dfs, along with FUSE as follows.

I assume that you have a HDFS cluster already up and running, and know the HDFS NameNode to connect to. I also assume that you would like to mount HDFS on a separate Debian/Ubuntu-based host.

On the host where you would like to mount HDFS, do the following.

First, install Java JDK.

Next, install fuse-dfs and all necessary dependencies as follows.

To install fuse-dfs on CentOS or RHEL 6:

$ wget http://archive.cloudera.com/redhat/6/x86_64/cdh/cdh3-repository-1.0-1.noarch.rpm
$ sudo yum --nogpgcheck localinstall cdh3-repository-1.0-1.noarch.rpm
$ sudo rpm --import http://archive.cloudera.com/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
$ sudo yum install hadoop-0.20-fuse

To install fuse-dfs on Debian or Ubuntu 10.10 and earlier:

$ wget http://archive.cloudera.com/one-click-install/$(lsb_release -cs)/cdh3-repository_1.0_all.deb
$ sudo dpkg -i cdh3-repository_1.0_all.deb
$ sudo apt-get update
$ sudo apt-get install hadoop-0.20-fuse

To install fuse-dfs on Ubuntu 12.04 and higher:

$ wget http://archive.cloudera.com/one-click-install/maverick/cdh3-repository_1.0_all.deb
$ sudo dpkg -i cdh3-repository_1.0_all.deb
$ sudo apt-get update
$ sudo apt-get install hadoop-0.20-fuse

Once fuse-dfs is installed, go ahead and mount HDFS using FUSE as follows.

$ sudo hadoop-fuse-dfs dfs://<name_node_hostname>:<namenode_port> <mount_point>

Once HDFS has been mounted at <mount_point>, you can use most of the traditional filesystem operations (e.g., cp, rm, cat, mv, mkdir, rmdir, more, scp). However, random write operations such as rsync, and permission related operations such as chmod, chown are not supported in FUSE-mounted HDFS.

 Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.