noticeable improvements over the previous stable 2.X.Y releases. This version has many improvements in HDFS and MapReduce. This how-to guide will help you to install Hadoop 2.6 on CentOS/RHEL 7/6/5, Ubuntu and other Debian-based operating system. This article doesn’t include the overall configuration to setup Hadoop, we have only basic configuration required to start working with Hadoop.
![Hadoop on Linux](https://tecadmin.net/wp-content/uploads/2014/05/hadoop-st.png)
Step 1: Installing Java
Java is the primary requirement to setup Hadoop on any system, So make sure you have Java installed on your system using the following command.
# java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
If you don’t have Java installed on your system, use one of the following links to install it first.
Install Java 8 on CentOS/RHEL 7/6/5
Install Java 8 on Ubuntu
Step 2: Creating Hadoop User
We recommend creating a normal (nor root) account for Hadoop working. So create a system account using the following command.
# adduser hadoop
# passwd hadoop
After creating an account, it also required to set up key-based ssh to its own account. To do this use execute following commands.
# su - hadoop
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
Let’s verify key based login. Below command should not ask for the password but the first time it will prompt for adding RSA to the list of known hosts.
$ ssh localhost
$ exit
Step 3. Downloading Hadoop 2.6.5
Now download hadoop 2.6.0 source archive file using below command. You can also select alternate download mirror for increasing download speed.
$ cd ~
$ wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz
$ tar xzf hadoop-2.6.5.tar.gz
$ mv hadoop-2.6.5 hadoop
Step 4. Configure Hadoop Pseudo-Distributed Mode
4.1. Setup Hadoop Environment Variables
First, we need to set environment variable uses by Hadoop. Edit ~/.bashrc file and append following values at end of file.
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
Now apply the changes in current running environment
$ source ~/.bashrc
Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOMEenvironment variable. Change the JAVA path as per install on your system.
export JAVA_HOME=/opt/jdk1.8.0_131/
4.2. Edit Configuration Files
Hadoop has many of configuration files, which need to configure as per requirements to setup Hadoop infrastructure. Let’s start with the configuration with basic Hadoop single node cluster setup. first, navigate to below location
$ cd $HADOOP_HOME/etc/hadoop
Edit core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
Edit hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>
Edit mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Edit yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
4.3. Format Namenode
Now format the namenode using the following command, make sure that Storage directory is
$ hdfs namenode -format
Sample output:
15/02/04 09:58:43 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = svr1.tecadmin.net/192.168.1.133
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.5
...
...
15/02/04 09:58:57 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.
15/02/04 09:58:57 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/02/04 09:58:57 INFO util.ExitUtil: Exiting with status 0
15/02/04 09:58:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at svr1.tecadmin.net/192.168.1.133
************************************************************/
Step 5. Start Hadoop Cluster
Now start your Hadoop cluster using the scripts provides by Hadoop. Just navigate to your Hadoop sbin directory and execute scripts one by one.
$ cd $HADOOP_HOME/sbin/
Now run start-dfs.sh script.
$ start-dfs.sh
Sample output:
15/02/04 10:00:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-svr1.tecadmin.net.out
localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-svr1.tecadmin.net.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is 3c:c4:f6:f1:72:d9:84:f9:71:73:4a:0d:55:2c:f9:43.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-svr1.tecadmin.net.out
15/02/04 10:01:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Now run start-yarn.sh script.
$ start-yarn.sh
Sample output:
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-svr1.tecadmin.net.out
localhost: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-svr1.tecadmin.net.out
Step 6. Access Hadoop Services in Browser
Hadoop NameNode started on port 50070 default. Access your server on port 50070 in your favorite web browser.
http://svr1.tecadmin.net:50070/
![hadoop single node namenode](https://tecadmin.net/wp-content/uploads/2014/05/hadoop-2-6single-node-namenode.png)
Now access port 8088 for getting the information about cluster and all applications
http://svr1.tecadmin.net:8088/
![hadoop single node applications](https://tecadmin.net/wp-content/uploads/2014/05/hadoop-2-6-single-node-applications.png)
Access port 50090 for getting details about secondary namenode.
http://svr1.tecadmin.net:50090/
![Hadoop single node secondary namenode](https://tecadmin.net/wp-content/uploads/2014/05/hadoop-2-6-single-node-secondary-namenode.png)
Access port 50075 to get details about DataNode
http://svr1.tecadmin.net:50075/
![hadoop-2-6-single-node-datanode](https://tecadmin.net/wp-content/uploads/2014/05/hadoop-2-6-single-node-datanode.png)
Step 7. Test Hadoop Single Node Setup
7.1 – Make the HDFS directories required using following commands.
$ bin/hdfs dfs -mkdir /user
$ bin/hdfs dfs -mkdir /user/hadoop
7.2 – Now copy all files from local file system /var/log/httpd to hadoop distributed file system using below command
$ bin/hdfs dfs -put /var/log/httpd logs
7.3 – Now browse hadoop distributed file system by opening below url in browser.
http://svr1.tecadmin.net:50070/explorer.html#/user/hadoop/logs
![hadoop-test-uploaded-files](https://tecadmin.net/wp-content/uploads/2014/05/hadoop-test-uploaded-files.png)
7.4 – Now copy logs directory for hadoop distributed file system to local file system.
$ bin/hdfs dfs -get logs /tmp/logs
$ ls -l /tmp/logs/
You can also check this tutorial to run wordcount mapreduce job example using command line.